Overview

FMiner is a visual web data extraction tool for web scraping and web screen scraping.

Its intuitive user interface permit you to quickly harness the software's powerful data mining engine to extract data from any site. The easy to use interface not only permits you to extract data from dynamic web site and JavaScript pages; it also supports site login, proxy server usage, HTTPS access, captcha decoding and plugins compatibility that make it the most powerful, most versatile and most easy to use product of its kind on the market.

In addition to web scraping, it can be as a web macro software recording and playing human actions on the web browser.

Getting to Know FMiner

Understanding the relationship and interaction among the how each of the main panels is critical to your designing effective data mining and web page scraping projects. And it's actually quite simple.

/media/fminercms/docsnag6/main_interface.jpg

The Project Design Panels

The Project Flow Chart combined with the Browser is where you instruct your project to goto, click, enter value, open links, iterate through pages and scrape page information. Each action is recorded and may just as easily be deleted as you step backward to edit the Process Map.

The Attribute and Browser Panels are where you identify, select and save the project's data elements. Think of this piece as the extension to the Project Process Map that allows you to operate on specific data elements after navigating to the desired pages. It enables you to narrow in on the data you need; define whether it is the value or code you'd like scraped; and specify where to save that data by naming tables and fields of your choosing.

There are four panels that are integral to this intuitive cognitive design model.

The Integrated Browser Panel

Having the browser integrated into the main screen admin screen of the software is critical to an cohesive design cycle. It permits you to click through your project to identify specific extraction elements on which you'd like the extraction code to operate. There is no coding involved. The browser permits you build your extraction project with the click of a mouse.

The Flowchart Process Map Panel

With each step you increment the flowchart and can follow the logical flow of your project. It's as simple as entering your target website url, clicking through links and selecting from preset record options to determine the scope of the data extraction for each page element. And best of all, through the whole process you will see a dynamic flowchart representation of your data extraction instruction set in an intuitive panel which guides you through project definition down to the most granular data element.

The Attribute Panel

In conjunction with the visual design interface, the associated attribute panel just below the flowchart window permits you to select specific or blocks of data elements by expanding or narrowing the scope of the DOM element being observed at each step of the diagram node. Further, the attribute panel's nuanced selection capabilities permits you to determine not only which parts to extract but also whether html tags should be included as well.

The Input and Output Panel

Having used the Browser, Process Map and Attribute Panels to define your project, you can view the resulting code, output results and input values tables in the Input Output Panel.

Note that you have four tabs at the bottom of this panel: Logs, Data, Selection, Variables.

The Log Tab gives you an under the hood view of the project as the program steps through your web scraping instruction set to extract data elements from your target web site.

  • The first is the Log Tab.

    The Log Tab gives you an under the hood view of the project as the program steps through your web scraping instruction set to extract data elements from your target web site.

  • The second is the Data Tables Tab.

    /media/fminercms/docsnag6/data.jpg

    This tab is used to access the table view of the extracted values and the form entry tables where other input variables such as target site URL lists, keywords for searching. Tables may be created and attributes defined, edited or deleted.

  • The third is the Selections Tab.

    Reflecting selections made throughout the browser and the attributes panel, this tab area displays the DOM codes you selected on the browser, thereby granting you a very precise lens into the DOM elements examined and selected at any step.

  • The fourth is the Run Code Variables Tab.

    /media/fminercms/docsnag6/variables.jpg

    This tab is for advanced usage and is used to define input parameters such to crawl the site. You may create an Input Table here that holds parameter values. A list of such values may be imported using the import values button located in the immediate right of the data tables tab window. Using a series of easy steps illustrated in the Input Variables List Tutorial you can instruct your project to iterated through any number of terms or values as text input search variables.