Web scraping

Introduce web scraping,and web scraping with FMiner

What is web scraping

Web scraping means crawling websites and extracting contents on the pages according to some rules, and storing the data into database or files.

In our new world of technology and easier access to information through the internet and the media we are now turning to technology once again to help us collect this information. That technology is called Web Scraping, sometimes called Web Harvesting, although Web Harvesting is slightly different. 

You need a software to scrape the web to pick out information you require from various URL’s giving you the data you need. Web scraping is likened to web browsing, however; instead of manually looking through a website to gain information that you can see, web scraping allows you not only to automate this, but to scrape data from the site that is hidden in background code we don’t see.

Examples of web scraping /web harvesting are the insurance comparison websites. If they were to search the net themselves for all the different types of home insurance, the information would be out of date before they could publish it. By using web scraping software, they can continually keep their site up to date giving the latest prices from several different companies.  

Many websites may post up information in formats that are basic texts at the forefront. However, there is hidden data in the background, which can only be accessed through code, HTML language, for instance. Consider a government official website, they might show that crime rate is decreased in a certain area to show that a certain methodology is working, However; in the background, the coding is still there showing that it increased in the next state along.  

web scraping introduce on wiki

web data extraction and FMiner 

Scrape a web with FMiner

There are many web scraping software, and users can use them to set extraction rules and extract the contents from the websites. FMiner is one of them. When the users want to extract information (such as weather, prices) from the a website, they can manually copy and paste the content from pages, or choose one of these software.   

FMiner is the web scraping software that can pick up that data and transfer it into a format which we can read and use. For instance, researchers may need a number of comparative statistics but don’t have the staffing to set up further research for this, so the software would be set up to find and gather results from set URL’s. So what is FMiner software?

FMiner is the web scraping software created and designed for ease of use to anyone. It was under development and rigorous testing for two years and is the very best in Scraper software today. The web scraping software is designed to scan through set sites, and scrape set data at your requirement. When you set up a scheduled and set the program, you can now extract the data and save in many formats such as .csv, Excel, access or an SQL Server. This makes the data easier to read through and in the case of comparative data this is a lot easier to work with.

Apart from web scraping, it also has other qualities such as web harvesting, web crawling, screen scraping and can also be used as a web extractor. All of these features add to the overall usage of the software. You can 'scrape' information from any type of page and the software is capable of dealing with many scripts such as HTTP being the more common, Java and Ajax, proxy, login and also plugins.

The web scraping software of FMiner gives you marketing information, competitor information and customer information. It helps in developing a closer relationship with customers through the discover of the products which are selling, what consumers like in a product, the product defects encountered, the specific group of customers who favor a product, etc. 

FMiner gives you direction as to making the right decisions as you are able to analyze how your company is standing in the market. If you would like to gain knowledge about current or upcoming trends, FMiner will always be there for you. Buying and selling trends, price comparisons, and consumer logistics are some of the things the software gathers, stores and analyzes for you. Overall, this web scraping software easy to use, very adaptable and has your answer to just about any equation you can think of making it the most versatile on the market today.  

For the software, usually more powerful, more complex to use. There are some web scraping software is powerful but difficult to get started; Some other is easy to get started, but with some functions missing. FMiner development began in 2009. Its goal is to complete an easy to use, real visual web scraping software, and can extract all the websites. Now, it has been very close to this goal. 

FMiner's main interface contains only the most basic controls, and users need a small number of operations to complete a general web extraction configuration. It also contains a wizard dialog to help users config the general extraction projects.

Javascript / ajax page's extraction is a challenge for the web scraping software. FMiner learns some macro software. It deals with the pages' javascript operations by recording and play backing user's actions. In this way, the users can extract the complex dynamic pages without understanding pages' code and writing scripts.complicated.

Screen scraping with FMiner

FMiner can scrape any site

After two years of development, fminer's functions is complete. It supports javascript/ajax (via macro), login, https, proxy, plugins; It can save the extracted results into csv, excel(xls), sqlite, access, sql server, mysql, postgres; It can extract complete data structures, include data's relation (eg, database foreign keys); it support schedule extraction and incremental extraction. In theory, it can crawl and scrape any website, and the operation is not complicated. 

Ways to use FMiner for web scraping

FMiner is a visual web scraper that can crawl a target website, scrape page content and convert scraped data into a structured format according to specified rules. Users need to specify the rules according to their needs and choose which pages to crawl, what data on these pages to extract and what format to store the extracted data in. Developing these rules is a very complex task for all web scraping software. We have worked hard to simplify this process and make FMiner easy to use. Now it is a real visual web scraping tool, enabling users to complete extraction just by clicking the mouse. There is no need to understand the code and writing scripts on webpages.

Extract google search result. 

Extract production pricing data, description, images. 

Gather news from multiple news site. 

Gather and monitor blogs for new articles. 

Extract job information. 

Gather user reviews and comments for productions. 

Download all software and scrape web data from a download site. 

Gather financial information from a financial site. 

Extract dom's source, tag and other thing from a page. 

Gather movie(top movie) information from imdb. 

Gather pages meta information(title, keywords, description). 

Log in a site and scrape data from it. 

Track auction prices. 

And many other web scraping efforts


About FMiner

FMiner is a real visual web scraping tool with a diagram designer, and you can use it to build a project with macro record. Goto video tutorials to see how to use it.



blog comments powered by Disqus