Optimize execution speed

To scrape a big site, maybe you need speed, here's some options you can set.

 

"Block files", "adblock" and "show image" on main interface

1. Show Image

You can uncheck "show image" to let the browser not show images on pages,  for most projects you can uncheck it when execute the project. 

 

2. Adblock

Block the URLs in adlist.ini file, this file include most of rules of advertisement.

 

3. Block Files

This is an important option to speed extraction, when check it, the program will block and not download these types files, the default files types are: css js jpg png gif ttf woff. And you can add more file types or remove one.  But you should carefully to use it, for some dynamic pages, you should not block js files, or the pages will not load correctly.

 

Project Settings

On project settings dialog, here's some options you can change to speed the extraction.

See here: project settings

1. Max Thread(browser) Count:

This is also an important option, the default count is 5, you can make it bigger to speed extraction. See more threads.

 

2. Action Interval

It's means how long time the program will wait when done an action. You can change it small.

 

Download Files after Extraction

Download files will spend much time, we can just scrape URLs of the files, and download them after extraction with downloader.