Avoid crash, and monitor execution for scraping big site

 

FMiner use a browser core to scrape pages, as a general browser, it may crash when accessed a lot of pages. This is the inevitable question.

To avoid and reduce crash, you should:

1. Check "block files" on main interface, this will block css, image, js ... files, to make execution stable. If the site need "js" file to load correctly, remove "js" from the blocked files. This will avoid most of crash problems.

2. Don't set "max thread(browser) count" too big. http://www.fminer.com/generaltimeout-interval-threads-count/ 

 

For small project, you can open the project again, and click "resume" continue to run it. And for big projects need run serval days, you should use a tool to monitor it, restart and resume run the project when it crash.

You can use this tool http://w-shadow.com/blog/2009/03/04/restart-on-crash/ to monitor FMiner. It's a freeware. Alternate Download Link

 

The steps:

1. Make a bat file and input text "mainwin.exe --resume your_project_path.fmpx", and copy it to FMiner's folder. Here we name it "runprj.bat". See Execute FMiner Task Externally

2.Config "Restart on Crash" like this:

3. Then run the project, it will restart and resume run when crash.