• You are not logged in.

#1 Oct. 11, 2012 17:48:02

dtwaddell
Registered: 2012-10-11
Posts: 1
Reputation: +  0  -
Profile   Send e-mail  

Can't see how to do this..

Hi, delighted to purchase this excellent product. It looks easy to use, but I am having some difficulty getting it to do what I want. Can anyone help me through this?

First I go to google, enter search string, click Search. This brings up a page of results.
Then I want to scrape all the titles from the 10 results listed, so I do Scrape Page, Select Multiple Sets, Select the block, Expand it and do group Select.

Then I resume recording to capture the content into the table.

I want to repeat this on subsequent pages so I right-click on Next and select Open Link Recursively -> Next.

Two problems.

first, when I run the results table only contains data from the first search result on each page.

Secondly, I would like to extend the operation so that, as well as scraping data from the results page, the script clicks each result URL to get to a detail page and then scrapes some data from here as well. The script needs to do this for each result. I cannot see any way to do this. I have tried quite a bit but no success yet.

I will keep trying, but if anyone can help in any way it would be much appreciated!

thanks,

David

Offline

#2 Oct. 14, 2012 21:53:26

admin
Registered: 2012-03-15
Posts: 289
Reputation: +  1  -
Profile   Send e-mail  

Can't see how to do this..

Thanks for choice FMiner.
For your questions, I think you should understand working principle of FMiner. FMiner will do actions follow diagrams in the scene, when do an action successfully, alone the bottom red line, when fail, alone the left black line, and if it has a right blue line, it will scrape contents when succeed, then alone the red line to do following actions. http://www.fminer.com/actions/

For your first question:
I think your project is like the attachment image. The “open links(s) recursively” node hasn't a right blue line, so FMiner will not scrape contents when done the opening “next” link action, you should drag a line from right joint of it to the right “scrape content” node.

For your second question:
You can continue record the actions, and add an “open link(s)” action after the “open links(s) recursively” node with all links you want to open, to select all links you should use “group select” feature. see this tutorial http://www.fminer.com/open-links-page-then-scrape-data-these-pages/, then add “scrape content” node and so on to scrape the pages.
There is one thing you should pay attention to, because the new “open link(s)” action is after the “open links(s) recursively” node, so it will just work for all the next pages (without the first page). You should check “Do follow actions on initial page” in the “recursive node” to make the initial page valid.





Edited admin (Oct. 14, 2012 22:09:22)

Attachments:
attachment 1.jpg (24.2 KB)

Offline

#3 Dec. 26, 2013 08:32:43

jameskona
Registered: 2013-12-26
Posts: 1
Reputation: +  0  -
Profile   Send e-mail  

Can't see how to do this..

Dear Admin,

The page you have shared is not working. Can you please help me. I am also having the same problem.
dtwaddell, did you get a solution for what you were looking for. If yes, please share with me as well.

Thanks,
James

Offline

#4 Dec. 26, 2013 22:46:10

admin
Registered: 2012-03-15
Posts: 289
Reputation: +  1  -
Profile   Send e-mail  

Can't see how to do this..

Please make sure the “next page” is a link or ajax button at first. I've made a new tutorial for “ajax button” flip, please see here: http://www.fminer.com/23-click-next-page-button-ajax-pages/

You can “step” to test your project, if the program can't select the targets correctly, please see here: http://www.fminer.com/skills-select-targets-correctly/

Offline

Board footer

Moderator control

Powered by DjangoBB