• You are not logged in.

#1 Nov. 30, 2012 03:37:41

alex
Registered: 2012-11-30
Posts: 13
Reputation: +  0  -
Profile   Send e-mail  

how to rip a website that has lazy loading feature

Hello, firstof, I am really glad that I purchased your program which I already used succesfully on a quite big project and it's also great that you update it so often making it more usuable.

I am now dealing with a website which has the article results page like this:
-there is no page 1, page 2 etc.
-instead, it will “lazy load” the rest of the content when I am scrolling the webpage (in a way pretty similar to Facebook)

How can I manage to rip a whole results page like that without having to manually scroll down a huge amount of results? thanks in advance for any input/recommendation!

Offline

#2 Dec. 1, 2012 19:39:30

admin
Registered: 2012-03-15
Posts: 289
Reputation: +  1  -
Profile   Send e-mail  

how to rip a website that has lazy loading feature

Thanks for using FMiner, FMiner hasn't an action of “scroll down”. But from this page, http://wiki.imacros.net/FAQ#Q:_How_to_scroll_down_a_page.3F I found a simple way to scroll a page. FMiner can run javascript in browser, so add an action “run javascript in browser”, and input code:

window.scrollBy(0,20000)

I've tested some pages, and it can work, please try your site, if you has any problem, please tell me.

Edited admin (Dec. 1, 2012 19:50:04)

Offline

#3 Dec. 15, 2012 12:49:31

alex
Registered: 2012-11-30
Posts: 13
Reputation: +  0  -
Profile   Send e-mail  

how to rip a website that has lazy loading feature

Great! thanks for the reply, this could have been the solution. Anyway, I've managed to find another workaround while studying better the site and it's navigation and I was lucky enough to find a next article button
Though, I guess it would be a nice idea to implement a scroll down feature in the default options of Fminer, because there are a lot of websites out there that use this lazy loading technique and they might not have some other options to skip through the articles. I've got away with it this time

Offline

#4 Dec. 16, 2012 23:36:50

admin
Registered: 2012-03-15
Posts: 289
Reputation: +  1  -
Profile   Send e-mail  

how to rip a website that has lazy loading feature

I've put this skill to the faq page: http://www.fminer.com/faq/ , This is not very important action, and achieve it with runjs is not complicated, I will not add this feature to new versions recently.

Offline

Board footer

Moderator control

Powered by DjangoBB