• You are not logged in.

#1 Oct. 12, 2013 07:04:21

alex
Registered: 2012-11-30
Posts: 13
Reputation: +  0  -
Profile   Send e-mail  

can fminer download

Hello I'm currently struggling with this:

I have an aspx site which displays a list of results with download links.
The downloads links must be clicked and they execute some ajax code which will bring up the save us dialog in the brower (the files are .doc's ).
I've tried to set the click action in FMiner and then scrape page/extract type/download-wait download.. but FMiner keeps telling me there's nothing to scrape.

Thank you for any help/advice!

Offline

#2 Oct. 13, 2013 20:42:32

admin
Registered: 2012-03-15
Posts: 289
Reputation: +  1  -
Profile   Send e-mail  

can fminer download

You can add a ‘scrape page’ action on ‘click’ action, then add ‘capture content’, and set ‘extract type’ to ‘download’ -> ‘wait download’.

Offline

#3 Oct. 15, 2013 02:36:59

alex
Registered: 2012-11-30
Posts: 13
Reputation: +  0  -
Profile   Send e-mail  

can fminer download

Yes, this is exactly what I did, but sadly FMiner would wait forever and nothing happens. I have uploaded my FMiner project here: fminer project, can you please have a look. I think this site does something that FMiner can't handle correctly (btw site's language is romanian, but all the actions are included in the project file)
Thank you!

Offline

#4 Oct. 20, 2013 21:05:44

admin
Registered: 2012-03-15
Posts: 289
Reputation: +  1  -
Profile   Send e-mail  

can fminer download

Sorry for reply you late, we went to Beijing days for a meeting.

Tested your project, it's no problem, but found a bug in FMiner of “wait download”, fixed in code. Please wait days for the new version to test your project.

Offline

#5 Oct. 22, 2013 21:34:58

admin
Registered: 2012-03-15
Posts: 289
Reputation: +  1  -
Profile   Send e-mail  

can fminer download

Fixed the bug, please upgrade to 9.03 to run your project.

Offline

#6 Nov. 29, 2013 00:32:56

alex
Registered: 2012-11-30
Posts: 13
Reputation: +  0  -
Profile   Send e-mail  

can fminer download

Thank you very much for the fast support and quick fix!

Now I have another question/suggestion I will post it here because it is also related to the download function, this time it's about image downloads.

I have this page where I need to download a lot of images, but image name is always the same as it's generated by the server (something like getimage.png). Now the problem is Fminer will download the image, but it will always overwrite the previous image with the new one as it has the same name. Is it possible to add a function to rename those files on the fly as they are downloaded or such function already exists and haven't noticed?

Offline

#7 Nov. 29, 2013 00:55:01

admin
Registered: 2012-03-15
Posts: 289
Reputation: +  1  -
Profile   Send e-mail  

can fminer download

I think this should not happen, because FMiner will check whether the file existing before save it, if the file existing, it will save to a new file like getimage_sdfsf.png.

If you really encounter this situation, please tell me the link, I will debug it.

Offline

#8 Nov. 29, 2013 01:52:25

alex
Registered: 2012-11-30
Posts: 13
Reputation: +  0  -
Profile   Send e-mail  

can fminer download

admin
I think this should not happen, because FMiner will check whether the file existing before save it, if the file existing, it will save to a new file like getimage_sdfsf.png.

If you really encounter this situation, please tell me the link, I will debug it.

Hello, I think it's a specific version bug yes because I tried the same link with an old version of Fminer (6.x) and the images are all getting names like -get-document-page5nfa2, get-document-pageonlWK and so on. I'll try to figure out if it's something on my side and comeback with information. Ps, it will be nice to have an alternative consecutive numbering to the documents besides the random numbers because it makes things easier for some tasks (eg if those images where a book and I would have to ocr the images after, it would be much easier to figure the page numbers/ordering..now I have to rely on timestamp information)

Offline

#9 Nov. 29, 2013 02:22:30

alex
Registered: 2012-11-30
Posts: 13
Reputation: +  0  -
Profile   Send e-mail  

can fminer download

The problem persists and it seems it's a possible bug, I sent you by e-mail the project. I would normally post the project on the forum to help others which would have this issue also, but this site is password protected. I attached the project which also contains login automation. The problems are:

1. images get overwritten and no random suffix is added at the end
2. the image file is empty (0 size)

Thank you for any help with this!

Offline

#10 Nov. 29, 2013 06:47:00

admin
Registered: 2012-03-15
Posts: 289
Reputation: +  1  -
Profile   Send e-mail  

can fminer download

Solve the problem, this is because the site limit the access, it will check the “Referer” in request, I added this value, now it can download files for this kind sites.

Please wait days for new version to scrape this site.

Offline

Board footer

Moderator control

Powered by DjangoBB