Skip sample if it takes too long to download about isic-archive-downloader HOT 3 OPEN

A temporary poor man's solution could be to identify the offset i of the last successfully downloaded sample (in my case 196) and manually rerun the script starting at offset i+2 (in my case 198), thus skipping the problematic sample.

This would be fine. However this does not work because it seems that filenames are not exactly relative to the requested offset and number of samples (I'm not exactly sure what's going on), and some samples end up being overwritten because they are saved with the same filename as previously downloaded samples.

from isic-archive-downloader.

For what it's worth, I've ran the script under various locations with my VPN (i.e. different IP address) and even asked a friend to run it for me. I have also tried running with --p 1 thinking that maybe all those processes were spamming the API too much. But the results are always the same, it always hangs at #160 for a while and then #197 for much longer. So I don't think the API is rate limitting us or anything like that.

You should be able to reproduce it the same way if you run this right now

python download_archive.py --num-images 250 --filter malignant

Do you understand why this is happening?

from isic-archive-downloader.

Hey! Thank you for this great idea! :)
And regarding the issues that you mention - I think some images are not downloadable for some reason.
You can even try to download them using the link in your browser and it won't be able to.
So I guess image #197 is one of these images. On the other hand I can't think of a reason yet for the hanging on image #160.
I will try to reproduce it myself when i'll have the time soon.

Btw, in order to skip the problematic samples in a more elegant way we could use the max_tries parameter which is present in some of the download functions in "download_single_item.py
", and add that parameter to the functions that don't have it.
I have put that parameter in some of the download functions with a default value of infinite tries, but never really given the user a way to specify another value in cases of problematic images such as you described.

So I guess that when i'll have more time, that will be the way to implement this request

from isic-archive-downloader.