Size estimation increases during download about moodle-dl HOT 8 OPEN

@masteoo Don't worry I have seen your issue. There is no need to ping me in another issue :D But you can contact me at any time via mail.

from moodle-dl.

I think we could introduce a feature to check the size of most external files. Currently we don't check the size of external files in advance, so we can't tell how big the files (mostly videos from external sources) are before we start downloading them. Sometimes we can not even tell the size if we start downloading the files, because some video formats or platforms do not tell how large the file is, so there is no way to estimate it in any way.
For now we could tell the user somehow that the estimated size is only for moodle files and not external files.

If someone is willing to implement this, we can have a discussion how I imagine this feature, and how it would be best to implement it. I personally would like to implement it, but have only little free time currently.

from moodle-dl.

After downloading 357GB which supposedly are 87% of 407 GB I had to kill it.
It says 246 Failed. So I wanted to look in MoodleDL.log which ones failed, but that file is empty. Is it supposed to be empty ?
Where can I learn which ones have failed so I can hunt them up manually and see if I can somehow get them ?

After freeing some space, can I just rerun 'moodle-dl' and it will resume where it left off without having to download again what's already downloaded, and fetching what's still missing and maybe retrying what has failed before ?

from moodle-dl.

moodle-dl was designed in such a way that it never downloads files that have already been downloaded. If it does, it is a bug, or the database has been changed manually.

Since a few versions, the log file is only used if you use the option --log-to-file.

Where can I learn which ones have failed so I can hunt them up manually and see if I can somehow get them ?

This information is only present in memory, if you kill moodle-dl you will not get informed about failed tasks. But you could scroll up the terminal output.
Failed tasks will be re-downloaded if you re-run moodle-dl

That also brings me to a good idea that we could implement. We could catch kill signals in the download loop, and stop downloading on that signal but inform and notify the user as it would do normally.

from moodle-dl.

Thank you

from moodle-dl.

If the size estimation is not really possible due to external sources, should I close this issue? Also, the percentage given there is clearly not 100 * downloaded / current_estimate.

from moodle-dl.

We can leave the issue open for now, until we find a proper way to tell the user what is going on. It will help users that see that the estimation is increasing while downloading. Also we can discuss here about this topic and find solutions / ideas that we could implement.

from moodle-dl.

The percentage is currently the best we could do (at least in a simple way). Some downloads like HLS Videos or other video formats, have irregular partition sizes, thats why sometimes we add bytes to the estimation, and sometimes we need to remove bytes from the estimation. Actually we should remove the total estimation at the end of a download of such a file and replace it with the actually downloaded size, that way the percentage should be a little better.

from moodle-dl.