Recommended way to deal with failed downloads? about nist-data-mirror HOT 3 OPEN

I've opened a PR #45 that implements downloading to a temporary directory.

from nist-data-mirror.

+1 to this issue. I've stumbled upon this project when issues with nvd.nist.gov broke CI in our projects. I intended to use the mirror to avoid this issue in the future. However, if connection issues can, indeed, break the mirror, it defeats the point.

from nist-data-mirror.

+1 We suffer the same issue with 1.5.3, and now upgraded to docker image nvd-mirror 1.6.0. But I expect the problem to show again in a few days. This is quite a problem for our CI system, as the maven dependency-check plugin caches the files itself but does not handle corrupt .json.gz files very well, i.e. it does not try to immediately fetch again when I've already fixed the nvd-mirror manually...

IMHO, nist-data-mirror should quarantine downloaded corrupt files. Or -- as suggested by @ghost -- download to a temporary file, before moving the integrity-checked file to the output directory, i.e. /tmp/nvd.

• The .meta needs a sanity check after download, e.g. last line must match "^sha256:[0-9A-F]{64}$
• The the .json and .json.gz have to be checked according to the length and checksum in the .meta file.
• As a result, the nvd-mirror will never update the htdocs folder with corrupt files from /tmp/nvd

from nist-data-mirror.