Avoid full file downloads in sources_process about cvexplore HOT 3 CLOSED

This would be a nice addition! Regarding saving the header to file; we could save it to the info collection of the database as well; saves that hassle of dealing with files.

from cvexplore.

I was thinking of a rather simple solution, if you don't find it too quick and dirty:

• Use a JSON based header cache saved as a file in working directory.
• Skip download if both Last-Modified and Content-Length headers are the same. No need for actually comparing the values even.
• If anything goes wrong, log the warnings/error and return True for "the file probably has changes" & let the current logic try the downloads and catch any errors. The same approach can cover both internal errors with the cache file loads, writes & contents as well as external errors with the HTTP communications.
• Only enable this check on certain data sources (CPE, CAPEC, VIA4) and keep the current functionality for EPSS.

Note to self:

• The repopulation probably needs to either delete or empty this file.

from cvexplore.

Thanks for the hint. Because the info collection already had this information cached we didn't even need additional caches, but could simply compare the value in there. That made adding the new functionality rather straightforward. Please review the pull request.

After this I can simply lower the update interval on CVE-Searche's SystemD timer from 2 hours to 1 hour, which would also be the same as the sleep time in the db_updater.py -v -l (loop mode). 👍

from cvexplore.