Comments (3)
This would be a nice addition! Regarding saving the header to file; we could save it to the info collection of the database as well; saves that hassle of dealing with files.
from cvexplore.
I was thinking of a rather simple solution, if you don't find it too quick and dirty:
- Use a JSON based header cache saved as a file in working directory.
- Skip download if both
Last-Modified
andContent-Length
headers are the same. No need for actually comparing the values even. - If anything goes wrong, log the warnings/error and return
True
for "the file probably has changes" & let the current logic try the downloads and catch any errors. The same approach can cover both internal errors with the cache file loads, writes & contents as well as external errors with the HTTP communications. - Only enable this check on certain data sources (CPE, CAPEC, VIA4) and keep the current functionality for EPSS.
Note to self:
- The repopulation probably needs to either delete or empty this file.
from cvexplore.
Thanks for the hint. Because the info
collection already had this information cached we didn't even need additional caches, but could simply compare the value in there. That made adding the new functionality rather straightforward. Please review the pull request.
After this I can simply lower the update interval on CVE-Searche's SystemD timer from 2 hours to 1 hour, which would also be the same as the sleep time in the db_updater.py -v -l
(loop mode). 👍
from cvexplore.
Related Issues (20)
- Make Sem-factor configurable and add random sleep time between requests
- Bug in bulk write mongodb HOT 8
- Logging to update_populate log stops when downloading cves
- Bug with OpenSSL versions HOT 5
- Database Updates fail when filtering out Deprecated CPEs HOT 4
- alembic check should not take place if datasource_type != "mysql"
- TypeError: unsupported operand type(s) for |: 'type' and 'type' HOT 3
- CPEs with ':' in the product or vendor are parsed incorrectly HOT 2
- Missing vulnerable_configurations on some some CVEs HOT 7
- Fix for #268 HOT 3
- Create possibility to populate a single entry (cpe or cve) for testing purposes HOT 3
- Update database model
- Background celery runners HOT 2
- Create cli functions for task creation, status checks etc
- Update package documentation HOT 1
- Celery Backend tasks should be documented and installation procedure for docker and systemd described HOT 1
- Create tasks for auto update
- Create tasks for reprocessing cve's on cpe update
- Create database schema documentation HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cvexplore.