"Add db for version maps" introduced massive performance regression about cve-bin-tool HOT 13 CLOSED

Putting it in the same database means you can't have any problems with the two being out of sync.

from cve-bin-tool.

Fuck, we'll get on this soon

from cve-bin-tool.

I will check this today.
But I think data is cached but function is called many times. The function is such that if called will not necessarily update the database.

from cve-bin-tool.

from cve-bin-tool.

from cve-bin-tool.

@rossburton I checked the code. There was nothing wrong with the code. The database exists and data is cached.
The current implementation waits 30 days from a release before downloading new data. Since 32 days have passed since the last release date and yet there is no release, it fetches the data again. It will continue to fetch as long as new release does not happen.
Probably setting it to 60 days may be safe.
So it does not cause any performance problems unless of course a new version is not released within the refresh time.

from cve-bin-tool.

I put a print() in sqlite.py:

diff --git a/cve_bin_tool/checkers/sqlite.py b/cve_bin_tool/checkers/sqlite.py
index 7b59a91..e818cc4 100644
--- a/cve_bin_tool/checkers/sqlite.py
+++ b/cve_bin_tool/checkers/sqlite.py
@@ -86,6 +86,7 @@ def get_version_map():

     changeurl = "https://www.sqlite.org/changes.html"
     try:
+        print("Fetching %s" % changeurl)
         response = request.urlopen(changeurl)
     except error.URLError as err:
         print("Could not fetch " + changeurl + ", " + err)

Then run cve-bin-tool:

$ cve-bin-tool -xv /usr/ -l debug
Connecting to NVD database and extracting the CVE list ... Please hold on.. This will take few minutes...
Last Update: 2019-03-11
Local database has been updated in the past 24h.
New data not downloaded.  Remove old files present at /home/ross/.cache/cve-bin-tool to force the update.
INFO:cve_bin_tool.Scanner:Checkers loaded: png, xml2, libnss, expat, libjpeg, libgcrypt, libcurl, curl, node, libtiff, openssl, kerberos, xerces, sqlite, zlib, systemd
DEBUG:cve_bin_tool.Scanner:Scanning file: '/usr/games/espdiff'
DEBUG:cve_bin_tool.Scanner:Scanning file: '/usr/sbin/biosdecode'
Fetching https://www.sqlite.org/changes.html
DEBUG:cve_bin_tool.Scanner:Scanning file: '/usr/sbin/grpck'
Fetching https://www.sqlite.org/changes.html
DEBUG:cve_bin_tool.Scanner:Scanning file: '/usr/sbin/tcpdchk'
Fetching https://www.sqlite.org/changes.html
DEBUG:cve_bin_tool.Scanner:Scanning file: '/usr/sbin/chroot.fakechroot'
DEBUG:cve_bin_tool.Scanner:Scanning file: '/usr/sbin/samba_kcc'
DEBUG:cve_bin_tool.Scanner:Scanning file: '/usr/sbin/blkmapd'
Fetching https://www.sqlite.org/changes.html
DEBUG:cve_bin_tool.Scanner:Scanning file: '/usr/sbin/logrotate'
Fetching https://www.sqlite.org/changes.html

It is downloading from sqlite.org for almost every file that is being scanned.

from cve-bin-tool.

I think you're misunderstanding. Almost every file, sqlite.py is downloading changes.html. This is not a working cache.

from cve-bin-tool.

Then the database has to be made more automatic.
Last update date must be saved and update should be done only after 30 days from when the database was last updated.

from cve-bin-tool.

You can set the number to say 40 and 60 in mapdb = VersionSignatureDb('sqlite', get_version_map, 40) in sqlite.py then it does not download everytime.

from cve-bin-tool.

Yes. The last release was 31 days ago, the code should be checking how long since it last checked instead.

Also, the refresh logic where it calculates the difference is pretty poor python. If you subtract one datetime from another the results is a timedelta which has a days attribute: no need to mess around with str() and split(). Just store the current datetime in the database, and compare that with now.

from cve-bin-tool.

The refresh logic is redundant, I did not know the available methods in datetime or timedelta. Thanks for the docs. 😊

New Lesson: Check the docs to see if a method already exists. Thanks

from cve-bin-tool.

@rossburton Should I store the current date (recent database update date) in a new database or store it in the current database as a singe entry ?

from cve-bin-tool.