cve-search / cvexplore Goto Github PK

Several options / fields from the CLI search options are not parsed correctly and do not show (correct) results

While testing further on our other issue I have setup CveXplore on a second system with a different connection. This time via a proxy system. The good ... I was able to update the CVE database mostly. :)

I encountered the following errors:

2023-11-17 17:29:17,352 - CveXplore.database.maintenance.Sources_process - ERROR    - Exception encountered during the download from: https://cwe.mitre.org/data/xml/cwec_latest.xml.zip. Error encountered: 'lastModified'
2023-11-17 17:29:18,935 - CveXplore.database.maintenance.Sources_process - ERROR    - Exception encountered during the download from: https://capec.mitre.org/data/xml/capec_latest.xml. Error encountered: 'lastModified'
2023-11-17 17:29:28,744 - CveXplore.database.maintenance.Sources_process - ERROR    - Exception encountered during the download from: https://www.cve-search.org/feeds/via4.json. Error encountered: 'lastModified'

One little thing is looking quite strange. If I search for some CVEs the "vulnerable config" area is sometimes missing:

Other CVEs have included them:

If I search on one of my old systems (pre CveXplore) for the first CVE it looks correct:

@P-T-I could you do a quick check if the same thing appears on your side? Not sure if this is also based on some weird state or connection stuff of my environment.

https://github.com/fkie-cad/nvd-json-data-feeds

• #235
• #236
• #237

Open question; is the HTTP API as a datasource still needed or can we retire this funcitonality in light of the #213 roadmap

Hi, when running the update the following error happens:

python3.10 sbin/db_updater.py -v -c
2023-12-17 14:15:00,961 - DBUpdater - INFO     - ==========================
2023-12-17 14:15:00,961 - DBUpdater - INFO     - Sun 17 December 2023 13:15
2023-12-17 14:15:00,961 - DBUpdater - INFO     - ==========================
2023-12-17 14:15:00,975 - CveXplore.database.maintenance.main_updater - INFO     - Starting Database update....
2023-12-17 14:15:00,975 - CveXplore.database.maintenance.main_updater - INFO     - Starting Database update....
2023-12-17 14:15:00,979 - CveXplore.database.maintenance.Sources_process - INFO     - CPE database update started
2023-12-17 14:15:00,979 - CveXplore.database.maintenance.Sources_process - INFO     - CPE database update started
2023-12-17 14:15:00,980 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
2023-12-17 14:15:00,980 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
2023-12-17 14:15:01,519 - CveXplore.database.maintenance.Sources_process - INFO     - Preparing to download 1902 CPE entries
2023-12-17 14:15:01,519 - CveXplore.database.maintenance.Sources_process - INFO     - Preparing to download 1902 CPE entries
Downloading and processing content:   0%|                                                                                                                                                             | 0/1902 [00:02<?, ?it/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pymongo/collection.py", line 566, in bulk_write
    request._add_to_bulk(blk)
AttributeError: 'NoneType' object has no attribute '_add_to_bulk'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cve-search/sbin/db_updater.py", line 118, in <module>
    cvex.database.update()
  File "/usr/local/lib/python3.10/site-packages/CveXplore/database/maintenance/main_updater.py", line 73, in update
    up.update()
  File "/usr/local/lib/python3.10/site-packages/CveXplore/database/maintenance/Sources_process.py", line 269, in update
    self.process_downloads()
  File "/usr/local/lib/python3.10/site-packages/CveXplore/database/maintenance/Sources_process.py", line 248, in process_downloads
    self._db_bulk_writer(processed_items)
  File "/usr/local/lib/python3.10/site-packages/CveXplore/database/maintenance/DownloadHandler.py", line 173, in _db_bulk_writer
    self.database[self.feed_type.lower()].bulk_write(batch, ordered=False)
  File "/usr/local/lib/python3.10/site-packages/pymongo/_csot.py", line 108, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pymongo/collection.py", line 568, in bulk_write
    raise TypeError(f"{request!r} is not a valid request")
TypeError: None is not a valid request

I need some help with the following output:

Traceback (most recent call last):
File "/home/benedikt/embark/emba/external/emba_venv/bin/cvexplore", line 8, in
sys.exit(main())
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 1666, in invoke
rv = super().invoke(ctx)
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/CveXplore/cli_cmds/db_cmds/commands.py", line 28, in initialize_cmd
ctx.obj["data_source"].database.initialize()
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/CveXplore/database/maintenance/main_updater.py", line 167, in initialize
cve_pop.populate()
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/CveXplore/database/maintenance/Sources_process.py", line 904, in populate
self.process_downloads()
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/CveXplore/database/maintenance/Sources_process.py", line 796, in process_downloads
for item in data_list["vulnerabilities"]
TypeError: 'ServerDisconnectedError' object is not subscriptable

Version is:
CveXplore==0.3.17

Within the logs I see some "...forbidden..." message
cvexplore-init.log

Any Idea what could cause this?

It seems that the downloader crashes with the following exception:

I've located where it crashes, and it seems that the following method "self.process_item" returns None values:

These None values are possibly generated because of this method:

These None values are then passed into the "_db_bulk_writer" method which in turn crashes because a None value doesn't have the expected methods/attributes, see here:

It might be a good idea to filter out all the None values before passing the list into _db_bulk_writer like this:

processed_items = [i for i in processed_items if i is not None]

• Populate CVE / CPE
• Populate other sources
• Update CVE/CPE
• Update other sources

Should we consider merging to couchDB, or making CVE-Search compatible with both?
I have heard a lot of negative comments about Mongo, and it would be neat to give multiple database options.
Also, the database layer should be abstracted a lot more as well. (I can do this)
My idea would be to make a database abstraction layer, which implements functions for both mongo, couch, postgres,... (we could further this if we see fit), and then we might be able to change databases with the configuration files.
Your thoughts? @adulau @wimremes

Current cve parsing doesn’t handle the cvss parsing correctly; needs to be appended to pr cve-search/cve-search#1010

Currently CveXplore does not support the schema versioning of CveSearch; this needs to be copied / incorporated into the code base.

Should address cve-search/cve-search#1040

Hi, is there a reason the cve["configurations"] field is not stored? We would like to do some additional queries on this filed, as NIST is missing some CPEs and we want to work around that by doing a lookup on the cpe stem and then filter with some additional logic if the cpe with its version would match the candidates we got by searching for the stem.

Check cve-search/cve-search#929

Hi CveXplore team/users,

could it be that the new NIST API is unstable or has a very sensitive flooding protection?
I was not able to update my database during the last days. The db init is always running into Requests forbidden errors:

2023-11-24 14:32:32,208 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Sending request to url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=116000
2023-11-24 14:32:32,209 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Request forbidden by administrative rules
2023-11-24 14:32:32,209 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Finished request to url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=116000
2023-11-24 14:32:32,509 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Current backoff: 3.975021817053997
2023-11-24 14:32:32,510 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Retrying 4/5
2023-11-24 14:32:32,510 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Sending request to url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=112000
2023-11-24 14:32:32,510 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Request forbidden by administrative rules
2023-11-24 14:32:32,511 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Finished request to url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=112000
2023-11-24 14:32:32,811 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Current backoff: 1.7708387408932822
2023-11-24 14:32:32,811 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Retrying 4/5
2023-11-24 14:32:32,812 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Sending request to url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=152000
2023-11-24 14:32:32,812 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Request forbidden by administrative rules
2023-11-24 14:32:32,812 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Finished request to url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=152000
2023-11-24 14:32:33,113 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Current backoff: 2.578926801668418
<snip>
2023-11-24 14:32:45,815 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=112000 failed....
2023-11-24 14:32:45,815 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=114000 failed....
2023-11-24 14:32:45,815 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=116000 failed....
2023-11-24 14:32:45,815 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=118000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=120000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=122000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=124000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=126000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=128000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=130000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=132000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=134000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=136000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=138000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=140000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=142000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=144000 failed....
2023-11-24 14:32:45,817 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=146000 failed....
2023-11-24 14:32:45,817 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=148000 failed....
2023-11-24 14:32:45,817 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=150000 failed....
2023-11-24 14:32:45,817 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=152000 failed....

Finished up with an unpopulated database.

Is this a general issue with the new API?

• #214
• #215
• #216
• #217
• #218
• #219
• #222
• #223
• #224

Hi, it seems like there is a bug when initializing the DB:

root@ubuntu-16gb-hel1-1:~/test/cve-search# cvexplore database initialize
2023-12-19 18:11:46,126 - CveXplore.core.database_maintenance.main_updater - INFO     - Starting Database initialization....
2023-12-19 18:11:46,129 - CveXplore.core.database_maintenance.sources_process - INFO     - CPE Database population started
2023-12-19 18:11:46,130 - CveXplore.core.database_maintenance.sources_process - INFO     - Starting download...
2023-12-19 18:11:46,621 - CveXplore.core.database_maintenance.sources_process - INFO     - Preparing to download 1219300 CPE entries
Downloading and processing content:  37%|██████████████████████████▏                                            | 450000/1219300 [01:00<00:46, 16385.36it/s]

Downloading and processing content: 100%|███████████████████████████████████████████████████████████████████████| 1219300/1219300 [02:32<00:00, 8017.15it/s]
2023-12-19 18:14:18,714 - CveXplore.core.database_maintenance.sources_process - INFO     - Duration: 0:02:32.583559                                         
2023-12-19 18:14:25,154 - CveXplore.core.database_indexer.db_indexer - ERROR    - Failed to create index cpe on [('id', 1)]: Index build failed: 963294aa-e336-4fbf-9f2d-1549beba0601: Collection cvedb.cpe ( 0b920ea1-a3c2-4ea5-a1a0-3aa8c7385d45 ) :: caused by :: E11000 duplicate key error collection: cvedb.cpe index: id dup key: { id: "00000e6c6e8ab2551731d92cb4ede47930235e22" }, full error: {'ok': 0.0, 'errmsg': 'Index build failed: 963294aa-e336-4fbf-9f2d-1549beba0601: Collection cvedb.cpe ( 0b920ea1-a3c2-4ea5-a1a0-3aa8c7385d45 ) :: caused by :: E11000 duplicate key error collection: cvedb.cpe index: id dup key: { id: "00000e6c6e8ab2551731d92cb4ede47930235e22" }', 'code': 11000, 'codeName': 'DuplicateKey', 'keyPattern': {'id': 1}, 'keyValue': {'id': '00000e6c6e8ab2551731d92cb4ede47930235e22'}}

Normally the initialization should empty the DB before? Is this reproducible on your side?

• #231
• #232
• #234
• #242

• DATASOURCE_DBAPI should be described per datasource type (which are installed by default)
• #282
• #286

Hi @P-T-I ... lets catch this up again - #196

I have some news for you. The following screenshot is from a fresh CveXplore installation and looks quite good:

Afterwards we needed to downgrade urllib3 and requests to get docker up and running again - see here

With the new (old) versions of requests and urllib3 CveXplore does not work anymore:

Is there a way for you to get CveXplore also working with urllib3<2 and requests<2.29.0?

Search with search.py and default output gives only access to CVSSv2 values. To access the v3 rating json output is the way to go.
It would be helpful if there will be a switch or some other possiblity to also access v3 CVSS rating on other output formats.

Example:

└─$ ./cve-search/bin/search.py -p busybox
<snip>
CVE     : CVE-2022-30065
DATE    : 2022-05-18 15:15:00
CVSS    : 6.8
<snip>

The CVSS value is v2 value. See also https://nvd.nist.gov/vuln/detail/CVE-2022-30065

As json output also includes v3 values it would be great if the other output formats also show these values.

I have 2 error messages which (I think) are related:

1:

��# MONGODB_HOST=172.36.0.1 cvexplore database initialize
2023-11-16 11:39:07,677 - CveXplore.api.nvd_nist.nvd_nist_api - WARNING - Could not find a NIST API Key in the '~/.cvexplore/.env' file; you could request one at: https://nvd.nist.gov/developers/request-an-api-key
2023-11-16 11:39:07,678 - CveXplore.database.maintenance.Sources_process - INFO - CPE Database population started
2023-11-16 11:39:07,700 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('id', 1)] on cpe
2023-11-16 11:39:07,704 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vendor', 1)] on cpe
2023-11-16 11:39:07,708 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('product', 1)] on cpe
2023-11-16 11:39:07,713 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cpeNameId', 1)] on cpe
2023-11-16 11:39:07,717 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('deprecated', 1)] on cpe
2023-11-16 11:39:07,722 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cpeName', 1)] on cpe
2023-11-16 11:39:07,727 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('title', 1)] on cpe
2023-11-16 11:39:07,728 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-16 11:39:08,331 - CveXplore.database.maintenance.Sources_process - INFO - Preparing to download 1195622 CPE entries
Downloading and processing content: 0%| | 0/1195622 [03:50<?, ?it/s]
Traceback (most recent call last):
File "/home/cylox/emba/external/emba_venv/bin/cvexplore", line 8, in
sys.exit(main())
^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1666, in invoke
rv = super().invoke(ctx)
^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/cli_cmds/db_cmds/commands.py", line 28, in initialize_cmd
ctx.obj["data_source"].database.initialize()
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 159, in initialize
cpe_pop.populate()
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 237, in populate
self.process_downloads()
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 141, in process_downloads
for item in data_list["products"]
~~~~~~~~~^^^^^^^^^^^^
TypeError: 'ServerTimeoutError' object is not subscriptable

And the second
2:

└─$ MONGODB_HOST=172.36.0.1 ./external/cve-search/sbin/db_updater.py -fv
Warning! Could not write log to /home/cylox/emba/emba/external/cve-search/log/update_populate.log. Disabling temporarily.
2023-11-15 04:22:41,284 - DBUpdater - INFO - Dropping metadata
2023-11-15 04:22:41,289 - DBUpdater - INFO - ==========================
2023-11-15 04:22:41,289 - DBUpdater - INFO - Wed 15 November 2023 14:22
2023-11-15 04:22:41,290 - DBUpdater - INFO - ==========================
2023-11-15 04:22:41,313 - CveXplore.database.maintenance.Sources_process - INFO - CPE Database population started
2023-11-15 04:22:41,313 - CveXplore.database.maintenance.Sources_process - INFO - CPE Database population started
2023-11-15 04:22:41,370 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('id', 1)] on cpe
2023-11-15 04:22:41,370 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('id', 1)] on cpe
2023-11-15 04:22:41,375 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vendor', 1)] on cpe
2023-11-15 04:22:41,375 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vendor', 1)] on cpe
2023-11-15 04:22:41,380 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('product', 1)] on cpe
2023-11-15 04:22:41,380 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('product', 1)] on cpe
2023-11-15 04:22:41,385 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cpeNameId', 1)] on cpe
2023-11-15 04:22:41,385 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cpeNameId', 1)] on cpe
2023-11-15 04:22:41,390 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('deprecated', 1)] on cpe
2023-11-15 04:22:41,390 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('deprecated', 1)] on cpe
2023-11-15 04:22:41,395 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cpeName', 1)] on cpe
2023-11-15 04:22:41,395 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cpeName', 1)] on cpe
2023-11-15 04:22:41,400 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('title', 1)] on cpe
2023-11-15 04:22:41,400 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('title', 1)] on cpe
2023-11-15 04:22:41,401 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-15 04:22:41,401 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-15 04:22:42,382 - CveXplore.database.maintenance.Sources_process - INFO - Preparing to download 1195362 CPE entries
2023-11-15 04:22:42,382 - CveXplore.database.maintenance.Sources_process - INFO - Preparing to download 1195362 CPE entries
Downloading and processing content: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1195362/1195362 [04:30<00:00, 4411.87it/s]
2023-11-15 04:27:13,352 - CveXplore.database.maintenance.Sources_process - INFO - Duration: 0:04:31.951544
2023-11-15 04:27:13,352 - CveXplore.database.maintenance.Sources_process - INFO - Duration: 0:04:31.951544
2023-11-15 04:27:13,634 - CveXplore.database.maintenance.Sources_process - INFO - Finished CPE database population
2023-11-15 04:27:13,634 - CveXplore.database.maintenance.Sources_process - INFO - Finished CPE database population
2023-11-15 04:27:13,634 - CveXplore.database.maintenance.main_updater - INFO - Sleeping for 30 seconds between CPE and CVE database population..
2023-11-15 04:27:13,634 - CveXplore.database.maintenance.main_updater - INFO - Sleeping for 30 seconds between CPE and CVE database population..
2023-11-15 04:27:43,640 - CveXplore.database.maintenance.Sources_process - INFO - CVE database population started
2023-11-15 04:27:43,640 - CveXplore.database.maintenance.Sources_process - INFO - CVE database population started
2023-11-15 04:27:43,640 - CveXplore.database.maintenance.Sources_process - INFO - Starting CVE database population starting from year: 2002
2023-11-15 04:27:43,640 - CveXplore.database.maintenance.Sources_process - INFO - Starting CVE database population starting from year: 2002
2023-11-15 04:27:43,757 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('id', 1)] on cves
2023-11-15 04:27:43,757 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('id', 1)] on cves
2023-11-15 04:27:43,762 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_configuration', 1)] on cves
2023-11-15 04:27:43,762 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_configuration', 1)] on cves
2023-11-15 04:27:43,766 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_product', 1)] on cves
2023-11-15 04:27:43,766 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_product', 1)] on cves
2023-11-15 04:27:43,772 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('modified', 1)] on cves
2023-11-15 04:27:43,772 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('modified', 1)] on cves
2023-11-15 04:27:43,776 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('published', 1)] on cves
2023-11-15 04:27:43,776 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('published', 1)] on cves
2023-11-15 04:27:43,781 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('lastModified', 1)] on cves
2023-11-15 04:27:43,781 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('lastModified', 1)] on cves
2023-11-15 04:27:43,786 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cvss', 1)] on cves
2023-11-15 04:27:43,786 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cvss', 1)] on cves
2023-11-15 04:27:43,791 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cvss3', 1)] on cves
2023-11-15 04:27:43,791 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cvss3', 1)] on cves
2023-11-15 04:27:43,796 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('summary', 'text')] on cves
2023-11-15 04:27:43,796 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('summary', 'text')] on cves
2023-11-15 04:27:43,804 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vendors', 1)] on cves
2023-11-15 04:27:43,804 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vendors', 1)] on cves
2023-11-15 04:27:43,810 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('products', 1)] on cves
2023-11-15 04:27:43,810 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('products', 1)] on cves
2023-11-15 04:27:43,816 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('assigner', 1)] on cves
2023-11-15 04:27:43,816 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('assigner', 1)] on cves
2023-11-15 04:27:43,821 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cwe', 1)] on cves
2023-11-15 04:27:43,821 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cwe', 1)] on cves
2023-11-15 04:27:43,826 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('status', 1)] on cves
2023-11-15 04:27:43,826 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('status', 1)] on cves
2023-11-15 04:27:43,831 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_product_stems', 1)] on cves
2023-11-15 04:27:43,831 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_product_stems', 1)] on cves
2023-11-15 04:27:43,836 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_configuration_stems', 1)] on cves
2023-11-15 04:27:43,836 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_configuration_stems', 1)] on cves
2023-11-15 04:27:43,836 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-15 04:27:43,836 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-15 04:27:44,991 - CveXplore.database.maintenance.Sources_process - INFO - Preparing to download 217132 CVE entries
2023-11-15 04:27:44,991 - CveXplore.database.maintenance.Sources_process - INFO - Preparing to download 217132 CVE entries
Downloading and processing content: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 217132/217132 [05:41<00:00, 636.35it/s]
2023-11-15 04:33:26,206 - CveXplore.database.maintenance.Sources_process - INFO - Duration: 0:05:42.370443
2023-11-15 04:33:26,206 - CveXplore.database.maintenance.Sources_process - INFO - Duration: 0:05:42.370443
2023-11-15 04:33:26,555 - CveXplore.database.maintenance.Sources_process - INFO - Finished CVE database population
2023-11-15 04:33:26,555 - CveXplore.database.maintenance.Sources_process - INFO - Finished CVE database population
2023-11-15 04:33:26,597 - CveXplore.database.maintenance.Sources_process - INFO - CPE database update started
2023-11-15 04:33:26,597 - CveXplore.database.maintenance.Sources_process - INFO - CPE database update started
2023-11-15 04:33:26,598 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-15 04:33:26,598 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
Traceback (most recent call last):
File "/home/cylox/emba/emba/./external/cve-search/sbin/db_updater.py", line 122, in
cvex.database.initialize()
File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 169, in initialize
self.update()
File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 67, in update
up.update()
File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 218, in update
self.process_downloads()
File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 168, in process_downloads
total_results = self.api_handler.get_count(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/api/nvd_nist/nvd_nist_api.py", line 196, in get_count
raise ApiDataRetrievalFailed
CveXplore.errors.apis.ApiDataRetrievalFailed

Generally I'm having problems to get the update/initialization to work on my systems.

My assumption at the moment is that we get a weird response from the nist api endpoint.

in get_count()
(or somewhere else?) I fail to receive the proper response

Regen search on field

Hi, I've encountered an inconsistency while using the CVE search command. Executing cvexplore cve search -c 2001-1382 -f id -f vulnerable_configuration yields results that include entries not directly relevant to the specified CVE, such as cpe:2.3:a:openbsd:openssh:8.9:-:*:*:*:*:*:*

To give you more context, these are the responses from NIST for comparison:

• CVE-2001-1382 Details
• Openssh 8.9 Entry Details

Inspected:

def padded_version(version: str):
        ret_list = []
        try:
            for v in version.split("."):
                ret_list.append(f"{int(v):05d}")
        except ValueError:
            return version
        return ".".join(ret_list)

returns "2.9.9p2" because of ValueError.

Afterwards in getCPEVersionInformation this matches lots of CPEs that should not be matched.

Hello

I install this tool now, how can I get last cves by vendor?

Coming from cve-search/cpe-guesser#1

During testing the new CveXpore plugin I received the following error for multiple times now:

└─$ cvexplore database update      
2023-11-16 07:05:36,258 - CveXplore.api.nvd_nist.nvd_nist_api - WARNING  - Could not find a NIST API Key in the '~/.cvexplore/.env' file; you could request one at: https://nvd.nist.gov/developers/request-an-api-key                                                                            
2023-11-16 07:05:36,261 - CveXplore.database.maintenance.Sources_process - INFO     - CPE database update started
2023-11-16 07:05:36,263 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
2023-11-16 07:05:37,609 - CveXplore.database.maintenance.Sources_process - INFO     - Preparing to download 0 CPE entries
Downloading and processing content: 0it [00:00, ?it/s]
2023-11-16 07:05:38,209 - CveXplore.database.maintenance.Sources_process - INFO     - Duration: 0:00:01.945437
2023-11-16 07:05:38,210 - CveXplore.database.maintenance.Sources_process - INFO     - Finished CPE database update
2023-11-16 07:05:38,224 - CveXplore.api.nvd_nist.nvd_nist_api - WARNING  - Could not find a NIST API Key in the '~/.cvexplore/.env' file; you could request one at: https://nvd.nist.gov/developers/request-an-api-key                                                                            
2023-11-16 07:05:38,225 - CveXplore.database.maintenance.Sources_process - INFO     - CVE database update started
2023-11-16 07:05:38,226 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
Traceback (most recent call last):
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/api/nvd_nist/nvd_nist_api.py", line 116, in _connect
    the_response = json.loads(r.text)
                   ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/api/nvd_nist/nvd_nist_api.py", line 121, in _connect
    if r.headers["content-type"] == "text/plain":
       ~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/requests/structures.py", line 52, in __getitem__
    return self._store[key.lower()][1]
           ~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'content-type'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kali/emba/external/emba_venv/bin/cvexplore", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1666, in invoke
    rv = super().invoke(ctx)
         ^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/cli_cmds/db_cmds/commands.py", line 22, in update_cmd
    ctx.obj["data_source"].database.update()
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 67, in update
    up.update()
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 851, in update
    self.process_downloads()
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 801, in process_downloads
    total_results = self.api_handler.get_count(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/api/nvd_nist/nvd_nist_api.py", line 188, in get_count
    ret_data = self.call(self.methods.GET, resource=resource, data=datasource)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/api/api_base_class.py", line 163, in call
    result = self._connect(
             ^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/api/nvd_nist/nvd_nist_api.py", line 131, in _connect
    raise Exception(err)
Exception: 'content-type'

The purpose of this issue is to propose an enhancement for the CveXplore project. The enhancement involves the integration of an exploit prediction scoring system, which would serve to enrich the CVE data with predictive insights regarding the potential for exploitation.

Advantages:

• Enhanced risk assessment capabilities for end-users, providing a clearer understanding of the exploitability of vulnerabilities.
• Improved prioritization of vulnerabilities to patch, based on a quantifiable exploitation likelihood score.

• #240
• #241
• #243
• #245

There are differences between the cli and the package; cli depends completely in .env file; package should depend on given parameters on lib instantiation; for the package those two should be merged and passed down to child classes