Giter VIP home page Giter VIP logo

cvexplore's Introduction

cve-search

Join the chat at https://gitter.im/cve-search/cve-search Build & Test Black formatting CodeQL

cve-search logo

cve-search is a tool to import CVE (Common Vulnerabilities and Exposures) and CPE (Common Platform Enumeration) into a MongoDB to facilitate search and processing of CVEs.

The main objective of the software is to avoid doing direct and public lookups into the public CVE databases. Local lookups are usually faster and you can limit your sensitive queries via the Internet.

cve-search includes a back-end to store vulnerabilities and related information, an intuitive web interface for search and managing vulnerabilities, a series of tools to query the system and a web API interface.

cve-search is used by many organizations including the public CVE services of CIRCL.

This document gives you basic information how to start with cve-search. For more information please refer to the documentation in the /doc folder of this project.

Getting started

Check the documentation to get you started

Usage

You can search the database using search.py.

usage: search.py [-h] [-q Q] [-p P [P ...]] [--only-if-vulnerable] [--strict_vendor_product] [--lax] [-f F] [-c C] [-o O]
                 [-l] [-n] [-r] [-a] [-v V] [-s S] [-t T] [-i I]

Search for vulnerabilities in the National Vulnerability DB. Data from http://nvd.nist.org.

options:
  -h, --help            show this help message and exit
  -q Q                  Q = search pip requirements file for CVEs, e.g. dep/myreq.txt
  -p P [P ...]          S = search one or more products, e.g. o:microsoft:windows_7 or o:cisco:ios:12.1 or
                        o:microsoft:windows_7 o:cisco:ios:12.1. Add --only-if-vulnerable if only vulnerabilities that
                        directly affect the product are wanted.
  --only-if-vulnerable  With this option, "-p" will only return vulnerabilities directly assigned to the product. I.e.
                        it will not consider "windows_7" if it is only mentioned as affected OS in an adobe:reader
                        vulnerability.
  --strict_vendor_product
                        With this option, a strict vendor product search is executed. The values in "-p" should be
                        formatted as vendor:product, e.g. microsoft:windows_7
  --lax                 Strict search for software version is disabled. Likely gives false positives for earlier
                        versions that were not yet vulnerable. Note that version comparison for non-numeric values
                        is done with simplifications.
  -f F                  F = free text search in vulnerability summary
  -c C                  search one or more CVE-ID
  -o O                  O = output format [csv|html|json|xml|cveid]
  -l                    sort in descending mode
  -n                    lookup complete cpe (Common Platform Enumeration) name for vulnerable configuration
  -r                    lookup ranking of vulnerable configuration
  -a                    Lookup CAPEC for related CWE weaknesses
  -v V                  vendor name to lookup in reference URLs
  -s S                  search in summary text
  -t T                  search in last n day
  -i I                  Limit output to n elements (default: unlimited)

Examples:

./bin/search.py -p cisco:ios:12.4
./bin/search.py -p cisco:ios:12.4 -o json
./bin/search.py -f nagios -n
./bin/search.py -p microsoft:windows_7 -o html

If you want to search all the WebEx vulnerabilities and only printing the official references from the supplier.

./bin/search.py -p webex: -o csv  -v "cisco"

You can also dump the JSON for a specific CVE ID.

./bin/search.py -c CVE-2010-3333 -o json

Or dump the last 2 CVE entries in RSS or Atom format.

./bin/dump_last.py -f atom -l 2

Or you can use the webinterface.

./web/index.py

Usage of the ranking database

There is a ranking database allowing to rank software vulnerabilities based on their common platform enumeration name. The ranking can be done per organization or department within your organization or any meaningful name for you.

As an example, you can add a partial CPE name like "sap:netweaver" which is very critical for your accounting department.

./sbin/db_ranking.py  -c "sap:netweaver" -g "accounting" -r 3

and then you can lookup the ranking (-r option) for a specific CVE-ID:

./bin/search.py -c CVE-2012-4341  -r  -n

Advanced usage

As cve-search is based on a set of tools, it can be used and combined with standard Unix tools. If you ever wonder what are the top vendors using the term "unknown" for their vulnerabilities:

python3 bin/search_fulltext.py -q unknown -f \
    | jq -c '. | .vulnerable_configuration[0]' \
    | cut -f5 -d: | sort  | uniq -c  | sort -nr | head -10

1500 oracle
381 sun
372 hp
232 google
208 ibm
126 mozilla
103 microsoft
100 adobe
 78 apple
 68 linux

You can compare CVSS (Common Vulnerability Scoring System ) values of some products based on their CPE name. Like comparing oracle:java versus sun:jre and using R to make some statistics about their CVSS values:

python3 bin/search.py -p oracle:java -o json \
  | jq -r '.cvss' | Rscript -e 'summary(as.numeric(read.table(file("stdin"))[,1]))'

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
1.800   5.350   9.300   7.832  10.000  10.000
python3 bin/search.py -p sun:jre -o json \
  | jq -r '.cvss' | Rscript -e 'summary(as.numeric(read.table(file("stdin"))[,1]))'

Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.000   5.000   7.500   7.333  10.000  10.000

Fulltext indexing

If you want to index all the CVEs from your current MongoDB collection:

./sbin/db_fulltext.py -l 0

and you query the fulltext index (to get a list of matching CVE-ID):

./bin/search_fulltext.py -q NFS -q Linux

or to query the fulltext index and output the JSON object for each CVE-ID:

./bin/search_fulltext.py -q NFS -q Linux -f

Fulltext visualization

The fulltext indexer visualization is using the fulltext indexes to build a list of the most common keywords used in CVE. NLTK is required to generate the keywords with the most common English stopwords and lemmatize the output. NTLK for Python 3 exists but you need to use the alpha version of NLTK.

./bin/search_fulltext.py  -g -s >cve.json

cve-search visualization

You can see a visualization on the demo site.

Web interface

The web interface is a minimal interface to see the last CVE entries and query a specific CVE. You'll need flask in order to run the website and Flask-PyMongo. To start the web interface:

cd ./web
./index.py

Then you can connect on http://127.0.0.1:5000/ to browser the last CVE.

Web API interface

The web interface includes a minimal JSON API to get CVE by ID, by vendor or product. A public version of the API is also accessible on cve.circl.lu.

List the know vendors in JSON

curl "http://127.0.0.1:5000/api/browse/"

Dump the product of a specific vendor in JSON

curl "http://127.0.0.1:5000/api/browse/zyxel"
{
  "product": [
    "n300_netusb_nbg-419n",
    "n300_netusb_nbg-419n_firmware",
    "p-660h-61",
    "p-660h-63",
    "p-660h-67",
    "p-660h-d1",
    "p-660h-d3",
    "p-660h-t1",
    "p-660h-t3",
    "p-660hw",
    "p-660hw_d1",
    "p-660hw_d3",
    "p-660hw_t3"
  ],
  "vendor": "zyxel"
}

Find the associated vulnerabilities to a vendor and a product.

curl "http://127.0.0.1:5000/api/search/zyxel/p-660hw" | jq .
[
  {
    "cwe": "CWE-352",
    "references": [
      "http://www.exploit-db.com/exploits/33518",
      "http://secunia.com/advisories/58513",
      "http://packetstormsecurity.com/files/126812/Zyxel-P-660HW-T1-Cross-Site-Request-Forgery.html",
      "http://osvdb.org/show/osvdb/107449"
    ],
    "vulnerable_configuration": [
      "cpe:/h:zyxel:p-660hw:_t1:v3"
    ],
    "Published": "2014-06-16T14:55:09.713-04:00",
    "id": "CVE-2014-4162",
    "Modified": "2014-07-17T01:07:29.683-04:00",
    "cvss": 6.8,
    "summary": "Multiple cross-site request forgery (CSRF) vulnerabilities in the Zyxel P-660HW-T1 (v3) wireless router allow remote attackers to hijack the authentication of administrators for requests that change the (1) wifi password or (2) SSID via a request to Forms/WLAN_General_1."
  },
  {
    "cwe": "CWE-20",
    "references": [
      "http://www.kb.cert.org/vuls/id/893726"
    ],
    "vulnerable_configuration": [
      "cpe:/h:zyxel:p-660h-63:-",
      "cpe:/h:zyxel:p-660h-t1:-",
      "cpe:/h:zyxel:p-660h-d3:-",
      "cpe:/h:zyxel:p-660h-t3:v2",
      "cpe:/h:zyxel:p-660h-t1:v2",
      "cpe:/h:zyxel:p-660h-d1:-",
      "cpe:/h:zyxel:p-660h-67:-",
      "cpe:/h:zyxel:p-660h-61:-",
      "cpe:/h:zyxel:p-660hw_t3:v2",
      "cpe:/h:zyxel:p-660hw_t3:-",
      "cpe:/h:zyxel:p-660hw_d3:-",
      "cpe:/h:zyxel:p-660hw_d1:v2",
      "cpe:/h:zyxel:p-660hw_d1:-",
      "cpe:/h:zyxel:p-660hw:_t1:v2",
      "cpe:/h:zyxel:p-660hw:_t1:-"
    ],

Software using cve-search

Docker versions

Official dockerized version of cve-search:

CVE-Search-Docker

There are some unofficial dockerized versions of cve-search (which are not maintained by us):

Changelog

You can find the changelog on GitHub Releases (legacy changelog).

License

cve-search is free software released under the "GNU Affero General Public License v3.0"

Copyright (c) 2012 Wim Remes - https://github.com/wimremes/
Copyright (c) 2012-2024 Alexandre Dulaunoy - https://github.com/adulau/
Copyright (c) 2015-2019 Pieter-Jan Moreels - https://github.com/pidgeyl/
Copyright (c) 2020-2024 Paul Tikken - https://github.com/P-T-I

cvexplore's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar

cvexplore's Issues

NIST API reachability

Hi CveXplore team/users,

could it be that the new NIST API is unstable or has a very sensitive flooding protection?
I was not able to update my database during the last days. The db init is always running into Requests forbidden errors:

2023-11-24 14:32:32,208 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Sending request to url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=116000
2023-11-24 14:32:32,209 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Request forbidden by administrative rules
2023-11-24 14:32:32,209 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Finished request to url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=116000
2023-11-24 14:32:32,509 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Current backoff: 3.975021817053997
2023-11-24 14:32:32,510 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Retrying 4/5
2023-11-24 14:32:32,510 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Sending request to url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=112000
2023-11-24 14:32:32,510 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Request forbidden by administrative rules
2023-11-24 14:32:32,511 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Finished request to url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=112000
2023-11-24 14:32:32,811 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Current backoff: 1.7708387408932822
2023-11-24 14:32:32,811 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Retrying 4/5
2023-11-24 14:32:32,812 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Sending request to url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=152000
2023-11-24 14:32:32,812 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Request forbidden by administrative rules
2023-11-24 14:32:32,812 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Finished request to url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=152000
2023-11-24 14:32:33,113 - CveXplore.api.nvd_nist.nvd_nist_api - DEBUG    - Current backoff: 2.578926801668418
<snip>
2023-11-24 14:32:45,815 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=112000 failed....
2023-11-24 14:32:45,815 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=114000 failed....
2023-11-24 14:32:45,815 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=116000 failed....
2023-11-24 14:32:45,815 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=118000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=120000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=122000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=124000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=126000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=128000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=130000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=132000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=134000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=136000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=138000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=140000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=142000 failed....
2023-11-24 14:32:45,816 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=144000 failed....
2023-11-24 14:32:45,817 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=146000 failed....
2023-11-24 14:32:45,817 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=148000 failed....
2023-11-24 14:32:45,817 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=150000 failed....
2023-11-24 14:32:45,817 - CveXplore.database.maintenance.Sources_process - ERROR    - Retrieval of api data on url: https://services.nvd.nist.gov/rest/json/cves/2.0/?noRejected&startIndex=152000 failed....

Finished up with an unpopulated database.

Is this a general issue with the new API?

Bug in bulk write mongodb

It seems that the downloader crashes with the following exception:
image

I've located where it crashes, and it seems that the following method "self.process_item" returns None values:

These None values are possibly generated because of this method:

def process_the_item(self, item: dict = None):

These None values are then passed into the "_db_bulk_writer" method which in turn crashes because a None value doesn't have the expected methods/attributes, see here:

self._db_bulk_writer(processed_items)

It might be a good idea to filter out all the None values before passing the list into _db_bulk_writer like this:

processed_items = [i for i in processed_items if i is not None]

Connection errors lead to failed initialization

I have 2 error messages which (I think) are related:

1:

��# MONGODB_HOST=172.36.0.1 cvexplore database initialize
2023-11-16 11:39:07,677 - CveXplore.api.nvd_nist.nvd_nist_api - WARNING - Could not find a NIST API Key in the '~/.cvexplore/.env' file; you could request one at: https://nvd.nist.gov/developers/request-an-api-key
2023-11-16 11:39:07,678 - CveXplore.database.maintenance.Sources_process - INFO - CPE Database population started
2023-11-16 11:39:07,700 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('id', 1)] on cpe
2023-11-16 11:39:07,704 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vendor', 1)] on cpe
2023-11-16 11:39:07,708 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('product', 1)] on cpe
2023-11-16 11:39:07,713 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cpeNameId', 1)] on cpe
2023-11-16 11:39:07,717 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('deprecated', 1)] on cpe
2023-11-16 11:39:07,722 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cpeName', 1)] on cpe
2023-11-16 11:39:07,727 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('title', 1)] on cpe
2023-11-16 11:39:07,728 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-16 11:39:08,331 - CveXplore.database.maintenance.Sources_process - INFO - Preparing to download 1195622 CPE entries
Downloading and processing content: 0%| | 0/1195622 [03:50<?, ?it/s]
Traceback (most recent call last):
File "/home/cylox/emba/external/emba_venv/bin/cvexplore", line 8, in
sys.exit(main())
^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1666, in invoke
rv = super().invoke(ctx)
^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/cli_cmds/db_cmds/commands.py", line 28, in initialize_cmd
ctx.obj["data_source"].database.initialize()
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 159, in initialize
cpe_pop.populate()
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 237, in populate
self.process_downloads()
File "/home/cylox/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 141, in process_downloads
for item in data_list["products"]
~~~~~~~~~^^^^^^^^^^^^
TypeError: 'ServerTimeoutError' object is not subscriptable

And the second
2:

└─$ MONGODB_HOST=172.36.0.1 ./external/cve-search/sbin/db_updater.py -fv
Warning! Could not write log to /home/cylox/emba/emba/external/cve-search/log/update_populate.log. Disabling temporarily.
2023-11-15 04:22:41,284 - DBUpdater - INFO - Dropping metadata
2023-11-15 04:22:41,289 - DBUpdater - INFO - ==========================
2023-11-15 04:22:41,289 - DBUpdater - INFO - Wed 15 November 2023 14:22
2023-11-15 04:22:41,290 - DBUpdater - INFO - ==========================
2023-11-15 04:22:41,313 - CveXplore.database.maintenance.Sources_process - INFO - CPE Database population started
2023-11-15 04:22:41,313 - CveXplore.database.maintenance.Sources_process - INFO - CPE Database population started
2023-11-15 04:22:41,370 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('id', 1)] on cpe
2023-11-15 04:22:41,370 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('id', 1)] on cpe
2023-11-15 04:22:41,375 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vendor', 1)] on cpe
2023-11-15 04:22:41,375 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vendor', 1)] on cpe
2023-11-15 04:22:41,380 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('product', 1)] on cpe
2023-11-15 04:22:41,380 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('product', 1)] on cpe
2023-11-15 04:22:41,385 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cpeNameId', 1)] on cpe
2023-11-15 04:22:41,385 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cpeNameId', 1)] on cpe
2023-11-15 04:22:41,390 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('deprecated', 1)] on cpe
2023-11-15 04:22:41,390 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('deprecated', 1)] on cpe
2023-11-15 04:22:41,395 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cpeName', 1)] on cpe
2023-11-15 04:22:41,395 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cpeName', 1)] on cpe
2023-11-15 04:22:41,400 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('title', 1)] on cpe
2023-11-15 04:22:41,400 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('title', 1)] on cpe
2023-11-15 04:22:41,401 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-15 04:22:41,401 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-15 04:22:42,382 - CveXplore.database.maintenance.Sources_process - INFO - Preparing to download 1195362 CPE entries
2023-11-15 04:22:42,382 - CveXplore.database.maintenance.Sources_process - INFO - Preparing to download 1195362 CPE entries
Downloading and processing content: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1195362/1195362 [04:30<00:00, 4411.87it/s]
2023-11-15 04:27:13,352 - CveXplore.database.maintenance.Sources_process - INFO - Duration: 0:04:31.951544
2023-11-15 04:27:13,352 - CveXplore.database.maintenance.Sources_process - INFO - Duration: 0:04:31.951544
2023-11-15 04:27:13,634 - CveXplore.database.maintenance.Sources_process - INFO - Finished CPE database population
2023-11-15 04:27:13,634 - CveXplore.database.maintenance.Sources_process - INFO - Finished CPE database population
2023-11-15 04:27:13,634 - CveXplore.database.maintenance.main_updater - INFO - Sleeping for 30 seconds between CPE and CVE database population..
2023-11-15 04:27:13,634 - CveXplore.database.maintenance.main_updater - INFO - Sleeping for 30 seconds between CPE and CVE database population..
2023-11-15 04:27:43,640 - CveXplore.database.maintenance.Sources_process - INFO - CVE database population started
2023-11-15 04:27:43,640 - CveXplore.database.maintenance.Sources_process - INFO - CVE database population started
2023-11-15 04:27:43,640 - CveXplore.database.maintenance.Sources_process - INFO - Starting CVE database population starting from year: 2002
2023-11-15 04:27:43,640 - CveXplore.database.maintenance.Sources_process - INFO - Starting CVE database population starting from year: 2002
2023-11-15 04:27:43,757 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('id', 1)] on cves
2023-11-15 04:27:43,757 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('id', 1)] on cves
2023-11-15 04:27:43,762 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_configuration', 1)] on cves
2023-11-15 04:27:43,762 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_configuration', 1)] on cves
2023-11-15 04:27:43,766 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_product', 1)] on cves
2023-11-15 04:27:43,766 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_product', 1)] on cves
2023-11-15 04:27:43,772 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('modified', 1)] on cves
2023-11-15 04:27:43,772 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('modified', 1)] on cves
2023-11-15 04:27:43,776 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('published', 1)] on cves
2023-11-15 04:27:43,776 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('published', 1)] on cves
2023-11-15 04:27:43,781 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('lastModified', 1)] on cves
2023-11-15 04:27:43,781 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('lastModified', 1)] on cves
2023-11-15 04:27:43,786 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cvss', 1)] on cves
2023-11-15 04:27:43,786 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cvss', 1)] on cves
2023-11-15 04:27:43,791 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cvss3', 1)] on cves
2023-11-15 04:27:43,791 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cvss3', 1)] on cves
2023-11-15 04:27:43,796 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('summary', 'text')] on cves
2023-11-15 04:27:43,796 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('summary', 'text')] on cves
2023-11-15 04:27:43,804 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vendors', 1)] on cves
2023-11-15 04:27:43,804 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vendors', 1)] on cves
2023-11-15 04:27:43,810 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('products', 1)] on cves
2023-11-15 04:27:43,810 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('products', 1)] on cves
2023-11-15 04:27:43,816 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('assigner', 1)] on cves
2023-11-15 04:27:43,816 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('assigner', 1)] on cves
2023-11-15 04:27:43,821 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cwe', 1)] on cves
2023-11-15 04:27:43,821 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('cwe', 1)] on cves
2023-11-15 04:27:43,826 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('status', 1)] on cves
2023-11-15 04:27:43,826 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('status', 1)] on cves
2023-11-15 04:27:43,831 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_product_stems', 1)] on cves
2023-11-15 04:27:43,831 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_product_stems', 1)] on cves
2023-11-15 04:27:43,836 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_configuration_stems', 1)] on cves
2023-11-15 04:27:43,836 - CveXplore.database.maintenance.Sources_process - INFO - Success to create index [('vulnerable_configuration_stems', 1)] on cves
2023-11-15 04:27:43,836 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-15 04:27:43,836 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-15 04:27:44,991 - CveXplore.database.maintenance.Sources_process - INFO - Preparing to download 217132 CVE entries
2023-11-15 04:27:44,991 - CveXplore.database.maintenance.Sources_process - INFO - Preparing to download 217132 CVE entries
Downloading and processing content: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 217132/217132 [05:41<00:00, 636.35it/s]
2023-11-15 04:33:26,206 - CveXplore.database.maintenance.Sources_process - INFO - Duration: 0:05:42.370443
2023-11-15 04:33:26,206 - CveXplore.database.maintenance.Sources_process - INFO - Duration: 0:05:42.370443
2023-11-15 04:33:26,555 - CveXplore.database.maintenance.Sources_process - INFO - Finished CVE database population
2023-11-15 04:33:26,555 - CveXplore.database.maintenance.Sources_process - INFO - Finished CVE database population
2023-11-15 04:33:26,597 - CveXplore.database.maintenance.Sources_process - INFO - CPE database update started
2023-11-15 04:33:26,597 - CveXplore.database.maintenance.Sources_process - INFO - CPE database update started
2023-11-15 04:33:26,598 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
2023-11-15 04:33:26,598 - CveXplore.database.maintenance.Sources_process - INFO - Starting download...
Traceback (most recent call last):
File "/home/cylox/emba/emba/./external/cve-search/sbin/db_updater.py", line 122, in
cvex.database.initialize()
File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 169, in initialize
self.update()
File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 67, in update
up.update()
File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 218, in update
self.process_downloads()
File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 168, in process_downloads
total_results = self.api_handler.get_count(
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cylox/emba/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/api/nvd_nist/nvd_nist_api.py", line 196, in get_count
raise ApiDataRetrievalFailed
CveXplore.errors.apis.ApiDataRetrievalFailed

Generally I'm having problems to get the update/initialization to work on my systems.

My assumption at the moment is that we get a weird response from the nist api endpoint.

in get_count()
(or somewhere else?) I fail to receive the proper response

Consider changing to/giving option of couchDB?

Should we consider merging to couchDB, or making CVE-Search compatible with both?
I have heard a lot of negative comments about Mongo, and it would be neat to give multiple database options.
Also, the database layer should be abstracted a lot more as well. (I can do this)
My idea would be to make a database abstraction layer, which implements functions for both mongo, couch, postgres,... (we could further this if we see fit), and then we might be able to change databases with the configuration files.
Your thoughts? @adulau @wimremes

[False Positive] Bug with update of CPEs

Hi, when running the update the following error happens:

python3.10 sbin/db_updater.py -v -c
2023-12-17 14:15:00,961 - DBUpdater - INFO     - ==========================
2023-12-17 14:15:00,961 - DBUpdater - INFO     - Sun 17 December 2023 13:15
2023-12-17 14:15:00,961 - DBUpdater - INFO     - ==========================
2023-12-17 14:15:00,975 - CveXplore.database.maintenance.main_updater - INFO     - Starting Database update....
2023-12-17 14:15:00,975 - CveXplore.database.maintenance.main_updater - INFO     - Starting Database update....
2023-12-17 14:15:00,979 - CveXplore.database.maintenance.Sources_process - INFO     - CPE database update started
2023-12-17 14:15:00,979 - CveXplore.database.maintenance.Sources_process - INFO     - CPE database update started
2023-12-17 14:15:00,980 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
2023-12-17 14:15:00,980 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
2023-12-17 14:15:01,519 - CveXplore.database.maintenance.Sources_process - INFO     - Preparing to download 1902 CPE entries
2023-12-17 14:15:01,519 - CveXplore.database.maintenance.Sources_process - INFO     - Preparing to download 1902 CPE entries
Downloading and processing content:   0%|                                                                                                                                                             | 0/1902 [00:02<?, ?it/s]
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/pymongo/collection.py", line 566, in bulk_write
    request._add_to_bulk(blk)
AttributeError: 'NoneType' object has no attribute '_add_to_bulk'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cve-search/sbin/db_updater.py", line 118, in <module>
    cvex.database.update()
  File "/usr/local/lib/python3.10/site-packages/CveXplore/database/maintenance/main_updater.py", line 73, in update
    up.update()
  File "/usr/local/lib/python3.10/site-packages/CveXplore/database/maintenance/Sources_process.py", line 269, in update
    self.process_downloads()
  File "/usr/local/lib/python3.10/site-packages/CveXplore/database/maintenance/Sources_process.py", line 248, in process_downloads
    self._db_bulk_writer(processed_items)
  File "/usr/local/lib/python3.10/site-packages/CveXplore/database/maintenance/DownloadHandler.py", line 173, in _db_bulk_writer
    self.database[self.feed_type.lower()].bulk_write(batch, ordered=False)
  File "/usr/local/lib/python3.10/site-packages/pymongo/_csot.py", line 108, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/pymongo/collection.py", line 568, in bulk_write
    raise TypeError(f"{request!r} is not a valid request")
TypeError: None is not a valid request

Duplicate CPE key during initialization

Hi, it seems like there is a bug when initializing the DB:

root@ubuntu-16gb-hel1-1:~/test/cve-search# cvexplore database initialize
2023-12-19 18:11:46,126 - CveXplore.core.database_maintenance.main_updater - INFO     - Starting Database initialization....
2023-12-19 18:11:46,129 - CveXplore.core.database_maintenance.sources_process - INFO     - CPE Database population started
2023-12-19 18:11:46,130 - CveXplore.core.database_maintenance.sources_process - INFO     - Starting download...
2023-12-19 18:11:46,621 - CveXplore.core.database_maintenance.sources_process - INFO     - Preparing to download 1219300 CPE entries
Downloading and processing content:  37%|██████████████████████████▏                                            | 450000/1219300 [01:00<00:46, 16385.36it/s]

Downloading and processing content: 100%|███████████████████████████████████████████████████████████████████████| 1219300/1219300 [02:32<00:00, 8017.15it/s]
2023-12-19 18:14:18,714 - CveXplore.core.database_maintenance.sources_process - INFO     - Duration: 0:02:32.583559                                         
2023-12-19 18:14:25,154 - CveXplore.core.database_indexer.db_indexer - ERROR    - Failed to create index cpe on [('id', 1)]: Index build failed: 963294aa-e336-4fbf-9f2d-1549beba0601: Collection cvedb.cpe ( 0b920ea1-a3c2-4ea5-a1a0-3aa8c7385d45 ) :: caused by :: E11000 duplicate key error collection: cvedb.cpe index: id dup key: { id: "00000e6c6e8ab2551731d92cb4ede47930235e22" }, full error: {'ok': 0.0, 'errmsg': 'Index build failed: 963294aa-e336-4fbf-9f2d-1549beba0601: Collection cvedb.cpe ( 0b920ea1-a3c2-4ea5-a1a0-3aa8c7385d45 ) :: caused by :: E11000 duplicate key error collection: cvedb.cpe index: id dup key: { id: "00000e6c6e8ab2551731d92cb4ede47930235e22" }', 'code': 11000, 'codeName': 'DuplicateKey', 'keyPattern': {'id': 1}, 'keyValue': {'id': '00000e6c6e8ab2551731d92cb4ede47930235e22'}}

Normally the initialization should empty the DB before? Is this reproducible on your side?

ServerDisconnect Error

I need some help with the following output:

Traceback (most recent call last):
File "/home/benedikt/embark/emba/external/emba_venv/bin/cvexplore", line 8, in
sys.exit(main())
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 1666, in invoke
rv = super().invoke(ctx)
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/CveXplore/cli_cmds/db_cmds/commands.py", line 28, in initialize_cmd
ctx.obj["data_source"].database.initialize()
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/CveXplore/database/maintenance/main_updater.py", line 167, in initialize
cve_pop.populate()
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/CveXplore/database/maintenance/Sources_process.py", line 904, in populate
self.process_downloads()
File "/home/benedikt/embark/emba/external/emba_venv/lib/python3.10/site-packages/CveXplore/database/maintenance/Sources_process.py", line 796, in process_downloads
for item in data_list["vulnerabilities"]
TypeError: 'ServerDisconnectedError' object is not subscriptable

Version is:
CveXplore==0.3.17

Within the logs I see some "...forbidden..." message
cvexplore-init.log

Any Idea what could cause this?

dependency issues - urllib3, requests packages

Hi @P-T-I ... lets catch this up again - #196

I have some news for you. The following screenshot is from a fresh CveXplore installation and looks quite good:

image

Afterwards we needed to downgrade urllib3 and requests to get docker up and running again - see here

With the new (old) versions of requests and urllib3 CveXplore does not work anymore:

image

Is there a way for you to get CveXplore also working with urllib3<2 and requests<2.29.0?

Integrate Exploit Prediction Scoring System to Enrich CVE Data

The purpose of this issue is to propose an enhancement for the CveXplore project. The enhancement involves the integration of an exploit prediction scoring system, which would serve to enrich the CVE data with predictive insights regarding the potential for exploitation.

Advantages:

  • Enhanced risk assessment capabilities for end-users, providing a clearer understanding of the exploitability of vulnerabilities.
  • Improved prioritization of vulnerabilities to patch, based on a quantifiable exploitation likelihood score.

Vulnerable Configs missing

While testing further on our other issue I have setup CveXplore on a second system with a different connection. This time via a proxy system. The good ... I was able to update the CVE database mostly. :)

I encountered the following errors:

2023-11-17 17:29:17,352 - CveXplore.database.maintenance.Sources_process - ERROR    - Exception encountered during the download from: https://cwe.mitre.org/data/xml/cwec_latest.xml.zip. Error encountered: 'lastModified'
2023-11-17 17:29:18,935 - CveXplore.database.maintenance.Sources_process - ERROR    - Exception encountered during the download from: https://capec.mitre.org/data/xml/capec_latest.xml. Error encountered: 'lastModified'
2023-11-17 17:29:28,744 - CveXplore.database.maintenance.Sources_process - ERROR    - Exception encountered during the download from: https://www.cve-search.org/feeds/via4.json. Error encountered: 'lastModified'

One little thing is looking quite strange. If I search for some CVEs the "vulnerable config" area is sometimes missing:

image

Other CVEs have included them:

image

If I search on one of my old systems (pre CveXplore) for the first CVE it looks correct:

image

@P-T-I could you do a quick check if the same thing appears on your side? Not sure if this is also based on some weird state or connection stuff of my environment.

Add additional field to the CVE

Hi, is there a reason the cve["configurations"] field is not stored? We would like to do some additional queries on this filed, as NIST is missing some CPEs and we want to work around that by doing a lookup on the cpe stem and then filter with some additional logic if the cpe with its version would match the candidates we got by searching for the stem.

db update connection error

During testing the new CveXpore plugin I received the following error for multiple times now:

└─$ cvexplore database update      
2023-11-16 07:05:36,258 - CveXplore.api.nvd_nist.nvd_nist_api - WARNING  - Could not find a NIST API Key in the '~/.cvexplore/.env' file; you could request one at: https://nvd.nist.gov/developers/request-an-api-key                                                                            
2023-11-16 07:05:36,261 - CveXplore.database.maintenance.Sources_process - INFO     - CPE database update started
2023-11-16 07:05:36,263 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
2023-11-16 07:05:37,609 - CveXplore.database.maintenance.Sources_process - INFO     - Preparing to download 0 CPE entries
Downloading and processing content: 0it [00:00, ?it/s]
2023-11-16 07:05:38,209 - CveXplore.database.maintenance.Sources_process - INFO     - Duration: 0:00:01.945437
2023-11-16 07:05:38,210 - CveXplore.database.maintenance.Sources_process - INFO     - Finished CPE database update
2023-11-16 07:05:38,224 - CveXplore.api.nvd_nist.nvd_nist_api - WARNING  - Could not find a NIST API Key in the '~/.cvexplore/.env' file; you could request one at: https://nvd.nist.gov/developers/request-an-api-key                                                                            
2023-11-16 07:05:38,225 - CveXplore.database.maintenance.Sources_process - INFO     - CVE database update started
2023-11-16 07:05:38,226 - CveXplore.database.maintenance.Sources_process - INFO     - Starting download...
Traceback (most recent call last):
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/api/nvd_nist/nvd_nist_api.py", line 116, in _connect
    the_response = json.loads(r.text)
                   ^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/api/nvd_nist/nvd_nist_api.py", line 121, in _connect
    if r.headers["content-type"] == "text/plain":
       ~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/requests/structures.py", line 52, in __getitem__
    return self._store[key.lower()][1]
           ~~~~~~~~~~~^^^^^^^^^^^^^
KeyError: 'content-type'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/kali/emba/external/emba_venv/bin/cvexplore", line 8, in <module>
    sys.exit(main())
             ^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1666, in invoke
    rv = super().invoke(ctx)
         ^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/cli_cmds/db_cmds/commands.py", line 22, in update_cmd
    ctx.obj["data_source"].database.update()
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/main_updater.py", line 67, in update
    up.update()
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 851, in update
    self.process_downloads()
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/database/maintenance/Sources_process.py", line 801, in process_downloads
    total_results = self.api_handler.get_count(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/api/nvd_nist/nvd_nist_api.py", line 188, in get_count
    ret_data = self.call(self.methods.GET, resource=resource, data=datasource)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/api/api_base_class.py", line 163, in call
    result = self._connect(
             ^^^^^^^^^^^^^^
  File "/home/kali/emba/external/emba_venv/lib/python3.11/site-packages/CveXplore/api/nvd_nist/nvd_nist_api.py", line 131, in _connect
    raise Exception(err)
Exception: 'content-type'

Add schema version into CveXplore

Currently CveXplore does not support the schema versioning of CveSearch; this needs to be copied / incorporated into the code base.

Vulnerable Configs - Issue with CVE-2001-1382: Search Results Including Irrelevant Entries

Hi, I've encountered an inconsistency while using the CVE search command. Executing cvexplore cve search -c 2001-1382 -f id -f vulnerable_configuration yields results that include entries not directly relevant to the specified CVE, such as cpe:2.3:a:openbsd:openssh:8.9:-:*:*:*:*:*:*

To give you more context, these are the responses from NIST for comparison:

Inspected:

def padded_version(version: str):
        ret_list = []
        try:
            for v in version.split("."):
                ret_list.append(f"{int(v):05d}")
        except ValueError:
            return version
        return ".".join(ret_list)

returns "2.9.9p2" because of ValueError.

Afterwards in getCPEVersionInformation this matches lots of CPEs that should not be matched.

CVSSv3 score

Search with search.py and default output gives only access to CVSSv2 values. To access the v3 rating json output is the way to go.
It would be helpful if there will be a switch or some other possiblity to also access v3 CVSS rating on other output formats.

Example:

└─$ ./cve-search/bin/search.py -p busybox
<snip>
CVE     : CVE-2022-30065
DATE    : 2022-05-18 15:15:00
CVSS    : 6.8
<snip>

The CVSS value is v2 value. See also https://nvd.nist.gov/vuln/detail/CVE-2022-30065

As json output also includes v3 values it would be great if the other output formats also show these values.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.