Giter VIP home page Giter VIP logo

intel / cve-bin-tool Goto Github PK

View Code? Open in Web Editor NEW
1.1K 29.0 445.0 537.37 MB

The CVE Binary Tool helps you determine if your system includes known vulnerabilities. You can scan binaries for over 200 common, vulnerable components (openssl, libpng, libxml2, expat and others), or if you know the components used, you can get a list of known vulnerabilities associated with an SBOM or a list of components and versions.

Home Page: https://cve-bin-tool.readthedocs.io/en/latest/

License: GNU General Public License v3.0

Python 99.53% CSS 0.17% JavaScript 0.21% Makefile 0.02% Perl 0.07%
cve security hacktoberfest vulnerabilities cvss swrepo system-tools devsecops security-automation security-tools python sbom vulnerability sbom-tool

cve-bin-tool's Introduction

CVE Binary Tool quick start / README

Build Status codecov Gitter On ReadTheDocs On PyPI Code style: black Imports: isort CII Best Practices OpenSSF Scorecard

The CVE Binary Tool is a free, open source tool to help you find known vulnerabilities in software, using data from the National Vulnerability Database (NVD) list of Common Vulnerabilities and Exposures (CVEs) as well as known vulnerability data from Redhat, Open Source Vulnerability Database (OSV), Gitlab Advisory Database (GAD), and Curl.

CVE Binary Tool uses the NVD API but is not endorsed or certified by the NVD.

The tool has two main modes of operation:

  1. A binary scanner which helps you determine which packages may have been included as part of a piece of software. There are 360 checkers. Our initial focus was on common, vulnerable open source components such as openssl, libpng, libxml2 and expat.

  2. Tools for scanning known component lists in various formats, including .csv, several linux distribution package lists, language specific package scanners and several Software Bill of Materials (SBOM) formats.

It is intended to be used as part of your continuous integration system to enable regular vulnerability scanning and give you early warning of known issues in your supply chain. It can also be used to auto-detect components and create SBOMs.

What CVE Binary Tool does when it runs:

Diagram of cve-bin-tool's workflow, described in text with more detail below.

  1. Download CVE Data (from NVD, Redhat, OSV, Gitlab, and Curl).
    • This happens once per day by default, not every time a scan is run.
    • On first run, downloading all data can take some time.
  2. Create/read a component list. There are two modes of operation:
    1. Creates a component list (including versions) using a combination of binary checkers and language component lists (such as python's requirements.txt).
    2. Read SBOM (use an existing component list in a standardized Software Bill of Materials format.)
  3. Create CVE List
    • This looks up all components found or read from an existing bill of materials and reports back any known issues associated with them
  4. Include triage/additional data
    • There are several options for adding triage/notes, information from previous reports to track vulnerability change over time, or known fix data
  5. Generate report in one or more formats (console, json, csv, html, pdf)

For more details, see our documentation or this quickstart guide

Installing CVE Binary Tool

CVE Binary Tool can be installed using pip:

pip install cve-bin-tool

If you want to try the latest code from the cve-bin-tool github or do development, you can also pip install --user -e . to install a local copy from a directory. The Contributor Documentation covers how to set up for local development in more detail.

Pip will install the python requirements for you, but for some types of extraction we use system libraries. If you have difficulties extracting files, you may want to look at our additional Requirements lists for Linux and Windows.

On first usage (and by default, once per day) The tool will download vulnerability data from a set of known vulnerability data sources. Due to reliability issues with NVD, as of release 3.3 we will be using our own NVD mirror at https://cveb.in/ by default rather than contacting NVD directly. If you wish to get data directly from the NVD servers you must provide your own NVD_API_KEY to use their API.

If you are using a release prior to 3.3 that does not use our mirror, please use an NVD_API_KEY as described above.

Most popular usage options

Finding known vulnerabilities using the binary scanner

To run the binary scanner on a directory or file:

cve-bin-tool <directory/file>

Note: That this option will also use any language specific checkers to find known vulnerabilities in components.

By default, the tool assumes you are attempting to scan a whole directory, but if you provide it with a single .csv or .json file that lists dependencies it will treat it as a bill of materials. You can also specify bill of materials files directly using the --input-file option or scan SBOMs with the instructions below.

Scanning an SBOM file for known vulnerabilities

To scan a software bill of materials file (SBOM):

cve-bin-tool --sbom <sbom_filetype> --sbom-file <sbom_filename>

Valid SBOM types are SPDX, CycloneDX, and SWID. Scanning of product names within an SBOM file is case insensitive.

The SBOM scanning how-to guide provides additional SBOM scanning examples.

Generating an SBOM

As well as scanning SBOMs, CVE Binary Tool can be used to generate an SBOM from a scan as follows:

cve-bin-tool  --sbom-type <sbom_type> --sbom-format <sbom-format> --sbom-output <sbom_filename> <other scan options as required>

Valid SBOM types are SPDX and CycloneDX.

The generated SBOM will include product name, version and supplier (where available). License information is not provided.

The SBOM generation how-to guide provides additional SBOM generation examples.

Generating a VEX

As well as scanning VEX, CVE Binary Tool can be used to generate an VEX from a scan as follows:

cve-bin-tool  --vex-type <vex_type> --vex-output <vex_filename> <other scan options as required>

Valid VEX types are CSAF, CycloneDX and OpenVEX.

The VEX generation how-to guide provides additional VEX generation examples.

Triaging vulnerabilities

The --triage-input-file option can be used to add extra triage data like remarks, comments etc. while scanning a directory so that output will reflect this triage data and you can save time of re-triaging (Usage: cve-bin-tool --triage-input-file test.vex /path/to/scan). The supported format is the CycloneDX VEX format which can be generated using the --vex option.

Typical usage:

  1. Generate triage file using cve-bin-tool /path/to/scan --vex triage.vex
  2. Edit triage.vex with your favourite text editor to provide triage information on the vulnerabilities listed.
  3. Use this triage file for future scans as follows: cve-bin-tool /path/to/scan --triage-input-file triage.vex

It should be possible to share triage data across different runs of cve-bin-tool or with other tools that support the CycloneDX VEX format. This would be particularly useful for teams that scan related products or containers, teams that need to use multiple tools for compliance reasons, companies that have a central security policy group that provides guidance on vulnerability triage, and more.

Using the tool offline

Specifying the --offline option when running a scan ensures that cve-bin-tool doesn't attempt to download the latest database files or to check for a newer version of the tool.

Note that you will need to obtain a copy of the vulnerability data before the tool can run in offline mode. The offline how-to guide contains more information on how to set up your database.

Using CVE Binary Tool in GitHub Actions

If you want to integrate cve-bin-tool as a part of your github action pipeline, you can use cve-bin-tool's official GitHub Action. Find more details here. The GitHub Action provide reports on the security tab, which is available to open source projects as well as GitHub customers who have paid for that access.

We also provide an example GitHub action if you wish to use the tool directly. This may be a good choice for teams who want to store reports in an evidence locker or those who don't have access to the GitHub Security tab.

Output Options

The CVE Binary Tool provides console-based output by default. If you wish to provide another format, you can specify this and a filename on the command line using --format. The valid formats are CSV, JSON, console, HTML and PDF. The output filename can be specified using the --output-file flag.

You can also specify multiple output formats by using comma (',') as separator:

cve-bin-tool file -f csv,json,html -o report

Note: You must not use spaces between the commas (',') and the output formats.

The reported vulnerabilities can additionally be reported in the Vulnerability Exchange (VEX) format by specifying --vex command line option. The generated VEX file can then be used as a --triage-input-file to support a triage process.

If you wish to use PDF support, you will need to install the reportlab library separately.

If you intend to use PDF support when you install cve-bin-tool you can specify it and report lab will be installed as part of the cve-bin-tool install:

pip install cve-bin-tool[PDF]

If you've already installed cve-bin-tool you can add reportlab after the fact using pip:

pip install --upgrade reportlab

Note that reportlab was taken out of the default cve-bin-tool install because it has a known CVE associated with it (CVE-2020-28463). The cve-bin-tool code uses the recommended mitigations to limit which resources added to PDFs, as well as additional input validation. This is a bit of a strange CVE because it describes core functionality of PDFs: external items, such as images, can be embedded in them, and thus anyone viewing a PDF could load an external image (similar to how viewing a web page can trigger external loads). There's no inherent "fix" for that, only mitigations where users of the library must ensure only expected items are added to PDFs at the time of generation.

Since users may not want to have software installed with an open, unfixable CVE associated with it, we've opted to make PDF support only available to users who have installed the library themselves. Once the library is installed, the PDF report option will function.

Configuration

You can use --config option to provide configuration file for the tool. You can still override options specified in config file with command line arguments. See our sample config files in the test/config

Auto-detection of components

CVE Binary tool attempts to do auto-detection of components using binary checkers, supported language component lists, and file extraction methods. The supported tools for auto-detection are listed below.

Binary checker list

The following checkers are available for finding components in binary files:

Available checkers
accountsservice acpid apache_http_server apcupsd apparmor asn1c assimp
asterisk atftp avahi axel bash bind binutils
bird bison bluez boinc botan bro bubblewrap
busybox bwm_ng bzip2 c_ares capnproto ceph chess
chrony civetweb clamav collectd commons_compress connman coreutils
cpio cronie cryptsetup cups curl cvs darkhttpd
dav1d davfs2 dbus debianutils dhclient dhcpcd dhcpd
dmidecode dnsmasq docker domoticz dosfstools dotnet dovecot
doxygen dpkg dropbear e2fsprogs ed elfutils emacs
enscript exfatprogs exim exiv2 f2fs_tools faad2 fastd
ffmpeg file firefox flac fluidsynth freeradius freerdp
fribidi frr gawk gcc gdal gdb gdk_pixbuf
gimp git glib glibc gmp gnomeshell gnupg
gnutls go gpgme gpsd graphicsmagick grep grub2
gstreamer gupnp gvfs gzip haproxy harfbuzz haserl
hdf5 heimdal hostapd hunspell hwloc i2pd icecast
icu iperf3 ipmitool ipsec_tools iptables irssi iucode_tool
iwd jack2 jacksondatabind janus jhead jq json_c
kbd keepalived kerberos kexectools kodi kubernetes ldns
lftp libarchive libass libbpg libcoap libconfuse libcurl
libdb libde265 libebml libevent libexpat libgcrypt libgd
libgit2 libheif libical libidn2 libinput libjpeg libjpeg_turbo
libksba liblas libmatroska libmemcached libmicrohttpd libmodbus libnss
libpcap libraw librsvg librsync libsamplerate libseccomp libsndfile
libsolv libsoup libsrtp libssh libssh2 libtasn1 libtiff
libtomcrypt libupnp libuv libvips libvirt libvncserver libvorbis
libvpx libxslt lighttpd linux_kernel lldpd logrotate lrzip
lua luajit lxc lynx lz4 mailx mariadb
mbedtls mdadm memcached micropython minetest mini_httpd minicom
minidlna miniupnpc miniupnpd moby modsecurity monit mosquitto
motion mpg123 mpv msmtp mtr mupdf mutt
mysql nano nasm nbd ncurses neon nessus
netatalk netdata netkit_ftp netpbm nettle nghttp2 nginx
ngircd nmap node ntfs_3g ntp ntpsec open_iscsi
open_vm_tools openafs opencv openjpeg openldap opensc openssh
openssl openswan openvpn p7zip pango patch pcre
pcre2 pcsc_lite perl php picocom pigz pixman
png polarssl_fedora poppler postgresql ppp privoxy procps_ng
proftpd protobuf_c pspp pure_ftpd putty python qemu
qpdf qt quagga radare2 radvd raptor rauc
rdesktop readline rpm rsync rsyslog rtl_433 rtmpdump
runc rust samba sane_backends sdl seahorse shadowsocks_libev
snapd sngrep snort socat sofia_sip speex spice
sqlite squashfs squid sslh stellarium strongswan stunnel
subversion sudo suricata sylpheed syslogng sysstat systemd
tar tcpdump tcpreplay terminology tesseract thrift thttpd
thunderbird timescaledb tinyproxy tor tpm2_tss traceroute transmission
trousers ttyd twonky_server u_boot udisks unbound unixodbc
upx util_linux varnish vim vlc vorbis_tools vsftpd
webkitgtk wget wireshark wolfssl wpa_supplicant xerces xml2
xscreensaver xwayland yasm zabbix zchunk zeek zlib
znc zsh zstandard

All the checkers can be found in the checkers directory, as can the instructions on how to add a new checker. Support for new checkers can be requested via GitHub issues.

Language Specific checkers

A number of checkers are available for finding vulnerable components in specific language packages.

Language Files supported
Dart pubspec.lock
Go Go.mod
Java pom.xml; JAR, WAR and EAR archives
JavaScript package-lock.json
Rust Cargo.lock
Ruby Gemfile.lock
R renv.lock
Swift Package.resolved
Python requirements.txt, PKG-INFO, METADATA; .whl and .egg files
Perl cpanfile

More information on language-specific checkers can be found in the CVE Binary Tool manual.

Supported Archive Formats

The following archive formats are currently supported by the auto-extractor:

Archive Format File Extension
zip .zip, .exe, .jar, .msi, .egg, .whl, .war, .ear
tar .tar, .tgz, .tar.gz, .tar.xz, .tar.bz2
deb .deb, .ipk
rpm .rpm
cab .cab
apk .apk
zst .zst
pkg .pkg

Additional Requirements

To use the auto-extractor, you may need the following utilities depending on the type of supported archive formats you need to extract.

The utilities below are required to run the full test suite on Linux:

  • file
  • strings
  • tar
  • unzip
  • rpm2cpio
  • cpio
  • ar
  • cabextract

Most of these are installed by default on many Linux systems, but cabextract and rpm2cpio in particular might need to be installed.

On windows systems, you may need:

  • ar
  • 7z
  • Expand
  • pdftotext

Windows has Expand installed by default, but ar and 7z might need to be installed. If you want to run our test-suite or scan a zstd compressed file, We recommend installing this 7-zip-zstd fork of 7zip. We are currently using 7z for extracting jar, apk, msi, exe and rpm files. To install ar you can install MinGW (which has binutils as a part of it) from here and run the downloaded .exe file.

If you get an error about building libraries when you try to install from pip, you may need to install the Windows build tools. The Windows build tools are available for free from https://visualstudio.microsoft.com/visual-cpp-build-tools/

If you get an error while installing brotlipy on Windows, installing the compiler above should fix it.

pdftotext is required for running tests. (users of cve-bin-tool may not need it, developers likely will.) The best approach to install it on Windows involves using conda (click here for further instructions).

You can check our CI configuration to see what versions of python we're explicitly testing.

Limitations

This scanner does not attempt to exploit issues or examine the code in greater detail; it only looks for library signatures and version numbers. As such, it cannot tell if someone has backported fixes to a vulnerable version, and it will not work if library or version information was intentionally obfuscated.

This tool is meant to be used as a quick-to-run, easily-automatable check in a non-malicious environment so that developers can be made aware of old libraries with security issues that have been compiled into their binaries.

The tool does not guarantee that any vulnerabilities reported are actually present or exploitable, neither is it able to find all present vulnerabilities with a guarantee.

Users can add triage information to reports to mark issues as false positives, indicate that the risk has been mitigated by configuration/usage changes, and so on.

Triage details can be re-used on other projects so, for example, triage on a Linux base image could be applied to multiple containers using that image.

For more information and usage of triage information with the tool kindly have a look here.

If you are using the binary scanner capabilities, be aware that we only have a limited number of binary checkers (see table above) so we can only detect those libraries. Contributions of new checkers are always welcome! You can also use an alternate way to detect components (for example, a bill of materials tool such as tern) and then use the resulting list as input to cve-bin-tool to get a more comprehensive vulnerability list.

The tool uses a vulnerability database in order to detect the present vulnerabilities, in case the database is not frequently updated (specially if the tool is used in offline mode), the tool would be unable to detect any newly discovered vulnerabilities. Hence it is highly advised to keep the database updated.

The tool does not guarantee that all vulnerabilities are reported as the tool only has access to a limited number of publicly available vulnerability databases. Contributions to introduce new sources of data to the tool are always welcome.

Whilst some validation checks are performed on the data within the vulnerability database, the tool is unable to assert the quality of the data or correct any discrepancies if the data is incomplete or inconsistent. This may result, for example, in some vulnerability reports where the severity is reported as UNKNOWN.

Feedback & Contributions

Bugs and feature requests can be made via GitHub issues. Be aware that these issues are not private, so take care when providing output to make sure you are not disclosing security issues in other products.

Pull requests are also welcome via git.

  • New contributors should read the contributor guide to get started.
  • Folk who already have experience contributing to open source projects may not need the full guide but should still use the pull request checklist to make things easy for everyone.

CVE Binary Tool contributors are asked to adhere to the Python Community Code of Conduct. Please contact Terri if you have concerns or questions relating to this code of conduct.

Security Issues

Security issues with the tool itself can be reported to Intel's security incident response team via https://intel.com/security.

If in the course of using this tool you discover a security issue with someone else's code, please disclose responsibly to the appropriate party.

Full option list

Usage: cve-bin-tool <directory/file to scan>

options:
  -h, --help            show this help message and exit
  -e EXCLUDE, --exclude EXCLUDE
                        Comma separated Exclude directory path
  -V, --version         show program's version number and exit
  --disable-version-check
                        skips checking for a new version
  --disable-validation-check
                        skips checking xml files against schema
  --offline             operate in offline mode
  --detailed            add CVE description in csv or json report (no effect on console, html or pdf)

CVE Data Download:
  Arguments related to data sources and Cache Configuration

  -n {api,api2,json,json-mirror,json-nvd}, --nvd {api,api2,json,json-mirror,json-nvd}
                        choose method for getting CVE lists from NVD
  -u {now,daily,never,latest}, --update {now,daily,never,latest}
                        update schedule for data sources and exploits database (default: daily)
  --nvd-api-key NVD_API_KEY
                        Specify NVD API key (used to improve NVD rate limit).
                        Set to `no` to ignore any keys in the environment.
  -d DISABLE_DATA_SOURCE, --disable-data-source DISABLE_DATA_SOURCE
                        comma-separated list of data sources (CURL, EPSS, GAD, NVD, OSV, REDHAT, RSD) to disable (default: NONE) 

  --use-mirror USE_MIRROR
                        use an mirror to update the database

Input:
  directory             directory to scan
  -i INPUT_FILE, --input-file INPUT_FILE
                        provide input filename
  --triage-input-file TRIAGE_INPUT_FILE
                        provide input filename for triage data
  -C CONFIG, --config CONFIG
                        provide config file
  -L PACKAGE_LIST, --package-list PACKAGE_LIST
                        provide package list
  --sbom {spdx,cyclonedx,swid}
                        specify type of software bill of materials (sbom) (default: spdx)
  --sbom-file SBOM_FILE
                        provide sbom filename
  --vex-file VEX_FILE
                        provide vex filename used for triage processing, the type of vex will be automatically detected.


Output:
  -q, --quiet           suppress output
  -l {debug,info,warning,error,critical}, --log {debug,info,warning,error,critical}
                        log level (default: info)
  -o OUTPUT_FILE, --output-file OUTPUT_FILE
                        provide output filename (default: output to stdout)
  --html-theme HTML_THEME
                        provide custom theme directory for HTML Report
  -f {csv,json,console,html,pdf}, --format {csv,json,console,html,pdf}
                        update output format (default: console)
                        specify multiple output formats by using comma (',') as a separator
                        note: don't use spaces between comma (',') and the output formats.
  --generate-config {yaml,toml,yaml,toml,toml,yaml}
                        generate config file for cve bin tool in toml and yaml formats.
  -c CVSS, --cvss CVSS  minimum CVSS score (as integer in range 0 to 10) to report (default: 0)
  -S {low,medium,high,critical}, --severity {low,medium,high,critical}
                        minimum CVE severity to report (default: low)
  --metrics             
                        check for metrics (e.g., EPSS) from found cves
  --epss-percentile EPSS_PERCENTILE
                        minimum epss percentile of CVE range between 0 to 100 to report
  --epss-probability EPSS_PROBABILITY
                        minimum epss probability of CVE range between 0 to 100 to report
  --no-0-cve-report     only produce report when CVEs are found
  -A [-], --available-fix [-]
                        Lists available fixes of the package from Linux distribution
  -b [-], --backport-fix [-]
                        Lists backported fixes if available from Linux distribution
  --affected-versions   Lists versions of product affected by a given CVE (to facilitate upgrades)
  --sbom-output SBOM_OUTPUT
                        provide software bill of materials (sbom) filename to generate
  --sbom-type {spdx,cyclonedx}
                        specify type of software bill of materials (sbom) to generate (default: spdx)
  --sbom-format {tag,json,yaml}
                        specify format of software bill of materials (sbom) to generate (default: tag)
  --vex-type {cyclonedx, csaf, openvex}
                        specify type of vulnerability exploitability exchange (vex) to generate (default: cyclonedx)
  --vex-ouptput VEX_OUTPUT
                        Provide vulnerability exploitability exchange (vex) filename to generate
Merge Report:
  Arguments related to Intermediate and Merged Reports

  -a [APPEND], --append [APPEND]
                        save output as intermediate report in json format
  -t TAG, --tag TAG     add a unique tag to differentiate between multiple intermediate reports
  -m MERGE, --merge MERGE
                        comma separated intermediate reports path for merging
  -F FILTER, --filter FILTER
                        comma separated tag string for filtering intermediate reports

Checkers:
  -s SKIPS, --skips SKIPS
                        comma-separated list of checkers to disable
  -r RUNS, --runs RUNS  comma-separated list of checkers to enable

Database Management:
  --import-json IMPORT_JSON
                        import database from json files chopped by years
  --ignore-sig          do not verify PGP signature while importing json data
  --log-signature-error
                        when the signature doesn't match log the error only instead of halting (UNSAFE)
  --verify PGP_PUBKEY_PATH
                        verify PGP sign while importing json files
  --export-json EXPORT_JSON
                        export database as json files chopped by years
  --pgp-sign PGP_PRIVATE_KEY_PATH
                        sign exported json files with PGP
  --passphrase PASSPHRASE
                        required passphrase for signing with PGP
  --export EXPORT       export database filename
  --import IMPORT       import database filename

Exploits:
  --exploits            check for exploits from found cves

Deprecated:
  -x, --extract         autoextract compressed files
  --report              Produces a report even if there are no CVE for the respective output format

For further information about all of these options, please see the CVE Binary Tool user manual.

cve-bin-tool's People

Contributors

anthonyharrison avatar b31ngd3v avatar breadgenie avatar chaitanyamogal avatar dependabot[bot] avatar ffontaine avatar freeshabh avatar github-actions[bot] avatar imsahil007 avatar inosmeet avatar joydeep049 avatar kishan3 avatar mastersans avatar metabiswadeep avatar molkree avatar nedsouza avatar niraj-kamdar avatar param211 avatar pdxjohnny avatar peb-peb avatar prajwalm2212 avatar rexbeast2 avatar rhythmrx9 avatar saurabhk122 avatar singhhrmn avatar stvml avatar terriko avatar wzao1515 avatar xdragon2002 avatar yashugarg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

cve-bin-tool's Issues

GSoC 2019 Project Idea: Add new checkers to the CVE Binary Tool

The CVE Binary tool team is hoping to participate in Google Summer of Code (GSoC) under the Python Software Foundation umbrella. You can read all about what this means at http://python-gsoc.org/. This issue, and any others tagged 'gsoc' are not generally available bugs, but related to project ideas for GSoC.

Project Idea : Add new checkers to the CVE Binary Tool

Project description: The CVE Binary Tool, at launch, only has checkers for 10 different open source libraries. We have a list of over 50 that are being added to our issues list (search for new checkers) that we'd like to have in our next few releases. For this project, you'll implement as many of those as you can, including producing tests.

(Don't worry that we're going to run out of checkers -- if we exhaust the current list, we can definitely find more to add.)

Skills: Python, git, experience reading source code for other languages a bonus

Difficulty level: Easy

Related Readings/Links: How to add new checkers

Potential mentors: @terriko @pdxjohnny @WhataTiberius

Getting Started: Python requires that all students submit a code sample as part of your application. For your first pull request, we recommend you write a new test case for an existing checker. The easiest one to do as a first pull request is a scanner test for a new version of a library we can already detect.

  • Existing tests are in test/
  • You can see the scanner tests in 'tests/test_scanner.py'
  • To add a new one, make a new test case that detects a few CVEs known to occur in this version, and a few that are known not to occur in that version. For example, here's what the current openssl test looks like:
    def test_openssl_1_0_2g(self):
        """Scanning test-openssl-1.0.2g.out"""
        self._binary_test(
            'test-openssl-1.0.2g.out',
            'openssl',
            '1.0.2g',
            [
                # Check for known cves in this version
                "CVE-2016-2107",
                "CVE-2016-2105",
                "CVE-2016-2106",
                "CVE-2016-2109",
                "CVE-2016-2176",
            ],
            [
                # Check to make sure an older CVE isn't included
                "CVE-2016-0800",
            ])
  • Not sure what CVEs apply to a version? The checkers themselves often have links, and have the vendor/product pair so you can find them in the national vulnerability database as well. Here's a link to the openssl checker for you to look at and the rest are in cve-bin-tool/checkers/
  • You'll also need to make a fake file designed to trick the checker into thinking it has found that version of the library. You can see these files in test/binaries/
  • Please note that sometimes the database we're using doesn't have perfect mapping between CVEs and product versions -- if you try to write a test that doesn't work because of that mapping but the description in the CVE says that version should be vulnerable, don't discard it! Instead, make a note of it in a comment here so we can investigate and maybe report it upstream.

Extra credit: Got your test working and want to try something more? You can also try adding a checker before the project starts. See the related readings above for instructions.

Aborts on broken symlinks

If a package to be extracted contains a broken symlink then the scan fails:

Traceback (most recent call last):
  File "/home/ross/.local/bin/cve-bin-tool", line 11, in <module>
    load_entry_point('cve-bin-tool==0.2.0', 'console_scripts', 'cve-bin-tool')()
  File "/home/ross/.local/lib/python3.5/site-packages/cve_bin_tool-0.2.0-py3.5.egg/cve_bin_tool/cli.py", line 250, in main
    args.extract, filepath)
  File "/home/ross/.local/lib/python3.5/site-packages/cve_bin_tool-0.2.0-py3.5.egg/cve_bin_tool/cli.py", line 160, in scan_and_or_extract_file
    filename)
  File "/home/ross/.local/lib/python3.5/site-packages/cve_bin_tool-0.2.0-py3.5.egg/cve_bin_tool/cli.py", line 160, in scan_and_or_extract_file
    filename)
  File "/home/ross/.local/lib/python3.5/site-packages/cve_bin_tool-0.2.0-py3.5.egg/cve_bin_tool/cli.py", line 162, in scan_and_or_extract_file
    scanner.scan_file(filepath)
  File "/home/ross/.local/lib/python3.5/site-packages/cve_bin_tool-0.2.0-py3.5.egg/cve_bin_tool/cli.py", line 99, in scan_file
    raise InvalidFileError(filename)
cve_bin_tool.cli.InvalidFileError: /tmp/tmpdh5ijsto/data.tar.xz.extracted/usr/lib/libnss_files.so

By disabling the temporary file deletion we can see that path is a broken symlink:

$ ls -l /tmp/tmpdh5ijsto/data.tar.xz.extracted/usr/lib/libnss_files.so
lrwxrwxrwx 1 ross ross 27 Mar  9  2018 /tmp/tmpdh5ijsto/data.tar.xz.extracted/usr/lib/libnss_files.so -> ../../lib/libnss_files.so.2

The package contains a symlink to a file in a dependent package.

As scanning any symlink is pointless (broken links can't be scanned, working links will be scanned at the target), the tool should skip symlinks. In fact, the iteration should probably skip everything that isn't a normal file.

NVD DB crash when initializting is interrupted

I noticed that when firstly running the unittest, NVDAutoUpdate.py would automatically install local database. However, if I interrupt the initialization (could be system crash or other reasons), the database is crash and it cannot be fixed, I have to delete all the json files manually and then run the command again. The error information is below:

python -m unittest test.test_scanner.TestScanner.test_expat_2_0_1
rm *.out
gcc -o test-curl-7.59.0.out test-curl-7.59.0.c
gcc -o test-expat-2.0.1.out test-expat-2.0.1.c
gcc -o test-nss-3.35.out test-nss-3.35.c
gcc -o test-png-1.6.26.out test-png-1.6.26.c
gcc -o test-libgcrypt-1.7.6.out test-libgcrypt-1.7.6.c
gcc -o test-libjpeg-turbo-2.0.1.out test-libjpeg-turbo-2.0.1.c
gcc -o test-png-1.6.36.out test-png-1.6.36.c
gcc -o test-zlib-1.2.8.out test-zlib-1.2.8.c
gcc -o test-png-1.4.11.out test-png-1.4.11.c
gcc -o test-sqlite-3.12.2.out test-sqlite-3.12.2.c
gcc -o test-xml2-2.9.2.out test-xml2-2.9.2.c
gcc -o test-xml2-2.9.0.out test-xml2-2.9.0.c
gcc -o test-xerces-3_1_1.out test-xerces-3_1_1.c
gcc -o test-node-9.3.0.out test-node-9.3.0.c
gcc -o test-tiff-4.0.9.out test-tiff-4.0.9.c
gcc -o test-curl-7.57.0.out test-curl-7.57.0.c
gcc -o test-curl-7.34.0.out test-curl-7.34.0.c
gcc -o test-openssl-1.0.2g.out test-openssl-1.0.2g.c
gcc -o test-systemd-239.out test-systemd-239.c
Last Update: 2019-02-19
Local database has been updated in the past 24h.
New data not downloaded.  Remove old files to force the update.
/home/linux/gsoc/cve-bin-tool/test/binaries/test-expat-2.0.1.out contains expat 2.0.1
F
======================================================================
FAIL: test_expat_2_0_1 (test.test_scanner.TestScanner)
Scanning test-expat-2.0.1.out
----------------------------------------------------------------------
Traceback (most recent call last):
  File "test/test_scanner.py", line 157, in test_expat_2_0_1
    'CVE-blahblah',
  File "test/test_scanner.py", line 81, in _binary_test
    self.assertIn(ensure_in, cves[package][version])
AssertionError: 'CVE-2012-1147' not found in {}

----------------------------------------------------------------------
Ran 1 test in 0.917s

FAILED (failures=1)

Thus I think there could be a checker to validate the local json file's correctness.

NVD Database does not match versions which are less than

Some versions are all vulnerable up until the one listed in NVD. Currently the import
and usage (scanner.get_cves) isn't set up to look for those CVEs.

Example:

https://libexpat.github.io/ - CVE-2017-9233

# "CVE-2016-4472",
# "CVE-2016-5300",
# "CVE-2012-6702",
# "CVE-2015-1283",
# 2.2
# "CVE-2017-9233",
# "CVE-2016-9063",

only shows one version (2.2.0) and has a versionEndIncluding key in the JSON
field for it's CPE which proabbly means any version before this.

We need a way to detect and use this info. Perhaps another table

{
  "cve" : {
    "data_type" : "CVE",
    "data_format" : "MITRE",
    "data_version" : "4.0",
    "CVE_data_meta" : {
      "ID" : "CVE-2017-9233",
      "ASSIGNER" : "[email protected]"
    },
    "affects" : {
      "vendor" : {
        "vendor_data" : [ {
          "vendor_name" : "libexpat_project",
          "product" : {
            "product_data" : [ {
              "product_name" : "libexpat",
              "version" : {
                "version_data" : [ {
                  "version_value" : "2.2.0"
                } ]
              }
            } ]
          }
        } ]
      }
    },
    "problemtype" : {
      "problemtype_data" : [ {
        "description" : [ {
          "lang" : "en",
          "value" : "CWE-611"
        } ]
      } ]
    },
    "references" : {
      "reference_data" : [ {
        "url" : "http://www.debian.org/security/2017/dsa-3898",
        "name" : "DSA-3898",
        "refsource" : "DEBIAN",
        "tags" : [ ]
      }, {
        "url" : "http://www.openwall.com/lists/oss-security/2017/06/17/7",
        "name" : "[oss-security] 20170618 Expat 2.2.1 security fixes",
        "refsource" : "MLIST",
        "tags" : [ "Mailing List", "VDB Entry" ]
      }, {
        "url" : "http://www.securityfocus.com/bid/99276",
        "name" : "99276",
        "refsource" : "BID",
        "tags" : [ "Third Party Advisory", "VDB Entry" ]
      }, {
        "url" : "http://www.securitytracker.com/id/1039427",
        "name" : "1039427",
        "refsource" : "SECTRACK",
        "tags" : [ ]
      }, {
        "url" : "https://github.com/libexpat/libexpat/blob/master/expat/Changes",
        "name" : "https://github.com/libexpat/libexpat/blob/master/expat/Changes",
        "refsource" : "CONFIRM",
        "tags" : [ "Release Notes", "Third Party Advisory" ]
      }, {
        "url" : "https://libexpat.github.io/doc/cve-2017-9233/",
        "name" : "https://libexpat.github.io/doc/cve-2017-9233/",
        "refsource" : "CONFIRM",
        "tags" : [ "Exploit", "Technical Description", "Vendor Advisory" ]
      }, {
        "url" : "https://support.apple.com/HT208112",
        "name" : "https://support.apple.com/HT208112",
        "refsource" : "CONFIRM",
        "tags" : [ ]
      }, {
        "url" : "https://support.apple.com/HT208113",
        "name" : "https://support.apple.com/HT208113",
        "refsource" : "CONFIRM",
        "tags" : [ ]
      }, {
        "url" : "https://support.apple.com/HT208115",
        "name" : "https://support.apple.com/HT208115",
        "refsource" : "CONFIRM",
        "tags" : [ ]
      }, {
        "url" : "https://support.apple.com/HT208144",
        "name" : "https://support.apple.com/HT208144",
        "refsource" : "CONFIRM",
        "tags" : [ ]
      } ]
    },
    "description" : {
      "description_data" : [ {
        "lang" : "en",
        "value" : "XML External Entity vulnerability in libexpat 2.2.0 and earlier (Expat XML Parser Library) allows attackers to put the parser in an infinite loop using a malformed external entity definition from an external DTD."
      } ]
    }
  },
  "configurations" : {
    "CVE_data_version" : "4.0",
    "nodes" : [ {
      "operator" : "OR",
      "cpe" : [ {
        "vulnerable" : true,
        "cpe22Uri" : "cpe:/a:libexpat_project:libexpat",
        "cpe23Uri" : "cpe:2.3:a:libexpat_project:libexpat:*:*:*:*:*:*:*:*",
        "versionEndIncluding" : "2.2.0"
      } ]
    } ]
  },
  "impact" : {
    "baseMetricV3" : {
      "cvssV3" : {
        "version" : "3.0",
        "vectorString" : "CVSS:3.0/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H",
        "attackVector" : "NETWORK",
        "attackComplexity" : "LOW",
        "privilegesRequired" : "NONE",
        "userInteraction" : "NONE",
        "scope" : "UNCHANGED",
        "confidentialityImpact" : "NONE",
        "integrityImpact" : "NONE",
        "availabilityImpact" : "HIGH",
        "baseScore" : 7.5,
        "baseSeverity" : "HIGH"
      },
      "exploitabilityScore" : 3.9,
      "impactScore" : 3.6
    },
    "baseMetricV2" : {
      "cvssV2" : {
        "version" : "2.0",
        "vectorString" : "(AV:N/AC:L/Au:N/C:N/I:N/A:P)",
        "accessVector" : "NETWORK",
        "accessComplexity" : "LOW",
        "authentication" : "NONE",
        "confidentialityImpact" : "NONE",
        "integrityImpact" : "NONE",
        "availabilityImpact" : "PARTIAL",
        "baseScore" : 5.0
      },
      "severity" : "MEDIUM",
      "exploitabilityScore" : 10.0,
      "impactScore" : 2.9,
      "obtainAllPrivilege" : false,
      "obtainUserPrivilege" : false,
      "obtainOtherPrivilege" : false,
      "userInteractionRequired" : false
    }
  },
  "publishedDate" : "2017-07-25T20:29Z",
  "lastModifiedDate" : "2017-11-04T01:29Z"
}

Debian extract is suboptimal

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now
WARNING:cve_bin_tool.TempDirExtractorContext:Failure extracting '/data/poky-tmp/master/deploy/ipk/corei7-64/linux-libc-headers-dbg_4.18-r0_corei7-64.ipk'

So the problem here is that data.tar could be compressed with xz:

$ ar x sln_2.28-r0_corei7-64.ipk
$ ls
control.tar.gz  data.tar.xz  debian-binary  sln_2.28-r0_corei7-64.ipk

However, the data is always called data.tar.* so there's no need to look into control.tar.gz at all.

"Add db for version maps" introduced massive performance regression

Commit 670808d introduced an auto-updating database for the sqlite versions, but this database is never actually cached. Putting some print() calls in:

$ cve-bin-tool  -xv /path/to/rootfs
Connecting to NVD database and extracting the CVE list ... Please hold on.. This will take few minutes...
Last Update: 2019-03-11
Local database has been updated in the past 24h.
New data not downloaded.  Remove old files present at /home/ross/.cache/cve-bin-tool to force the update.
update required
get_version_map()
update required
get_version_map()
update required
get_version_map()
update required
get_version_map()
update required
get_version_map()
update required
get_version_map()
update required
get_version_map()
update required
get_version_map()
update required
get_version_map()
update required
get_version_map()
update required
get_version_map()
update required
get_version_map()
bin/systemctl contains systemd
[ ... ]

It is constantly downloading the sqlite index and rebuilding the version database.

Improve automation/caching of sqlite release ids

In #63 I added an sqlite checker that includes sqlite release ids gleaned from their website. The code is there to grab them from the live website, but there's a local copy in an array rather than having that run all the time which would quickly have gotten ridiculous inside the checker.

We need to investigate a better way to handle this with some sort of caching, probably another sqlite database in .cache/cve-bin-tool, possibly with a longer refresh cycle or a refresh cycle defined by when nvd has a new sqlite issue. We might also want to try to future-proof things so that other checkers can have similar cached data.

Add false test for inpath

As mentioned in #39, we could use a False test for inpath for code coverage. This should be just a line similar to the test that was added, only assertFalse instead of assertTrue and with a completely unlikely filename to test.

Enable/disable individual checkers

As we add more checkers, we're going to hit the point where users may want to enable/disable certain ones to reduce the time taken to run the CVE Binary Tool. We might as well make sure we've got the ability to do that before it's a problem.

I'm thinking we should probably handle this the way Bandit or Pylint does, with options for command line flags AND an option for a config file.

Bandit the option of either --tests <list of tests> or --skip <list of tests> on the command line, or -c <configfile> to use a config file instead. We might want to use --checkers rather than --tests.
Relevant link: https://github.com/PyCQA/bandit/blob/master/README.rst

Cache packages for test suite

deb, rpm, etc. packages downloaded during / for testing are not cached and are re-downloaded each time. Testing could be speed up by caching them. Make sure to add to travis cachedir as well.

Related: #9

Improve NVD db cache-clearing message

The NVD database only downloads if you have a "stale" copy that's more than a day old, and spits out a message warning you that it's not going to update as follows:

        print("Local database has been updated in the past 24h.")
        print("New data not downloaded.  Remove old files to force the update.")

We should replace this message with one that also tells you where those files are, since it's likely not obvious to the user. (They're in ~/.cache/cve-bin-tool but you should make the message print the actual
variable used, self.disk_location)

Test failing on Ubuntu 16.04

Steps to Reproduce

  1. $ python setup.py install
  2. $ python setup.py test

Output

======================================================================
ERROR: test_extract_file_cab (test.test_extract.TestExtractFileCab)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bksahu/cve-bin-tool/test/test_extract.py", line 122, in test_extract_file_cab
    self.extractor.file_extractors[self.extractor.extract_file_cab]]):
  File "/home/bksahu/cve-bin-tool/test/test_extract.py", line 43, in extract_files
    yield ectx.extract(os.path.join(self.tempdir, filename))
  File "/home/bksahu/cve-bin-tool/cve_bin_tool/extractor.py", line 111, in extract
    if extractor(filename, extracted_path) != 0:
  File "/home/bksahu/cve-bin-tool/cve_bin_tool/extractor.py", line 82, in extract_file_cab
    ["cabextract", "-d", extraction_path, filename])
  File "/home/bksahu/anaconda3/envs/cve/lib/python3.6/subprocess.py", line 287, in call
    with Popen(*popenargs, **kwargs) as p:
  File "/home/bksahu/anaconda3/envs/cve/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/home/bksahu/anaconda3/envs/cve/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'cabextract': 'cabextract'

======================================================================
ERROR: test_extract_file_rpm (test.test_extract.TestExtractFileRpm)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bksahu/cve-bin-tool/test/test_extract.py", line 98, in test_extract_file_rpm
    self.extractor.file_extractors[self.extractor.extract_file_rpm]]):
  File "/home/bksahu/cve-bin-tool/test/test_extract.py", line 43, in extract_files
    yield ectx.extract(os.path.join(self.tempdir, filename))
  File "/home/bksahu/cve-bin-tool/cve_bin_tool/extractor.py", line 111, in extract
    if extractor(filename, extracted_path) != 0:
  File "/home/bksahu/cve-bin-tool/cve_bin_tool/extractor.py", line 64, in extract_file_rpm
    with PopenCTX(["rpm2cpio", filename], stdout=subprocess.PIPE) as proc:
  File "/home/bksahu/anaconda3/envs/cve/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/bksahu/cve-bin-tool/cve_bin_tool/extractor.py", line 23, in PopenCTX
    proc = subprocess.Popen(*args, **kwargs)
  File "/home/bksahu/anaconda3/envs/cve/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/home/bksahu/anaconda3/envs/cve/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'rpm2cpio': 'rpm2cpio'

======================================================================
ERROR: test_binary_curl_7_20_0 (test.test_cli.TestCLI)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bksahu/cve-bin-tool/test/test_cli.py", line 32, in test_binary_curl_7_20_0
    CURL_7_20_0_RPM))
  File "/home/bksahu/cve-bin-tool/cve_bin_tool/extractor.py", line 111, in extract
    if extractor(filename, extracted_path) != 0:
  File "/home/bksahu/cve-bin-tool/cve_bin_tool/extractor.py", line 64, in extract_file_rpm
    with PopenCTX(["rpm2cpio", filename], stdout=subprocess.PIPE) as proc:
  File "/home/bksahu/anaconda3/envs/cve/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/bksahu/cve-bin-tool/cve_bin_tool/extractor.py", line 23, in PopenCTX
    proc = subprocess.Popen(*args, **kwargs)
  File "/home/bksahu/anaconda3/envs/cve/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/home/bksahu/anaconda3/envs/cve/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'rpm2cpio': 'rpm2cpio'

======================================================================
ERROR: test_extract_curl_7_20_0 (test.test_cli.TestCLI)
Scanning curl-7.20.0
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/bksahu/cve-bin-tool/test/test_cli.py", line 27, in test_extract_curl_7_20_0
    self.tempdir]), 0)
  File "/home/bksahu/cve-bin-tool/cve_bin_tool/cli.py", line 263, in main
    args.extract, filepath)
  File "/home/bksahu/cve-bin-tool/cve_bin_tool/cli.py", line 165, in scan_and_or_extract_file
    for filename in walker([ectx.extract(filepath)]):
  File "/home/bksahu/cve-bin-tool/cve_bin_tool/extractor.py", line 111, in extract
    if extractor(filename, extracted_path) != 0:
  File "/home/bksahu/cve-bin-tool/cve_bin_tool/extractor.py", line 64, in extract_file_rpm
    with PopenCTX(["rpm2cpio", filename], stdout=subprocess.PIPE) as proc:
  File "/home/bksahu/anaconda3/envs/cve/lib/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/home/bksahu/cve-bin-tool/cve_bin_tool/extractor.py", line 23, in PopenCTX
    proc = subprocess.Popen(*args, **kwargs)
  File "/home/bksahu/anaconda3/envs/cve/lib/python3.6/subprocess.py", line 729, in __init__
    restore_signals, start_new_session)
  File "/home/bksahu/anaconda3/envs/cve/lib/python3.6/subprocess.py", line 1364, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'rpm2cpio': 'rpm2cpio'

----------------------------------------------------------------------
Ran 38 tests in 150.582s

FAILED (errors=4, skipped=1)
Test failed: <unittest.runner.TextTestResult run=38 errors=4 failures=0>
error: Test failed: <unittest.runner.TextTestResult run=38 errors=4 failures=0>

System Specs

Python 3.6.8
Ubuntu 16.04.

HTML reporter [ GSoC : Stretch goal ]

This is one of the stretch goal ideas I have come up with to add to my GSoC proposal.

The current output formats supported by cve-bin-tool are command line and a csv output file.
These are good reporter tools. But we could also support a html format so that users ( both programmers and non-programmers) can easily visualise/understand the results of the analysis done by cve-bin-tool . We could add a new command line option for this.

This is an example of a html report generated by https://github.com/pyta-uoft/pyta
pyta_report.html.zip

I will add the details for implementation here in the coming days.

Add flag/tests for python strings/file implementation

In #100 I switched the logic so that we use system strings/file if they're available, but this makes it a pain to test that code without messing with your system path. It would be nice to have a command line flag for folk who want to run it to do tests/check performance using that code.

Similarly, it'd be nice to have tests (probably hidden behind the LONG_TESTS flag for now) that provided code coverage in this area.

New checker: berkeleydb (often just known as db)

New checker request: berkeleydb (often known just as 'db')

Source code: https://www.oracle.com/technetwork/database/database-technologies/berkeleydb/downloads/index.html
Note that this is downloaded as individual versions and there is no public repository for you to view history. You might have to dig a bit through older versions to find appropriate signatures that work.

CVE data: https://www.cvedetails.com/vulnerability-list/vendor_id-93/product_id-32070/Oracle-Berkeley-Db.html

Instructions: How to add a new checker to the CVE Binary Tool

Assumes .exe are self-extracted zip, mostly this isn't true

The Zip extractor is used for any .exe files. Python's distutils module ships .exe files.

warning [/home/ross/Local/mess/03/usr/lib/python3.5/distutils/command/wininst-7.1.exe]:  zipfile claims to be last disk of a multi-part archive;
  attempting to process anyway, assuming all parts have been concatenated
  together in order.  Expect "errors" and warnings...true multi-part support
  doesn't exist yet (coming soon).
error [/home/ross/Local/mess/03/usr/lib/python3.5/distutils/command/wininst-7.1.exe]:  missing 4070319391 bytes in zipfile
  (attempting to process anyway)
error [/home/ross/Local/mess/03/usr/lib/python3.5/distutils/command/wininst-7.1.exe]:  attempt to seek before beginning of zipfile
  (please check that you have transferred or created the zipfile in the
  appropriate BINARY mode and that you have compiled UnZip properly)
WARNING:cve_bin_tool.TempDirExtractorContext:Failure extracting '/home/ross/Local/mess/03/usr/lib/python3.5/distutils/command/wininst-7.1.exe'

Find out why travis for python 3.3 is passing things that shouldn't pass

Noticed weird behaviour when pushing WIP code that intentionally would fail tests. One of them passed! This issue is to remind us to look into why.

Passing travis build was with python 3.3, and it looks like it's not running tests at all:
https://travis-ci.com/intel/cve-bin-tool/jobs/175591508

For comparison, here's one of the builds that was failing as expected:
https://travis-ci.com/intel/cve-bin-tool/jobs/175591507
(libjpeg test case should fail due to missing file)

Current git cve-bin-tool is very slow

Using the 0.2 release I can scan an entire rootfs in 107 seconds.

With current git master, I got bored and gave up after five minutes. A quick look at strace shows that it has been spending the last few minutes slowly working through libc.so (17M) and is still going.

I'm sure there's a good reason why the use of file/strings was removed and scanning in Python has been done but this is far too slow. Python is slower than C but this is clearly an algorithmic problem.

Add performance tests

As per #97, we could probably use some performance tests so that travis will warn us if there's a huge performance regression we should look at. What the threshold for "failing" should be and how they work can be a topic of discussion here before an implementation is written.

New Checker: sqlite

This is an issue to go with work I have already in progress, opened as a place to put notes.

GSoC 2019 Project Idea : Windows support for CVE Binary Tool

The CVE Binary tool team is hoping to participate in Google Summer of Code (GSoC) under the Python Software Foundation umbrella. You can read all about what this means at http://python-gsoc.org/. This issue, and any others tagged 'gsoc' are not generally available bugs, but related to project ideas for GSoC.

Project Idea : Windows support for CVE Binary Tool

Project description: The CVE Binary Tool was designed for use on Linux, and thus makes assumptions about the availability of command line utilities, but it doesn't have to be that way. The two utilities it uses for parsing files are file (gives you file type information) and strings (gives you a list of strings found in a given binary). These can be written in pure python, allowing the CVE Binary Tool to be architecture independent.

(Note that it is actually quite possible to run the CVE Binary Tool on Windows right now, if you have those utilities installed through something like cygwin or windows subsystem for linux, but we're hoping for this task that you could pretty much run it on a fresh windows install, and that we'd have the tests to prove it.)

The CVE Binary Tool also uses a number of system utilties for extracting files from various archive formats (from apk to zip files!). These utilities may also have different names on different platforms. Investigate how to deal with those more smoothly. It's possible this could also be done in pure python, we could use utilities that are platform specific and do appropriate checks to make sure they're installed (or suggest them to the user).

Skills: Python, git, multi-platform development

Difficulty level: Intermediate

Related Readings/Links: None at this time.

Potential mentors: @terriko @pdxjohnny @WhataTiberius

Getting Started: There's no "easy" issue that makes a good first commit here, so see the "Getting started" instructions in #24 for setting up your first test.

Another possible good first test is a "real file" test of the checkers. Details on how to add one are are available in #107. Short version, your test will look like this:

    @unittest.skipUnless(os.getenv('LONG_TESTS') == '1', 'Skipping long tests')
    def test_rpm_curl_7_32_0(self):
        """
        test to see if we detect a real copy of curl 7.32.0
        """
        self._file_test(
            'https://archives.fedoraproject.org/pub/archive/fedora/linux/releases/20/Everything/x86_64/os/Packages/c/',
            'curl-7.32.0-3.fc20.x86_64.rpm',
            'curl',
            '7.32.0')

And when you test it locally, you'll need to make sure you have LONG_TESTS enabled, so this one would have to be run as follows:

LONG_TESTS=1 python -m unittest test.test_scanner.TestScanner.test_rpm_curl_7_32_0

What we want to see in your application: Describe how you intend to solve the problem, and give us some "stretch goals" of other platform work you could do once windows is working. Don't forget to include some time for building appropriate tests. (e.g. would you tackle Mac OS next? Improve test coverage? Do you have a feature you'd want to add once this is done?) We think that in an ideal situation, Windows support won't take the full summer, so there's a good chance you'd get to work on the "stretch goal" once the main project is complete.

New checker: systemd

New checker request: systemd

Source code: Please add links in the comments so this issue can be updated

CVE data: Please add links in the comments so this issue can be updated

Confused by debug symbols

If cve-bin-tools scans a package repository that contains split debug packages, it gets confused:

/tmp/cve-bin-tool-qk93dy83/libexpat-dbg_2.2.6-r0_corei7-64.ipk.extracted/usr/lib/.debug/libexpat.so.1.6.8 is expat 
Known CVEs in version 
CVE-2012-6702, CVE-2016-5300, CVE-2016-0718

Because guess_expat_version_from_content couldn't find any strings, the version is ''.

RPM and Deb Extractors don't work if path to scan is relative

The starting scan path is used to construct the paths to the filenames being passed to extract_file_rpm and extract_file_deb but as those commands set cwd in subprocess calls the paths are not valid. The starting scan path should be resolved on startup.

Add packages to test suite

The test suite should have at least a few packages of each format to scan. To avoid distributing binaries, maybe it should download a few curated packages from distro feeds?

Enhance NVD data with CVE information from curl

As described in #1 and noted in some of our test cases (which have missing CVEs that should be detected but aren't due to flaws in the data) the NVD database is a convenient source of information but not a very high quality one. We could enhance it by using vulnerability information from other sources.

One that's particularly good is curl, which has a website set up with information about the vulnerabilities in each version:

For example:
https://curl.haxx.se/docs/vuln-7.63.0.html

We could definitely use this data and add it to the local copy of the nvd database. I have a script that used to pull all this data (from before nvd was added and before this was open sourced) and I'll cut and paste it into a comment in case it's useful.

Note for GSoC participants: this probably isn't a good first issue for you to tackle but may be fine for students who've got a few commits already.

Implementing strings in python

Hi, I am interested in adding Windows support for part of my GSoC goal and I want to start with implementing strings and file in pure python. Intuitively this is not so hard, but I have a question for strings.

I think the naive way is to open a file, and then process it. The overhead of this method is obviously the memory consumption since it would read the whole file into memory at once. Therefore, my question is whether I should take large file into consideration or not worry about that. Thanks.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.