stevespringett / nist-data-mirror Goto Github PK

View Code? Open in Web Editor NEW

206.0 15.0 93.0 217 KB

A simple Java command-line utility to mirror the CVE JSON data from NIST.

License: Apache License 2.0

Java 73.43% Dockerfile 7.22% Shell 10.82% Mustache 8.52%

appsec nvd software-security nist cpe cve java software-composition-analysis sca

nist-data-mirror's People

Contributors

Stargazers

Watchers

Forkers

kevinhcross ncredinburgh pombredanne efenderbosch sellersj bwhitmore amenezes danmaxic quentinkhoo skyun2017 jonathanaustin jeremylong ediri kumarchatla htetthiha motusllc tiandiyixian adian98 dnozay mmaker82 gcaracuel 8x8cloud chocpanda camelpunch leizhen groie masonluo anpanag sean-heller uipko cassionoronha kolachoor xuxinping kenankurdic aamen110 mookkiah phxql kalyanaraopilli burningice2012 toddq tudormi darthmanwe lzgfather yashomer1994 zhushizhou xintao2017 mng222n jssellars finaris ibit-tech good-man-1998 xbw1220 eailoo aikebah zhm022726 tauonico paulcormier strawberrybiscuits falco9 srlk kingnaoufal tomkuipers functicons conor-yang-cn veadan qstesiro luohao518 s3dse tbruketa jingshouyan qweraqq luyulong bathulasai rubenqba lbreuss syncron-oss wuyongc officernordberg rian-gomes eugenmayer nontster daer-71 gdfhgjhg praphull-purohit cegeka-everesst kuinfu zhangchangwei

nist-data-mirror's Issues

Some downloads fail with ZipException

After starting the nist-data-mirror Docker container (docker run -it --rm -p 80:80 sspringett/nvdmirror), several file downloads fail with the following error message:

2019-10-18 19:50:12,011 INFO success: initialize_htdocs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
java.util.zip.ZipException: Not in GZIP format
	at java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:165)
	at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79)
	at java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:91)
	at us.springett.nistdatamirror.NistDataMirror.uncompress(NistDataMirror.java:225)
	at us.springett.nistdatamirror.NistDataMirror.doDownload(NistDataMirror.java:214)
	at us.springett.nistdatamirror.NistDataMirror.mirror(NistDataMirror.java:128)
	at us.springett.nistdatamirror.NistDataMirror.main(NistDataMirror.java:70)

nvdcve-2.0-2005 is not downloaded

Hello

I use the last release of nist-data-mirror but I'm facing a strange issue. The file from nvdcve-2.0-2005 to nvdcve-2.0-2007.xml.gz and nvdcve-2.0-2009.xml.gz ar not downloaded.

Any idea please ?
-rwxrwxrwx 1 dependency dependency 12M juin 14 11:43 nvdcve-2.0-2004.xml
-rwxrwxrwx 1 dependency dependency 849K juin 14 11:43 nvdcve-2.0-2004.xml.gz
-rwxrwxrwx 1 dependency dependency 31M juin 14 11:43 nvdcve-2.0-2008.xml
-rwxrwxrwx 1 dependency dependency 2,2M juin 14 11:43 nvdcve-2.0-2008.xml.gz
-rwxrwxrwx 1 dependency dependency 46M juin 14 11:43 nvdcve-2.0-2010.xml
-rwxrwxrwx 1 dependency dependency 2,9M juin 14 11:43 nvdcve-2.0-2010.xml.gz
-rwxrwxrwx 1 dependency dependency 109M juin 14 11:43 nvdcve-2.0-2011.xml
-rwxrwxrwx 1 dependency dependency 6,1M juin 14 11:43 nvdcve-2.0-2011.xml.gz
-rwxrwxrwx 1 dependency dependency 44M juin 14 11:44 nvdcve-2.0-2012.xml
-rwxrwxrwx 1 dependency dependency 2,6M juin 14 11:44 nvdcve-2.0-2012.xml.gz
-rwxrwxrwx 1 dependency dependency 45M juin 14 11:44 nvdcve-2.0-2013.xml
-rwxrwxrwx 1 dependency dependency 2,7M juin 14 11:44 nvdcve-2.0-2013.xml.gz
-rwxrwxrwx 1 dependency dependency 44M juin 14 11:44 nvdcve-2.0-2014.xml
-rwxrwxrwx 1 dependency dependency 2,8M juin 14 11:44 nvdcve-2.0-2014.xml.gz
-rwxrwxrwx 1 dependency dependency 40M juin 14 11:44 nvdcve-2.0-2015.xml
-rwxrwxrwx 1 dependency dependency 2,4M juin 14 11:44 nvdcve-2.0-2015.xml.gz
-rwxrwxrwx 1 dependency dependency 55M juin 14 11:44 nvdcve-2.0-2016.xml
-rwxrwxrwx 1 dependency dependency 3,3M juin 14 11:44 nvdcve-2.0-2016.xml.gz
-rwxrwxrwx 1 dependency dependency 161M juin 14 11:44 nvdcve-2.0-2017.xml
-rwxrwxrwx 1 dependency dependency 8,9M juin 14 11:44 nvdcve-2.0-2017.xml.gz
-rwxrwxrwx 1 dependency dependency 221M juin 14 11:44 nvdcve-2.0-2018.xml
-rwxrwxrwx 1 dependency dependency 13M juin 14 11:44 nvdcve-2.0-2018.xml.gz
-rwxrwxrwx 1 dependency dependency 76M juin 14 11:44 nvdcve-2.0-2019.xml
-rwxrwxrwx 1 dependency dependency 4,0M juin 14 11:44 nvdcve-2.0-2019.xml.gz
-rwxrwxrwx 1 dependency dependency 8,0M juin 14 17:10 nvdcve-2.0-modified.xml
-rwxrwxrwx 1 dependency dependency 457K juin 14 17:10 nvdcve-2.0-modified.xml.gz

Thanks

Feature Request: Only download if resource needs updating

Somehow Dependency Check does this (I haven't examined the source closely enough to see exactly how). It would be a nice optimization, so my cron job doesn't have be wasteful of network resources.

Cron running twice

When I set the crond command to debug crond -s /var/spool/cron/crontabs -f -d -l 8 in src/docker/conf/supervisord.conf the logs show that the mirror script is run twice: once for user root and once for user mirror. Is this behaviour intended?

I can confirm that the user mirror is not necessary for the setup. I build the image without the user mirror and the cron is running only for user root. The files are served the same way as before. I also deployed the helm chart to k8s and can confirm that it works, too. I can target the k8s mirror when using dependency-check without any issues.

From the Dockerfile (line 24-33):

# obsolete # ENV user=mirror

RUN apk update                                               && \
    apk add --no-cache openjdk8-jre dcron nss supervisor     && \
    # obsolete # addgroup -S $user                           && \
    # obsolete # adduser -S $user -G $user                   && \
    mkdir -p /tmp/nvd                                        && \
    # obsolete # chown -R $user:$user /tmp/nvd               && \
    # obsolete # chown -R $user:$user /usr/local/apache2/htdocs   && \
    rm -v /usr/local/apache2/htdocs/index.html

From src/docker/conf/supervisord.conf (line 17-23):

[program:initialize_htdocs]
command=/mirror.sh
autorestart=false
stdout_logfile=/dev/fd/1
stdout_logfile_maxbytes=0
redirect_stderr=true
# obsolete # user=mirror

Tomcat serving NVD mirror files, depceck client reporting "Unable to update Cached Web DataSource"

mirrored in a local folder with java -jar nist-data-mirror.jar by tomcat (static content)
executed depcheck cli with parameters :

--cveUrl12Modified http://server:8090/nvd-mirror/nvdcve-Modified.xml.gz --cveUrl20Modified http://server:8090/nvd-mirror/nvdcve-2.0-Modified.xml.gz --cveUrl12Base http://server:8090/nvd-mirror/nvdcve-%d.xml.gz --cveUrl20Base http://server:8090/nvd-mirror/nvdcve-2.0-%d.xml.gz

execution partially complete, error regarding "update Cached Web DataSource": maybe need to clear H2 client database?

Here the datails from execution:

2018-03-19T14:43:02.9352768Z [INFO] Checking for updates
2018-03-19T14:43:02.9352768Z [INFO] starting getUpdatesNeeded() ...
2018-03-19T14:43:03.1384018Z [INFO] Download Started for NVD CVE - Modified
2018-03-19T14:43:03.3727768Z [INFO] Download Complete for NVD CVE - Modified (234 ms)
2018-03-19T14:43:03.3727768Z [INFO] Processing Started for NVD CVE - Modified
2018-03-19T14:43:04.0915268Z [WARN] Unable to update Cached Web DataSource, using local data instead. Results may not include recent vulnerabilities.
2018-03-19T14:43:04.0915268Z [INFO] Analysis Started
2018-03-19T14:43:07.4040268Z [INFO] Finished Archive Analyzer (3 seconds)
2018-03-19T14:43:07.4196518Z [INFO] Finished File Name Analyzer (0 seconds)
2018-03-19T14:43:10.3102768Z [INFO] Finished Assembly Analyzer (2 seconds)
2018-03-19T14:43:10.3259018Z [INFO] Finished Dependency Merging Analyzer (0 seconds)
2018-03-19T14:43:10.3259018Z [INFO] Finished Version Filter Analyzer (0 seconds)
2018-03-19T14:43:10.4352768Z [INFO] Finished Hint Analyzer (0 seconds)
2018-03-19T14:43:13.1384018Z [INFO] Created CPE Index (2 seconds)
2018-03-19T14:43:13.1384018Z [INFO] Skipping CPE Analysis for npm
2018-03-19T14:43:14.0446518Z [INFO] Finished CPE Analyzer (3 seconds)
2018-03-19T14:43:14.1384018Z [INFO] Finished False Positive Analyzer (0 seconds)
2018-03-19T14:43:14.2790268Z [INFO] Finished Cpe Suppression Analyzer (0 seconds)
2018-03-19T14:43:14.3884018Z [INFO] Finished NVD CVE Analyzer (0 seconds)
2018-03-19T14:43:14.4352768Z [INFO] Finished Vulnerability Suppression Analyzer (0 seconds)
2018-03-19T14:43:14.4821518Z [INFO] Finished Dependency Bundling Analyzer (0 seconds)
2018-03-19T14:43:14.5915268Z [INFO] Analysis Complete (10 seconds)
2018-03-19T14:43:14.9040268Z [ERROR] org.xml.sax.SAXException: Error updating 'CVE-2004-0558'
2018-03-19T14:43:14.9040268Z org.owasp.dependencycheck.data.nvdcve.DatabaseException: Error updating 'CVE-2004-0558'

entry.sh creates faulty httpd.conf

On the startup of the container the entry.sh tries to replace the default port 80 with the internal used port 8080. On restart it tries to do that again and breaks the config by doing so.
At the Moment on second startup the 'Listen' line is 'Listen 808080'

I think the sed command should be changed to:
sed -i 's/^Listen 80$/Listen 8080/g' /usr/local/apache2/conf/httpd.conf

Docker image appears to have been pushed

Is this reliable?

https://hub.docker.com/r/sspringett/nvdmirror

Recommended way to deal with failed downloads?

In the light of issues like #38: What is the recommended way to use this tool when downloads can fail?

Is it safe to use this tool without risking to corrupt an existing mirror directory? If I have used the tool successfully to mirror the JSON files into a directory nist/, will this directory still contain the data in a usable way even if running this tool again fails?

Edit: If I look at the code

nist-data-mirror/src/main/java/us/springett/nistdatamirror/NistDataMirror.java

Line 196 in 5e5ef4a

file = new File(outputDir, filename);

the target file is directly opened for writing. This can potentially corrupt it. Maybe a temporary file could be used instead?

HELM Chart - Allow http proxy values to be supplied through values.yaml.

The mirror script uses below command which takes http proxy host and port. But the HELM template is not providing option to supply these environment variables.

java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -jar -Dhttp.proxyHost="${proxy_host}" -Dhttp.proxyPort="${proxy_port}" /usr/local/bin/nist-data-mirror.jar /tmp/nvd

Please allow these values to be supplied through values.yaml. It would be good if we even add additional env variables if consumer needed it.

Docker container doesn't mirror all files

I recently opened an issue for the DependencyCheck, and therefore found that I've been facing issues with the nist-data-mirror.

After following the instructions for the Docker container, when I run curl http://localhost I see the following:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<html>
 <head>
  <title>Index of /</title>
 </head>
 <body>
<h1>Index of /</h1>
<ul><li><a href="nvdcve-1.1-2002.meta"> nvdcve-1.1-2002.meta</a></li>
<li><a href="nvdcve-1.1-2003.meta"> nvdcve-1.1-2003.meta</a></li>
<li><a href="nvdcve-1.1-2004.meta"> nvdcve-1.1-2004.meta</a></li>
<li><a href="nvdcve-1.1-2005.meta"> nvdcve-1.1-2005.meta</a></li>
<li><a href="nvdcve-1.1-2006.meta"> nvdcve-1.1-2006.meta</a></li>
<li><a href="nvdcve-1.1-2007.meta"> nvdcve-1.1-2007.meta</a></li>
<li><a href="nvdcve-1.1-2008.meta"> nvdcve-1.1-2008.meta</a></li>
<li><a href="nvdcve-1.1-2009.meta"> nvdcve-1.1-2009.meta</a></li>
<li><a href="nvdcve-1.1-2010.meta"> nvdcve-1.1-2010.meta</a></li>
<li><a href="nvdcve-1.1-2011.meta"> nvdcve-1.1-2011.meta</a></li>
<li><a href="nvdcve-1.1-2012.meta"> nvdcve-1.1-2012.meta</a></li>
<li><a href="nvdcve-1.1-2013.meta"> nvdcve-1.1-2013.meta</a></li>
<li><a href="nvdcve-1.1-2014.meta"> nvdcve-1.1-2014.meta</a></li>
<li><a href="nvdcve-1.1-2015.meta"> nvdcve-1.1-2015.meta</a></li>
<li><a href="nvdcve-1.1-2016.meta"> nvdcve-1.1-2016.meta</a></li>
<li><a href="nvdcve-1.1-2017.json"> nvdcve-1.1-2017.json</a></li>
<li><a href="nvdcve-1.1-2017.json.gz"> nvdcve-1.1-2017.json.gz</a></li>
<li><a href="nvdcve-1.1-2017.meta"> nvdcve-1.1-2017.meta</a></li>
<li><a href="nvdcve-1.1-2018.meta"> nvdcve-1.1-2018.meta</a></li>
<li><a href="nvdcve-1.1-2019.json"> nvdcve-1.1-2019.json</a></li>
<li><a href="nvdcve-1.1-2019.json.gz"> nvdcve-1.1-2019.json.gz</a></li>
<li><a href="nvdcve-1.1-2019.meta"> nvdcve-1.1-2019.meta</a></li>
<li><a href="nvdcve-1.1-2020.json"> nvdcve-1.1-2020.json</a></li>
<li><a href="nvdcve-1.1-2020.json.gz"> nvdcve-1.1-2020.json.gz</a></li>
<li><a href="nvdcve-1.1-2020.meta"> nvdcve-1.1-2020.meta</a></li>
<li><a href="nvdcve-1.1-2021.json"> nvdcve-1.1-2021.json</a></li>
<li><a href="nvdcve-1.1-2021.json.gz"> nvdcve-1.1-2021.json.gz</a></li>
<li><a href="nvdcve-1.1-2021.meta"> nvdcve-1.1-2021.meta</a></li>
<li><a href="nvdcve-1.1-2022.json"> nvdcve-1.1-2022.json</a></li>
<li><a href="nvdcve-1.1-2022.json.gz"> nvdcve-1.1-2022.json.gz</a></li>
<li><a href="nvdcve-1.1-2022.meta"> nvdcve-1.1-2022.meta</a></li>
<li><a href="nvdcve-1.1-modified.json"> nvdcve-1.1-modified.json</a></li>
<li><a href="nvdcve-1.1-modified.json.gz"> nvdcve-1.1-modified.json.gz</a></li>
<li><a href="nvdcve-1.1-modified.meta"> nvdcve-1.1-modified.meta</a></li>
</ul>
</body></html>

However, on the Docker container itself, I see the following files:

/tmp/nvd # ls
nvdcve-1.1-2002.json         nvdcve-1.1-2009.json.gz      nvdcve-1.1-2016.meta
nvdcve-1.1-2002.json.gz      nvdcve-1.1-2009.meta         nvdcve-1.1-2017.json
nvdcve-1.1-2002.meta         nvdcve-1.1-2010.json         nvdcve-1.1-2017.json.gz
nvdcve-1.1-2003.json         nvdcve-1.1-2010.json.gz      nvdcve-1.1-2017.meta
nvdcve-1.1-2003.json.gz      nvdcve-1.1-2010.meta         nvdcve-1.1-2018.json
nvdcve-1.1-2003.meta         nvdcve-1.1-2011.json         nvdcve-1.1-2018.json.gz
nvdcve-1.1-2004.json         nvdcve-1.1-2011.json.gz      nvdcve-1.1-2018.meta
nvdcve-1.1-2004.json.gz      nvdcve-1.1-2011.meta         nvdcve-1.1-2019.json
nvdcve-1.1-2004.meta         nvdcve-1.1-2012.json         nvdcve-1.1-2019.json.gz
nvdcve-1.1-2005.json         nvdcve-1.1-2012.json.gz      nvdcve-1.1-2019.meta
nvdcve-1.1-2005.json.gz      nvdcve-1.1-2012.meta         nvdcve-1.1-2020.json
nvdcve-1.1-2005.meta         nvdcve-1.1-2013.json         nvdcve-1.1-2020.json.gz
nvdcve-1.1-2006.json         nvdcve-1.1-2013.json.gz      nvdcve-1.1-2020.meta
nvdcve-1.1-2006.json.gz      nvdcve-1.1-2013.meta         nvdcve-1.1-2021.json
nvdcve-1.1-2006.meta         nvdcve-1.1-2014.json         nvdcve-1.1-2021.json.gz
nvdcve-1.1-2007.json         nvdcve-1.1-2014.json.gz      nvdcve-1.1-2021.meta
nvdcve-1.1-2007.json.gz      nvdcve-1.1-2014.meta         nvdcve-1.1-2022.json
nvdcve-1.1-2007.meta         nvdcve-1.1-2015.json         nvdcve-1.1-2022.json.gz
nvdcve-1.1-2008.json         nvdcve-1.1-2015.json.gz      nvdcve-1.1-2022.meta
nvdcve-1.1-2008.json.gz      nvdcve-1.1-2015.meta         nvdcve-1.1-modified.json
nvdcve-1.1-2008.meta         nvdcve-1.1-2016.json         nvdcve-1.1-modified.json.gz
nvdcve-1.1-2009.json         nvdcve-1.1-2016.json.gz      nvdcve-1.1-modified.meta

Why would the container not be fully mirroring the files in /tmp/nvd? I'd expect to see all these files available when I reach localhost.

Insufficient validation of proxy settings

The check for proxy system properties is not sufficient. There should not only be a null check but also a check for empty strings:

nist-data-mirror/src/main/java/us/springett/nistdatamirror/NistDataMirror.java

Line 96 in 5e5ef4a

if (proxyHost != null && proxyPort != null) {

This is because in the mirror.sh script which is executed in the Docker environment the proxy properties are always being set. In case no proxy_hostor proxy_port environment variable is set the http.proxyHost or http.proxyPort JVM properties are set to an empty string:

nist-data-mirror/src/docker/scripts/mirror.sh

Line 3 in 5e5ef4a

 files=`java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -jar -Dhttp.proxyHost="${proxy_host}" -Dhttp.proxyPort="${proxy_port}" /usr/local/bin/nist-data-mirror.jar /tmp/nvd | grep -Eo '(Download succeeded|Uncompressed).*' | grep -Eo '[^ ]*\.(gz|meta|json|xml)'` 

GitHub action - Unable to access jarfile nist-data-mirror.jar

Hi, i am trying to incorporate the commands into a GitHub actions script but i am getting the "Unable to access jarfile nist-data-mirror.jar" error, did anyone manage to run it on GitHub actions please?

    - name: build the .jar file
      run: |
        mkdir  mirror-dir
        mvn clean package
        java -jar nist-data-mirror.jar mirror-dir
        ls -al    
      shell: bash

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  32.450 s
[INFO] Finished at: 2022-06-03T13:03:31Z
[INFO] ------------------------------------------------------------------------
Error: Unable to access jarfile nist-data-mirror.jar

Unable to run nist-data-mirror on OpenShift

As OpenShift does not support using the root user, it is not possible to deploy the Docker image to OpenShift. It fails with the followig error:

It seems that supervisord is configured to run as root:
https://github.com/stevespringett/nist-data-mirror/blob/master/src/docker/conf/supervisord.conf#L3

Introduce a logger

I see that the project uses System.out to print out messages, for example:

nist-data-mirror/src/main/java/us/springett/nistdatamirror/NistDataMirror.java

Line 116 in 8f1faec

System.out.println("Downloading files at " + currentDate);

As a result, an application that uses the library doesn't have much control over this messages. It would be better to use a logger. If this improvement sounds good, I can open a pull request that introduces log4j. Please let me know what you think.

No .jar file for release 1.5.0 in releases

Please can you provide the .jar for version 1.5.0 under releases?

Allow to add volume and configure DNS in HELM chart.

Problem 1:
The java command which runs to update the mirror is failing due to certificates issue. We have our internal proxy which uses internal CA which needs to be installed in host machine.

#java -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -jar -Dhttp.proxyHost="${proxy_host}" -Dhttp.proxyPort="${proxy_port}" /usr/local/bin/nist-data-mirror.jar /tmp/nvd 
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2011.meta
Download failed : java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

Solutions:

- extend the image and add certificate.
- mount the /etc/pki/java/cacerts file.

For solution 2, helm chart should support additional volume.

Problem 2:
The java command for downloading mirror failed. The reason for this, the nvd.nist.gov is not resolvable. This is due to the nature of base image

Not able to resolve hostname
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2020.json.gz
Download failed : nvd.nist.gov
bash-4.4# nslookup nvd.nist.gov
nslookup: can't resolve '(null)': Name does not resolve

nslookup: can't resolve 'nvd.nist.gov': Name does not resolve
bash-4.4# nslookup nvd.nist.gov.
nslookup: can't resolve '(null)': Name does not resolve

Name:      nvd.nist.gov.
Address 1: 54.85.30.225 ec2-54-85-30-225.compute-1.amazonaws.com
Address 2: 2600:1f18:268d:1d01:f609:5e91:8a48:f546

Solution:
Change dns config (/etc/resolve.conf) with proper ndots value.
HELM chart should support configuring dns config

Download and uncompression are slower than expected.

Downloading and uncompressing from scratch takes almost 3 minutes on my machine. Initial profiling showed that InputStreams and OutputStreams are not buffered.

Allow option of downloading JSON or XML files only

To cut down download sizes and to speed up download times, please could the ability to download JSON or XML files only be added?

release 1.4.0 compiled against newer jdk

11:11:27  Exception in thread "main" java.lang.UnsupportedClassVersionError: us/springett/nistdatamirror/NistDataMirror : Unsupported major.minor version 52.0
11:11:27  	at java.lang.ClassLoader.defineClass1(Native Method)
11:11:27  	at java.lang.ClassLoader.defineClass(ClassLoader.java:808)
11:11:27  	at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
11:11:27  	at java.net.URLClassLoader.defineClass(URLClassLoader.java:443)
11:11:27  	at java.net.URLClassLoader.access$100(URLClassLoader.java:65)
11:11:27  	at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
11:11:27  	at java.net.URLClassLoader$1.run(URLClassLoader.java:349)
11:11:27  	at java.security.AccessController.doPrivileged(Native Method)
11:11:27  	at java.net.URLClassLoader.findClass(URLClassLoader.java:348)
11:11:27  	at java.lang.ClassLoader.loadClass(ClassLoader.java:430)
11:11:27  	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:323)
11:11:27  	at java.lang.ClassLoader.loadClass(ClassLoader.java:363)
11:11:27  	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)```

Hello,
OpenJDK7 is still supported until June 2020
https://access.redhat.com/articles/1299013
Would you mind releasing a jar that still works with 1.7 ?

Thank you

Realease of recent NVD Metadata Changes

Could you consider in preparation for DependencyCheck v5.0.0 release to prepare a relase of nist-data-mirror as well?
#16

json 1.0 feeds no longer supported?

It looks like the json 1.0 feeds are no longer supported. This was announced here.
Looks like they finally decided to take the old feeds offline.
Since nist-data-mirror tries to download both versions, an error level is reported once it has finished.

nvdcve-1.1-2013.json.gz corrupt with nist-data-mirror 1.6.0

Hello,
we are currently facing a weird issue with our nist-data-mirror for approx. a week.
The file nvdcve-1.12013.json.gz gets corrupt and cannot be extracted:

$ gunzip nvdcve-1.1-2013.json.gz

gzip: nvdcve-1.1-2013.json.gz: unexpected end of file

When the file is corrupt and we run nist-data-mirror 1.6.0 on our data folder we are getting the following output:

[Thu Oct  6 09:16:08 CEST 2022]
Downloading files at Thu Oct 06 09:16:08 CEST 2022
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.meta
Download succeeded nvdcve-1.1-modified.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-recent.meta
Download succeeded nvdcve-1.1-recent.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2002.meta
Download succeeded nvdcve-1.1-2002.meta
File 2002 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2003.meta
Download succeeded nvdcve-1.1-2003.meta
File 2003 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2004.meta
Download succeeded nvdcve-1.1-2004.meta
File 2004 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2005.meta
Download succeeded nvdcve-1.1-2005.meta
File 2005 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2006.meta
Download succeeded nvdcve-1.1-2006.meta
File 2006 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2007.meta
Download succeeded nvdcve-1.1-2007.meta
File 2007 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2008.meta
Download succeeded nvdcve-1.1-2008.meta
File 2008 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2009.meta
Download succeeded nvdcve-1.1-2009.meta
File 2009 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2010.meta
Download succeeded nvdcve-1.1-2010.meta
File 2010 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2011.meta
Download succeeded nvdcve-1.1-2011.meta
File 2011 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2012.meta
Download succeeded nvdcve-1.1-2012.meta
File 2012 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.meta
Download succeeded nvdcve-1.1-2013.meta
File 2013 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.meta
Download succeeded nvdcve-1.1-2013.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.meta
Download succeeded nvdcve-1.1-2013.meta
The File 2013 is corrupted
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2014.meta
Download succeeded nvdcve-1.1-2014.meta
File 2014 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2015.meta
Download succeeded nvdcve-1.1-2015.meta
File 2015 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2016.meta
Download succeeded nvdcve-1.1-2016.meta
File 2016 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2017.meta
Download succeeded nvdcve-1.1-2017.meta
File 2017 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2018.meta
Download succeeded nvdcve-1.1-2018.meta
File 2018 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2019.meta
Download succeeded nvdcve-1.1-2019.meta
File 2019 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2020.meta
Download succeeded nvdcve-1.1-2020.meta
File 2020 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2021.meta
Download succeeded nvdcve-1.1-2021.meta
File 2021 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2022.meta
Download succeeded nvdcve-1.1-2022.meta
File 2022 is valid.

So it reports that the file is corrupted; but I have no idea what has caused the corruption and how to solve it.
We have a cronjob that is running the nist-data-mirror every hour.

I already deleted all nvdcve-1.1-2013* files and rerun the nist-data-mirror:

...
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2011.meta
Download succeeded nvdcve-1.1-2011.meta
File 2011 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2012.meta
Download succeeded nvdcve-1.1-2012.meta
File 2012 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.meta
Download succeeded nvdcve-1.1-2013.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.json.gz
Download succeeded nvdcve-1.1-2013.json.gz
java.io.EOFException: Unexpected end of ZLIB input stream
        at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:245)
        at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
        at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
        at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
        at us.springett.nistdatamirror.NistDataMirror.uncompress(NistDataMirror.java:263)
        at us.springett.nistdatamirror.NistDataMirror.doDownload(NistDataMirror.java:249)
        at us.springett.nistdatamirror.NistDataMirror.downloadVersionForYear(NistDataMirror.java:191)
        at us.springett.nistdatamirror.NistDataMirror.mirror(NistDataMirror.java:155)
        at us.springett.nistdatamirror.NistDataMirror.main(NistDataMirror.java:87)
File 2013 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.meta
Download succeeded nvdcve-1.1-2013.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.meta
Download succeeded nvdcve-1.1-2013.meta
The File 2013 is corrupted
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2014.meta
Download succeeded nvdcve-1.1-2014.meta
File 2014 is valid.
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2015.meta
Download succeeded nvdcve-1.1-2015.meta
File 2015 is valid.
...

So the download fron the nist-data-mirror failed. I did a manual download from the same server and it worked without problem:

$ wget https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.json.gz
--2022-10-06 09:33:35--  https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2013.json.gz
Resolving nvd.nist.gov (nvd.nist.gov)... 18.235.227.114, 2600:1f18:268d:1d01:f609:5e91:8a48:f546
Connecting to nvd.nist.gov (nvd.nist.gov)|18.235.227.114|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 2449748 (2.3M) [application/x-gzip]
Saving to: ‘nvdcve-1.1-2013.json.gz’

100%[=======================================================================================================================>] 2,449,748   2.38MB/s   in 1.0s

2022-10-06 09:33:37 (2.38 MB/s) - ‘nvdcve-1.1-2013.json.gz’ saved [2449748/2449748]

Any advice and help would be appreciated.

Greetings, Rainer

java.io.EOFException: Unexpected end of ZLIB input stream

hi ,

How can I solve this? I use the latest version of NVD-Data-Mirror.

C:\Users\Administrator\Downloads>java -Djdk.http.auth.tunneling.disabledSchemes=
"" -Djdk.http.auth.proxying.disabledSchemes="" -Dhttps.proxyHost=10.1.54.181 -Dh
ttps.proxyPort=1080 -jar nist-data-mirror.jar C:\nvd\
Downloading files at Mon Feb 22 16:41:35 ULAT 2021
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.meta
Download succeeded nvdcve-1.1-modified.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.json.gz
Download succeeded nvdcve-1.1-modified.json.gz
java.io.EOFException: Unexpected end of ZLIB input stream
        at java.util.zip.InflaterInputStream.fill(Unknown Source)
        at java.util.zip.InflaterInputStream.read(Unknown Source)
        at java.util.zip.GZIPInputStream.read(Unknown Source)
        at java.io.FilterInputStream.read(Unknown Source)
        at us.springett.nistdatamirror.NistDataMirror.uncompress(NistDataMirror.
java:232)
        at us.springett.nistdatamirror.NistDataMirror.doDownload(NistDataMirror.
java:218)
        at us.springett.nistdatamirror.NistDataMirror.mirror(NistDataMirror.java
:139)
        at us.springett.nistdatamirror.NistDataMirror.main(NistDataMirror.java:8
4)
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2002.meta

Thanks!

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Currently the downlod fails:

$ java -version
openjdk version "13" 2019-09-17
OpenJDK Runtime Environment AdoptOpenJDK (build 13+33)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 13+33, mixed mode, sharing)
$ java -jar nist-data-mirror-1.3.0.jar nist-mirror json
Downloading files at Mon Sep 30 09:03:50 CEST 2019
Downloading https://nvd.nist.gov/feeds/json/cve/1.0/nvdcve-1.0-modified.meta
Download failed : PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
...

Firefox 69.0.1 has no problems accessing this and other URLs.

NVD JSON data feed version changed to 1.1

Hello,

the NVD recently changed their JSON data feed to version 1.1 to support CVSSv3.1, as per this announcement: https://nvd.nist.gov/General/News/JSON-1-1-Vulnerability-Feed-Release

The announcement claims, that the 1.0 feeds are no longer available. They can still be reached for now, but contain incomplete CVE data (e.g.: CVE-2019-16942, which only contains CVSSv2 data in the 1.0 feed as per the time of my issue request).

Therefore it would be great if this library would support the mirroring of the new JSON data feed.

Best regards

Regarding nist-data-mirror and dependency-check

I have build nist-data-mirror docker container. All nvdcve*.json and nvdcve*.xml files downloaded in container under /tmp/nvd

Now question is : How can use this data with OWASP Dependency Check?

In which config file i need to mentioned below settings? is it on my nist-data-mirror container or client side?

cveUrl12Modified=http://hostname/mirror/nvd/nvdcve-modified.xml.gz
cveUrl20Modified=http://hostname/mirror/nvd/nvdcve-2.0-modified.xml.gz
cveUrl12Base=http://hostname/mirror/nvd/nvdcve-%d.xml.gz
cveUrl20Base=http://hostname/mirror/nvd/nvdcve-2.0-%d.xml.gz

NVD mirror to dependency check

Hello,
Usually I have this error : Fatal exception(s) analyzing Core RH: Unable to continue dependency-check analysis.
[ERROR] Unable to connect to the database
So I think to mirrir the CPE/CVE to have local access
But I don't unterstand how can I add it ! help please
(what is the utility of the database H2 in dependency check , ii contain just the CVE to check the vulnerability ?? )

Add Retire JS Repository?

The NIST data mirror is often used in conjunction with dependency-check. Now that dependency-check utilizes RetireJS to analyze JS files - should the data-mirror be updated to also mirror the RetireJS Repository?

The only reason I am posting the question as opposed to just submitting a PR is that this is titled "NIST" data mirror. Thoughts?

Feature Request: Only download if resource needs updating

Somehow Dependency Check does this (I haven't examined the source closely enough to see exactly how). It would be a nice optimization, so my cron job doesn't have be wasteful of network resources.

Container running sspringett/nvdmirror:1.3.0 seems to fail in recent updates.

Hi,

I am running the mirror via sspringett/nvdmirror:1.3.0. It was running fine at first, but it seems to not function properly now (i.e the cron task is running on schedule, but it constantly exits with a failure).

This is the log file from the container:

Jun  6 07:26:01 36971edcecc1 crond: crond 4.5 dillon's cron daemon, started with loglevel notice
Jun  6 08:00:16 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
/tmp/nvd/nvdcve-1.0-modified.meta
/tmp/nvd/nvdcve-1.0-2002.meta
...trimmed...
/tmp/nvd/nvdcve-1.0-2019.json
Jun  6 12:00:16 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
/tmp/nvd/nvdcve-1.0-modified.meta
/tmp/nvd/nvdcve-modified.xml.gz
...trimmed...
/tmp/nvd/nvdcve-1.0-2019.meta
Jun  6 16:00:11 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  6 20:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
/tmp/nvd/nvdcve-1.0-modified.meta
/tmp/nvd/nvdcve-modified.xml.gz
...trimmed...
/tmp/nvd/nvdcve-1.0-2019.meta
Jun  7 00:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
/tmp/nvd/nvdcve-1.0-modified.meta
/tmp/nvd/nvdcve-modified.xml.gz
...trimmed...
/tmp/nvd/nvdcve-1.0-2019.meta
Jun  7 04:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
/tmp/nvd/nvdcve-1.0-modified.meta
/tmp/nvd/nvdcve-modified.xml.gz
...trimmed...
/tmp/nvd/nvdcve-1.0-2019.meta
Jun  7 08:00:16 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  7 12:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  7 16:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  7 20:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  8 00:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  8 04:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  8 08:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  8 12:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  8 16:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  8 20:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  9 00:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  9 04:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  9 08:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  9 12:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  9 16:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun  9 20:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 10 00:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 10 04:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 10 08:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 10 12:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 10 16:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 10 20:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 11 00:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 11 04:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 11 08:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 11 12:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 11 16:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 11 20:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 12 00:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
Jun 12 04:00:01 36971edcecc1 crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1

Listing of files in the temp directory used by the mirror:

/tmp/nvd # ls -hlrt
total 2758696
-rw-r--r--    1 root     root      600.7K Jun  6 07:20 nvdcve-2004.xml.gz
-rw-r--r--    1 root     root        4.6M Jun  6 07:20 nvdcve-2004.xml
-rw-r--r--    1 root     root      848.2K Jun  6 07:20 nvdcve-2.0-2004.xml.gz
-rw-r--r--    1 root     root       11.3M Jun  6 07:20 nvdcve-2.0-2004.xml
-rw-r--r--    1 root     root      961.4K Jun  6 07:20 nvdcve-1.0-2004.json.gz
-rw-r--r--    1 root     root       15.0M Jun  6 07:20 nvdcve-1.0-2004.json
-rw-r--r--    1 root     root      938.7K Jun  6 07:20 nvdcve-2005.xml.gz
-rw-r--r--    1 root     root        7.3M Jun  6 07:20 nvdcve-2005.xml
-rw-r--r--    1 root     root        1.3M Jun  6 07:20 nvdcve-2.0-2005.xml.gz
-rw-r--r--    1 root     root       17.4M Jun  6 07:20 nvdcve-2.0-2005.xml
-rw-r--r--    1 root     root        1.5M Jun  6 07:20 nvdcve-1.0-2005.json.gz
-rw-r--r--    1 root     root       23.1M Jun  6 07:20 nvdcve-1.0-2005.json
-rw-r--r--    1 root     root        1.5M Jun  6 07:20 nvdcve-2006.xml.gz
-rw-r--r--    1 root     root       12.1M Jun  6 07:20 nvdcve-2006.xml
-rw-r--r--    1 root     root        2.0M Jun  6 07:20 nvdcve-2.0-2006.xml.gz
-rw-r--r--    1 root     root       27.3M Jun  6 07:20 nvdcve-2.0-2006.xml
-rw-r--r--    1 root     root        2.3M Jun  6 07:21 nvdcve-1.0-2006.json.gz
-rw-r--r--    1 root     root       34.7M Jun  6 07:21 nvdcve-1.0-2006.json
-rw-r--r--    1 root     root        1.5M Jun  6 07:21 nvdcve-2007.xml.gz
-rw-r--r--    1 root     root       11.2M Jun  6 07:21 nvdcve-2007.xml
-rw-r--r--    1 root     root        1.9M Jun  6 07:21 nvdcve-2.0-2007.xml.gz
-rw-r--r--    1 root     root       25.0M Jun  6 07:21 nvdcve-2.0-2007.xml
-rw-r--r--    1 root     root        2.3M Jun  6 07:21 nvdcve-1.0-2007.json.gz
-rw-r--r--    1 root     root       33.1M Jun  6 07:21 nvdcve-1.0-2007.json
-rw-r--r--    1 root     root        1.3M Jun  6 07:21 nvdcve-2009.xml.gz
-rw-r--r--    1 root     root       11.3M Jun  6 07:21 nvdcve-2009.xml
-rw-r--r--    1 root     root        2.1M Jun  6 07:21 nvdcve-2.0-2009.xml.gz
-rw-r--r--    1 root     root       30.6M Jun  6 07:21 nvdcve-2.0-2009.xml
-rw-r--r--    1 root     root        2.4M Jun  6 07:21 nvdcve-1.0-2009.json.gz
-rw-r--r--    1 root     root       42.5M Jun  6 07:21 nvdcve-1.0-2009.json
-rw-r--r--    1 root     root        1.3M Jun  6 07:21 nvdcve-2010.xml.gz
-rw-r--r--    1 root     root       14.9M Jun  6 07:21 nvdcve-2010.xml
-rw-r--r--    1 root     root        2.8M Jun  6 07:21 nvdcve-2.0-2010.xml.gz
-rw-r--r--    1 root     root       45.6M Jun  6 07:21 nvdcve-2.0-2010.xml
-rw-r--r--    1 root     root        3.2M Jun  6 07:22 nvdcve-1.0-2010.json.gz
-rw-r--r--    1 root     root       67.0M Jun  6 07:22 nvdcve-1.0-2010.json
-rw-r--r--    1 root     root      159.9M Jun  6 08:00 nvdcve-1.0-2017.json
-rw-r--r--    1 root     root        5.5M Jun  6 08:01 nvdcve-2018.xml.gz
-rw-r--r--    1 root     root       61.4M Jun  6 08:01 nvdcve-2018.xml
-rw-r--r--    1 root     root       11.8M Jun  6 08:01 nvdcve-2.0-2018.xml.gz
-rw-r--r--    1 root     root      215.6M Jun  6 08:01 nvdcve-2.0-2018.xml
-rw-r--r--    1 root     root        8.4M Jun  6 08:01 nvdcve-1.0-2018.json.gz
-rw-r--r--    1 root     root      209.4M Jun  6 08:01 nvdcve-1.0-2018.json
-rw-r--r--    1 root     root        1.7M Jun  6 08:01 nvdcve-2019.xml.gz
-rw-r--r--    1 root     root       20.5M Jun  6 08:01 nvdcve-2019.xml
-rw-r--r--    1 root     root        3.7M Jun  6 08:01 nvdcve-2.0-2019.xml.gz
-rw-r--r--    1 root     root       71.7M Jun  6 08:01 nvdcve-2.0-2019.xml
-rw-r--r--    1 root     root        2.6M Jun  6 08:01 nvdcve-1.0-2019.json.gz
-rw-r--r--    1 root     root       63.9M Jun  6 08:02 nvdcve-1.0-2019.json
-rw-r--r--    1 root     root        1.1M Jun  6 12:00 nvdcve-2002.xml.gz
-rw-r--r--    1 root     root        8.4M Jun  6 12:00 nvdcve-2002.xml
-rw-r--r--    1 root     root        1.4M Jun  6 12:00 nvdcve-2.0-2002.xml.gz
-rw-r--r--    1 root     root       18.7M Jun  6 12:00 nvdcve-2.0-2002.xml
-rw-r--r--    1 root     root        1.6M Jun  6 12:00 nvdcve-1.0-2002.json.gz
-rw-r--r--    1 root     root       24.7M Jun  6 12:00 nvdcve-1.0-2002.json
-rw-r--r--    1 root     root      312.8K Jun  6 12:00 nvdcve-2003.xml.gz
-rw-r--r--    1 root     root        2.3M Jun  6 12:00 nvdcve-2003.xml
-rw-r--r--    1 root     root      428.2K Jun  6 12:00 nvdcve-2.0-2003.xml.gz
-rw-r--r--    1 root     root        5.4M Jun  6 12:00 nvdcve-2.0-2003.xml
-rw-r--r--    1 root     root      476.2K Jun  6 12:00 nvdcve-1.0-2003.json.gz
-rw-r--r--    1 root     root        6.9M Jun  6 12:00 nvdcve-1.0-2003.json
-rw-r--r--    1 root     root        1.5M Jun  6 12:00 nvdcve-2008.xml.gz
-rw-r--r--    1 root     root       12.8M Jun  6 12:00 nvdcve-2008.xml
-rw-r--r--    1 root     root        2.2M Jun  6 12:00 nvdcve-2.0-2008.xml.gz
-rw-r--r--    1 root     root       30.9M Jun  6 12:00 nvdcve-2.0-2008.xml
-rw-r--r--    1 root     root        2.5M Jun  6 12:00 nvdcve-1.0-2008.json.gz
-rw-r--r--    1 root     root       40.6M Jun  6 12:00 nvdcve-1.0-2008.json
-rw-r--r--    1 root     root        3.0M Jun  6 12:00 nvdcve-2011.xml.gz
-rw-r--r--    1 root     root       29.8M Jun  6 12:00 nvdcve-2011.xml
-rw-r--r--    1 root     root        6.1M Jun  6 12:00 nvdcve-2.0-2011.xml.gz
-rw-r--r--    1 root     root      108.8M Jun  6 12:00 nvdcve-2.0-2011.xml
-rw-r--r--    1 root     root        6.6M Jun  6 12:00 nvdcve-1.0-2011.json.gz
-rw-r--r--    1 root     root      182.1M Jun  6 12:00 nvdcve-1.0-2011.json
-rw-r--r--    1 root     root        1.2M Jun  6 12:00 nvdcve-2012.xml.gz
-rw-r--r--    1 root     root       14.3M Jun  6 12:00 nvdcve-2012.xml
-rw-r--r--    1 root     root        2.5M Jun  6 12:00 nvdcve-2.0-2012.xml.gz
-rw-r--r--    1 root     root       43.2M Jun  6 12:00 nvdcve-2.0-2012.xml
-rw-r--r--    1 root     root        2.9M Jun  6 12:00 nvdcve-1.0-2012.json.gz
-rw-r--r--    1 root     root        1.3M Jun  6 12:00 nvdcve-2013.xml.gz
-rw-r--r--    1 root     root       61.9M Jun  6 12:00 nvdcve-1.0-2012.json
-rw-r--r--    1 root     root       15.1M Jun  6 12:00 nvdcve-2013.xml
-rw-r--r--    1 root     root        2.6M Jun  6 12:00 nvdcve-2.0-2013.xml.gz
-rw-r--r--    1 root     root       43.5M Jun  6 12:00 nvdcve-2.0-2013.xml
-rw-r--r--    1 root     root        3.0M Jun  6 12:00 nvdcve-1.0-2013.json.gz
-rw-r--r--    1 root     root       61.6M Jun  6 12:00 nvdcve-1.0-2013.json
-rw-r--r--    1 root     root        1.6M Jun  6 12:00 nvdcve-2014.xml.gz
-rw-r--r--    1 root     root       16.5M Jun  6 12:00 nvdcve-2014.xml
-rw-r--r--    1 root     root        2.7M Jun  6 12:00 nvdcve-2.0-2014.xml.gz
-rw-r--r--    1 root     root       43.3M Jun  6 12:00 nvdcve-2.0-2014.xml
-rw-r--r--    1 root     root        3.0M Jun  6 12:00 nvdcve-1.0-2014.json.gz
-rw-r--r--    1 root     root       59.6M Jun  6 12:00 nvdcve-1.0-2014.json
-rw-r--r--    1 root     root        1.5M Jun  6 12:00 nvdcve-2015.xml.gz
-rw-r--r--    1 root     root       15.8M Jun  6 12:00 nvdcve-2015.xml
-rw-r--r--    1 root     root        2.3M Jun  6 12:00 nvdcve-2.0-2015.xml.gz
-rw-r--r--    1 root     root       39.4M Jun  6 12:00 nvdcve-2.0-2015.xml
-rw-r--r--    1 root     root        2.6M Jun  6 12:00 nvdcve-1.0-2015.json.gz
-rw-r--r--    1 root     root       52.4M Jun  6 12:00 nvdcve-1.0-2015.json
-rw-r--r--    1 root     root         165 Jun  7 04:00 nvdcve-1.0-2019.meta
-rw-r--r--    1 root     root         166 Jun  7 04:00 nvdcve-1.0-2018.meta
-rw-r--r--    1 root     root         162 Jun  7 08:00 nvdcve-1.0-modified.meta
-rw-r--r--    1 root     root      296.6K Jun  7 08:00 nvdcve-modified.xml.gz
-rw-r--r--    1 root     root        2.7M Jun  7 08:00 nvdcve-modified.xml
-rw-r--r--    1 root     root      541.8K Jun  7 08:00 nvdcve-2.0-modified.xml.gz
-rw-r--r--    1 root     root        9.1M Jun  7 08:00 nvdcve-2.0-modified.xml
-rw-r--r--    1 root     root      432.3K Jun  7 08:00 nvdcve-1.0-modified.json.gz
-rw-r--r--    1 root     root        8.9M Jun  7 08:00 nvdcve-1.0-modified.json
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2006.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2005.meta
-rw-r--r--    1 root     root         163 Jun  7 08:00 nvdcve-1.0-2004.meta
-rw-r--r--    1 root     root         162 Jun  7 08:00 nvdcve-1.0-2003.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2002.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2010.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2009.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2008.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2007.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2014.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2013.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2012.meta
-rw-r--r--    1 root     root         166 Jun  7 08:00 nvdcve-1.0-2011.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2016.meta
-rw-r--r--    1 root     root         165 Jun  7 08:00 nvdcve-1.0-2015.meta
-rw-r--r--    1 root     root        1.7M Jun  7 08:00 nvdcve-2016.xml.gz
-rw-r--r--    1 root     root       20.0M Jun  7 08:00 nvdcve-2016.xml
-rw-r--r--    1 root     root        3.3M Jun  7 08:00 nvdcve-2.0-2016.xml.gz
-rw-r--r--    1 root     root       54.6M Jun  7 08:00 nvdcve-2.0-2016.xml
-rw-r--r--    1 root     root        3.4M Jun  7 08:00 nvdcve-1.0-2016.json.gz
-rw-r--r--    1 root     root       71.7M Jun  7 08:00 nvdcve-1.0-2016.json
-rw-r--r--    1 root     root         166 Jun  7 08:00 nvdcve-1.0-2017.meta
-rw-r--r--    1 root     root        4.1M Jun  7 08:00 nvdcve-2017.xml.gz
-rw-r--r--    1 root     root       46.1M Jun  7 08:00 nvdcve-2017.xml
-rw-r--r--    1 root     root        8.8M Jun  7 08:00 nvdcve-2.0-2017.xml.gz
-rw-r--r--    1 root     root      160.1M Jun  7 08:00 nvdcve-2.0-2017.xml
-rw-r--r--    1 root     root        4.5M Jun  7 08:01 nvdcve-1.0-2017.json.gz

Using wget to download works ok (just confirm it's not my network issue):

/dev/shm # wget https://nvd.nist.gov/feeds/xml/cve/1.2/nvdcve-modified.xml.gz
Connecting to nvd.nist.gov (18.235.227.114:443)
nvdcve-modified.xml. 100% |***********************************************************************************************************************************************|  282k  0:00:00 ETA

I understand that NVD does not have unlimited bandwidth and will throttle or block.

Is there any workaround I can use for a more reliable mirroring?

Thank you for your time.

Docker Image 1.4.0 is throwing "Permission denied" issues

Hi,

I'm having an issue when using version 1.4.0 of the docker image.

After populating /tmp/nvd, the application is unable to copy the files over to /usr/local/apache2/htdocs.

Running the container

mkdir -p /tmp/target/docs/

docker run -dit \
    --name mirror \
    -p 80:80 \
    --mount type=bind,source=/tmp/target/docs/,target=/usr/local/apache2/htdocs \
    sspringett/nvdmirror:1.4.0

Logs

2020-01-07 10:15:55,354 INFO Set uid to user 0 succeeded
2020-01-07 10:15:55,355 INFO supervisord started with pid 1
2020-01-07 10:15:56,357 INFO spawned: 'httpd' with pid 8
2020-01-07 10:15:56,358 INFO spawned: 'crond' with pid 9
2020-01-07 10:15:56,359 INFO spawned: 'initialize_htdocs' with pid 10
Updating...
crond 4.5 dillon's cron daemon, started with loglevel notice
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.17.0.3. Set the 'ServerName' directive globally to suppress this message
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 172.17.0.3. Set the 'ServerName' directive globally to suppress this message
[Tue Jan 07 10:15:56.384458 2020] [mpm_event:notice] [pid 8:tid 140323194153832] AH00489: Apache/2.4.39 (Unix) configured -- resuming normal operations
[Tue Jan 07 10:15:56.384489 2020] [core:notice] [pid 8:tid 140323194153832] AH00094: Command line: 'httpd -D FOREGROUND'
2020-01-07 10:15:57,385 INFO success: httpd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-01-07 10:15:57,385 INFO success: crond entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2020-01-07 10:15:57,385 INFO success: initialize_htdocs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
/tmp/nvd/nvdcve-1.0-modified.meta
cp: can't create '/usr/local/apache2/htdocs/nvdcve-1.0-modified.meta': Permission denied

... snipped ...

/tmp/nvd/nvdcve-1.1-2020.json
cp: can't create '/usr/local/apache2/htdocs/nvdcve-1.1-2020.json': Permission denied
2020-01-07 10:22:44,156 INFO exited: initialize_htdocs (exit status 1; not expected)

Running processes

# ps -ef
PID   USER     TIME  COMMAND
    1 root      0:00 {supervisord} /usr/bin/python2 /usr/bin/supervisord -n -c /etc/supervisor/conf.d/supervisord.conf -l
    8 root      0:00 httpd -DFOREGROUND
    9 root      0:00 crond -s /var/spool/cron/crontabs -f
   25 daemon    0:00 httpd -DFOREGROUND
   26 daemon    0:00 httpd -DFOREGROUND
   27 daemon    0:00 httpd -DFOREGROUND
  122 root      0:00 /bin/bash
  323 root      0:00 ps -ef

Files in /tmp/

# ls -lah /tmp/
total 20
drwxrwxrwt    6 root     root        4.0K Jan  7 10:15 .
drwxr-xr-x   51 root     root        4.0K Jan  7 10:19 ..
drwxr-xr-x    2 root     root        4.0K Jan  7 10:30 cron.fObHDd
drwxr-xr-x    2 mirror   mirror      4.0K Jan  7 10:22 hsperfdata_mirror
drwxr-xr-x    2 mirror   mirror      4.0K Jan  7 10:22 nvd

Files in /usr/local/apache2/

total 48
drwxr-xr-x   15 www-data www-data    4.0K Jan  7 10:15 .
drwxr-xr-x   15 root     root        4.0K Jan  7 10:15 ..
drwxr-xr-x    2 root     root        4.0K May 11  2019 bin
drwxr-xr-x    2 root     root        4.0K May 11  2019 build
drwxr-xr-x    2 root     root        4.0K May 11  2019 cgi-bin
drwxr-xr-x    4 root     root        4.0K Oct 18 19:50 conf
drwxr-xr-x    3 root     root        4.0K May 11  2019 error
drwxrwxr-x    2 1000     1000        4.0K Jan  7 10:15 htdocs
drwxr-xr-x    3 root     root        4.0K May 11  2019 icons
drwxr-xr-x    2 root     root        4.0K May 11  2019 include
drwxr-xr-x    2 root     root        4.0K Jan  7 10:15 logs
drwxr-xr-x    2 root     root        4.0K May 11  2019 modules

Thank you for your time.

Help needed: Are the XMLs supposed to be uploaded in Nexus?

I had a successful execution but nothing gets uploaded to my Nexus repo.

My init.gradle

allprojects  {
  apply plugin: 'maven'

  repositories {
    maven { url "http://localhost:8081/nexus/content/groups/public" }
    mavenCentral()
  }

  configurations {
      deployerJars
  }

  dependencies {
      deployerJars "org.apache.maven.wagon:wagon-http:2.5"
  }

  uploadArchives {
      repositories.mavenDeployer {
          configuration = configurations.deployerJars
          repository(url: "http://localhost:8081/nexus/content/repositories/NVD") {
              authentication(userName: "admin", password: "admin123")
          }
          snapshotRepository(url: "http://localhost:8081/nexus/content/repositories/NVD") {
              authentication(userName: "admin", password: "admin123")
          }
      }
  }
}

$ gradle -I init.gradle clean UploadArchives
:clean
:compileJava
warning: [options] bootstrap class path not set in conjunction with -source 1.7
1 warning
:processResources UP-TO-DATE
:classes
:jar
:uploadArchives
BUILD SUCCESSFUL

Unable to build with IBM Java 8

I get the following error when I try to build this project locally with IBM Java 8

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project nist-data-mirror: Compilation failure
[ERROR] /tmp/nist-data-mirror/src/main/java/us/springett/nistdatamirror/NistDataMirror.java:[19,52] package com.sun.org.apache.xpath.internal.operations does not exist

I can work around it by removing the unused import of com.sun.org.apache.xpath.internal.operations.Bool in the NistDataMirror.java file. Since it's not actually used, I would think it could simply be removed in the main codebase and then the project would again be compilable with any jdk.

Using proxy with account

Hello,

This is not a real issue. I just wated to share a tip so jar works on java 8 when using a proxy with authentication (thanks release 1.5.0).
Just add -Djdk.http.auth.tunneling.disabledSchemes="" in the command line before calling jar file.

NistDataMirror.java has hardcoded Windows file separators

Lines 289 and 293 have hardcoded Windows file separators that are causing dependency check jobs to fail on OSX and Linux environments.

Cron not running in Docker container

After building the Docker container and starting it as described in the readme the log prints the following line which states that the mirror crontab setting is being ignored:

Oct 1 08:27:54 nist-data-mirror.somewhere.net crond: ignoring /etc/crontabs/mirror (non-existent user)

Usage with Dependency Check

Hi Steve,

I would like to enquire, with regards to nist-data-mirror, how can one make use of the downloaded xml/json files with OWASP Dependency Check?
From what I understand, Dependency Check makes use of a h2.db file to run the scan but this project(nist-data-mirror) gets a local copy of NIST vulnerability data in the form of xml/json files. How are the 2 projects tied together and what steps must i take in order to integrate the downloaded xml/json files via this project into Dependency Check?

I hope to hear from you soon.

Artifact downloads

One question about mirroring. What is the final artifact from this step? Is it a h2.db file?

I would like to mirror the DB with this tool, and then, run Dependency check against it.

If the artifact that comes after this step is not a h2.db file, how do i run Dependency check against this artifact?

Thanks

What is the recommended frequency for nist data mirror?

Hi @stevespringett
what do you recommend as a the frequency for the nist data mirror. Should i be running the script once every 24 hrs or more frequently?

Release the binary

Is there any possibility of releasing this as a binary to bintray or github releases?
Just FYI, bintray has free releases for open source projects and is a good way of getting JARs / WARs etc. into maven central (this is optional).

Thanks!

Don't start the update process if one is already running

As of now, the update process is started once every 15 minutes. If nvd.nist.gov slows down for some reason (I've seen dramatic slowdowns a couple days ago), we can have two or more update processes running concurrently. I suggest doing something like this instead:

flock -n /tmp/nvd.lock /mirror.sh

so that the spawned process immediately exits if there is another one running.

k8s deprecation 'networking.k8s.io/v1beta1' in v1.22

In chart

nist-data-mirror/k8s/nist-data-mirror/templates/nist-data-mirror-ingress.yaml

Line 21 in d76fbe8

apiVersion: networking.k8s.io/v1beta1

api version 'networking.k8s.io/v1beta1' is used. This is scheduled to be removed in k8s v1.22. Please update to be able to keep using the chart in the future. Replacement is 'networking.k8s.io/v1'.

stevespringett / nist-data-mirror Goto Github PK

nist-data-mirror's People

Contributors

Stargazers

Watchers

Forkers

nist-data-mirror's Issues

Running the container

Logs

Running processes

Files in /tmp/

Files in /usr/local/apache2/

Recommend Projects

Recommend Topics

Recommend Org