Comments (4)
chris-gillatt
commented on August 23, 2024
I use nist-data-mirror to get my artefacts into a jFrog Artifactory instance (Nexus OSS also works) and then point my builds at that where I can control the reliablity and bandwidth. I do that by running a docker container which contains the jFrog CLI and then schedule it through my CI/CD tool to run every night. The cli will skip failures and try again later, but the previous files will always exist if something was to go majorly wrong meaning I always have a level of cover.
from nist-data-mirror.
Muzamri
commented on August 23, 2024
Hi @chris-gillatt,
Thank you for sharing your setup.
If I understand correctly -- you used the nist-data-mirror to get the files from NVD and then move it to jFrog to actually serve the file?
I actually have no problem using dependency-check.sh to mirror from the container -- the problem is getting the mirror updated from the actual NVD servers.
It's kind of intermittent, for example, I went ahead and restarted the container:
$ docker logs -f 6199f155757e
Starting mirror
/tmp/nvd/nvdcve-1.0-modified.meta
<...trimmed...>
/tmp/nvd/nvdcve-1.0-2019.json
Mirror started
Jun 12 07:53:58 6199f155757e crond: crond 4.5 dillon's cron daemon, started with loglevel notice
AH00557: httpd: apr_sockaddr_info_get() failed for 6199f155757e
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
AH00557: httpd: apr_sockaddr_info_get() failed for 6199f155757e
AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
[Wed Jun 12 07:54:14.015875 2019] [mpm_event:notice] [pid 198:tid 140351906831208] AH00489: Apache/2.4.39 (Unix) configured -- resuming normal operations
[Wed Jun 12 07:54:14.015904 2019] [core:notice] [pid 198:tid 140351906831208] AH00094: Command line: 'httpd -D FOREGROUND'
Jun 12 08:00:16 6199f155757e crond: exit status 1 from user mirror /mirror.sh >> /var/log/cron.log 2>&1
/tmp/nvd/nvdcve-1.0-modified.meta
/tmp/nvd/nvdcve-1.0-2002.meta
<...trimmed...>
/tmp/nvd/nvdcve-1.0-2019.meta
and list the files in the temp directory again:
/tmp/nvd # ls -hlrt
total 2776720
-rw-r--r-- 1 root root 228.3K Jun 12 07:47 nvdcve-modified.xml.gz
-rw-r--r-- 1 root root 2.2M Jun 12 07:47 nvdcve-modified.xml
-rw-r--r-- 1 root root 407.6K Jun 12 07:48 nvdcve-2.0-modified.xml.gz
-rw-r--r-- 1 root root 7.3M Jun 12 07:48 nvdcve-2.0-modified.xml
-rw-r--r-- 1 root root 350.9K Jun 12 07:48 nvdcve-1.0-modified.json.gz
-rw-r--r-- 1 root root 7.5M Jun 12 07:48 nvdcve-1.0-modified.json
-rw-r--r-- 1 root root 1.1M Jun 12 07:48 nvdcve-2002.xml.gz
-rw-r--r-- 1 root root 8.4M Jun 12 07:48 nvdcve-2002.xml
-rw-r--r-- 1 root root 1.4M Jun 12 07:48 nvdcve-2.0-2002.xml.gz
-rw-r--r-- 1 root root 18.7M Jun 12 07:48 nvdcve-2.0-2002.xml
-rw-r--r-- 1 root root 1.6M Jun 12 07:48 nvdcve-1.0-2002.json.gz
-rw-r--r-- 1 root root 24.7M Jun 12 07:48 nvdcve-1.0-2002.json
-rw-r--r-- 1 root root 312.8K Jun 12 07:48 nvdcve-2003.xml.gz
-rw-r--r-- 1 root root 2.3M Jun 12 07:48 nvdcve-2003.xml
-rw-r--r-- 1 root root 428.2K Jun 12 07:48 nvdcve-2.0-2003.xml.gz
-rw-r--r-- 1 root root 5.4M Jun 12 07:48 nvdcve-2.0-2003.xml
-rw-r--r-- 1 root root 600.7K Jun 12 07:48 nvdcve-2004.xml.gz
-rw-r--r-- 1 root root 4.6M Jun 12 07:48 nvdcve-2004.xml
-rw-r--r-- 1 root root 476.2K Jun 12 07:48 nvdcve-1.0-2003.json.gz
-rw-r--r-- 1 root root 6.9M Jun 12 07:48 nvdcve-1.0-2003.json
-rw-r--r-- 1 root root 848.2K Jun 12 07:48 nvdcve-2.0-2004.xml.gz
-rw-r--r-- 1 root root 11.3M Jun 12 07:48 nvdcve-2.0-2004.xml
-rw-r--r-- 1 root root 961.4K Jun 12 07:48 nvdcve-1.0-2004.json.gz
-rw-r--r-- 1 root root 15.0M Jun 12 07:48 nvdcve-1.0-2004.json
-rw-r--r-- 1 root root 938.7K Jun 12 07:48 nvdcve-2005.xml.gz
-rw-r--r-- 1 root root 7.3M Jun 12 07:48 nvdcve-2005.xml
-rw-r--r-- 1 root root 1.3M Jun 12 07:48 nvdcve-2.0-2005.xml.gz
-rw-r--r-- 1 root root 17.4M Jun 12 07:48 nvdcve-2.0-2005.xml
-rw-r--r-- 1 root root 1.5M Jun 12 07:48 nvdcve-1.0-2005.json.gz
-rw-r--r-- 1 root root 23.1M Jun 12 07:48 nvdcve-1.0-2005.json
-rw-r--r-- 1 root root 1.5M Jun 12 07:48 nvdcve-2006.xml.gz
-rw-r--r-- 1 root root 12.1M Jun 12 07:48 nvdcve-2006.xml
-rw-r--r-- 1 root root 2.0M Jun 12 07:48 nvdcve-2.0-2006.xml.gz
-rw-r--r-- 1 root root 27.3M Jun 12 07:48 nvdcve-2.0-2006.xml
-rw-r--r-- 1 root root 2.3M Jun 12 07:48 nvdcve-1.0-2006.json.gz
-rw-r--r-- 1 root root 34.7M Jun 12 07:48 nvdcve-1.0-2006.json
-rw-r--r-- 1 root root 1.5M Jun 12 07:48 nvdcve-2007.xml.gz
-rw-r--r-- 1 root root 11.2M Jun 12 07:48 nvdcve-2007.xml
-rw-r--r-- 1 root root 2.0M Jun 12 07:48 nvdcve-2.0-2007.xml.gz
-rw-r--r-- 1 root root 25.2M Jun 12 07:48 nvdcve-2.0-2007.xml
-rw-r--r-- 1 root root 2.3M Jun 12 07:48 nvdcve-1.0-2007.json.gz
-rw-r--r-- 1 root root 33.1M Jun 12 07:48 nvdcve-1.0-2007.json
-rw-r--r-- 1 root root 1.5M Jun 12 07:48 nvdcve-2008.xml.gz
-rw-r--r-- 1 root root 12.8M Jun 12 07:48 nvdcve-2008.xml
-rw-r--r-- 1 root root 2.2M Jun 12 07:48 nvdcve-2.0-2008.xml.gz
-rw-r--r-- 1 root root 30.9M Jun 12 07:48 nvdcve-2.0-2008.xml
-rw-r--r-- 1 root root 2.5M Jun 12 07:49 nvdcve-1.0-2008.json.gz
-rw-r--r-- 1 root root 40.6M Jun 12 07:49 nvdcve-1.0-2008.json
-rw-r--r-- 1 root root 1.3M Jun 12 07:49 nvdcve-2009.xml.gz
-rw-r--r-- 1 root root 11.3M Jun 12 07:49 nvdcve-2009.xml
-rw-r--r-- 1 root root 2.1M Jun 12 07:49 nvdcve-2.0-2009.xml.gz
-rw-r--r-- 1 root root 30.6M Jun 12 07:49 nvdcve-2.0-2009.xml
-rw-r--r-- 1 root root 2.4M Jun 12 07:49 nvdcve-1.0-2009.json.gz
-rw-r--r-- 1 root root 42.5M Jun 12 07:49 nvdcve-1.0-2009.json
-rw-r--r-- 1 root root 1.3M Jun 12 07:49 nvdcve-2010.xml.gz
-rw-r--r-- 1 root root 14.9M Jun 12 07:49 nvdcve-2010.xml
-rw-r--r-- 1 root root 2.8M Jun 12 07:49 nvdcve-2.0-2010.xml.gz
-rw-r--r-- 1 root root 45.6M Jun 12 07:49 nvdcve-2.0-2010.xml
-rw-r--r-- 1 root root 3.2M Jun 12 07:49 nvdcve-1.0-2010.json.gz
-rw-r--r-- 1 root root 67.0M Jun 12 07:49 nvdcve-1.0-2010.json
-rw-r--r-- 1 root root 3.0M Jun 12 07:49 nvdcve-2011.xml.gz
-rw-r--r-- 1 root root 29.8M Jun 12 07:49 nvdcve-2011.xml
-rw-r--r-- 1 root root 6.1M Jun 12 07:50 nvdcve-2.0-2011.xml.gz
-rw-r--r-- 1 root root 108.8M Jun 12 07:50 nvdcve-2.0-2011.xml
-rw-r--r-- 1 root root 6.6M Jun 12 07:50 nvdcve-1.0-2011.json.gz
-rw-r--r-- 1 root root 182.1M Jun 12 07:50 nvdcve-1.0-2011.json
-rw-r--r-- 1 root root 1.2M Jun 12 07:50 nvdcve-2012.xml.gz
-rw-r--r-- 1 root root 14.3M Jun 12 07:50 nvdcve-2012.xml
-rw-r--r-- 1 root root 2.5M Jun 12 07:50 nvdcve-2.0-2012.xml.gz
-rw-r--r-- 1 root root 43.2M Jun 12 07:50 nvdcve-2.0-2012.xml
-rw-r--r-- 1 root root 2.9M Jun 12 07:50 nvdcve-1.0-2012.json.gz
-rw-r--r-- 1 root root 61.9M Jun 12 07:50 nvdcve-1.0-2012.json
-rw-r--r-- 1 root root 1.3M Jun 12 07:50 nvdcve-2013.xml.gz
-rw-r--r-- 1 root root 15.2M Jun 12 07:50 nvdcve-2013.xml
-rw-r--r-- 1 root root 2.7M Jun 12 07:50 nvdcve-2.0-2013.xml.gz
-rw-r--r-- 1 root root 44.1M Jun 12 07:50 nvdcve-2.0-2013.xml
-rw-r--r-- 1 root root 3.0M Jun 12 07:50 nvdcve-1.0-2013.json.gz
-rw-r--r-- 1 root root 62.0M Jun 12 07:50 nvdcve-1.0-2013.json
-rw-r--r-- 1 root root 1.6M Jun 12 07:51 nvdcve-2014.xml.gz
-rw-r--r-- 1 root root 16.6M Jun 12 07:51 nvdcve-2014.xml
-rw-r--r-- 1 root root 2.8M Jun 12 07:51 nvdcve-2.0-2014.xml.gz
-rw-r--r-- 1 root root 43.9M Jun 12 07:51 nvdcve-2.0-2014.xml
-rw-r--r-- 1 root root 3.0M Jun 12 07:51 nvdcve-1.0-2014.json.gz
-rw-r--r-- 1 root root 60.0M Jun 12 07:51 nvdcve-1.0-2014.json
-rw-r--r-- 1 root root 1.5M Jun 12 07:51 nvdcve-2015.xml.gz
-rw-r--r-- 1 root root 15.9M Jun 12 07:51 nvdcve-2015.xml
-rw-r--r-- 1 root root 2.3M Jun 12 07:51 nvdcve-2.0-2015.xml.gz
-rw-r--r-- 1 root root 39.8M Jun 12 07:51 nvdcve-2.0-2015.xml
-rw-r--r-- 1 root root 2.6M Jun 12 07:51 nvdcve-1.0-2015.json.gz
-rw-r--r-- 1 root root 52.6M Jun 12 07:51 nvdcve-1.0-2015.json
-rw-r--r-- 1 root root 1.8M Jun 12 07:53 nvdcve-2019.xml.gz
-rw-r--r-- 1 root root 21.4M Jun 12 07:53 nvdcve-2019.xml
-rw-r--r-- 1 root root 3.9M Jun 12 07:53 nvdcve-2.0-2019.xml.gz
-rw-r--r-- 1 root root 75.0M Jun 12 07:53 nvdcve-2.0-2019.xml
-rw-r--r-- 1 root root 2.6M Jun 12 07:53 nvdcve-1.0-2019.json.gz
-rw-r--r-- 1 root root 66.0M Jun 12 07:53 nvdcve-1.0-2019.json
-rw-r--r-- 1 root root 162 Jun 12 08:00 nvdcve-1.0-modified.meta
-rw-r--r-- 1 root root 162 Jun 12 08:00 nvdcve-1.0-2003.meta
-rw-r--r-- 1 root root 165 Jun 12 08:00 nvdcve-1.0-2002.meta
-rw-r--r-- 1 root root 165 Jun 12 08:00 nvdcve-1.0-2007.meta
-rw-r--r-- 1 root root 165 Jun 12 08:00 nvdcve-1.0-2006.meta
-rw-r--r-- 1 root root 165 Jun 12 08:00 nvdcve-1.0-2005.meta
-rw-r--r-- 1 root root 163 Jun 12 08:00 nvdcve-1.0-2004.meta
-rw-r--r-- 1 root root 166 Jun 12 08:00 nvdcve-1.0-2011.meta
-rw-r--r-- 1 root root 165 Jun 12 08:00 nvdcve-1.0-2010.meta
-rw-r--r-- 1 root root 165 Jun 12 08:00 nvdcve-1.0-2009.meta
-rw-r--r-- 1 root root 165 Jun 12 08:00 nvdcve-1.0-2008.meta
-rw-r--r-- 1 root root 165 Jun 12 08:00 nvdcve-1.0-2016.meta
-rw-r--r-- 1 root root 165 Jun 12 08:00 nvdcve-1.0-2015.meta
-rw-r--r-- 1 root root 165 Jun 12 08:00 nvdcve-1.0-2014.meta
-rw-r--r-- 1 root root 165 Jun 12 08:00 nvdcve-1.0-2013.meta
-rw-r--r-- 1 root root 165 Jun 12 08:00 nvdcve-1.0-2012.meta
-rw-r--r-- 1 root root 1.7M Jun 12 08:00 nvdcve-2016.xml.gz
-rw-r--r-- 1 root root 20.0M Jun 12 08:00 nvdcve-2016.xml
-rw-r--r-- 1 root root 3.3M Jun 12 08:00 nvdcve-2.0-2016.xml.gz
-rw-r--r-- 1 root root 54.6M Jun 12 08:00 nvdcve-2.0-2016.xml
-rw-r--r-- 1 root root 3.4M Jun 12 08:00 nvdcve-1.0-2016.json.gz
-rw-r--r-- 1 root root 71.7M Jun 12 08:00 nvdcve-1.0-2016.json
-rw-r--r-- 1 root root 166 Jun 12 08:00 nvdcve-1.0-2017.meta
-rw-r--r-- 1 root root 4.1M Jun 12 08:00 nvdcve-2017.xml.gz
-rw-r--r-- 1 root root 46.2M Jun 12 08:00 nvdcve-2017.xml
-rw-r--r-- 1 root root 8.9M Jun 12 08:00 nvdcve-2.0-2017.xml.gz
-rw-r--r-- 1 root root 160.3M Jun 12 08:00 nvdcve-2.0-2017.xml
-rw-r--r-- 1 root root 6.7M Jun 12 08:00 nvdcve-1.0-2017.json.gz
-rw-r--r-- 1 root root 160.8M Jun 12 08:00 nvdcve-1.0-2017.json
-rw-r--r-- 1 root root 166 Jun 12 08:00 nvdcve-1.0-2018.meta
-rw-r--r-- 1 root root 5.6M Jun 12 08:00 nvdcve-2018.xml.gz
-rw-r--r-- 1 root root 62.4M Jun 12 08:01 nvdcve-2018.xml
-rw-r--r-- 1 root root 12.0M Jun 12 08:01 nvdcve-2.0-2018.xml.gz
-rw-r--r-- 1 root root 220.1M Jun 12 08:01 nvdcve-2.0-2018.xml
-rw-r--r-- 1 root root 8.5M Jun 12 08:01 nvdcve-1.0-2018.json.gz
-rw-r--r-- 1 root root 211.8M Jun 12 08:01 nvdcve-1.0-2018.json
-rw-r--r-- 1 root root 165 Jun 12 08:01 nvdcve-1.0-2019.meta
It seems like it's working as intended.
I'm planning to let the container run for another few more days and see what turns up in the logs.
Thank you again.
from nist-data-mirror.
chris-gillatt
commented on August 23, 2024
@Muzamri yes your understanding is correct. The source is really unreliable (The NVD servers) which is why I started using the nist-data-mirror in the first place. Getting the files into my company firewall was the easy part - making them a reliable source for owasp at the build stage was the problem for me. nist-data-mirror solves it!
from nist-data-mirror.
Muzamri
commented on August 23, 2024
Ah, alright -- I have a feeling back then, that the NVD servers are not (that) reliable -- the sequential errors that was logged, got me worried though.
I will be closing this issue.
Thank you again for your time. Cheers!
from nist-data-mirror.
Related Issues (20)
- Docker container doesn't mirror all files HOT 1
- k8s deprecation 'networking.k8s.io/v1beta1' in v1.22
- NistDataMirror.java has hardcoded Windows file separators HOT 2
- Download and uncompression are slower than expected. HOT 7
- Unable to build with IBM Java 8 HOT 2
- GitHub action - Unable to access jarfile nist-data-mirror.jar HOT 1
- Cron running twice
- nvdcve-1.1-2013.json.gz corrupt with nist-data-mirror 1.6.0 HOT 7
- Some downloads fail with ZipException HOT 2
- Unable to run nist-data-mirror on OpenShift HOT 3
- release 1.4.0 compiled against newer jdk HOT 2
- Docker Image 1.4.0 is throwing "Permission denied" issues HOT 1
- Allow to add volume and configure DNS in HELM chart. HOT 3
- HELM Chart - Allow http proxy values to be supplied through values.yaml. HOT 1
- Don't start the update process if one is already running HOT 1
- Introduce a logger HOT 1
- json 1.0 feeds no longer supported? HOT 5
- No .jar file for release 1.5.0 in releases HOT 1
- Using proxy with account
- java.io.EOFException: Unexpected end of ZLIB input stream HOT 8
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nist-data-mirror.