Comments (8)
other Linux Server without proxy also not work..
XXXX@XXX-25-201:~/nvd-mirror$ java -jar /home/XXXX/nvd-mirror/nist-data-mirror2.jar /home/XXXX/nvd-mirror/data/
Downloading files at Mon Feb 22 17:03:56 CST 2021
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.meta
Download succeeded nvdcve-1.1-modified.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2002.meta
Download succeeded nvdcve-1.1-2002.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2003.meta
Download succeeded nvdcve-1.1-2003.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2004.meta
Download succeeded nvdcve-1.1-2004.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2004.json.gz
Download succeeded nvdcve-1.1-2004.json.gz
Uncompressed nvdcve-1.1-2004.json
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2005.meta
Download succeeded nvdcve-1.1-2005.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2005.json.gz
Download succeeded nvdcve-1.1-2005.json.gz
java.io.EOFException: Unexpected end of ZLIB input stream
at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:245)
at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
at us.springett.nistdatamirror.NistDataMirror.uncompress(NistDataMirror.java:232)
at us.springett.nistdatamirror.NistDataMirror.doDownload(NistDataMirror.java:218)
at us.springett.nistdatamirror.NistDataMirror.downloadVersionForYear(NistDataMirror.java:160)
at us.springett.nistdatamirror.NistDataMirror.mirror(NistDataMirror.java:142)
at us.springett.nistdatamirror.NistDataMirror.main(NistDataMirror.java:84)
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2006.meta
Download succeeded nvdcve-1.1-2006.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2006.json.gz
Download succeeded nvdcve-1.1-2006.json.gz
java.io.EOFException: Unexpected end of ZLIB input stream
at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:245)
at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
at us.springett.nistdatamirror.NistDataMirror.uncompress(NistDataMirror.java:232)
at us.springett.nistdatamirror.NistDataMirror.doDownload(NistDataMirror.java:218)
at us.springett.nistdatamirror.NistDataMirror.downloadVersionForYear(NistDataMirror.java:160)
at us.springett.nistdatamirror.NistDataMirror.mirror(NistDataMirror.java:142)
at us.springett.nistdatamirror.NistDataMirror.main(NistDataMirror.java:84)
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2007.meta
from nist-data-mirror.
Same issue, I think we need to check the original file form nist database.
from nist-data-mirror.
from nist-data-mirror.
@daxin09pp
After compare the gz file form https://nvd.nist.gov
and local, I can be sure that some error bytes be inserted into the correct file.
It's kind of little weird, but I'm not sure how this happened.
Maybe it's a network problem.
So I use proxy by adding -Dhttp.proxyHost=YOUR_HTTP_PROXY_HOST -Dhttp.proxyPort=YOUT_HTTP_PROXY_PORT
to redownload from https://nvd.nist.gov
That works again
from nist-data-mirror.
@FavorMylikes @sellersj
Thanks guys. It's really a network issue. When I use the proxy or the network is better, it is not a issue.
from nist-data-mirror.
We are also running into this issue from time to time.
NVD provides the Size and the SHA256 Hash with the meta information. Would it be possible to add at least a check of the gz file size. At best uncompress the gz and compare the hash of the json file. If there is a different, download the file again.
from nist-data-mirror.
@stevespringett would be great to merge the validCheck implemented by @falco9 as it should solve our problem with invalid files coming up once a week.
from nist-data-mirror.
Fix has been merged. Thanks @falco9 for the PR.
from nist-data-mirror.
Related Issues (20)
- Docker container doesn't mirror all files HOT 1
- k8s deprecation 'networking.k8s.io/v1beta1' in v1.22
- NistDataMirror.java has hardcoded Windows file separators HOT 2
- Download and uncompression are slower than expected. HOT 7
- Unable to build with IBM Java 8 HOT 2
- GitHub action - Unable to access jarfile nist-data-mirror.jar HOT 1
- Cron running twice
- nvdcve-1.1-2013.json.gz corrupt with nist-data-mirror 1.6.0 HOT 7
- Some downloads fail with ZipException HOT 2
- Unable to run nist-data-mirror on OpenShift HOT 3
- release 1.4.0 compiled against newer jdk HOT 2
- Docker Image 1.4.0 is throwing "Permission denied" issues HOT 1
- Allow to add volume and configure DNS in HELM chart. HOT 3
- HELM Chart - Allow http proxy values to be supplied through values.yaml. HOT 1
- Don't start the update process if one is already running HOT 1
- Introduce a logger HOT 1
- json 1.0 feeds no longer supported? HOT 5
- No .jar file for release 1.5.0 in releases HOT 1
- Using proxy with account
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nist-data-mirror.