Comments (8)
daxin09pp
commented on August 23, 2024
1
other Linux Server without proxy also not work..
XXXX@XXX-25-201:~/nvd-mirror$ java -jar /home/XXXX/nvd-mirror/nist-data-mirror2.jar /home/XXXX/nvd-mirror/data/
Downloading files at Mon Feb 22 17:03:56 CST 2021
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-modified.meta
Download succeeded nvdcve-1.1-modified.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2002.meta
Download succeeded nvdcve-1.1-2002.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2003.meta
Download succeeded nvdcve-1.1-2003.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2004.meta
Download succeeded nvdcve-1.1-2004.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2004.json.gz
Download succeeded nvdcve-1.1-2004.json.gz
Uncompressed nvdcve-1.1-2004.json
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2005.meta
Download succeeded nvdcve-1.1-2005.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2005.json.gz
Download succeeded nvdcve-1.1-2005.json.gz
java.io.EOFException: Unexpected end of ZLIB input stream
at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:245)
at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
at us.springett.nistdatamirror.NistDataMirror.uncompress(NistDataMirror.java:232)
at us.springett.nistdatamirror.NistDataMirror.doDownload(NistDataMirror.java:218)
at us.springett.nistdatamirror.NistDataMirror.downloadVersionForYear(NistDataMirror.java:160)
at us.springett.nistdatamirror.NistDataMirror.mirror(NistDataMirror.java:142)
at us.springett.nistdatamirror.NistDataMirror.main(NistDataMirror.java:84)
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2006.meta
Download succeeded nvdcve-1.1-2006.meta
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2006.json.gz
Download succeeded nvdcve-1.1-2006.json.gz
java.io.EOFException: Unexpected end of ZLIB input stream
at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:245)
at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159)
at java.base/java.util.zip.GZIPInputStream.read(GZIPInputStream.java:118)
at java.base/java.io.FilterInputStream.read(FilterInputStream.java:107)
at us.springett.nistdatamirror.NistDataMirror.uncompress(NistDataMirror.java:232)
at us.springett.nistdatamirror.NistDataMirror.doDownload(NistDataMirror.java:218)
at us.springett.nistdatamirror.NistDataMirror.downloadVersionForYear(NistDataMirror.java:160)
at us.springett.nistdatamirror.NistDataMirror.mirror(NistDataMirror.java:142)
at us.springett.nistdatamirror.NistDataMirror.main(NistDataMirror.java:84)
Downloading https://nvd.nist.gov/feeds/json/cve/1.1/nvdcve-1.1-2007.meta
from nist-data-mirror.
FavorMylikes
commented on August 23, 2024
Same issue, I think we need to check the original file form nist database.
from nist-data-mirror.
sellersj
commented on August 23, 2024
Even though it's downloading httpS files, try to use the http (no S) proxy
config. That is what has worked for me in the past.
Jim
…On Tue, Apr 13, 2021 at 8:12 AM FavorMylikes ***@***.***> wrote:
Same issue, I think we need to check the original file form nist database.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#78 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAABCJQWQR3LX5Z43R5NEGDTIQYKVANCNFSM4YADEVJQ>
.
from nist-data-mirror.
FavorMylikes
commented on August 23, 2024
@daxin09pp
After compare the gz file form https://nvd.nist.gov and local, I can be sure that some error bytes be inserted into the correct file.
It's kind of little weird, but I'm not sure how this happened.
Maybe it's a network problem.
So I use proxy by adding -Dhttp.proxyHost=YOUR_HTTP_PROXY_HOST -Dhttp.proxyPort=YOUT_HTTP_PROXY_PORT to redownload from https://nvd.nist.gov
That works again
from nist-data-mirror.
daxin09pp
commented on August 23, 2024
@FavorMylikes @sellersj
Thanks guys. It's really a network issue. When I use the proxy or the network is better, it is not a issue.
from nist-data-mirror.
PascalTurbo
commented on August 23, 2024
We are also running into this issue from time to time.
NVD provides the Size and the SHA256 Hash with the meta information. Would it be possible to add at least a check of the gz file size. At best uncompress the gz and compare the hash of the json file. If there is a different, download the file again.
from nist-data-mirror.
PascalTurbo
commented on August 23, 2024
@stevespringett would be great to merge the validCheck implemented by @falco9 as it should solve our problem with invalid files coming up once a week.
from nist-data-mirror.
stevespringett
commented on August 23, 2024
Fix has been merged. Thanks @falco9 for the PR.
from nist-data-mirror.
Related Issues (20)
- Docker container doesn't mirror all files HOT 1
- k8s deprecation 'networking.k8s.io/v1beta1' in v1.22
- NistDataMirror.java has hardcoded Windows file separators HOT 2
- Download and uncompression are slower than expected. HOT 7
- Unable to build with IBM Java 8 HOT 2
- GitHub action - Unable to access jarfile nist-data-mirror.jar HOT 1
- Cron running twice
- nvdcve-1.1-2013.json.gz corrupt with nist-data-mirror 1.6.0 HOT 7
- Some downloads fail with ZipException HOT 2
- Unable to run nist-data-mirror on OpenShift HOT 3
- release 1.4.0 compiled against newer jdk HOT 2
- Docker Image 1.4.0 is throwing "Permission denied" issues HOT 1
- Allow to add volume and configure DNS in HELM chart. HOT 3
- HELM Chart - Allow http proxy values to be supplied through values.yaml. HOT 1
- Don't start the update process if one is already running HOT 1
- Introduce a logger HOT 1
- json 1.0 feeds no longer supported? HOT 5
- No .jar file for release 1.5.0 in releases HOT 1
- Using proxy with account
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nist-data-mirror.