Dependency-Track NVD API mirroring for Dependency-Check about dependency-track HOT 24 OPEN

Sorry folks, the cache implementation did not make it to 4.10.1. I'll assign this to 4.11; We can still decide to pre-pone the release to 4.10.x once it's fully implemented.

from dependency-track.

For those not following the linked DependencyCheck thread, what we'll do on the DT side is to provide an optional feature which can be used to emulate the NVD's REST API: jeremylong/DependencyCheck#6277 (comment)

Dependency-Check 9.0.5 now supports customization of the REST API URL, so once this is implemented in DT, you'll be able to point DC to DT's emulated API endpoint.

from dependency-track.

@aggeboe Thank you for referencing the dependency-check issue. We have the same scenario indeed.

from dependency-track.

It might be best that DT adopts to working with the cache created by vulnz, which is what DC is already doing or will be doing. That way you could host your cache centrally, and serve both DT and DC with it.

But that would mean the introduction of another piece of infrastructure. I would rather like DT to function as this central cache, just like before with dependency-track 4.9.x and dependency-check 8.x.
DT claims to feature NVD 2.0 API mirroring, although it is not well-documented how to consume the cache. From what I can tell the DT codebase it is already relying on vulnz client library. The vulnz cli tool uses the same library (also by @jeremylong):

<dependency>
   <groupId>io.github.jeremylong</groupId>
   <artifactId>open-vulnerability-clients</artifactId>
   <version>5.1.0</version>
</dependency>

Class org/dependencytrack/tasks/NistApiMirrorTask.java (https://github.com/DependencyTrack/dependency-track/blob/master/src/main/java/org/dependencytrack/tasks/NistApiMirrorTask.java) has dependencies on io.github.jeremylong.openvulnerability.client.*

from dependency-track.

On Moday I installed 4.10.0 and set up the new NIST Rest API.
As far as i see, DT has not set a Last Modification Time yet and therefore pulls every day the complete list.
I see this in my logs:

dependency-track.log:2023-12-14 09:27:32,686 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 2000/233111 CVEs (0%)
dependency-track.log:2023-12-14 09:27:49,277 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 4000/233111 CVEs (1%)
dependency-track.log:2023-12-14 09:28:04,956 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 6000/233111 CVEs (2%)
dependency-track.log:2023-12-14 09:28:21,147 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 8000/233111 CVEs (3%)

is this related to the OP issue?

from dependency-track.

@Kretikus Does it ever complete successfully? There should be a log line stating:

Mirroring of CVEs completed in

Unfortunately due to the way the API was designed, we cannot store intermediate last modification timestamps, it's only possible at the very end of the mirror operation...

@officerNordberg Can you share your configuration in the UI? What toggles did you enable / disable? I've gotten some feedback via Slack already that the way it's structured is confusing so I'll definitely get that improved.

As an FYI we did get confirmation from the NVD via email that feeds will not be retired as planned on 18th. I asked about a public statement but haven't yet gotten feedback. But I want you to know that, as per my current knowledge, you will not be cut off from NVD updates if the REST API integration doesn't work for you right now.

from dependency-track.

@nscuro got the same heads up on the predictable further delay of the feed retirement through yocto/linux foundation this morning.

I have all 3 toggles enabled, I have tried just enabling API as well as just the first two without the "Additionally..." toggle.

from dependency-track.

@Kretikus try restarting, not only did the restart fix my missing EventSubscription for the ApiTask but once my mirror completed, the Last updated fields remained empty until I restarted yet again.

from dependency-track.

An official update has been posted here: https://groups.google.com/a/list.nist.gov/g/nvd-news/c/aofnAd3HP2g

Due to feedback received from many different downstream data consumer groups after our previous reminder, we will again be extending the retirement date for the Legacy Data Feed files. However, we will still be retiring the 1.0 APIs.
Going forward, we will be improving capabilities to allow for bulk download of the NVD dataset. The legacy data feeds will remain available until this effort is completed.

For everyone experiencing issues with the new REST API integration, you can safely switch back to the previous feed-based way of mirroring.

from dependency-track.

@tomkuipers To my understanding, dependency-check 9.x will not work with dependencytrack's NVD mirror functionality, because only NVD API 1.1 (data feeds) are made available for other tools.

I suspect I'm struggling with the same scenario

• dependencytrack 4.9.x
• dependencycheck 8.x getting NVD API 1.1 updates from /mirror/nvd

and now want to switch to (not working or haven't found a solution yet)

• dependencytrack 4.10.0
• dependency-check 9.x and get NVD API 2.0 updates from dependencytrack

This issue maybe relevant for you as well: jeremylong/DependencyCheck#6277

from dependency-track.

It might be best that DT adopts to working with the cache created by vulnz, which is what DC is already doing or will be doing. That way you could host your cache centrally, and serve both DT and DC with it.

from dependency-track.

If DT is concerned to avoid EOL-ing the NVD API 1.1 (data feeds) at https://my.dependency-track.host/mirror/nvd
for existing downstream consumers, it may be a good feature to have an additional url
https://my.dependency-track.host/mirror/nvd-v2 (or equiv),
so legacy downstream consumers are not being disrupted and downstream consumers such as dependency-check V9+
could use a different path mirror/nvd-v2 to have vulnz/dependency-check 9+ compatible feeds.

CI implementations of dependency-check V9+ have to change its use anyway, as dependency-check V9+'s use of command line arguments changed with the support of NIST NVD apiv2 support,
so having a new endpoint URL in the local DT server would be a good compromise IMHO.

from dependency-track.

@tomkuipers +1
We used to have a central caching mechanism in our organisation (H2 pre-built NVD image) to speed up dependency-check in CI/CD. When DT was introduced in our organisation with the NVD 1.1 data feed mirror, we immediately switched to the DT cache to utilize the mirror/cache already in place here.

Thus, we really hoped that a similiar mechanism would be in place for the NVD API 2.0 in DT 4.10.0, for it to continue to function as a central cache.

from dependency-track.

@nscuro Thanks for all your hard work, really looking forward to release 4.10.1. Any idea when it migth be released?

from dependency-track.

I'm also looking forward to version v.4.10.1, thank you @nscuro for implementing the feature and collaborating with dependency-check's @jeremylong to find a solution.

from dependency-track.

Yesterday I installed 4.10.0 and set up the new NIST Rest API.
As far as i see, DT has not set a Last Modification Time yet NOR does my log contain any NistApiMirrorTask lines or any apparent errors. I've changed the Task Schedule from 24 hours to 1 hour, waited and no dice. I've disabled and reenabled a few times. I've been tempted to set the Last Mod variables but have heretofore headed the Configuration warning. Not sure what to do next.

from dependency-track.

@nscuro @officerNordberg Thanks for the heads up, regarding the feed retirement delay! Much appreciated!

from dependency-track.

@nscuro I should have realized that configuring it requires a restart afterward. I don't know why I didn't expect that but, yup, that did it.

from dependency-track.

The NVD setting should not require a restart. The decision of which source to use is happening dynamically at runtime.

... but I think change of schedule does, because tasks are scheduled on startup based on what is configured at that moment in time. So yeah that doesn't make testing this any easier because impact can't be observed immediately. Yet another reason for adding the ability to manually trigger tasks.

from dependency-track.

Sounds like (I hope), they're gonna reinstate some kind of batch download feeds.

from dependency-track.

@Kretikus Does it ever complete successfully? There should be a log line stating:

Mirroring of CVEs completed in

Yes, here is the log from today:

2023-12-18 09:55:30,067 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 196000/233564 CVEs (83%)
2023-12-18 09:55:44,206 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 198000/233564 CVEs (84%)
2023-12-18 09:55:59,579 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 200000/233564 CVEs (85%)
2023-12-18 09:56:13,136 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 202000/233564 CVEs (86%)
2023-12-18 09:56:23,415 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 204000/233564 CVEs (87%)
2023-12-18 09:56:37,305 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 206000/233564 CVEs (88%)
2023-12-18 09:56:54,020 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 208000/233564 CVEs (89%)
2023-12-18 09:57:08,287 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 210000/233564 CVEs (89%)
2023-12-18 09:57:20,495 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 212000/233564 CVEs (90%)
2023-12-18 09:57:36,939 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 214000/233564 CVEs (91%)
2023-12-18 09:57:51,644 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 216000/233564 CVEs (92%)
2023-12-18 09:58:07,079 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 218000/233564 CVEs (93%)
2023-12-18 09:58:23,707 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 220000/233564 CVEs (94%)
2023-12-18 09:58:38,319 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 222000/233564 CVEs (95%)
2023-12-18 09:58:53,594 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 224000/233564 CVEs (95%)
2023-12-18 09:59:06,151 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 226000/233564 CVEs (96%)
2023-12-18 09:59:22,012 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 228000/233564 CVEs (97%)
2023-12-18 09:59:35,948 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 230000/233564 CVEs (98%)
2023-12-18 09:59:50,941 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirrored 232000/233564 CVEs (99%)
2023-12-18 10:00:01,439 [] INFO [org.dependencytrack.tasks.NistApiMirrorTask] Mirroring of 233564 CVEs completed in PT32M46.652815377S

A restart showed now a date, but with a wrong time. The times above are CET (UTC+1) and the screenshot in the front-end shows:

from dependency-track.

My last update seems to only update on restarts read from the Config loaded at restart.

2023-12-15 17:16:59,679 INFO [NistApiMirrorTask] Mirroring of 233501 CVEs completed in PT24M51.368955359S
2023-12-15 19:56:29,793 INFO [NistApiMirrorTask] Mirroring CVEs that were modified since 2023-12-15T16:53:57Z
2023-12-15 19:56:33,445 INFO [NistApiMirrorTask] Mirroring of 49 CVEs completed in PT3.655197639S
2023-12-16 19:56:29,784 INFO [NistApiMirrorTask] Mirroring CVEs that were modified since 2023-12-15T16:53:57Z
2023-12-16 19:56:34,960 INFO [NistApiMirrorTask] Mirroring of 219 CVEs completed in PT5.17599173S
2023-12-17 19:56:29,784 INFO [NistApiMirrorTask] Mirroring CVEs that were modified since 2023-12-15T16:53:57Z
2023-12-17 19:56:35,536 INFO [NistApiMirrorTask] Mirroring of 247 CVEs completed in PT5.752890702S

from dependency-track.

My last update seems to only update on restarts read from the Config loaded at restart.

Checking. Looks like the property in question is loaded from the ORMs cache even though it shouldn't, and cache has been explicitly disabled for it...

@Kretikus A restart showed now a date, but with a wrong time. The times above are CET (UTC+1) and the screenshot in the front-end shows: [...]

The time in the frontend does not represent the time when Mirroring of 233564 CVEs completed was logged, but the latest "last modification" timestamp of all mirrored CVEs in that run. So even though your mirroring operation completed "now", the frontend may show a time from (way) earlier in the day.

from dependency-track.

Yup, the last modification timestamp requiring a restart to take effect is caused by inconsistencies of the ORM's L2 cache. Fixed in #3322 and backported to 4.10.1 in #3323.

If anyone is feeling adventurous, you can disable the L2 cache system-wide via alpine.datanucleus.cache.level2.type=none / ALPINE_DATANUCLEUS_CACHE_LEVEL2_TYPE=none. Other than resolving this particular issue, it also improves memory footprint quite a bit. We disabled it in Hyades and only saw benefits. I am planning to disable it per default for DT v4.x as well eventually, but would love some input whether others are running into issues with L2 disabled.

from dependency-track.