Comments (5)
I am interested in working on this one. Please assign me if possible. Thanks.
from dependency-track.
Hi, thank you for your contribute. There is any new about this issue?
from dependency-track.
Note that in all
CacheStampedeBlocker
warnings following theMetaAnalyzer
error, the reported key is actually properly URL-encoded.
The PURL specification requires URL encoding of special characters. However, when accessing individual parts of the PURL (say the package's name), you URL-decode to get the original value.
Analyzers will decode the PURL in order to assemble repository URLs, whereas warning / error logs will simply use the entire PURL, that's the reason why you're seeing this difference.
I'm not a programmer, but wouldn't it make sense to simply URL-encode any URL before passing it on to its respective MetaAnalyzer? 🤔 🤷♂️
Yes. I guess the original implementation assumed that PURLs would contain valid namespaces and names according to the respective ecosystem's conventions, but in the examples you shared that is clearly not the case.
To my knowledge, neither Maven, nor Python, nor NuGet packages are allowed to contain spaces.
We labeled this issue as good first issue
since it's easy to resolve. Seems like @laurentiu-ghergu did not end up working on it. If no one picks it up prior to the 4.12 release, I'll do it.
from dependency-track.
Similar issue was fixed for NPM in v4.11: #3456
from dependency-track.
Same problem here with Maven (https://repo1.maven.org/maven2/), NuGet (https://api.nuget.org/) and Python (https://pypi.org/). URLs passed to MetaAnalyzers do not get URL-encoded, so any URL with a space character (" "
) causes a Java error, and the project's risk analysis is aborted.
Maven example:
2024-08-13 08:58:19,086 ERROR [MavenMetaAnalyzer] Request failure
java.net.URISyntaxException: Illegal character in path at index 57: https://repo1.maven.org/maven2/org/ops4j/pax/url/mvn/mvn; singleton:=true/maven-metadata.xml
at java.base/java.net.URI$Parser.fail(Unknown Source)
at java.base/java.net.URI$Parser.checkChars(Unknown Source)
at java.base/java.net.URI$Parser.parseHierarchical(Unknown Source)
at java.base/java.net.URI$Parser.parse(Unknown Source)
at java.base/java.net.URI.<init>(Unknown Source)
at org.apache.http.client.utils.URIBuilder.<init>(URIBuilder.java:82)
at org.dependencytrack.tasks.repositories.AbstractMetaAnalyzer.processHttpRequest(AbstractMetaAnalyzer.java:98)
at org.dependencytrack.tasks.repositories.MavenMetaAnalyzer.analyze(MavenMetaAnalyzer.java:81)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2024-08-13 08:58:19,086 WARN [CacheStampedeBlocker] An error occurred while populating cache repositoryMetaCache for key pkg:maven/org.ops4j.pax.url.mvn/mvn%3B%20singleton%3A%[email protected] : java.lang.NullPointerException
org.dependencytrack.exception.MetaAnalyzerException: java.lang.NullPointerException
at org.dependencytrack.tasks.repositories.MavenMetaAnalyzer.analyze(MavenMetaAnalyzer.java:110)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException: null
NuGet example:
2024-08-13 08:33:49,448 ERROR [NugetMetaAnalyzer] Request failure
java.net.URISyntaxException: Illegal character in path at index 43: https://api.nuget.org/v3-flatcontainer/mono common language infrastructure/index.json
at java.base/java.net.URI$Parser.fail(Unknown Source)
at java.base/java.net.URI$Parser.checkChars(Unknown Source)
at java.base/java.net.URI$Parser.parseHierarchical(Unknown Source)
at java.base/java.net.URI$Parser.parse(Unknown Source)
at java.base/java.net.URI.<init>(Unknown Source)
at org.apache.http.client.utils.URIBuilder.<init>(URIBuilder.java:82)
at org.dependencytrack.tasks.repositories.AbstractMetaAnalyzer.processHttpRequest(AbstractMetaAnalyzer.java:98)
at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.performVersionCheck(NugetMetaAnalyzer.java:108)
at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.analyze(NugetMetaAnalyzer.java:99)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2024-08-13 08:33:49,448 WARN [CacheStampedeBlocker] An error occurred while populating cache repositoryMetaCache for key pkg:nuget/Mono%20Common%20Language%[email protected] : java.lang.NullPointerException
org.dependencytrack.exception.MetaAnalyzerException: java.lang.NullPointerException
at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.performVersionCheck(NugetMetaAnalyzer.java:124)
at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.analyze(NugetMetaAnalyzer.java:99)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException: null
Python/PyPi example:
2024-08-13 08:59:54,474 ERROR [PypiMetaAnalyzer] Request failure
java.net.URISyntaxException: Illegal character in path at index 27: https://pypi.org/pypi/magic file extensions/json
at java.base/java.net.URI$Parser.fail(Unknown Source)
at java.base/java.net.URI$Parser.checkChars(Unknown Source)
at java.base/java.net.URI$Parser.parseHierarchical(Unknown Source)
at java.base/java.net.URI$Parser.parse(Unknown Source)
at java.base/java.net.URI.<init>(Unknown Source)
at org.apache.http.client.utils.URIBuilder.<init>(URIBuilder.java:82)
at org.dependencytrack.tasks.repositories.AbstractMetaAnalyzer.processHttpRequest(AbstractMetaAnalyzer.java:98)
at org.dependencytrack.tasks.repositories.PypiMetaAnalyzer.analyze(PypiMetaAnalyzer.java:75)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
2024-08-13 08:59:54,474 WARN [CacheStampedeBlocker] An error occurred while populating cache repositoryMetaCache for key pkg:pypi/magic%20file%[email protected] : java.lang.NullPointerException
org.dependencytrack.exception.MetaAnalyzerException: java.lang.NullPointerException
at org.dependencytrack.tasks.repositories.PypiMetaAnalyzer.analyze(PypiMetaAnalyzer.java:107)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException: null
Note that in all CacheStampedeBlocker
warnings following the MetaAnalyzer
error, the reported key is actually properly URL-encoded. I'm not a programmer, but wouldn't it make sense to simply URL-encode any URL before passing it on to its respective MetaAnalyzer
? 🤔 🤷♂️
We are currently using syft-1.8.0
and DependencyTrack-4.9.1
, so if this problem is already fixed in a later version please let me know (but then this issue would already be closed, I guess 😇).
Thanks for listening! 🙂
Thomas
from dependency-track.
Related Issues (20)
- API response from GET /api/v1/team/{id} is missing data
- Importing SBOM, results in Exception HOT 2
- BOM upload fails when BOM XML contains multiple namespace declarations HOT 3
- cpe:2.3:a:perl:perl is incorrectly mapped to debian package perl instead of perl-base HOT 4
- Add Conan repository support for C++ projects HOT 2
- `TrivyAnalysisTaskIntegrationTest` is failing for Trivy v0.54.0 HOT 4
- Error while processing bom: Duplicate key HOT 23
- Components dashboard HOT 2
- high load due to massive updates on DEPENDENCYMETRICS and VULNERABILITY
- display CVEs from github the same way as the CVEs from NVD
- Trivy analyzer can't detect SLES OS HOT 1
- Migrate Trivy integration to use Trivy's gRPC API HOT 2
- NVD mirroring error HOT 4
- Add support for multiple non expression licenses
- Dependency Graphs & External References Not Displayed After Upgrade to 4.1.1.6 HOT 1
- OSS License Retrieval HOT 1
- Uncaught internal server error after successful user login HOT 2
- getting error 403 forbidden on ui and on the network tab "We're sorry but this app doesn't work properly without JavaScript enabled. Please enable it to continue." HOT 13
- StackOverflowError when processing sbom for the second time
- Package version matching fails for pre-release versions (e.g. rc1)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dependency-track.