Giter VIP home page Giter VIP logo

Comments (5)

laurentiu-ghergu avatar laurentiu-ghergu commented on September 22, 2024 3

I am interested in working on this one. Please assign me if possible. Thanks.

from dependency-track.

lameirat avatar lameirat commented on September 22, 2024 3

Hi, thank you for your contribute. There is any new about this issue?

from dependency-track.

nscuro avatar nscuro commented on September 22, 2024 2

Note that in all CacheStampedeBlocker warnings following the MetaAnalyzer error, the reported key is actually properly URL-encoded.

The PURL specification requires URL encoding of special characters. However, when accessing individual parts of the PURL (say the package's name), you URL-decode to get the original value.

Analyzers will decode the PURL in order to assemble repository URLs, whereas warning / error logs will simply use the entire PURL, that's the reason why you're seeing this difference.

I'm not a programmer, but wouldn't it make sense to simply URL-encode any URL before passing it on to its respective MetaAnalyzer? 🤔 🤷‍♂️

Yes. I guess the original implementation assumed that PURLs would contain valid namespaces and names according to the respective ecosystem's conventions, but in the examples you shared that is clearly not the case.

To my knowledge, neither Maven, nor Python, nor NuGet packages are allowed to contain spaces.

We labeled this issue as good first issue since it's easy to resolve. Seems like @laurentiu-ghergu did not end up working on it. If no one picks it up prior to the 4.12 release, I'll do it.

from dependency-track.

nscuro avatar nscuro commented on September 22, 2024

Similar issue was fixed for NPM in v4.11: #3456

from dependency-track.

z1atk0 avatar z1atk0 commented on September 22, 2024

Same problem here with Maven (https://repo1.maven.org/maven2/), NuGet (https://api.nuget.org/) and Python (https://pypi.org/). URLs passed to MetaAnalyzers do not get URL-encoded, so any URL with a space character (" ") causes a Java error, and the project's risk analysis is aborted.

Maven example:

2024-08-13 08:58:19,086 ERROR [MavenMetaAnalyzer] Request failure
java.net.URISyntaxException: Illegal character in path at index 57: https://repo1.maven.org/maven2/org/ops4j/pax/url/mvn/mvn; singleton:=true/maven-metadata.xml
        at java.base/java.net.URI$Parser.fail(Unknown Source)
        at java.base/java.net.URI$Parser.checkChars(Unknown Source)
        at java.base/java.net.URI$Parser.parseHierarchical(Unknown Source)
        at java.base/java.net.URI$Parser.parse(Unknown Source)
        at java.base/java.net.URI.<init>(Unknown Source)
        at org.apache.http.client.utils.URIBuilder.<init>(URIBuilder.java:82)
        at org.dependencytrack.tasks.repositories.AbstractMetaAnalyzer.processHttpRequest(AbstractMetaAnalyzer.java:98)
        at org.dependencytrack.tasks.repositories.MavenMetaAnalyzer.analyze(MavenMetaAnalyzer.java:81)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
        at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
        at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
        at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
2024-08-13 08:58:19,086 WARN [CacheStampedeBlocker] An error occurred while populating cache repositoryMetaCache for key pkg:maven/org.ops4j.pax.url.mvn/mvn%3B%20singleton%3A%[email protected] : java.lang.NullPointerException
org.dependencytrack.exception.MetaAnalyzerException: java.lang.NullPointerException
        at org.dependencytrack.tasks.repositories.MavenMetaAnalyzer.analyze(MavenMetaAnalyzer.java:110)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
        at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
        at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
        at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException: null

NuGet example:

2024-08-13 08:33:49,448 ERROR [NugetMetaAnalyzer] Request failure
java.net.URISyntaxException: Illegal character in path at index 43: https://api.nuget.org/v3-flatcontainer/mono common language infrastructure/index.json
        at java.base/java.net.URI$Parser.fail(Unknown Source)
        at java.base/java.net.URI$Parser.checkChars(Unknown Source)
        at java.base/java.net.URI$Parser.parseHierarchical(Unknown Source)
        at java.base/java.net.URI$Parser.parse(Unknown Source)
        at java.base/java.net.URI.<init>(Unknown Source)
        at org.apache.http.client.utils.URIBuilder.<init>(URIBuilder.java:82)
        at org.dependencytrack.tasks.repositories.AbstractMetaAnalyzer.processHttpRequest(AbstractMetaAnalyzer.java:98)
        at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.performVersionCheck(NugetMetaAnalyzer.java:108)
        at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.analyze(NugetMetaAnalyzer.java:99)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
        at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
        at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
        at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
2024-08-13 08:33:49,448 WARN [CacheStampedeBlocker] An error occurred while populating cache repositoryMetaCache for key pkg:nuget/Mono%20Common%20Language%[email protected] : java.lang.NullPointerException
org.dependencytrack.exception.MetaAnalyzerException: java.lang.NullPointerException
        at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.performVersionCheck(NugetMetaAnalyzer.java:124)
        at org.dependencytrack.tasks.repositories.NugetMetaAnalyzer.analyze(NugetMetaAnalyzer.java:99)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
        at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
        at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
        at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException: null

Python/PyPi example:

2024-08-13 08:59:54,474 ERROR [PypiMetaAnalyzer] Request failure
java.net.URISyntaxException: Illegal character in path at index 27: https://pypi.org/pypi/magic file extensions/json
        at java.base/java.net.URI$Parser.fail(Unknown Source)
        at java.base/java.net.URI$Parser.checkChars(Unknown Source)
        at java.base/java.net.URI$Parser.parseHierarchical(Unknown Source)
        at java.base/java.net.URI$Parser.parse(Unknown Source)
        at java.base/java.net.URI.<init>(Unknown Source)
        at org.apache.http.client.utils.URIBuilder.<init>(URIBuilder.java:82)
        at org.dependencytrack.tasks.repositories.AbstractMetaAnalyzer.processHttpRequest(AbstractMetaAnalyzer.java:98)
        at org.dependencytrack.tasks.repositories.PypiMetaAnalyzer.analyze(PypiMetaAnalyzer.java:75)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
        at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
        at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
        at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
2024-08-13 08:59:54,474 WARN [CacheStampedeBlocker] An error occurred while populating cache repositoryMetaCache for key pkg:pypi/magic%20file%[email protected] : java.lang.NullPointerException
org.dependencytrack.exception.MetaAnalyzerException: java.lang.NullPointerException
        at org.dependencytrack.tasks.repositories.PypiMetaAnalyzer.analyze(PypiMetaAnalyzer.java:107)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:174)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.lambda$analyze$0(RepositoryMetaAnalyzerTask.java:121)
        at io.github.resilience4j.retry.Retry.lambda$decorateCallable$5(Retry.java:237)
        at io.github.resilience4j.retry.Retry.executeCallable(Retry.java:373)
        at org.dependencytrack.util.CacheStampedeBlocker.readThroughOrPopulateCache(CacheStampedeBlocker.java:201)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.analyze(RepositoryMetaAnalyzerTask.java:126)
        at org.dependencytrack.tasks.repositories.RepositoryMetaAnalyzerTask.inform(RepositoryMetaAnalyzerTask.java:104)
        at alpine.event.framework.BaseEventService.lambda$publish$0(BaseEventService.java:107)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.lang.NullPointerException: null

Note that in all CacheStampedeBlocker warnings following the MetaAnalyzer error, the reported key is actually properly URL-encoded. I'm not a programmer, but wouldn't it make sense to simply URL-encode any URL before passing it on to its respective MetaAnalyzer? 🤔 🤷‍♂️

We are currently using syft-1.8.0 and DependencyTrack-4.9.1, so if this problem is already fixed in a later version please let me know (but then this issue would already be closed, I guess 😇).

Thanks for listening! 🙂
Thomas

from dependency-track.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.