[ingestion/data-quality issue] GUAC ingestion failing for SBOM file generated from blackduck scanning tool about guac HOT 15 CLOSED

Thanks for opening the issue @pradyumnaparalikar! Looks like this is the first time we are dealing with Yocto recipe packages based on the error log. If you have an example SBOM we can test with, we can get the fix out relatively soon. Most likely the issue is here:

from guac.

Thanks @pradyumnaparalikar. Will look into this and have a fix out soon!

from guac.

@lumjjb is purlConvert still required with the fixes made to the upstream library and our updates: 00d978b#diff-9e2e4581a29812be036a73925744bcf86eb233ff3998ef0e8f2c661b5555718aL84

Also, types like "alpm", "apk", "huggingface", "githubactions", "mlflow", are defined by the purl library: purl.TypeYocto, purl.TypeHuggingface, purl.TypeApk, purl.TypeMLFlow, purl.TypeAlpm, was there a thought to do something different?

Edit: For the OCI and Docker usecase it makes sense to keep. For the others, we just default to using the packageurl-go?

from guac.

The initial thinking around this was to be intentional, so that we pick up things that we don't recognize, in this case i think we can add an exception and keep track of that exception list. i.e. defining a list of GUAC exception PURLs.

Also, @JPEWdev may be able to chime in on the YOCTO PURL.

from guac.

Should be all set to test again @pradyumnaparalikar. Let us know if you run into any issues.

from guac.

Hi, @pxp928 , how can I test the changes? do we have an image to test?
I am seeing some issues when I am trying to run "make" in guac directory
How it can be tested via binary?

from guac.

Make sure to pull the latest on main and you can run make build to get the updated binaries in the bin directory.

To build a new local image you can run make container which will build a local image called local-organic-guac.

Run:

export GUAC_IMAGE=local-organic-guac

docker-compose -f container_files/guac-demo-compose.yaml up --force-recreate

That will bring up the latest changes, and test like you did before via guacone in the bin directory.

from guac.

@pxp928

[ec2-user@ip-10-195-1-1 guac]$ pwd
/home/ec2-user/golang/guac

`[ec2-user@ip-10-195-1-1 guac]$ make build
goreleaser build --clean --snapshot --single-target
• starting build...
• loading path=.goreleaser.yaml
• skipping validate...
• loading environment variables
• getting and validating git state
• git state commit=3577d4dc0f26e3d826c540bee5f7142a0ee6a693 branch=main current_tag=v0.5.2 previous_tag=v0.5.1 dirty=false
• pipe skipped reason=disabled during snapshot mode
• parsing tag
• setting defaults
• partial
• snapshotting
• building snapshot... version=SNAPSHOT-3577d4dc
• running before hooks
• running hook=go mod tidy
• running hook=go generate ./...
• took: 18s
• checking distribution directory
• cleaning dist
• setting up metadata
• storing release metadata
• writing file=dist/metadata.json
• loading go mod information
• build prerequisites
• writing effective config file
• writing config=dist/config.yaml
• building binaries
• partial build match=target=linux_amd64
• partial build match=target=linux_amd64
• partial build match=target=linux_amd64
• partial build match=target=linux_amd64
• partial build match=target=linux_amd64
• building binary=dist/guaccollect_linux_amd64_v1/guaccollect-linux-amd64
• building binary=dist/guacgql_linux_amd64_v1/guacgql-linux-amd64
• building binary=dist/guacingest_linux_amd64_v1/guacingest-linux-amd64
• building binary=dist/guaccsub_linux_amd64_v1/guaccsub-linux-amd64
• partial build match=target=linux_amd64
• building binary=dist/guacone_linux_amd64_v1/guacone-linux-amd64
• building binary=dist/guacrest_linux_amd64_v1/guacrest-linux-amd64
• took: 8s
• universal binaries
• pipe skipped reason=5 errors occurred:
* no darwin binaries found with ids: guaccollect
* no darwin binaries found with ids: guacingest
* no darwin binaries found with ids: guaccsub
* no darwin binaries found with ids: guacone
* no darwin binaries found with ids: guacgql

• storing artifacts metadata
• writing file=dist/artifacts.json
• build succeeded after 25s
• thanks for using goreleaser!
cp: cannot create regular file './bin/dist/guaccsub_linux_amd64_v1/guaccsub-linux-amd64': No such file or directory
cp: cannot stat 'guaccsub': No such file or directory
cp: cannot stat 'guaccollect': No such file or directory
cp: cannot stat 'guacgql': No such file or directory
cp: cannot stat 'guacingest': No such file or directory
cp: cannot stat 'guacrest': No such file or directory
cp: cannot stat 'guacone': No such file or directory
make: *** [Makefile:119: build] Error 123`

Getting this error, and when I am trying with the root below error is coming

[ec2-user@ip-10-195-1-1 guac]$ sudo make build goreleaser is not installed. Please install goreleaser and try again. make: *** [Makefile:284: check-goreleaser-tool-check] Error 1

from guac.

@pradyumnaparalikar, can you show us your dist/artifacts.json?
It looks like goreleaser finished the build here and it's the cp command in the Markfile that run into issues - https://github.com/guacsec/guac/blob/main/Makefile#L119.
The cp command takes input from dist/artifacts.json to source and copy the binaries from goreleaser's dist dir to a bin dir at where the make command was run.

Can you also help us look at whether these binaries exist?
• building binary=dist/guaccollect_linux_amd64_v1/guaccollect-linux-amd64
• building binary=dist/guacgql_linux_amd64_v1/guacgql-linux-amd64
• building binary=dist/guacingest_linux_amd64_v1/guacingest-linux-amd64
• building binary=dist/guaccsub_linux_amd64_v1/guaccsub-linux-amd64
• building binary=dist/guacone_linux_amd64_v1/guacone-linux-amd64
• building binary=dist/guacrest_linux_amd64_v1/guacrest-linux-amd64

from guac.

[{"name":"metadata.json","path":"dist/metadata.json","internal_type":30,"type":"Metadata"},{"name":"guaccsub-linux-amd64","path":"dist/guaccsub_linux_amd64_v1/guaccsub-linux-amd64","goos":"linux","goarch":"amd64","goamd64":"v1","internal_type":4,"type":"Binary","extra":{"Binary":"guaccsub-linux-amd64","Ext":"","ID":"guaccsub"}},{"name":"guaccollect-linux-amd64","path":"dist/guaccollect_linux_amd64_v1/guaccollect-linux-amd64","goos":"linux","goarch":"amd64","goamd64":"v1","internal_type":4,"type":"Binary","extra":{"Binary":"guaccollect-linux-amd64","Ext":"","ID":"guaccollect"}},{"name":"guacgql-linux-amd64","path":"dist/guacgql_linux_amd64_v1/guacgql-linux-amd64","goos":"linux","goarch":"amd64","goamd64":"v1","internal_type":4,"type":"Binary","extra":{"Binary":"guacgql-linux-amd64","Ext":"","ID":"guacgql"}},{"name":"guacingest-linux-amd64","path":"dist/guacingest_linux_amd64_v1/guacingest-linux-amd64","goos":"linux","goarch":"amd64","goamd64":"v1","internal_type":4,"type":"Binary","extra":{"Binary":"guacingest-linux-amd64","Ext":"","ID":"guacingest"}},{"name":"guacrest-linux-amd64","path":"dist/guacrest_linux_amd64_v1/guacrest-linux-amd64","goos":"linux","goarch":"amd64","goamd64":"v1","internal_type":4,"type":"Binary","extra":{"Binary":"guacrest-linux-amd64","Ext":"","ID":"guacrest"}},{"name":"guacone-linux-amd64","path":"dist/guacone_linux_amd64_v1/guacone-linux-amd64","goos":"linux","goarch":"amd64","goamd64":"v1","internal_type":4,"type":"Binary","extra":{"Binary":"guacone-linux-amd64","Ext":"","ID":"guacone"}}]

This is the content of dist/artifacts.json.

I can see all the below binaries except linux_amd64

• building binary=dist/guaccollect_linux_amd64_v1/guaccollect-linux-amd64
• building binary=dist/guacgql_linux_amd64_v1/guacgql-linux-amd64
• building binary=dist/guacingest_linux_amd64_v1/guacingest-linux-amd64
• building binary=dist/guaccsub_linux_amd64_v1/guaccsub-linux-amd64
• partial build match=target=linux_amd64
• building binary=dist/guacone_linux_amd64_v1/guacone-linux-amd64
• building binary=dist/guacrest_linux_amd64_v1/guacrest-linux-amd64

from guac.

There's your bins...not sure why the Makefile would try to copy from ./bin/[rest of the bin path] which caused the error.

Do you want to try building the guac container and use the bins at that goreleaser dist location while we're figuring this out?

from guac.

@pradyumnaparalikar, if you are still having issues, we will be doing a release next week that will include the fixes.

from guac.

@sunnyyip, thanks for your help. I built the container locally by commenting on the copy line.

@pxp928 Thanks for following up; yesterday, I tried to build a container, and it was successful; data was also ingested.
Would you like to help me with the use case?
For example, we use Blackduck to scan XYZ products; Blackduck will report all the vuln info for various product components.
Now, I generated an SBOM file from BD and ingested the same into GUAC; what additional things can GUAC offer compared to BD?

from guac.

@pradyumnaparalikar It's going to be hard to compare a suite of vendor tools to just GUAC as there will be some overlap between some of the Synopsys tools. I haven't used Black Duck in a few years but I think the main thing GUAC does is it provides observability for you software supply chain. So as data comes in from any of a number of sources, could be black duck, could be open source tools, could be public databases, you get a better understanding of what's going on in your software supply chain.

The main difference from just a one time SCA/Vulnerability scan is that we hold onto that data so as new data comes in we can trigger on new vulnerabilities or issues as they arise as opposed to needing to run another scan. In addition we track the vulnerabilities across everywhere they might be not just in one project but across all your projects. Certain other tools might do something similar but we think our approach is pretty powerful as it allows you to easily understand both the state of the security in your software and its dependencies as well as how that changes over time.

from guac.

As you ingest more data about other projects, GUAC grows and understands the relationship between these projects (and their dependencies). So when the time comes when a new vulnerability or a troublesome dependency is discovered, GUAC will be able to notify you where you are affected (without having to re-scan all your projects again). It will also help you determine the most efficient patch plan based on an aggregation of all the data and where you are affected. An example of this is in our blog post here: https://www.kusari.dev/blog/terror-of-curl.

from guac.