Description

SBOM Analysis family status is missing from the AssetScan drill down

Expected Behavior

SBOM Analysis family status should be present the same as the other families in the drill down.

Actual Behavior

Affected Version

main

Steps to Reproduce

1. Go to drill down page for a specific asset scan

Checklist

• I have read the contributing guidelines
• I have verified this does not duplicate an existing issue

Problem Statement

If a user accidentially exits the scan configuration step with say "all" this could start a large amount of scans. There does not seem a safety net and/or way to cancel these mistakenly triggered scans

Proposed Solution

Add a "cancel/stop scan" button in the below screen that terminates the underlying instance performing the scan

Alternatives Considered

Perhaps don't auto initiate the scans on exit from configuration definition and have the user go to a screen to to say "Go" - this could also be a useful API endpoint if it doesn't already exist.

Additional Context

The ability to cancel could save end users real world cost and overhead

Problem Statement

It would be cool if we were able to create a docker provider (not just for testing) which can discovery running container assets and images, as well as snapshot the running containers them through docker save and scan both the snapshots and images.

Proposed Solution

A new provider like the other cloud providers, which talks to the docker daemon to get a list of running containers and container images. These are reported as assets to the control plane. When an asset is chosen to be scanned the provider will receive the asset, determine if it is a runtime container or an image, then follow this flow:

If the asset is a runtime container:

• A snapshot will be taken through docker commit saving the containers file system to a new container image

Then:

• A scanner container will be booted with the docker socket mounted, and then the container image (either the original or the runtime snapshot) is configured as the input to the VMClarity CLI as a local container image.

Alternatives Considered

None

Additional Context

This can also be used as a good candidate for e2e testing because it should not require anything more than a machine running docker.

Problem Statement

Support additional clouds (e.g.: Openstack) so that VMClarity can be deployed and integrated into other public or private clouds.

Proposed Solution

Implement Cloud Provider model plugin so that cloud integrations are abstracted using exiting vendor-provided so that VMs/PVs volume snapshots can be scanned in such environments:

• https://github.com/topics/k8s-sig-cloud-provider
• https://github.com/topics/k8s-sig-storage

Alternatives Considered

None

Additional Context

N/A

As a user of VMClarity I would like to be able to install VMClarity control plane once, and use that instance of VMClarity to discover and scan assets from multiple cloud accounts, kubernetes clusters or docker daemons.

Subtasks:

• #581
• #637
• #780
• #640
• #641
• #642
• #643
• #644
• #645

Windows is a supported operating system on many of the public clouds, for example AWS and Azure both have support for Windows images. Today the VMClarity scanner is only capable of scanning Linux/Unix style volumes for issues. There are several reasons for this:

a) A number of the tools/scanners used by VMClarity only look for issues in Linux/Unix file locations
b) The VMClarity scanner will only find and mount common linux volume formats (ext4, ext3 etc) and not NTFS.

Operating system awareness (manual input / detected) should be added to the VMClarity scanning infrastructure such that scanners can decide:

a) if they are capable of scanning the provided input
b) if they need to do anything different to scan the target input

We should document what Operating Systems each scanner in all the families support so that users can create scan configurations which perform successfully if targeted at windows nodes.

We should look to see if there is at least one sbom and vulnerability scanner that supports a windows file system, and if there isn't we should add one to provide the minimum level of support for windows in VMClarity.

Description

The Scanner may stuck on volume mounting indefinitely (actually up until the Scan timeout) in case there is underlying operating system issue.

Expected Behavior

Scanner should fail if volume mounting is taking too long (>5mins).

Actual Behavior

Scanner stuck on volume mounting indefinitely.

Affected Version

All

Steps to Reproduce

There are no exact steps to reproduce as it requires the OS to be in a weird state where it blocks mount calls targeting volumes with issue indefinitely.

Checklist

• I have read the contributing guidelines
• I have verified this does not duplicate an existing issue

Description

Call to /api/scans API always returns a null targetIDs for all returned Scan items.
/api/scanResults for the same scan correctly populates the target.id property.

Expected Behavior

List of targets present in the output from /api/scans

Actual Behavior

(see description)

Affected Version

Version: latest
Commit: 3e45aa3bbd2689c6c12547943b6d1fa674307f88

Steps to Reproduce

1. Have at least 1 scan finished
2. Call <host>:<port>/api/scans trough Postman or similar tool

Checklist

• I have read the contributing guidelines
• I have verified this does not duplicate an existing issue

Problem Statement

I'm the author of cdxgen and dep-scan.

cdxgen is a polyglot sbom generation tool from the CycloneDX community.
dep-scan is an advanced OSS audit tool with automatic prioritization

Both the tools are available as single binary executables, oci images and support both cli and REST-based integration.

https://github.com/CycloneDX/cdxgen#sbom-server
https://github.com/AppThreat/dep-scan#server-mode

Proposed Solution

cdxgen would increase the number of languages and frameworks supported. depscan would add the much needed triaging and prioritization capabilities

Alternatives Considered

Please briefly describe which alternatives, if any, have been considered, including merits of alternate approaches and
tradeoffs being made.

Additional Context

Please provide any other information that may be relevant.

Re-enable dupword and wastedassign linters in golangci-lint configuration and fix reported errors.

See: #331 (comment)

Description

Not all the UI queries are using the $select, as a result, the waiting time for the page to load is longer than expected.
For example, the following should improved

http://localhost:8889/api/scanResults/00ac1da0-cc08-49fb-b3b2-9a097da3523d?$expand=scan,scan/scanConfig,target

You can select the info you need

$select=status,summary

Or even sub-select from an expended field

$expand=scan/scanConfig($select=name,scheduled),scan($select=scanConfigSnapshot) 

There is a need to go over all UI queries to see how they can be improved.

Expected Behavior

UI queries should not GET information the UI doesn't need

Checklist

• I have read the contributing guidelines
• I have verified this does not duplicate an existing issue

There are a couple of packages that rely on cgo. In the long run, it would be good to know:

• which packages require cgo
• are there cgo-free alternatives for such packages
• possibilities of porting functionalities as cgo-free modules

Is your feature request related to a problem? Please describe.
The VMClarity readme.md provides an AWS CloudFormation Template for easy spin-up in an AWS account. The file is hosted in GitHub vs. the AWS supported location of an S3 bucket. Per the AWS guidelines the templateURL must be in an S3 bucket or you will receive an error of "TemplateURL must be a supported URL." Reference documentation: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-stack.html

Describe the solution you'd like
Place the https://raw.githubusercontent.com/openclarity/vmclarity/main/installation/aws/VmClarity.cfn file into a public S3 bucket (once we move the project to public).

Description

Failed to get riskiest asset widget info from the backend.
The following error is shown in the logs.

Jun 11 14:46:52 ip-10-0-0-205 docker[8436]: time="2023-06-11T14:46:52Z" level=warning msg="Failed to get asset info, skipping target: failed to get asset type: unsupported provider: aws"

Checklist

• I have read the contributing guidelines
• I have verified this does not duplicate an existing issue

Today "findings" are represented in VMClarity as a flat structure with a lot of duplicated information. For example if a vulnerability is found in two different assets then that is represented as two "findings" one for each asset.

A much more efficient and industry standard way to represent this sort of information is using a graph with nodes representing things like assets, packages, vulnerabilities; and edges represent relationships between those nodes.

Using a structure like this great reduces the duplication when storing this information as a vulnerability will appear once in the system, and so will a package. In addition this graph becomes much easier to traverse in order to answer user queries like "how many assets are affected by vulnerability X".

There are a number of ways to build/store a graph, but its most likely the to be most performant by using a DB implementation that is built for it for example Neo4j.

A example of the graph finding structure might look something like:
Note: not all finding types are shown here

location
  /\
  ||
  \/
asset <--> package <--> vulnerability <--> exploit
  /\
  ||
  \/
malware

It is likely that this graph should be a separate microservice built on top of the existing scanning API, effectively separating the data collection (scanning layer) and the post processed searchable view.

However it could also be interesting to investigate representing all the objects in the system in a graph for example:

scan
  /\
  ||
  \/
assetScan
  /\
  ||
  \/
asset <--> package <--> vulnerability <--> exploit
  /\
  ||
  \/
malware

We should first investigate graph DB options, their advantages and disadvantages, what the query language looks like, and how it will either interact with or replace ODATA in the API. We should also consider a home grown graph (nodes and links presented in SQLite/Postgres) and how that compares.

Once we choose an option, we should draw up a design for how to represent the data we have today in that system, what changes needs to be made etc.

After that we can start opening implementation tasks.

#377 (review)
.golangci.yml:

      # Replace for GORM update
      # TODO(chrisgacsal): remove when https://github.com/anchore/sqlite/pull/3 is merged
      - github.com/anchore/sqlite

Problem Statement

Currently, VMClarity performs a full scan of all libraries during each scan, regardless of whether or not a bug has been detected in a specific library (e.g., PyTorch). This approach can be time-consuming and inefficient, especially when dealing with large libraries. It would be beneficial to optimize the scanning process by introducing a fast-mode scan that targets only the libraries that need to be checked for bug fixes.

Proposed Solution

Implement a fast-mode scan in VMClarity that selectively scans only the libraries where bugs have been detected in previous scans. The fast-mode scan will significantly improve the performance of VMClarity during subsequent scans because it eliminates the need to scan every library from scratch to check whether the bug is fixed or not. By selectively scanning only the libraries that require bug checks, VMClarity can save time and computing resources, resulting in faster and more efficient scanning processes.

However, it is important to note that the proposed fast-mode scan should be optional for users. Some users may still prefer to conduct a full scan even if bugs have been detected in specific libraries.

Alternatives Considered

Selective Scanning: Another alternative is to implement selective scanning based on user-defined configurations. Users could manually specify which libraries should be scanned during each scan. However, this approach would require constant manual intervention and might not be feasible for large codebases with numerous libraries.

The proposed fast-mode scan strikes a balance by automatically targeting the specific libraries that require bug checks, while still allowing users to opt for a full scan if desired.

Additional Context

By introducing this feature, VMClarity will provide users with a more streamlined scanning process, allowing them to identify and address bugs more effectively. Furthermore, the optional nature of the fast-mode scan ensures that users have the flexibility to choose between speed and thoroughness based on their specific needs.

Description

It seems that all the scans of VMs that runs on different regions from where the VMClarity server is running failed due to

failed to create volume snapshot from target volume. TargetVolumeID=XXX: failed to create snapshot for volume. VolumeID=XXX: operation error EC2: CreateSnapshot, https response error StatusCode: 400, RequestID: 6c1d0f0f-4536-4114-bf88-c0dd616821b6, api error InvalidVolume.NotFound: The volume 'XXX' does not exist.

Expected Behavior

VMClarity should be able to scan VMs that running on any region

Actual Behavior

Scan failed

Affected Version

main (1652f7e)

Steps to Reproduce

1. Deploy the stack on region X with sqlite DB (the use of sqlite is due to #395 )
2. Run a scan with scope on region Y (not X)
3. Check scan results

Checklist

• I have read the contributing guidelines
• I have verified this does not duplicate an existing issue

In order to untangle VMClarity and KubeClarity, improve maintainability, and make the OpenClarity multi-scanner CLI reusable we should move the CLI components to a separate repo and reorganise them.

The repo should be called ""

The components that should move from KubeClarity are:

• shared/pkg/job_manager
• shared/pkg/scanner
• shared/pkg/analyzer
• shared/pkg/converter
• shared/pkg/config
• shared/pkg/utils/cyclonedx_helper
• shared/pkg/utils/trivy
• shared/pkg/utils/image_helper
• shared/pkg/utils/sbom
• shared/pkg/utils/vulnerabiity
• shared/pkg/utils/input.go

The components that should move from VMClarity are:

• pkg/shared/families

The code once moved it should be reorganised to ensure that its a cohesive package, some examples are:

• the code in kubeclarity/pkg/analyzer should be moved into pkg/families/sbom and aligned with the other families
• the code in kubeclarity/pkg/scanner should be moved into pkg/families/vulnerabilities and aligned with the other families
• the code in shared/pkg/config/trivy.go should be split into pkg/families/sbom/trivy/config.go and pkg/families/vulnerabilties/trivy/config.go

After the move is performed the code that has been moved should be removed from KubeClarity and VMClarity, and the new repo should be used as a golang library to replace it.

Outside of this task some enhancements should be done to all the code that is now unified:

• Add context support all the way from the family manager to the scanners
• Refactor the job_manager to make it responsible for running the scanner in go routines, instead of the scanners themselves.

These should be opened and issues on the new repo.

Is your feature request related to a problem? Please describe.
The current readme.md document provides a "To install vmclarity in your AWS account [Click Here]" reference. It currently goes to eu-centrial-1. We should provide links for multiple regions AND reference a public S3 bucket (See issue: #96.

Describe the solution you'd like
Update the installation steps to provide AWS CloudFormation links for multiple regions and reference a public S3 bucket. Additionally, we should just provide a link directly to the AWS CF template file in S3 as many people don't trust stack links.

Create a new Controller which adopts the reconcile logic from common package for handling events related to Scan objects.

To support arm architecture with the whole project we need to update the docker images in the following repositories:

• https://github.com/openclarity/vmclarity
• https://github.com/openclarity/vmclarity-tools-base
• https://github.com/openclarity/exploit-db-server
• https://github.com/openclarity/grype-server
• https://github.com/openclarity/freshclam-mirror

As well as to upgrade the cloudformation file to support AWS.

Need to create a Terraform plan to provision the VMClarity control plan using the supported Terraform providers (AWS, Azure, GCP, K8s). Need also to allow the user to configure and provision the VMClarity cloud providers (that allow to discover and scan assets):

// Defines values for CloudProvider.
const (
	AWS        CloudProvider = "AWS"
	Azure      CloudProvider = "Azure"
	Docker     CloudProvider = "Docker"
	External   CloudProvider = "External"
	GCP        CloudProvider = "GCP"
	Kubernetes CloudProvider = "Kubernetes"
)

Thats a great point, the AWS DiscoverAssets flow paginates the AWS call but it still results in all the models.AssetInfo's in memory. We should probably change that into a channel so the pagination is passed through and each Asset is handled one at a time.

Originally posted by @sambetts-cisco in #438 (comment)

Description

After deploying the CFN stack from the main branch (commit 1652f7e), getting a UI error and the following errors in VMClarity logs upon scan drill down in the UI:

Jun 12 07:12:27 ip-10-0-0-11 docker[5115]: {"time":"2023-06-12T07:12:27.596313084Z","id":"","remote_ip":"172.18.0.1","host":"10.0.0.11:8888","method":"GET","uri":"/api/scans/048da54d-620f-4a07-9265-6d062912e0cd?%24expand=scanConfig","user_agent":"Go-http-client/1.1","status":500,"error":"","latency":656940,"latency_human":"656.94µs","bytes_in":0,"bytes_out":165}
Jun 12 07:12:27 ip-10-0-0-11 docker[5115]: time="2023-06-12T07:12:27Z" level=error msg="failed to get scan from db. id=048da54d-620f-4a07-9265-6d062912e0cd: failed to query DB: ERROR: cannot extract elements from a scalar (SQLSTATE 22023)"
Jun 12 07:12:27 ip-10-0-0-11 docker[5115]: time="2023-06-12T07:12:27Z" level=error msg="Failed to reconcile item: failed to fetch Scan. ScanID=048da54d-620f-4a07-9265-6d062912e0cd: failed to get a scan status. status code=500: failed to get scan from db. id=048da54d-620f-4a07-9265-6d062912e0cd: failed to query DB: ERROR: cannot extract elements from a scalar (SQLSTATE 22023)" controller=ScanWatcher
Jun 12 07:12:27 ip-10-0-0-11 docker[4634]: 2023-06-12 07:12:27.596 GMT [168] ERROR:  cannot extract elements from a scalar

Affected Version

(commit 1652f7e)

Steps to Reproduce

1. Install CFN from main (commit 1652f7e)
2. Create a scan configuration
3. Click on the scan once it's created from the config.

Description

The asset scan status is missing.
Following #379 the TargetScanState was updated but the corresponding code in the UI was not.

Checklist

• I have read the contributing guidelines
• I have verified this does not duplicate an existing issue

Description

Today the AWS install configures VMClarity's DB data to be stored in /opt/vmclarity on the root volume of the VMClarity control plane VM. If something happens to that VM (replaced by cloud formation or terminated by mistake) this can result in data loss. This should not happen and the VMClarity data needs to be more resilient.

Expected Behavior

Data should survive a complete VMClarity control plane server replacement.

Actual Behavior

Data is lost along with the VM.

Affected Version

0.4.0

Steps to Reproduce

1. Terminate and recover the VMClarity server VM through cloudformation, all the data is reset.

Checklist

• I have read the contributing guidelines
• I have verified this does not duplicate an existing issue

We have to be able to write end to end tests to the project so that we can ensure the quality of the software.
After this issue solved we will be able write end to end tests for new features and also cover existing use cases.

Create a new Controller which adopts the reconcile logic from common package for handling events related to ScanResult objects.

Description

Riskiest regions widget is showing region + vpc

Expected Behavior

The widget should show only regions without the vpc name.

Checklist

• I have read the contributing guidelines
• I have verified this does not duplicate an existing issue

Support odatasql for SQL variants, as well as
updating the gorm database driver to connect to postgres when configured
to do so.

Description

Per OpenAPI specs here, target property of TargetScanResult should contain targetInfo field (ref), but the property over 13 scan results and 3 scans does not exist, and only the id of the target is available.

I have tried passing the ?expand=True and ?$expand=True, but no change in behavior.

Expected Behavior

Field is present.

Actual Behavior

(see description)

Affected Version

Version: latest
Commit: 3e45aa3bbd2689c6c12547943b6d1fa674307f88

Steps to Reproduce

1. Have at least 1 scan finished
2. Call <host>:<port>/api/scanResults trough Postman or similar tool

Checklist

• I have read the contributing guidelines
• I have verified this does not duplicate an existing issue

We need to document the azure installer better from both a deployer perspective and from a developer perspective (such as how to compile the bicep into the vmclarity.json ARM template)

YARA is a tool aimed at (but not limited to) helping malware researchers to identify and classify malware samples.

YARA is a popular tool used by malware researchers and provides an extremely flexible way to search for malware across a file system. We should add YARA as a scanner in the malware family, it should follow a similar structure to the other scanners in VMClarity and we should also add a remote server to handle the YARA rule download and sync in a centralised way.

Subtasks:

• #576

@pbalogh-sa Can you please open some new issues and link them here as subtasks for this "epic". You can make sure Github links them together properly by adding them into the list in the format:

- [ ] #<issue number>

Break the backend monolith into the following microservices with separate containers:

• Orchestrator
• REST API Server (DB)
• UI Backend
• UI

This will allow us to build, deploy and scale these components separately.

The Lynis scanner currently runs a sub-set of the possible tests that Lynis supports. The majority of the tests were disabled because they evaluate things outside of the inputted root directory (for example looking at the running kernel).

However some tests were disabled because they had prerequisite tests or configurations that where needed we should re-evaulate these tests and see which ones can be enabled with some additional work.

Output of this task should be a Github Discussion document describing which tests can be re-enabled with which additional configurations for Lynis.

Here is the list of tests to be re-evaluated with the notes that were taken at the time:

Need to make all the stats in the dashboard linkable to allow the user to navigate to the corresponding screen in order to inspect the affected objects and the relevant findings, when all the needed filters are applied to the selected dashboard widget.

Add infrastructure to VMClarity to discover and scan Virtual Machine images. This will require a new AssetType to be added. Once this is in place then any providers which can discover and scan virtual machines images can be updated to support it.

VMClarity supports encrypted volumes, we need to document how to configure the environment and VMClarity to scan encrypted volumes for the different providers (AWS, Azure and GCP).