oss-review-toolkit / ort-ci-gitlab Goto Github PK

View Code? Open in Web Editor NEW

13.0 5.0 15.0 287 KB

Use ORT in your GitLab pipelines

License: Apache License 2.0

gitlab-ci gitlab sbom-generator sbom license-checking spdx cyclonedx ospo ci

ort-ci-gitlab's People

Stargazers

Watchers

Forkers

zhernovs janderssonse yanndd1 driveblocks siddhant-k-code unclejay80 cumucore kokolanako bhairavdhanwade daissi shirosweets verebelyin gaybro8777 fviernau sandrociceros-globalhitss

ort-ci-gitlab's Issues

Evaluate error: Unable to list the remote branches, not authorized on git

Hello
I would like to refer to the closed issue oss-review-toolkit/ort#6400
I have similar case but in EVALUATION phase.
I tried to pass the credentials in .netrc file or even pass to evaluate coammand --package-types PACKAGE but
the oprion is not present in EVALUATION Error: no such option --package-types. same error:
Without the option I encounter following:

10:17:05.008 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Trying to download ':::' sources to '/tmp/ort-Companion1745841932715318541' from VCS...
10:17:05.008 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Using processed VcsInfo(type=Git, url=https://gitlab-ci-token@************/opensource/oss-compliance-sample-projects/maven-01.git, revision=8f2a20ce55b2f6633637fe3020aef1511b2b8ed2, path=). Original was VcsInfo(type=, url=, revision=, path=).
10:17:05.052 [main] INFO  org.ossreviewtoolkit.utils.ort.OrtProxySelector - Proxy selector was successfully installed.
10:17:05.066 [main] INFO  org.ossreviewtoolkit.utils.ort.OrtAuthenticator - Authenticator was successfully installed.
10:17:05.197 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Detected VCS type 'Git' from type name 'Git'.
10:17:05.399 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - No Git revision for package '' and version '' found: IOException: Cannot guess a revision name from a blank version.
10:17:06.122 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - Metadata has invalid Git revision '8f2a20ce55b2f6633637fe3020aef1511b2b8ed2': IOException: Unable to list the remote branches.
Caused by: TransportException: https://gitlab-ci-token@************/opensource/oss-compliance-sample-projects/maven-01.git: not authorized
    Caused by: TransportException: https://gitlab-ci-token@************/opensource/oss-compliance-sample-projects/maven-01.git: not authorized
10:17:06.123 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Falling back to trying to download from https://************/opensource/oss-compliance-sample-projects/maven-01.git which has credentials removed.
10:17:06.208 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - No Git revision for package '' and version '' found: IOException: Cannot guess a revision name from a blank version.
10:17:06.917 [main] INFO  org.ossreviewtoolkit.downloader.VersionControlSystem - Metadata has invalid Git revision '8f2a20ce55b2f6633637fe3020aef1511b2b8ed2': IOException: Unable to list the remote branches.
Caused by: TransportException: https://************/opensource/oss-compliance-sample-projects/maven-01.git: not authorized
    Caused by: TransportException: https://************/opensource/oss-compliance-sample-projects/maven-01.git: not authorized
10:17:07.036 [main] INFO  org.ossreviewtoolkit.downloader.Downloader - Failed attempt to download source code for ':::' from VcsInfo(type=Git, url=https://gitlab-ci-token@************/opensource/oss-compliance-sample-projects/maven-01.git, revision=8f2a20ce55b2f6633637fe3020aef1511b2b8ed2, path=) took 2.028289033s.
10:17:07.047 [main] INFO  org.ossreviewtoolkit.utils.scripting.ScriptRunner - Evaluating the script took 9.916247556s.
Exception in thread "main" org.ossreviewtoolkit.downloader.DownloadException: Download failed for ':::'.
	at org.ossreviewtoolkit.downloader.Downloader.download(Downloader.kt:75)
	at org.ossreviewtoolkit.downloader.Downloader.download$default(Downloader.kt:70)
	at org.ossreviewtoolkit.evaluator.SourceTreeResolver$Companion$forRemoteRepository$1.invoke(SourceTreeResolver.kt:42)
	at org.ossreviewtoolkit.evaluator.SourceTreeResolver$Companion$forRemoteRepository$1.invoke(SourceTreeResolver.kt:36)

Is there an option to pass credentials to ORT for gitlab or tell evaluator not to review again the sources that are already checkout in the working directory

the project I want to evaluate is located in my internal gitlab

In general I would suggest you delete this project and push a docker image to docker hub which can be used

Imagine following gitlab-ci:

compliance:ort:
  stage: compliance
  image: ort-analyzer:1.2.3
  variables:
      INPUT_DIR: $CI_PROJECT_DIR
      OUTPUT_DIR: $CI_PROJECT_DIR/ort-output
      FORMAT : JSON
  script:
    - ort analyze -i $INPUT_DIR -o $OUTPUT_DIR -f $FORMAT --info

The solution provided on this project is overcomplicated as fuck to be honest

Feature proposal: Add support for the -c flag in the scripts (the ort general conf file)

We use both gitlab and github in our org for different projects, because of history. I'm was playing around with ORT, creating a small github action for abstracting it - https://github.com/janderssonse/ort-action ,, which basically works. for example i used it here - https://github.com/janderssonse/gradle-versions-filter-plugin/blob/ort-ci/.github/workflows/ort-report.yml . But before putting more effort and making that more than a POC I had a look your ort-gitlab-ci, and Instead of me writing a something of less kind-of-copy of the ort-gitlab-ci bash scripts, I thought I should have a go at forking and resuse the core work for the ort-gitlab-ci. I can currently only see benefits with having the same scrips code base, and letting my intended gh-action be more of a specialized wrapper for the ort-gitlab-ci scripts code base, instead for cooking up my own "almost the same"- bash code versions.
Having a look at the scripts and what I want to do, I think the only one option i'm missing in the ort-config-ci bash-scripts the -c flag for ort, ie I would on occasion need to (in an action) serve an ort config file for ORT itself. Would it be possible to either add support for that setting that flag with a path option in the scripts or alternatively would you accept a PR which adds support for that flag as option?

(Note: In the poc action code I linked too, it is just a code scratchblock, so I haven't added creds, licenses and stuff yet neither done any cleanups really. I started adding options for the input, using the same variables names used in the ort-gitlab-ci btw).

Stage to build and tag docker image

What exactly is missing from this job when run from GitLab CI. The dockerfile is first build and tagged with $CI_REGISTRY/ort:latest. SO when the job is being run, it will use that cache. It however still compains about #ERROR: invalid tag "/ort:c1c053e": invalid reference format. Any solutions to this

Switch Trigger Implementation

Since GitLab version 14.9 the pipeline trigger implementation got expanded by the strategy keyword. To my understanding this could replace the polling logic implemented in ort-gitlab-ci.yml ,

Building ORT Docker Images within GitLab/Kubernetes

Due to the fact that kubernetes deprecated dockershim it is not possible to run docker in docker configurations in kubernetes anymore. GitLab recommends to use google/kaniko. Kaniko is not yet able to build buildkit images and therefore not able to build ORT docker images in kubernetes.

TL;DR: For GitLab/Kubernetes alternative container image build tools need to be supported.

Ort-scan build failed in Mime types

Hello,
I trying to include .gitlab-ci.yml into mime types repository.

Here is my .gitlab-ci.yml file

 include:
  - project: Karippery/ort-gitlab-ci
    file: ort-gitlab-ci.yml

image: node:latest

stages:
  - build
  - test
  - lint

cache:
  paths:
    - node_modules/

install-dependencies:
  stage: build
  script:
    - npm install
  artifacts:
    paths:
      - node_modules/

unit-tests:
  stage: test
  script: npm test
  
lint:
  stage: test
  script: npm run lint

ort-scan:
  stage: test
  retry: 2
  variables:
    SW_NAME: "Mime Types"
    SW_VERSION: "2.1.32"
    VCS_URL: "https://gitlab.com/Karippery/mime-types.git"
    ORT_ALLOW_DYNAMIC_VERSIONS: "true"
  artifacts:
    reports:
      license_scanning: ort-results/gl-license-scanning-report.json

error I get

Executing busybox-1.35.0-r13.trigger
Executing ca-certificates-20211220-r0.trigger
OK: 9 MiB in 21 packages
�[32;1m$ ORT_GITLAB_CI_REVISION=${OSS_GITLAB_CI_REVISION:-main} # collapsed multi-line command�[0;m
curl: (22) The requested URL returned error: 401
section_end:1657101364:step_script
�[0Ksection_start:1657101364:upload_artifacts_on_failure
�[0K�[0K�[36;1mUploading artifacts for failed job�[0;m�[0;m
�[32;1mUploading artifacts...�[0;m
�[0;33mWARNING: ort-results/scan-report-web-app.html: no matching files. Ensure that the artifact path is relative to the working directory�[0;m 
�[0;33mWARNING: ort-results: no matching files. Ensure that the artifact path is relative to the working directory�[0;m 
�[31;1mERROR: No files to upload                         �[0;m 
�[32;1mUploading artifacts...�[0;m
�[0;33mWARNING: ort-results/gl-license-scanning-report.json: no matching files. Ensure that the artifact path is relative to the working directory�[0;m 
�[31;1mERROR: No files to upload                         �[0;m 
section_end:1657101365:upload_artifacts_on_failure
�[0Ksection_start:1657101365:cleanup_file_variables
�[0K�[0K�[36;1mCleaning up project directory and file based variables�[0;m�[0;m
section_end:1657101366:cleanup_file_variables
�[0K�[31;1mERROR: Job failed: exit code 22
�[0;m

Ort-scan Error: no such option: "--performance"

Hi,
I got some errors while running the pipeline.

import Source Code to my Gitlab

Ort for Gitlab: https://gitlab.com/tsteenbe/ort-gitlab-ci.git
Ort-config: https://github.com/oss-review-toolkit/ort-config.git
Mime-Types: https://gitlab.com/tsteenbe/mime-types.git

Variables I add to the pipeline are:

SW_NAME: Mime Types
SW_VERSION: 2.1.35
VCS_TYPE: git
VCS_URL: shh://[email protected]:Karippery/mime-types
VCS_REVISION: 63435958fd26d94832b11cd916db1eaa06987f5e
ORT_ALLOW_DYNAMIC_VERSIONS: true

.gitlab-ci.yml in Mime Types

include:
  - project: Karippery/ort-gitlab-ci
    file: ort-gitlab-ci.yml

image: node:latest

stages:
  - build
  - test
  - lint
  - ort-scan

cache:
  paths:
    - node_modules/

install-dependencies:
  stage: build
  script:
    - npm install
  artifacts:
    paths:
      - node_modules/

unit-tests:
  stage: test
  script: npm test
  
lint:
  stage: test
  script: npm run lint

ort-scan:
  stage: ort-scan
  retry: 2
  variables:
    SW_NAME: "Mime Types"
    SW_VERSION: "2.1.27"
    VCS_URL: "ssh://[email protected]:Karippery/mime-types.git"
    ORT_ALLOW_DYNAMIC_VERSIONS: "true"
  artifacts:
    reports:
      license_scanning: ort-results/gl-license-scanning-report.json

ort-scan failed on mime type

show complete raw file is here raw.txt

ort-scan failed in Ort for Gitlab

show complete raw file is here raw.txt

metadata.json

metadata.json.txt

please help me to build ort tool

Error in ORT_RESULTS_SCANNER_FILE configuration

It seems the specification of the ORT_RESULTS_SCANNER_FILE variable is incorrect, which leads to empty web reports in case the evaluater is disabled.

https://github.com/driveblocks/ort-gitlab-ci/blob/main/.gitlab-ci.yml#L166

In addition, we faced the challenge that the ORT_DISABLE_ADVISOR, ORT_DISABLE_EVALUATOR and ORT_DISABLE_SCANNER variables have to be set explicitly to false for the toolkit to work correctly. This is hard to grasp from the description of the variables as they do not have a default value.

I'll prepare a pull request to propose a fix for both topics.

Can the ORT Gitlab CI pipeline checkout private repositories ?

I have setup the pipeline and required tokens, but when trying to checkout a private repository, the ORT git downloader fails with :

Downloading from Git VCS at https://gitlab.cern.ch/acc-co/myprivaterepo.git...
org.ossreviewtoolkit.downloader.DownloadException: Download failed for 'Downloader::myprivaterepo:'.
	at org.ossreviewtoolkit.downloader.Downloader.download(Downloader.kt:77)
	at org.ossreviewtoolkit.plugins.commands.downloader.DownloaderCommand.downloadFromProjectUrl(DownloaderCommand.kt:419)
	at org.ossreviewtoolkit.plugins.commands.downloader.DownloaderCommand.run(DownloaderCommand.kt:208)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:306)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:319)
	at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:40)
(...)
	at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:513)
	at org.ossreviewtoolkit.cli.OrtMainKt.main(OrtMain.kt:66)
	Suppressed: org.ossreviewtoolkit.downloader.DownloadException: Git failed to get revisions from URL 'https://gitlab.cern.ch/myprivaterepo.git'.
		at org.ossreviewtoolkit.downloader.VersionControlSystem.download(VersionControlSystem.kt:249)
		at org.ossreviewtoolkit.downloader.Downloader.downloadFromVcs(Downloader.kt:271)
		at org.ossreviewtoolkit.downloader.Downloader.downloadFromVcs$default(Downloader.kt:184)
		at org.ossreviewtoolkit.downloader.Downloader.handleVcsDownload(Downloader.kt:[109](https://gitlab.cern.ch/acc-co/devops/dependency-management/ort-gitlab-ci/-/jobs/31863565#L109))
		at org.ossreviewtoolkit.downloader.Downloader.download(Downloader.kt:81)
		... 10 more
	Caused by: org.ossreviewtoolkit.downloader.DownloadException: Unable to determine a revision to checkout.
		at org.ossreviewtoolkit.downloader.VersionControlSystem.getRevisionCandidates-0E7RQCE(VersionControlSystem.kt:302)
		at org.ossreviewtoolkit.downloader.VersionControlSystem.download(VersionControlSystem.kt:248)
		... 14 more
		Suppressed: java.io.IOException: Cannot guess a revision name from a blank version.
			at org.ossreviewtoolkit.downloader.WorkingTree.guessRevisionName(WorkingTree.kt:89)
			at org.ossreviewtoolkit.downloader.VersionControlSystem.getRevisionCandidates_0E7RQCE$addGuessedRevision(VersionControlSystem.kt:306)
			at org.ossreviewtoolkit.downloader.VersionControlSystem.getRevisionCandidates-0E7RQCE(VersionControlSystem.kt:353)
			... 15 more
		Suppressed: java.io.IOException: Unable to list the remote branches.
			at org.ossreviewtoolkit.downloader.vcs.GitWorkingTree$listRemoteBranches$1.invoke(GitWorkingTree.kt:149)
			at org.ossreviewtoolkit.downloader.vcs.GitWorkingTree$listRemoteBranches$1.invoke(GitWorkingTree.kt:143)
			at org.ossreviewtoolkit.downloader.vcs.GitWorkingTree.useRepo(GitWorkingTree.kt:60)
			at org.ossreviewtoolkit.downloader.vcs.GitWorkingTree.listRemoteBranches(GitWorkingTree.kt:143)
			at org.ossreviewtoolkit.downloader.VersionControlSystem.isFixedRevision-gIAlu-s(VersionControlSystem.kt:404)
			at org.ossreviewtoolkit.downloader.VersionControlSystem.getRevisionCandidates_0E7RQCE$addMetadataRevision(VersionControlSystem.kt:328)
			at org.ossreviewtoolkit.downloader.VersionControlSystem.getRevisionCandidates-0E7RQCE(VersionControlSystem.kt:363)
			... 15 more
		Caused by: org.eclipse.jgit.api.errors.TransportException: https://gitlab.cern.ch/myprivaterepo.git: not authorized
			at org.eclipse.jgit.api.LsRemoteCommand.execute(LsRemoteCommand.java:192)
			at org.eclipse.jgit.api.LsRemoteCommand.call(LsRemoteCommand.java:131)
			at org.ossreviewtoolkit.downloader.vcs.GitWorkingTree$listRemoteBranches$1.invoke(GitWorkingTree.kt:145)
			... 21 more
		Caused by: org.eclipse.jgit.errors.TransportException: https://gitlab.cern.ch/myprivaterepo.git: not authorized
			at org.eclipse.jgit.transport.TransportHttp.connect(TransportHttp.java:696)
			at org.eclipse.jgit.transport.TransportHttp.openFetch(TransportHttp.java:465)
			at org.eclipse.jgit.api.LsRemoteCommand.execute(LsRemoteCommand.java:170)
			... 23 more
	Suppressed: org.ossreviewtoolkit.downloader.DownloadException: No source artifact URL provided.
		at org.ossreviewtoolkit.downloader.Downloader.downloadSourceArtifact(Downloader.kt:311)
		at org.ossreviewtoolkit.downloader.Downloader.handleSourceArtifactDownload(Downloader.kt:150)
		at org.ossreviewtoolkit.downloader.Downloader.download(Downloader.kt:82)
		... 10 more
The download took 796.761771ms.
07:26:00.198 [main] ERROR org.ossreviewtoolkit.plugins.commands.downloader.DownloaderCommand - The following failure(s) occurred:
DownloadException: Download failed for 'Downloader::myprivaterepo:'.
Suppressed: DownloadException: Git failed to get revisions from URL 'https://gitlab.cern.ch/myprivaterepo.git'.
    Caused by: DownloadException: Unable to determine a revision to checkout.
        Suppressed: IOException: Cannot guess a revision name from a blank version.
        Suppressed: IOException: Unable to list the remote branches.
            Caused by: TransportException: https://gitlab.cern.ch/myprivaterepo.git: not authorized
                Caused by: TransportException: https://gitlab.cern.ch/myprivaterepo.git: not authorizedSuppressed: DownloadException: No source artifact URL provided.

Does the Git downloader need to be provided with credentials ? How does one configure the authenticator ?
In particular, oss-review-toolkit/ort#2580

I would expect that Gitlab provides all required environment including authentication, but does JGit pick up on that ?

Link in README doesn't work

I'm following instructions in the README and I can't open a link for ORT configuration repo. I get an error 404. Probably, this repository is not public. However, seems that this link is redundant as I was able to set up the scanning job without this repo.