Acceptance criteria:

Upon, merge to https://github.com/redhat-appstudio/build-definitions , everything inside https://github.com/redhat-appstudio/build-definitions/tree/main/pipelines/build-templates-bundle should be built into a new bundle as deemed relevant.

Implementation notes

• Add a /.tekon directory in https://github.com/redhat-appstudio/infra-deployments/tree/main/components/build
• Add manifests to setup a webhook that would be triggered from https://github.com/redhat-appstudio/build-definitions
• The pipeline that is run upon triggering the webhook should build the bundle and push to Quay https://quay.io/repository/redhat-appstudio/build-templates-bundle?tab=tags

I have a component that has a etc-pki-secret for accessing RHEL repositories but no activation-key secret.

With the d4262b9 version of the buildah task, I see in my logs:

[build] Adding the entitlement to the build
[build] Adding activation key to the build

With 2099988 (failing PR for my component), I see only:

[build] Adding activation key to the build

I believe the checks in the code:

      if [ -d "$ACTIVATION_KEY_PATH" ]; then
        cp -r --preserve=mode "$ACTIVATION_KEY_PATH" /tmp/activation-key
        mkdir /shared/rhsm-tmp
        VOLUME_MOUNTS="${VOLUME_MOUNTS} --volume /tmp/activation-key:/activation-key -v /shared/rhsm-tmp:/etc/pki/entitlement:Z"
        echo "Adding activation key to the build"
      elif [ -d "$ENTITLEMENT_PATH" ]; then
        cp -r --preserve=mode "$ENTITLEMENT_PATH" /tmp/entitlement
        VOLUME_MOUNTS="${VOLUME_MOUNTS} --volume /tmp/entitlement:/etc/pki/entitlement"
        echo "Adding the entitlement to the build"
      fi

are faulty because if the optional secrets are missing, the mount points are still created, they just are empty directories. The Kubernetes documentation doesn't specify what happens for a missing optional secret, but see this stack exchange question.

This is a request to include oras binaries in the appstudio-utils image from https://github.com/oras-project/oras/releases
Not sure if this is too experimental or if this is not the right place to include it but open to feedback.

The task starts by downloading an SBOM to figure out the base images, but this doesn't work for multi-arch images unless you specify which SBOM to download.

It should probably download all of them and check all of them if its a multiarch image.

For RHTAS prodsec compliance, we have been asked to provide links to systems or documents where it is shown how SAST findings are analyzed and processed by the team. With the current Konflux configuration, it is not straightforward to do this. Snyk Code results are stored as pipeline results, but are not shipped anywhere else during a pipelinerun. It would be ideal for us if the snyk results could be published to snyk.io using the --report option for snyk code test. This feature appears to be in private beta, but it has been for about a year. Perhaps we could use it?

There's a long arc of changes in container tooling to support zstd:chunked compression that supports better compression and better download capabilities in clients. However, old versions of docker (pre-2023) don't support it, which has held up adoption.

The request here is to add zstd:chunked compression in addition to the normal gzip compression for images when we push an OCI image index - so that new clients can take advantage of it, and old clients don't break. To illustrate, for an ordinary multi-arch image index, you would have 4 entries, one for each arch. The idea here is to have 8 entries, one for each arch with gzip compression, and one for each arch with zstd:chunked compression.

Idea number 1: I looked, and buildah manifest push has a --add-compression zstd:chunked option which (I think) will automatically add the extra image manifests on push.

However, I think we might have issues with image provenance. In a Konflux multi-arch build pipeline, we build each arch separately in a buildah task. Those produce image manifests, and their digests are recorded and provenance information is associated with them. We then produce the image index, record its digest, and record stuff there too.

When we validate an image at release time, we ensure that we can find provenance for both the image index as well as all of the referenced image manifests.

If we "magically" add an extra four manifests when we create the image index, then our policy check will fail unless their digests and pullspecs are somehow also exposed, so that chains can attest to their origin.

Idea number 2: Instead, perhaps our buildah tasks should be modified to first push the image with gzip compression, and then call buildah push again with a --compression-format zstd:chunked option to generate the second image manifest then, earlier in the process. Our buildah task would need to expose both digests.. hm.

Idea number 3: Perhaps we should add a new intermediary task that accepts the digest of the image-manifest producing buildah task as a param, pulls it, and then re-pushes it with zstd:chunked. This new task would expose the pullspec and digest of the zstd:chunked manifest as a result, which will induce chains to record provenance for it. The task that builds the image index would then take 8 inputs instead of 4. All 8 would be explicitly added to the image index.

Issue

The task "source-build" fails using our PipelineRun - https://github.com/redhat-buildpacks/builder-ubi-base/blob/main/.tekton/pipelinerun-builder-ubi-base.yaml#L438-L457 and reports such an error:

step-get-base-images
BASE_IMAGES param received:
sha256:cf6762b050e84d6c5cf31542f339edf312d9cb10a85bea92c27b1a86bb437d2d

step-build
2024-09-13 10:08:33,969:source-build:DEBUG:workspace directory /var/source-build
2024-09-13 10:08:33,969:source-build:DEBUG:working directory /var/source-build/source-build
2024-09-13 10:08:33,975:build-source.source-archive:DEBUG:Stashing any changes to working repo ['git', 'stash']
No local changes to save
2024-09-13 10:08:34,007:build-source.source-archive:DEBUG:Collecting timestamp of the commit at HEAD ['git', 'show', '-s', '--format=%cI']
2024-09-13 10:08:34,010:build-source.source-archive:DEBUG:Generate source repo file list ['git', 'ls-files', '--recurse-submodules', '-z']
2024-09-13 10:08:34,013:build-source.source-archive:DEBUG:Generate source archive ['tar', 'caf', '/var/source-build/source-build/source_archive/builder-ubi-base-c44a5c1d37588e85733dc49878b5ff9e8a2c1dd5.tar.gz', '--mtime', '2024-09-13T12:03:44+02:00', '--transform', 's,^,builder-ubi-base-c44a5c1d37588e85733dc49878b5ff9e8a2c1dd5/,', '--null', '-T-']
2024-09-13 10:08:34,541:build-source.source-archive:DEBUG:Popping any stashed changes to working repo ['git', 'stash', 'pop']
No stash entries found.
2024-09-13 10:08:34,543:build-source.source-archive:INFO:add source archive directory to sources for bsi: /var/source-build/source-build/source_archive
2024-09-13 10:08:34,543:source-build:INFO:Image sha256:cf6762b050e84d6c5cf31542f339edf312d9cb10a85bea92c27b1a86bb437d2d does not come from supported allowed registry.
2024-09-13 10:08:34,791:source-build:ERROR:failed to build source image
Traceback (most recent call last):
  File "/opt/source_build/source_build.py", line 1070, in main
    build_result = build(build_args)
                   ^^^^^^^^^^^^^^^^^
  File "/opt/source_build/source_build.py", line 1036, in build
    source_image = resolve_source_image(base_image, args.registry_allowlist)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/source_build/source_build.py", line 465, in resolve_source_image
    return resolve_source_image_by_manifest(binary_image)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/source_build/source_build.py", line 444, in resolve_source_image_by_manifest
    source_image = generate_konflux_source_image(image)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/source_build/source_build.py", line 400, in generate_konflux_source_image
    digest = fetch_image_manifest_digest(cleaned_image)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/source_build/source_build.py", line 192, in fetch_image_manifest_digest
    return run(cmd, check=True, text=True, capture_output=True).stdout.strip()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['skopeo', 'inspect', '--format', '{{.Digest}}', '--no-tags', '--retry-times', '5', 'docker://sha256:cf6762b050e84d6c5cf31542f339edf312d9cb10a85bea92c27b1a86bb437d2d']' returned non-zero exit status 1.
2024-09-13 10:08:34,792:source-build:INFO:build result {"status": "failure", "message": "Command '['skopeo', 'inspect', '--format', '{{.Digest}}', '--no-tags', '--retry-times', '5', 'docker://sha256:cf6762b050e84d6c5cf31542f339edf312d9cb10a85bea92c27b1a86bb437d2d']' returned non-zero exit status 1."}
2024-09-13 10:08:34,792:source-build:INFO:write build result into file /tekton/results/BUILD_RESULT

I am trying to use the build-definitions with https://github.com/openshift-pipelines/pipelines-service.

With some fixes in pipelines-service (https://github.com/openshift-pipelines/pipelines-service/compare/main...guillaumerose:builddef?expand=1), the PR is triggered but I hit an issue: ClusterTask doesn't exist on KCP and all definitions are using them.
How could we solve that?

Events:
  Type     Reason         Age   From         Message
  ----     ------         ----  ----         -------
  Normal   Started        29s   PipelineRun  
  Warning  Failed         27s   PipelineRun  Pipeline /noop can't be Run; it contains Tasks that don't exist: Couldn't retrieve Task "openshift-client": the server could not find the requested resource (get clustertasks.tekton.dev openshift-client)
  Warning  InternalError  27s   PipelineRun  1 error occurred:
           * Couldn't retrieve Task "openshift-client": the server could not find the requested resource (get clustertasks.tekton.dev openshift-client)

With the buildah-remote task, there are problems if several are run at the same time with the same workspace. For example, I saw:

• arch1 completes, $(workspaces.source.path)/rhtap-final-image is rsynced back from the arch1 remote
• arch2 starts, rhtap-final-image is rsynced to the arch2 remote
• The arch2 image is added to rhtap-final-image
• arch1 completes, rhtap-final-image is rsynced back from the arch2 remote - with both architectures
• buildah pull oci:rhtap-final-image fails because the directory now has two images and buildah doesn't know which to use

But there are all sorts of other failures and corruptions that can occur. The workaround people have used for this is to use separate workspaces and duplicate the prefetch-dependencies steps. That works, with the downsides:

• You have to know to use that workaround and why
• A more complex, harder to maintain pipeline
• Right now, cachi2 can't prefetch for only one architecture, making this approach much less efficient.

As far as I can tell, this could be fixed pretty simply by making the buildah/buildah-remote tasks align more closely with the buildah-remote-oci-tasks - use an emptydir workdir for temporary files rather than doing that in the workspace.

Issue

The task "step-show-sbom" fails and reports the following error. SBOM produced is available here: https://gist.githubusercontent.com/cmoulliard/6f69b60c201a2cddc0372c156213f6b0/raw/2e1b99a02cabe21673a27758815064ebf525bbac/gistfile1.txt

jq: error (at <stdin>:1): Cannot iterate over null (null)

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Repository problems

These problems occurred while renovating this repository.

• WARN: No docker auth found - returning

⚠ Dependency Lookup Warnings ⚠

• Renovate failed to look up the following dependencies: Failed to look up docker package registry.redhat.io/openshift4/ose-tools-rhel8, Failed to look up docker package registry.redhat.io/openshift4/ose-cli, Could not determine new digest for update (datasource: docker).

Files affected: .tekton/pull-request.yaml, task/init/0.1/init.yaml, task/s2i-java/0.1/s2i-java.yaml, task/s2i-nodejs/0.1/s2i-nodejs.yaml, task/summary/0.1/summary.yaml

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• chore(deps): update quay.io/containerbuildsystem/cachi2 docker digest to 9036f59

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

This repo creates multiple images for delivery into the app studio platform.
Default pipelines are delivered into quay.io/redhat-appstudio/build-templates-bundle
Base Image for the ClusterTasks are in quay.io/redhat-appstudio/appstudio-utils

The versions for these images are maintained separately in each directory which leads to confusion
Options

1. We change the release build to build both images at the same version. This requires the cluster tasks which are defined in the app studio environment to get version bumped every time any new release occurs.
2. We write some update scripts which update the infra-deployment repository contents with the correct versions of the two (or more) images to prevent the wrong images updates.

I think we should do both -- and embed the automation in the .ci script for building and then installing new pipelines into app studio.

In various contexts, we want to install packages into a sub directory of the root directory, and then use them as the root directory of a subsequent stage. Examples include:

• Creating a standard base image from a Containerfile
• Creating a bootc base image using rpm-ostree
• Creating a Flatpak runtime

In all of these cases, we really don't want to install into a bare directory - this will result in weirdness like redirections to /dev/null going into a actual file. The directory at least needs a skeleton /dev populated and would ideally have /proc and /sys as well.

There are two basic ways that this could be enabled:

• Allow running buildah build with --cap-add=all --security-opt=label=disable ; this is sufficient to allow the Containerfile to create its own mount namespace and populate the root directory itself.

• Instead specify the install directory as a task parameter, and have the buildah task add something like:

         --device=/dev/null:"$INSTALL_ROOT"/dev/null
         --device=/dev/random:"$INSTALL_ROOT"/dev/random
         --device=/dev/urandom:"$INSTALL_ROOT"/dev/urandom
         --device=/dev/zero:"$INSTALL_ROOT"/dev/zero
  

The advantage of this is that it is more flexible and satisfies some other needs for nested sandboxing¹. It may the only way that actually works for the bootc case [@cgwalters - is that right?]. On the other hand, the second approach is more obviously secure. ²

There are lots of challenges to producing bit-wise reproducible builds. One of them is to get all tools all the way through the build chain to use the same time. See https://reproducible-builds.org/docs/ and in particular https://reproducible-builds.org/docs/source-date-epoch/

• We should expose commit-timestamp as SOURCE_BUILD_EPOCH in all of our tasks. Not all tools respect that, but it is the standard environment variable that they should respect.
• Notably, buildah doesn't respect it, but it does accept a --timestamp argument that will help produce the same bits, assuming that nothing inside the containerfile being built is not sensitive to the time.
• We should also provide a --build-arg SOURCE_BUILD_EPOCH=$SOURCE_BUILD_EPOCH to each buildah build, so that if a containerfile accepts that as an ARG, it can take advantage of that.

Doing these things won't give us all 100% bit-wise reproducible builds, but it will close some gaps in that direction.

RUN <<EOF
mkdir -p /etc/foo
cp -aR /src/foo /etc/foo
EOF

gets turned into:

RUN . /cachi2/cachi2.env && <<EOF
mkdir -p /etc/foo
cp -aR /src/foo /etc/foo
EOF

which is weirdly not a syntax error, but a no-op. The right transformation would be something like:

RUN . /cachi2/cachi2.env && /bin/sh <<EOF
mkdir -p /etc/foo
cp -aR /src/foo /etc/foo
EOF

Once you consider RUN --mount=type=bin,src=/a,dest=/b <<EOF wedging this into the current code gets a little tricky.

I came across this bit of code that runs after the just-built image is mounted (to be passed to scanners):

https://github.com/konflux-ci/build-definitions/blame/38c6cd3f4733ed1ee638ce43bacd1096e3e5076d/task/buildah-remote/0.2/buildah-remote.yaml#L487

What would be a lot less ugly than just blowing away all symbolic links is using Linux's openat2 system call has RESOLVE_IN_ROOT which allows a process to safely inspect a distinct root and resolve any symlinks as if they're in that root.

Or perhaps simpler often, just...don't follow symlinks in whatever is doing this scanning. (Why would it traverse symlinks?)

redhat-appstudio/infra-deployments#53 (comment)

Acceptance Criteria

The following tekton manifests should consume pipeline definitions from https://quay.io/repository/redhat-appstudio/build-templates-bundle?tab=tags which are built off https://github.com/redhat-appstudio/build-definitions/tree/main/pipelines/build-templates-bundle

• https://github.com/redhat-appstudio/infra-deployments/tree/main/components/gitops/.tekton
• https://github.com/redhat-appstudio/infra-deployments/tree/main/components/has/.tekton