We should consider making chains configurable so it can use a different set of criteria for making decisions based on some characteristics of the TaskRun.

A concrete example here is the Tekton CI set-up. We store everything right now in rekor, but we probably only care about release builds.

main...solarwinds:ite6 is a draft branch.

The ITE-6 code is here: https://github.com/in-toto/in-toto-golang/tree/ite-6

A list of things we want to do before releasing 1.0:

• #204
• #203
• #191
• #190
• #166
• #156
• #87
• Tekton CLI
• #217
• #236

Right now we only sign TaskRuns and OCI Images, but it would be cool to give users a way to sign other types of results (binaries, files, whatever)

Feature request

Use case

we probably need to:

• update this doc: https://github.com/tektoncd/community/blob/main/working-groups.md
• add the meeting to the tekton community cal!

maybe more?

We've started storing attestations for images in cosign using a ".att" suffix. We should figure out how to do that here in chains (right now we don't support storing attestations in OCI). I think the biggest challenge will be that the storage backends don't know what they're storing (signatures vs. attestations), and we hardcoded the ".sig" suffix.

cc @priyawadhwa any ideas?

Also maybe run it in a different namespace from the pipelines controller. Right now we piggy-back off both.

This should be done to limit ACLs on any signing keys better.

This seems similar to tektoncd/plumbing#241, which the tektoncd/pipeline repo faced in early 2020.

Seems like the fix was adding more nodes to the prow cluster and upgrading golangci-lint to a version which used less memory (1.27).

Our current version already incorporates those changes (1.41) so we could look into adding more nodes to the prow cluster.

For now, I'll try increasing the timeout to 10m and see if that resolves things.

Right now signatures and payloads are stored in annotations on the TaskRuns:

chains.tekton.dev/payload
chains.tekton.dev/signature

But, as we start storing multiple types of payloads (one for the overall TaskRun, one for each image/binary produced, etc.), we need a way to store multiple. Some ideas:

Store a JSON object under each annotation.

Kubectl apply does something similar:

    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"chains","app.kubernetes.io/name":"tekton-pipelines","pipeline.tekton.dev/release":"devel","version":"devel"},"name":"tekton-chains-controller","namespace":"tekton-pipelines"},"spec":{"replicas":1,"selector":{"matchLabels":{"app":"tekton-chains-controller"}},"template":{"metadata":{"annotations":{"cluster-autoscaler.kubernetes.io/safe-to-evict":"false"},"labels":{"app":"tekton-chains-controller","app.kubernetes.io/component":"controller","app.kubernetes.io/name":"tekton-pipelines","version":"devel"}},"spec":{"containers":[{"env":[{"name":"SYSTEM_NAMESPACE","valueFrom":{"fieldRef":{"fieldPath":"metadata.namespace"}}},{"name":"METRICS_DOMAIN","value":"tekton.dev/chains"}],"image":"gcr.io/dlorenc-vmtest2/controller-92006fd957c0afd31de6a40b3e33b39f@sha256:0924a9571260ac4892f2c9a52bfd1c1775683cddd7cf1725ca8ec816bdb32f40","name":"tekton-chains-controller","volumeMounts":[{"mountPath":"/etc/config-logging","name":"config-logging"},{"mountPath":"/etc/signing-secrets","name":"signing-secrets"}]}],"serviceAccountName":"tekton-pipelines-controller","volumes":[{"configMap":{"name":"config-logging"},"name":"config-logging"},{"name":"signing-secrets","secret":{"secretName":"signing-secrets"}}]}}}}

This could look like:

chains.tekton.dev/payloads: |
  {
    "artifacts": [
      {
        "uri": "tekton://$namespace/$name/$uid",
        "payload": "$payload",
      },
      {
        "uri": "oci://gcr.io/foo/bar",
        "payload": "$payload",
      },
    ]

Problems: The modification operation here is "read-modify-write", so we risk losing writes or races.

A new annotation per signed artifact

Instead of one with all payloads, each artifact would get its own. Something like:

chains.tekton.dev/taskrun-payload
chains.tekton.dev/image1-payload
chains.tekton.dev/binary1-payload

The logic to retrieve all and validate them all is a little tougher - we need to loop through all annotations matching a pattern.
But, the write operation is merge-safe.

I'm leaning toward #2.

An easy way to start this would be to turn on experimental mode for cosign, which will upload OCI signatures to the tlog for us.

Eventually, we could store ITE6 attestations in rekor as well!

We add a UUID to annotations for payload-taskrun and signature-taskrun

"annotations": {
      "chains.tekton.dev/payload-taskrun-8c0e9bbc-192b-4ea3-8a6d-26d2ac1e0024": "eyJjb25kaXRpb25zIjpbeyJ0eXBlIjoiU3VjY2VlZGVkIiwic3RhdHVzIjoiVHJ1ZSIsImxhc3RUcmFuc2l0aW9uVGltZSI6IjIwMjEtMDUtMjFUMDk6NDI6MTFaIiwicmVhc29uIjoiU3VjY2VlZGVkIiwibWVzc2FnZSI6IkFsbCBTdGVwcyBoYXZlIGNvbXBsZXRlZCBleGVjdXRpbmcifV0sInBvZE5hbWUiOiJzYW1wbGUtcGlwZWxpbmUtcnVuLWVjaG8tc29tZXRoaW5nLWJqOHI3LXBvZC1zd3J3cyIsInN0YXJ0VGltZSI6IjIwMjEtMDUtMjFUMDk6NDI6MDRaIiwiY29tcGxldGlvblRpbWUiOiIyMDIxLTA1LTIxVDA5OjQyOjExWiIsInN0ZXBzIjpbeyJ0ZXJtaW5hdGVkIjp7ImV4aXRDb2RlIjowLCJyZWFzb24iOiJDb21wbGV0ZWQiLCJzdGFydGVkQXQiOiIyMDIxLTA1LTIxVDA5OjQyOjEwWiIsImZpbmlzaGVkQXQiOiIyMDIxLTA1LTIxVDA5OjQyOjEwWiIsImNvbnRhaW5lcklEIjoiY29udGFpbmVyZDovL2JkNzBkNDBkMGU4NThkNjc0ZDM3ZjIyMDg3NGIzODI3NzExZmE3YzEyYzBmM2I0Yjk0YTE4YTkwNDQxZTc2NjYifSwibmFtZSI6ImVjaG8iLCJjb250YWluZXIiOiJzdGVwLWVjaG8iLCJpbWFnZUlEIjoicmVnaXN0cnkuYWNjZXNzLnJlZGhhdC5jb20vdWJpOC1taW5pbWFsQHNoYTI1NjpmYzc1NTMyYTIwYzFlN2ViMDUxMmEwM2ZlYWM0NjU1NDI3OGJjZTk0NmNmNDU0YTc4ZTExNDMzZTM5YTY2ZDJkIn1dLCJ0YXNrU3BlYyI6eyJzdGVwcyI6W3sibmFtZSI6ImVjaG8iLCJpbWFnZSI6InJlZ2lzdHJ5LmFjY2Vzcy5yZWRoYXQuY29tL3ViaTgtbWluaW1hbCIsImNvbW1hbmQiOlsiZWNobyJdLCJhcmdzIjpbImhlbGxvIHdvcmxkIl0sInJlc291cmNlcyI6e319XX19",
      "chains.tekton.dev/signature-taskrun-8c0e9bbc-192b-4ea3-8a6d-26d2ac1e0024": "MEQCIHOOgvkybfaea7moBVHfhyH+BZTBVtdDkxrqiBghWkWFAiBOLuNtvR6nML7TsgddcfKRB+fnbpBpjcr+NLn/vGrFrA==",
      "chains.tekton.dev/signed": "true",
      "pipeline.tekton.dev/release": "devel"
    },

I guess there is a reason for this? I ask as it might it more cumbersome for folks to parse out the values as they will need to find the UUID or use some sort of regex or wildcard to ignore the appended uuid.

I'm imagining a Tekton Cat but with the chains necklace on, hanging out with the Falco logo. The cat will be holding some kind of shield with the SPIFFE logo on it, and we can throw in some sigstore/cosign logos too when we get them.

Ref cncf/tag-security#625 (comment)

But anyway - the real is to try to put together an end to end hardened build system using Falco policies to detect runtime issues. Use something like SPIRE to tie builds to specific machines, and some kind of TPM attestations for integrity of the build system itself and the individual build instances.

In-toto, sigstore and hermekton can provide a secure end to end provenance of build artifacts, which could include these build system and build node attestations all the way back to a hardware root of trust.

The fulcio one most likely can't run in our integration test cluster because I'm guessing Workload Identity isn't set up

It could still be nice to have the test though, and we could manually run it before releases or something

We should probably clean up the configmap code a bit, the defaults are set in yaml which means they get overwritten every time you deploy.

We can probably do this better with actual defaults, and maybe even use reflection!

Right now, we will continuously retry if some part of the reconcile loops fails. We could retry up to 3 times, and track the number of retries via annotations on the TaskRun:

annotations:
  chains.tekton.dev/retries: 2

If all three retries fail, we can set the signed annotation on the TaskRun to “failed”:

annotations:
  chains.tekton.dev/signed: failed

If we set it to "failed", we should leave the "retry" annotation so that it's clear all three retries occurred and failed.

This may well not be a bug, but figured I would use this to track my findings.

I am following the readme steps, but there are no signing annotations such as payload / signature. I have tried this with gpg, cosign and x509.

Steps followed

1. Generated keys

gpg --gen-key
gpg --export-secret-key --armor $keyname > pgp.private-key
gpg --export --armor $keyname > pgp.public-key

2. Load the secrets

kubectl create secret generic signing-secrets -n tekton-chains --from-file=pgp.passphrase --from-file=pgp.private-key --from-file=pgp.public-key

3. Run the example taskrun

kubectl create -f examples/task-output-image.yaml

NAME                                SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME
build-push-run-output-image-fmjmx   True        Succeeded   64m         64m

tkn taskrun list
NAME                                STARTED      DURATION     STATUS
build-push-run-output-image-fmjmx   1 hour ago   23 seconds   Succeeded

Resulting taskrun data, note "chains.tekton.dev/signed": "true", yet no signature or payload annotations (its the same with cosign keys, x509)

kubectl get taskrun build-push-run-output-image-fmjmx -o=json | jq
{
  "apiVersion": "tekton.dev/v1beta1",
  "kind": "TaskRun",
  "metadata": {
    "annotations": {
      "chains.tekton.dev/signed": "true",
      "pipeline.tekton.dev/release": "devel"
    },
    "creationTimestamp": "2021-05-19T09:57:49Z",
    "generateName": "build-push-run-output-image-",
    "generation": 1,
    "labels": {
      "app.kubernetes.io/managed-by": "tekton-pipelines"
    },
    "name": "build-push-run-output-image-fmjmx",
    "namespace": "default",
    "resourceVersion": "134736",
    "uid": "d2f71d21-73fa-4957-a08c-88a1729d3c84"
  },
  "spec": {
    "resources": {
      "inputs": [
        {
          "name": "sourcerepo",
          "resourceSpec": {
            "params": [
              {
                "name": "revision",
                "value": "v0.32.0"
              },
              {
                "name": "url",
                "value": "https://github.com/GoogleContainerTools/skaffold"
              }
            ],
            "type": "git"
          }
        }
      ],
      "outputs": [
        {
          "name": "builtImage",
          "resourceSpec": {
            "params": [
              {
                "name": "url",
                "value": "gcr.io/foo/bar"
              }
            ],
            "type": "image"
          }
        }
      ]
    },
    "serviceAccountName": "default",
    "taskSpec": {
      "resources": {
        "inputs": [
          {
            "name": "sourcerepo",
            "type": "git"
          }
        ],
        "outputs": [
          {
            "name": "builtImage",
            "targetPath": "/workspace/sourcerepo",
            "type": "image"
          }
        ]
      },
      "steps": [
        {
          "image": "busybox",
          "name": "build-and-push",
          "resources": {},
          "script": "set -e\ncat <<EOF > $(inputs.resources.sourcerepo.path)/index.json\n{\n\"schemaVersion\": 2,\n\"manifests\": [\n    {\n    \"mediaType\": \"application/vnd.oci.image.index.v1+json\",\n    \"size\": 314,\n    \"digest\": \"sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b5\"\n    }\n]\n}\n"
        },
        {
          "image": "busybox",
          "name": "echo",
          "resources": {},
          "script": "cat $(inputs.resources.sourcerepo.path)/index.json"
        }
      ]
    },
    "timeout": "1h0m0s"
  },
  "status": {
    "completionTime": "2021-05-19T09:58:12Z",
    "conditions": [
      {
        "lastTransitionTime": "2021-05-19T09:58:12Z",
        "message": "All Steps have completed executing",
        "reason": "Succeeded",
        "status": "True",
        "type": "Succeeded"
      }
    ],
    "podName": "build-push-run-output-image-fmjmx-pod-7f7br",
    "resourcesResult": [
      {
        "key": "commit",
        "resourceName": "sourcerepo",
        "resourceRef": {
          "name": "sourcerepo"
        },
        "value": "6ed7aad5e8a36052ee5f6079fc91368e362121f7"
      },
      {
        "key": "url",
        "resourceName": "sourcerepo",
        "resourceRef": {
          "name": "sourcerepo"
        },
        "value": "https://github.com/GoogleContainerTools/skaffold"
      },
      {
        "key": "digest",
        "resourceName": "builtImage",
        "resourceRef": {
          "name": "builtImage"
        },
        "value": "sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b5"
      },
      {
        "key": "url",
        "resourceName": "builtImage",
        "resourceRef": {
          "name": "builtImage"
        },
        "value": "gcr.io/foo/bar"
      }
    ],
    "startTime": "2021-05-19T09:57:49Z",
    "steps": [
      {
        "container": "step-create-dir-builtimage-fhhzh",
        "imageID": "gcr.io/distroless/base@sha256:aa4fd987555ea10e1a4ec8765da8158b5ffdfef1e72da512c7ede509bc9966c4",
        "name": "create-dir-builtimage-fhhzh",
        "terminated": {
          "containerID": "containerd://7bebd525bd9369eed146acc55179d856a1ee05652ad236945b8a8a2281b2b0b7",
          "exitCode": 0,
          "finishedAt": "2021-05-19T09:58:01Z",
          "reason": "Completed",
          "startedAt": "2021-05-19T09:58:01Z"
        }
      },
      {
        "container": "step-git-source-sourcerepo-7tvf6",
        "imageID": "localhost:5000/mypipeline/git-init-4874978a9786b6625dd8b6ef2a21aa70@sha256:b873705b6716384afd08e485dffd112a77d038e06cda95e64d759273d11e1f7f",
        "name": "git-source-sourcerepo-7tvf6",
        "terminated": {
          "containerID": "containerd://d1a09b1b3daebd019834bc22152848211f37a3833530da3e0944c1eb5b5fbea6",
          "exitCode": 0,
          "finishedAt": "2021-05-19T09:58:10Z",
          "message": "[{\"key\":\"commit\",\"value\":\"6ed7aad5e8a36052ee5f6079fc91368e362121f7\",\"resourceName\":\"sourcerepo\",\"resourceRef\":{\"name\":\"sourcerepo\"}},{\"key\":\"url\",\"value\":\"https://github.com/GoogleContainerTools/skaffold\",\"resourceName\":\"sourcerepo\",\"resourceRef\":{\"name\":\"sourcerepo\"}}]",
          "reason": "Completed",
          "startedAt": "2021-05-19T09:58:01Z"
        }
      },
      {
        "container": "step-build-and-push",
        "imageID": "docker.io/library/busybox@sha256:b5fc1d7b2e4ea86a06b0cf88de915a2c43a99a00b6b3c0af731e5f4c07ae8eff",
        "name": "build-and-push",
        "terminated": {
          "containerID": "containerd://fbf8538daf6a215cfbabd1109a47935a971c03766608dbcebde250f95707de1a",
          "exitCode": 0,
          "finishedAt": "2021-05-19T09:58:11Z",
          "reason": "Completed",
          "startedAt": "2021-05-19T09:58:11Z"
        }
      },
      {
        "container": "step-echo",
        "imageID": "docker.io/library/busybox@sha256:b5fc1d7b2e4ea86a06b0cf88de915a2c43a99a00b6b3c0af731e5f4c07ae8eff",
        "name": "echo",
        "terminated": {
          "containerID": "containerd://5451255769fcb3973e6c1893f6f50d9064057859a0c1e397787a119175accf15",
          "exitCode": 0,
          "finishedAt": "2021-05-19T09:58:11Z",
          "reason": "Completed",
          "startedAt": "2021-05-19T09:58:11Z"
        }
      },
      {
        "container": "step-image-digest-exporter-h6hxl",
        "imageID": "localhost:5000/mypipeline/imagedigestexporter-6e7c518e6125f31761ebe0b96cc63971@sha256:70b6715e478899bcc4a2c2c65db2fad26413fc1865bce368c48930cdc9c05eb9",
        "name": "image-digest-exporter-h6hxl",
        "terminated": {
          "containerID": "containerd://b723d8ee2bc4c909668c00025eda98c7ca39d665a455c1ce38bccfa6b09fb993",
          "exitCode": 0,
          "finishedAt": "2021-05-19T09:58:12Z",
          "message": "[{\"key\":\"digest\",\"value\":\"sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b5\",\"resourceName\":\"builtImage\",\"resourceRef\":{\"name\":\"builtImage\"}},{\"key\":\"url\",\"value\":\"gcr.io/foo/bar\",\"resourceName\":\"builtImage\",\"resourceRef\":{\"name\":\"builtImage\"}}]",
          "reason": "Completed",
          "startedAt": "2021-05-19T09:58:12Z"
        }
      }
    ],
    "taskSpec": {
      "resources": {
        "inputs": [
          {
            "name": "sourcerepo",
            "type": "git"
          }
        ],
        "outputs": [
          {
            "name": "builtImage",
            "targetPath": "/workspace/sourcerepo",
            "type": "image"
          }
        ]
      },
      "steps": [
        {
          "image": "busybox",
          "name": "build-and-push",
          "resources": {},
          "script": "set -e\ncat <<EOF > $(inputs.resources.sourcerepo.path)/index.json\n{\n\"schemaVersion\": 2,\n\"manifests\": [\n    {\n    \"mediaType\": \"application/vnd.oci.image.index.v1+json\",\n    \"size\": 314,\n    \"digest\": \"sha256:05f95b26ed10668b7183c1e2da98610e91372fa9f510046d4ce5812addad86b5\"\n    }\n]\n}\n"
        },
        {
          "image": "busybox",
          "name": "echo",
          "resources": {},
          "script": "cat $(inputs.resources.sourcerepo.path)/index.json"
        }
      ]
    }
  }
}

Expected Behavior

All TaskRuns are signed.

Actual Behavior

TaskRuns that are quickly cleaned up might not get signed.

Steps to Reproduce the Problem

I see this sometimes when running tasks with mink run task, which pretty aggressively cleans things up upon completion.

I think the fix is to have chains attach a finalizer to the TaskRun, which entails:

1. Customize the finalizer name in controller.Options here:

   chains/pkg/reconciler/taskrun/controller.go

   Line 45 in 488f011

   return controller.Options{

   . We want to customize the name in case other controllers start to finalize TaskRun and to clearly identify that it is chains that hasn't processed the resource.
2. Implement FinalizeKind, which can probably just call ReconcileKind. You can see an example of this here: https://github.com/knative/serving/blob/c6d95c0cfcb27b6cb8c7cc82bf30e3de661a67f5/pkg/reconciler/labeler/labeler.go#L40

Even with these two things, there is a chance that a TaskRun won't be signed, if the TaskRun completes and is removed before the chains controller can even process the resource a single time, but this seems fairly unlikely to happen in practice.

Things we should definitely do:

• #100
• #84
• #157
• #158
• #155

Right now, we parse through TaskRun annotations to set this field. However, annotations aren't copied over into the TaskRun spec from the Task itself, so this often doesn't work.

We'll need a different way for users to let Chains know if a build is reproducible.

Chains currently watches the outputs (resource or task) from TaskRuns, and signs these. Using a separate system to capture the final signed payload is an intentional decision to avoid the need to make cluster-level secrets accessible to the build process. This does require placing some trust in the Kubernetes control plane/data layer. Anyone with edit permission on pods in that namespace could forge or falsify taskrun outputs.

We could tighten this up a bit by adding another level of security, allowing Tasks to sign their own outputs, then validating those signatures before accepting the final payload.

This could look like:

• a new config option to require signed task outputs
• integration with SPIRE to provide workload credentials suitable for signing
• Chains can place a status back on the taskrun indicating that the results were all verified before the final output was signed, and indicate this in the resulting signature payloads.

https://github.com/sigstore/cosign/blob/main/SPEC.md

When we sign OCI artifacts with "simplesigning" format we should make sure it works with this spec. I think it does already, should just double check and add a test.

We should be able to copy cosign for oci repository replacement for storing image signatures

ref: https://github.com/sigstore/cosign/blob/main/cmd/cosign/cli/sign.go#L305

Placeholder issue for in-toto integration tasks

@SantiagoTorres , hopefully you can see this.

• Await on #9 landing first
• Create simple python package, place onto test pypi
• Construct tasks and pipeline
• How will we query the link file?

Will need their own issues.

• Storage adapter
• Support more signing keys

Right now chains supports "Tekton" as a storage backend, which means signatures and payloads are stored in Tekton objects via annotations. We would also like to store these objects in an OCI registry.

Notary v2 work is ongoing, which should make it possible to store signatures for container images in a native manner, but that is still a long way off from being supported widely. Additionally, this would be limited to signatures for artifacts also stored in a registry.

We have a few options:

• Wait for notary v2 support
• Make our own storage system, using the OCI artifacts work. (see tektoncd/pipeline#2137 for an example of this in Tekton already)

I propose we start with our own storage system on OCI. At a high level it'll look like this:

• An image object will be created for every signature/payload we generate.
  • This object will contain:
    • A manifest, with an empty config and two layers
    • One layer will contain the signature
    • One layer will contain the payload that was signed
• These images will be uploaded to an OCI registry, configured via a ConfigMap for the chains installation.
• They will be named in a pattern corresponding to the TaskRun Name/Namespace pair for easy retrieval later
• The TaskRun can be annotated to contain a reference to this object as well

cc @jonjohnsonjr is there anything silly I'm missing here?

Remove PGP? We added it as the first signer, because it's all that was really in use. Since, it's been deprecated from the go crypto library, and we have other options.

If no one is using PGP, we should remove it.

ref golang/go#44226

See here: in-toto/in-toto-golang#101

When generating ITE-6 payloads we can use this format (and the JSON wrapper) for signatures.

It might mean we have to change the interfaces around a bit, but now's the time!

I'm talking about this code in particular:

There are two things about this that are bugging me:

1. NewInCluster should be avoided, and this should probably be calling New with a kube client passed through (this will also make testing easier because you can pass a fake client though).
2. Hard-coding the service account name ("tekton-chains-controller") feels wrong to me for a variety of reasons (what if I want to reconfigure things to run differently? Why is this using the controller's SA vs. the task's SA?)

Figured I'd open an issue for discussion rather than just slinging a PR for this one.

cc @dlorenc @priyawadhwa

We probably want some docs around :

• enabling transparency log support
• enabling fulcio, with additional options
• tekton-provenance custom type
• the new dynamodb storage backend is supported

Expected Behavior

Deleting a namespace containing taskruns completes successfully.

Actual Behavior

I am seeing the chains finalization failing because the service account gets cleaned up, which results in a reconciliation error, and the TaskRun sits around forever (keeping the namespace forever).

Steps to Reproduce the Problem

I noticed this because I was running the pipelines e2e tests, which clean a lot of things up by deleting namespaces, and when I dug into why so much stuff was being left around, it looks a lot like this.

Additional Info

Perhaps the finalization path should log, but not return errors? 🤷

cc @priyawadhwa @dlorenc

This is about to land in cosign: sigstore/cosign#335

Fulcio can issue certs for OIDC accounts. The prod instance supports GCP ones, local instances can do any.

We can get a new cert for every signature, or get one for 5-10m and rotate before it expires.

We'll need to add on other "verification bundles" into the signatures/payloads for all the storage backends.

This way the controller can push signatures to a registry! I got this to work by creating a dockerregistry secret with credentials, and applying it to the tekton-chains-controller service account as an imagePullSecret

$ kubectl create secret docker-registry registry-credentials \
  --docker-server=gcr.io \
  --docker-username=_json_key \
  [email protected] \
  --docker-password="$(cat credentials.json)"


$ kubectl patch serviceaccount tekton-chains-controller \
  -p "{\"imagePullSecrets\": [{\"name\": \"registry-credentials\"}]}" -n tekton-chains

I think some sort of simple hello world walk through / example would be useful, so that folks new to chains can follow something along and get an idea of how everything rigs together.

The current DEVELOPER.md lists how to load with ko. It would be nice to then point to a simple tutorial of how to setup secrets and then get signing to occur against a pipeline or task. As per title, it could be as simple as an echo output.

Right now, we're skipping these tests because we can't access GCR. Some options we have to fix this:

1. Use an insecure registry running in the cluster (this can only be accessed from pods because the LoadBalancer doesn't work)
2. Figure out a way to inject a secret with GCR credentials into the test cluster

If we're storing multiple payloads on multiple backends, we'll retry if any of the storage uploads failed. This means we may retry storing something that was successfully uploaded the first time, which some of our backends don't like.

We could either make all the operations safe to retry, or be more specific about what we retry.

I don't think this is enforced right now because some of our files have headers and some don't ~

/*
Copyright 2021 The Tekton Authors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/

we get this error:

{"level":"info","ts":"2021-08-12T15:12:00.776Z","logger":"watcher.event-broadcaster","caller":"record/event.go:282","msg":"Event(v1.ObjectReference{Kind:\"TaskRun\", Namespace:\"default\", Name:\"build-push-run-output-image-l4mjf\", UID:\"8628d8a4-2ca6-401f-a15a-c310d2cbd562\", APIVersion:\"tekton.dev/v1beta1\", ResourceVersion:\"91241155\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' 1 error occurred:\n\t* getting digest: digest must be between 71 and 71 runes in length: \n\n"}

We should reorganize the readme a bit to make this more approachable. Some ideas:

• Start with the description of what chains is for, and then a quick list of all the features
• Then show installation and a quickstart
• Move exhaustive config documentation to it's own page
• Move tutorials for each key/signing type to their own page
• A fancy gif/screenshot never hurts!

They're signed, so maybe we can stick them in the same .sig image with a different media type?

We should include these in the next release:

• Add instructions for authenticating to a registry for the controller #86
• ITE-6 as a formatter #65
• Docs/Readme improvements #102

and it would be nice to have these as well!

• Figure out ITE-5/SSL Signing Spec integration #75
• Sign chains releases with chains #84

PR #22 will add a config store we can use to configure the chains system. Here's a rough sketch for what that config might look like.

Roughly in chains, we have a few subsystems:

• Things to watch and sign
• Metadata formats to sign (in-toto, grafeas, etc.)
• Signing strategies (PGP, x509, etc.)
• Storage systems to store signatures in

Some storage systems and metadata formats may cross artifact types, but others won't. Here's my proposal for how to organize config, starting with things to watch and sign.

Things to sign will include taskruns, oci images, binaries, wheels, etc.

taskruns.formats.enabled-formats=in-toto,tekton,etc.
taskruns.signing.enabled-strategies=pgp,x509,etc.
taskruns.storage.backends=tekton,gcs,oci,etc.

We can then repeat this for the other artifact types:

oci.formats.enabled-formats=in-toto
oci.signing.enabled-strategies=pgp
oci.storage.backends=oci

to indicate we want to sign OCI images using PGP, with metadata in the in-toto format, and store that in an OCI registry.

Some of these may require their own custom sections as well:

storage.gcs.location=gs://foo/bar

would indicate where we want to store signatures in GCS.

For OCI, we could either store signatures next to the image we sign (notary v2 style), or we'd want to specify a repository to store signatures in (for other types):

storage.oci.repository=gcr.io/foo/bar

We need to update the release Pipeline to store the Git commit and url as results, so that Chains picks it up and includes it in the final provenance.

Right now, the git repo/commit isn't in there

$ rekor-cli get --uuid 3873a54462deab6320d1cac993b31b36bb28ff5c2f0d16993909b61907235ec6 --format json | jq -r .Attestation | base64 --decode | jq

{
  "_type": "publish-chains-release",
  "predicateType": "https://tekton.dev/chains/provenance",
  "subject": [
    {
      "name": "gcr.io/tekton-releases/github.com/tektoncd/chains/cmd/controller",
      "digest": {
        "sha256": "1189a2207be3e93e91aaeff323dc2804576f188527afe3cc2e9a9a0c688344df"
      }
    }
  ],
  "predicate": {
    "invocation": {
      "parameters": [
        "package={string github.com/tektoncd/chains []}",
        "versionTag={string v0.3.0 []}",
        "imageRegistry={string gcr.io []}",
        "imageRegistryPath={string tekton-releases []}",
        "releaseAsLatest={string true []}",
        "platforms={string linux/amd64,linux/arm64 []}",
        "serviceAccountPath={string release.json []}",
        "package=github.com/tektoncd/chains",
        "images=controller",
        "imageRegistry=gcr.io",
        "imageRegistryRegions=us eu asia",
        "releaseAsLatest=true",
        "platforms=linux/amd64,linux/arm64,linux/s390x,linux/ppc64le"
      ],
      "recipe_uri": "task://publish-chains-release",
      "event_id": "c26f0ca6-1a00-4313-ac6b-ee098ad859da",
      "builder.id": "tekton-chains"
    },
    "recipe": {
      "steps": [
        {
          "entryPoint": "#!/busybox/sh\nset -ex\n\n# Login to the container registry\nDOCKER_CONFIG=$(cat ${CONTAINER_REGISTY_CREDENTIALS} | \\\n  crane auth login -u _json_key --password-stdin $(params.imageRegistry) 2>&1 | \\\n  sed 's,^.*logged in via \\(.*\\)$,\\1,g')\n\n# Auth with account credentials for all regions.\nfor region in ${REGIONS}\ndo\n  HOSTNAME=${region}.$(params.imageRegistry)\n  cat ${CONTAINER_REGISTY_CREDENTIALS} | crane auth login -u _json_key --password-stdin ${HOSTNAME}\ndone\ncp ${DOCKER_CONFIG} /workspace/docker-config.json\n",
          "arguments": null,
          "environment": {
            "container": "container-registy-auth",
            "image": "docker-pullable://gcr.io/go-containerregistry/crane@sha256:3095b8be43318d89e593a7d067430b79070cd9da32b4c9484438cef117a31a98"
          },
          "annotations": null
        },
        {
          "entryPoint": "#!/usr/bin/env sh\nset -ex\n\n# Setup docker-auth\nDOCKER_CONFIG=~/.docker\nmkdir -p ${DOCKER_CONFIG}\ncp /workspace/docker-config.json ${DOCKER_CONFIG}/config.json\n\n# Change to directory with our .ko.yaml\ncd ${PROJECT_ROOT}\n\n# For each cmd/* directory, include a full gzipped tar of all source in\n# vendor/. This is overkill. Some deps' licenses require the source to be\n# included in the container image when they're used as a dependency.\n# Rather than trying to determine which deps have this requirement (an(params.imageRegistryd\n# probably get it wrong), we'll just targz up the whole vendor tree and\n# include it. As of 9/20/2019, this amounts to about 11MB of additional\n# data in each image.\nTMPDIR=$(mktemp -d)\ntar cfz ${TMPDIR}/source.tar.gz vendor/\nfor d in cmd/*; do\n  if [ -d ${d}/kodata/ ]; then\n    ln -s ${TMPDIR}/source.tar.gz ${d}/kodata/\n  fi\ndone\n\n# Rewrite \"devel\" to params.versionTag\nsed -i -e 's/\\(chains.tekton.dev\\/release\\): \"devel\"/\\1: \"$(params.versionTag)\"/g' -e 's/\\(app.kubernetes.io\\/version\\): \"devel\"/\\1: \"$(params.versionTag)\"/g' -e 's/\\(version\\): \"devel\"/\\1: \"$(params.versionTag)\"/g' ${PROJECT_ROOT}/config/*.yaml\n\n# Publish images and create release.yaml\nmkdir -p $OUTPUT_RELEASE_DIR\n\nko resolve --platform=$(params.platforms) --preserve-import-paths -t $(params.versionTag) -f ${PROJECT_ROOT}/config/ > $OUTPUT_RELEASE_DIR/release.yaml\n\n# Publish images and create release.notags.yaml\n# This is useful if your container runtime doesn't support the `image-reference:tag@digest` notation\n# This is currently the case for `cri-o` (and most likely others)\nko resolve --platform=$(params.platforms) --preserve-import-paths -t $(params.versionTag) -f ${PROJECT_ROOT}/config/ > $OUTPUT_RELEASE_DIR/release.notags.yaml\n",
          "arguments": null,
          "environment": {
            "container": "run-ko",
            "image": "docker-pullable://gcr.io/tekton-releases/dogfooding/ko@sha256:ff918ec2c8bbe416d5a9b6f9d25dfe9012dce673922fe7b2d5d69a99b02df0ac"
          },
          "annotations": null
        },
        {
          "entryPoint": "set -ex\n\nIMAGES_PATH=${CONTAINER_REGISTRY}/$(params.package)\n\nfor cmd in $(params.images)\ndo\n  IMAGES=\"${IMAGES} ${IMAGES_PATH}/cmd/${cmd}:$(params.versionTag)\"\ndone\n\n# Parse the built images from the release.yaml generated by ko\nkoparse \\\n  --path $OUTPUT_RELEASE_DIR/release.yaml \\\n  --base ${IMAGES_PATH} --images ${IMAGES} > /workspace/built_images\n",
          "arguments": null,
          "environment": {
            "container": "koparse",
            "image": "docker-pullable://gcr.io/tekton-releases/dogfooding/koparse@sha256:5945f709f5533347e2fac2f7e757a2acde2ce25418a7193489bf49027aa0497f"
          },
          "annotations": null
        },
        {
          "entryPoint": "#!/busybox/sh\nset -ex\n\n# Setup docker-auth\nDOCKER_CONFIG=~/.docker\nmkdir -p ${DOCKER_CONFIG}\ncp /workspace/docker-config.json ${DOCKER_CONFIG}/config.json\n\nREGIONS=\"us eu asia\"\n\n# Tag the images and put them in all the regions\nfor IMAGE in $(cat /workspace/built_images)\ndo\n  IMAGE_WITHOUT_SHA=${IMAGE%%@*}\n  IMAGE_WITHOUT_SHA_AND_TAG=${IMAGE_WITHOUT_SHA%%:*}\n  IMAGE_WITH_SHA=${IMAGE_WITHOUT_SHA_AND_TAG}@${IMAGE##*@}\n\n  echo $IMAGE_WITH_SHA, >> $(results.IMAGES.path)\n\n  if [[ \"$(params.releaseAsLatest)\" == \"true\" ]]\n  then\n    crane cp ${IMAGE_WITH_SHA} ${IMAGE_WITHOUT_SHA_AND_TAG}:latest\n  fi\n\n  for REGION in ${REGIONS}\n  do\n    if [[ \"$(params.releaseAsLatest)\" == \"true\" ]]\n    then\n      for TAG in \"latest\" $(params.versionTag)\n      do\n        crane cp ${IMAGE_WITH_SHA} ${REGION}.${IMAGE_WITHOUT_SHA_AND_TAG}:$TAG\n      done\n    else\n      TAG=\"$(params.versionTag)\"\n      crane cp ${IMAGE_WITH_SHA} ${REGION}.${IMAGE_WITHOUT_SHA_AND_TAG}:$TAG\n      echo ${REGION}.$IMAGE_WITH_SHA, >> $(results.IMAGES.path)\n    fi\n  done\ndone\n",
          "arguments": null,
          "environment": {
            "container": "tag-images",
            "image": "docker-pullable://gcr.io/go-containerregistry/crane@sha256:3095b8be43318d89e593a7d067430b79070cd9da32b4c9484438cef117a31a98"
          },
          "annotations": null
        }
      ]
    },
    "metadata": {
      "buildStartedOn": "2021-07-28T15:23:18Z",
      "buildFinishedOn": "2021-07-28T15:29:12Z",
      "reproducible": false
    }
  }
}

Let's release v0.1.0 of chains! Things we should do before then:

• Clarify and document release process #72

And it would be nice to have:

• Better defaults/validation for the configmap #71
• Create a separate k8s service account for chains #68
• Run the integration tests that require a registry #73
• Support cosign generated private keys for signing #79

This will probably be something similar to what triggers does, described here: https://github.com/tektoncd/triggers/blob/main/tekton/release-cheat-sheet.md

We can just setup registry:2 or something from Dockerhub in a local service as part of the e2e setup, then push there during tests.

We should come up with some tests for all the tasks in the catalog, or the pipelines examples: https://github.com/tektoncd/pipeline/tree/main/examples

We generate package URLs for the intoto formatter:

Instead of creating the string ourselves, it would be nice if we could use the library to do it for us: https://github.com/package-url/packageurl-go

I briefly tried to add it in myself, but the library returns a percent-encoded string, and I'm not totally sure if that's what we're supposed to be using (right now we aren't percent-encoding strings)

https://github.com/package-url/packageurl-go/blob/0789fa562603bb38a897134d735e7fc7ddf44b1e/packageurl.go#L83-L84

This would probably be a type of x509 signing, which supports the COSIGN ENCRYPTED PRIVATE KEY type