Right now we have one root CA configured in GCP Private CA. We could improve this a bit:

Some high-level suggestions:

Add an intermediary, and keep the root disabled.

The root could then be "enterprise" which comes with auditing.

Have multiple roots, across multiple providers/HSMs.

A majority would be required to sign any new intermediate for it to be trusted.
This is TUF-style and allows us to survive a root compromise.

It requires us to implement our own cert bundle checking logic though, rather than relying on the simple standard ones.

Rotation

We'll need to figure out our root rotation strategy anyway.
This will be also affect how we intend to get our root trusted and known to clients.

After #57 trying to build the project I encountered an error:

$ make
swagger generate server -f openapi.yaml -q -r COPYRIGHT.txt -t pkg/generated --exclude-main -A fulcio_server --exclude-spec --flag-strategy=pflag -P github.com/coreos/go-oidc/v3/oidc.IDToken
swagger generate client -f openapi.yaml -q -r COPYRIGHT.txt -t pkg/generated -P github.com/coreos/go-oidc/v3/oidc.IDToken
go build ./cmd/server
pkg/generated/restapi/configure_fulcio_server.go:30:2: missing go.sum entry needed to verify package github.com/coreos/go-oidc/v3/oidc (imported by github.com/sigstore/fulcio/pkg/generated/restapi) is provided by exactly one module; to add:
	go get github.com/sigstore/fulcio/pkg/generated/restapi
Makefile:24: recipe for target 'server' failed
make: *** [server] Error 1

Easy workaround is to run the provided command (go get github.com/sigstore/fulcio/pkg/generated/restapi) but I was wondering if this is expected or should be fixed.

I'm a bit confused about go getting part of the package I'm trying to build locally and what side effects this may produce.

Description

The work to surface public issuer URLs is tracked here: Azure/AKS#2327

Currently fulcio returns the cert chain as two whitespace separated blocks.

It would be more useful to folks if we returned in json:

{
  "rootca": {-----BEGIN CERTIFICATE-----abcde...},
  "client_cert": {-----BEGIN CERTIFICATE-----abcde...},
}

There's no canonical mapping of email address to authoritative OIDC provider. This means we can't just let users provide their own configs. For example:

• Someone could stand up their own OIDC instance and issue tokens for any email address, if we trusted all providers blindly we would mis-issue certs.
  • We would be able to detect this and recover from it with transparency logs, but it's still not good.

So, we'll need a way to add OIDC providers that we trust one by one to cover most email addresses.

For users that truly want to run their own email systems, we have a couple options:

• ACME-style DNS challenges on MX records to ensure the user has control of email routing for that domain
• Email-based challenges - we send a token, they enter it. Email is wildly insecure, but if we want to support complete federation we don't have too many other options. This is hard to automate as well.
• Our own login system
  • I don't love this idea, but it does give us flexibility. Login to sigstore with an account (OIDC is fine too), and strengthen that account with other challenges (similar to keybase). We can verify access to an email address here however we need to once a week, month, etc. manually, then automatically issue certs using our own oauth challenges.

Description

As mentioned in #187 (comment) we should re-work this to store the rootCA in memory when the process is initialized rather than load it over and over on a per-transaction basis.

Description

Release of fulcio v0.1.0

• Modular CA
• Fuclio Client #95

• planned date
  • 07/Jul/2021

#96

Right now we're fine with a single "audience" or "client id" field across our issuers, but it's possible we might need different ones (and even client secrets) for each individual one.

Don''t have the resolution just yet, but opening this to track

COSIGN_EXPERIMENTAL=1 go run cmd/cosign/main.go sign -fulcio-server http://127.0.0.1:5555 ghcr.io/lukehinds/sigstore-test-three:latest 
Generating ephemeral keys...
Retrieving signed certificate...
Your browser will now be opened to:
https://oauth2.sigstore.dev/auth/auth?access_type=online&client_id=sigstore&code_challenge=dH3cg1zLx7XIsreUP79k9zYDU_-BJuPBkr8zaV_8QZI&code_challenge_method=S256&nonce=1vqTxBpal7N6wZiPE13kAjUtMa2&redirect_uri=http%3A%2F%2Flocalhost%3A5556%2Fauth%2Fcallback&response_type=code&scope=openid+email&state=1vqTx4qnjHmmOASyiY15WY8kIB8
warning: uploading to the transparency log at https://rekor.sigstore.dev for a private image, please confirm [Y/N]: Y
tlog entry created with index:  11028
Pushing signature to: ghcr.io/lukehinds/sigstore-test-three:sha256-568999d4aedd444465c442617666359ddcd4dc117b22375983d2576c3847c9ba.sig


COSIGN_EXPERIMENTAL=1 go run cmd/cosign/main.go verify ghcr.io/lukehinds/sigstore-test-three:latest
error: no matching signatures:
x509: certificate specifies an incompatible key usage
exit status 1

Same stuff we have in rekor:

• Structured logs to stdout, ingested into stackdriver with working error levels
• Metrics on latency, new certs issued, etc.
• Alerts on all of the above.
• Test probers running against prod

Google has now implemented 1.0 with various changes, this means we need to update the googleca code to 1.0

The MVP is established now and other projects are leveraging fulcio, so its time to plan for 1.0

• #194
• #187
• #195
• #13

cc @cpanato

Setup deployment flows and staging environments.

We have this TODO to actually return the SCT when a cert is created:

If we started doing this, we could use the SCT to verify the cert is in the CT log. This would be the final piece of validation for cosign bundling to be finished!

Now that d643aae is available, we need some integration test coverage.

Fulcio will need to log certs to a CT log. Given our non-standard usecase, it probably makes sense for this to be a new one rather than an existing one (although a cert is a cert...). Has anyone given thought to this yet? We could run the log as part of rekor (same DB, different log) or a different service.

It will need a public API, probably https://tools.ietf.org/html/rfc6962

• https:///ct/v1/get-roots would return the root CA cert for fulcio itself
• https:///ct/v1/add-chain would be the call to add a new entry
  ...

We need tool to load test fulcio and report those results.

About

Skills

Interest areas

The OIDC flow used by fulcio / cosign appears to be limited to public OAuth2 clients. That is, cosign is both the OAuth2 client and the user client, acquires the OIDC ID token, and sends it to fulcio (which is the relying party).

This is a bit limiting as it limits fulcio to issuers that support public clients (and PKCE).

Was there ever any consideration of alternate OIDC setups? For example, if fulcio itself is the OIDC client, then it can maintain confidentiality and use a wider range of issuers.

Thanks!

Here's one idea on how we can federate - https://twitter.com/gustavonarea/status/1369563159556669442?s=19

We'd grant specific OIDC providers that we trust an intermediate cert. We can manage them automatically for the big OIDC providers we trust, and establish criteria for other ones we want to delegate to.

Establish how we can provide a method for verifiers to attest fulcio grants actually come from a real OpenID Provider.

Currently we store the signers email address into the signing certificates, but there is no way for a verify / attest that email was provided by an OpenID Provider and not entered by fuclio with no grant being performed.

I am not sure if it possible to do this with OAUTH at present, so wanted to open this up for discussion to see if anyone knows of a mechanism.

Outline the OIDC protocol flow in UML or similar. This should/could include various client operations and values such as the oidc-client-id & oidc-client-secret, how the email challenge occurs (PubK[email]), how fulcio interacts with the various IDPs , what are the redirect urls, how are the scopes requested and utilised.

This should be sufficent enough in description so that anyone wanting to develop their own sigstore signing client can understand the flow sequence.

Description

It would be good to have information on the sha that the builder is requesting a certificate for in the case of a github workflow. this would allow a user to authenticate when the workflow was running, and trust that a builder is signing on a specific sha (if branch protection is off at some point in a repo, anyone can make a malicious signed artifact with a builder in their malicious branch)

This will help getting repos up to SLSA 2!

Related:
#204

PR #180 missed the modular CA implementation and does not work for fulcioca.

from a cursory look we need a challenges.GithubWorkflowValue to manage subject.Value

cc @mattmoor

I'm trying to run fulcio locally with

go run ./cmd/server/ serve --port 5555 --gcp_private_ca_parent=projects/<project>/locations/<location>/certificateAuthorities/<name>

as described in DEVELOPMENT.md.

I'm able to get the server running, but running the client is failing with this bug:

$ go run ./cmd/client/
Your browser will now be opened to:
<browser url>
Error: json: cannot unmarshal object into Go value of type string
Usage:
  fulcio-client [flags]

Flags:
      --fulcio_address string       address of fulcio server (default "http://127.0.0.1:5555")
  -h, --help                        help for fulcio-client
      --oidc-client-id string       client ID for application (default "237800849078-rmntmr1b2tcu20kpid66q5dbh1vdt7aj.apps.googleusercontent.com")
      --oidc-client-secret string   client secret for application (default "CkkuDoCgE2D_CCRRMyF_UIhS")
      --oidc-issuer string          OIDC provider to be used to issue ID token (default "https://accounts.google.com")

2021/03/15 15:01:11 json: cannot unmarshal object into Go value of type string
exit status 1

Interestingly, it seems to pass with fulcio prod but fail with fulcio locally, so I'm guessing something was merged recently which broke this behavior. Seems like the error is coming from the call to SigningCert

I'm checking it out right now, but opened an issue in case anybody might already know what's going on :)

We changed to the new dex-enabled one by default, we should delete the old one after the next cosign release goes out.

Description

tl;dr We should allow Fulcio to accept OIDC tokens from some (see below) federated Kubernetes OIDC issuers.

Background and High-level proposal

Kubernetes has a relatively new feature called "Service Account Projected Volumes", which allows a workload to generate short-lived, audience-scoped OIDC tokens for a given serviceaccount, e.g.

{
  "header": {
    "alg": "RS256",
    "kid": "oDK51cHaX3V8crffIaSQG8Sx1Vys3xHLD6moTkivqaM"
  },
  "payload": {
    "aud": [
      "something"
    ],
    "exp": 1634744688,
    "iat": 1634744088,
    "iss": "https://container.googleapis.com/v1/projects/mattmoor-credit/locations/us-west1-b/clusters/tenant-cluster",
    "kubernetes.io": {
      "namespace": "default",
      "pod": {
        "name": "oidc-test",
        "uid": "49ad3572-b3dd-43a6-8d77-5858d3660275"
      },
      "serviceaccount": {
        "name": "default",
        "uid": "f5720c1d-e152-4356-a897-11b07aff165d"
      }
    },
    "nbf": 1634744088,
    "sub": "system:serviceaccount:default:default"
  },
  "signature": "..."
}

At least GKE and EKS expose public issuer endpoints for these, which enable these tokens to be used with OIDC federation to authenticate with off-cluster services.

Corollary: if we support this, then we can actually start to leverage KinD to run e2e tests for keyless flows

Broken assumptions and work needed

In general, the subject (sub) of an OIDC flow is only unique within the scoping of the issuer (iss), but today Fulcio assumes a handful of fairly central OIDC issuers with disjoint subjects. This assumption manifests in the subject being the only piece of information included in the issued certificate. If we don't address this assumption, then we can't meaningfully add support for tokens issued by clusters because any cluster could sign things as system:serviceaccount:default:default!

AI: So the first order of business is to make the implicit issuer URLs we have today explicit. I'd propose we start to unconditionally include the iss as the first URI in the URIs section of the certificate we issue.

AI: Once it is safe to start including Kubernetes OIDC in the issuer configuration, I propose that we add support for Kubernetes OIDC issuers (not configured in the public instance!), and use it to stand up some basic KinD e2e testing for Fulcio following: https://github.com/mattmoor/kind-oidc. Once this lands, we can also use this same setup to start doing presubmit testing for keyless flows downstream in cosign (cc @dekkagaijin).

AI: Once we have support for directly configured Kubernetes OIDC issuers, I propose that we refactor the current issuer configuration to support a degree of fuzzy matching, and start adding well-known public prefixes like GKE and EKS.

AI: Once we have support for Kubernetes OIDC issuers, I propose we augment the cosign OIDC "magic" to support a particular mount path (e.g. /var/run/sigstore/cosign/oidc-token)

I believe that with the sum total of these features, we should be able to support fairly broad K8s "workload identity" based flows (both public Fulcio w/ public cloud vendor, and private Fulcios on-prem), and start supporting "keyless" presubmit testing (upstream and downstream in tools 🤩).

cc @dlorenc @lukehinds

This would be really useful to get a full "keyless" experience in automated CICD systems like Tekton Chains.

I can think of a few distinct use cases:

Internal Identity

For artifacts built in internal systems, only intended for internal consumption, the identity of a machine is a little bit simpler.

We can do something like SPIFFE and manage machine identity within an internal, local "trust domain". A chains server would start up, get an identity cert with SPIFFE, and use this against an internal CA to get an ACME style code signing cert with a configurable validity period (one week seems reasonable?). Chains would automatically create and rotate certs, using something like cert-manager.

Trusted Timestamps or a transparency log would still be required here to attest that signatures were created when certificates were valid, but these could be internal (in a separate trust domain) or external.
https://spiffe.io/docs/latest/spiffe-about/spiffe-concepts/

cc @asraa - maybe the TSA server component of Rekor would make sense as a standalone component (or an easy to run Rekor config) that doesn't require a database.

External Identity

For artifacts built in either internal or external systems, where the artifact is intended for external consumption (published docker images, pypi uploads, etc.), we need a way to attest system identity to Fulcio in a meaningful way. I'm not aware of anything that makes sense here other than ACME, but maybe at the URI level rather than at the DNS hostname level.

Then, for example, a system that builds public artifacts (let's say GitHub Actions or the Prow cluster for Kubernetes) would need a public hostname. It would create a challenge to Fulcio, which would issue a (short-lived) code signing cert (maybe 14 days) after verifying the challenge.

cc @SantiagoTorres for any other ideas.

The external identity of a CICD system would be represented by the URI verified in the challenge. Timestamp servers or tlogs would again be required here, but they should be public for public artifacts.

The simplest way seems to be the "precertificate" method, here: https://letsencrypt.org/2018/04/04/sct-encoding.html

We request two certs from the CA - a "precertificate" and the real one. The precertificate is "poisoned" with a special bit set: https://github.com/google/certificate-transparency-go/blob/v1.1.1/x509/x509.go#L1208

Then that is included in the log. When that is finished, the SCT (signed cert timestamp) is embedded in the final cert. The certs are identical, except the poison bit is removed and the SCT is added.

That way clients can verify the cert is in the log without making any requests.

Basically, resolve my TODO here: https://github.com/sigstore/fulcio/blob/development/pkg/ca/ca.go#L63

We should write this up into a basic spec.

• What fields must be set for something to be a valid code signing cert by sigstore clients
• How does expiry work?
• What "timestamp" proofs are acceptable?
• How does chain validation work? What can intermediate certs do?

And probably much more!

I know @mmalone talked about this a bit in slack, but can't remember if we got an issue filed somewhere.

We now have support for SPIFFE IDs, and federation through #107!

This means we can issue certs for subjects like spiffe://somedomain.com/foo/bar, and authenticate them against an OIDC endpoint. Right now we require that the OIDC endpoint be at or above the same (sub)domain of the SVID. An OIDC endpoint of foo.com can grant SVIDs for bar.foo.com, but not the other way.

We can also grant subordinate certificates (certificates signed by our certificate, that are also permitted to act as CAs) to other nested Fulcio instances.

What we're still missing is process and automation for adding these domains to our config and issuing subordinate certs.

Proposal

Allowlisting

Let's start manual. We can use a pull-request based, "gitops-style" enrollment process. We'll create a config file in this repo to map to our production config. Users can open a pull request (or maybe issue first?) requesting that we add their domain to our production config.

We can do basic due-diligence that the requester is actually authorized and intends to enable this, and codify this process over time. This could be automated longer-term with something like ACME.

We can automatically "expire" these entries by setting them for removal after some time period unless the requester wants to remain in the list. Let's start with 6 months.

We'll need at least the following:

• Contact info (email address)
• Verify this email address
• OIDC endpoint
• Intended sub-domains or any other restriction on the SVIDs
• What else?

Subordinate Certs

Same as above, let's start manual. We can grant certs for 3 months and setup automation to renew. These should ONLY be used for signing certs with SVIDs matching the domain, similar to above. Let's look into how to enforce this.

We can do stronger proofs here like ACME, but this is probably a bigger challenge at first than manual issues. Here we'll need:

• Data for the CSR (public key)
• Contact info
• Domain to allow SVIDs under

We support a config file (default being `/etc/fulcio-config/config.json)

Provide examples of how to utilise (schema structure, flags that can be used etc)

We need to find a way to handle credentials in CI

https://github.com/sigstore/fulcio/pull/32/checks?check_run_id=2088067630#step:9:12

Perhaps a github secret

We should log who said the challenge was correct somewhere. So instead of just a cert issued to an email, we say "google validated the identity token", in case of an issue with the OIDC providers.

Question

--help calls itself fulcio-server but the release is fulcio, which should it be?

we're currently renaming it fulcio-server in nixpkgs
https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/security/fulcio/default.nix#L21

Question

Description

When launching fulcio as described in the config/DEVELOPMENT.md file, it will use default configuration, and the lru does not get correctly initialized, which results in a panic.

Example invocation:

go run main.go serve --port 5555 --ca googleca --gcp_private_ca_parent=<my gcp stuff here>

And then invoking cosign like so:

COSIGN_EXPERIMENTAL=1 ./cosign sign --fulcio-url=http://localhost:5555 ghcr.io/vaikas/task-0d6334dfa6713aace72701018aa72314@sha256:bea0dff7f02e43b0d56b3f8c6cf3ffc5de7d16c4ea4536ef2ba4ef79dc390640

Hilarity ensues because the lru is nil.

For some reason, the OIDC proxy provider https://oauth2.sigstore.dev/auth requests profile scope against Google and Microsoft. It shouldn't do that, openid email is enough.

Side-note: Is this proxy provider going to stay? It seems like it would complicate the whole attestation story even more, see #80.

Right now it's hard to search for certs by public key. You can search by hash or timestamp in the normal protocol. We could add a new index to the CT log or write our own index/searcher.

Same as: sigstore/rekor#191

The fulcio client code recently moved from this repo to the sigstore cli in the sigstore repo. I think fulcio is useful on its own as well, independently of rekor. The sigstore cli (correct me if I'm wrong) currently doesn't allow to just retrieve a fulcio cert without signing and sending to rekor.

Is fulcio intended to stay as an independent service? If so, does it make sense to have a stand-alone client here (or elsewhere) for it? Or alternatively extend the sigstore cli to retrieve a cert for custom usage?

In accordance with sigstore/community#16, implement a version flag

Figure out our strategy for log rotation. Ct logs get cut off and rotated each year.

The document at https://github.com/sigstore/community/blob/main/docs/zero-trust-supply-chains.pdf says:

For publicly distributed artifacts, organizations have a few options. An internal build service can
request short-lived certificates from the public Fulcio instance directly. The organization
operating the build service can register with the public Fulcio by uploading their root trust
domain certificate and the SPIFFE ID corresponding to the workload for the build server,
something like spiffe://prod.acme.com/build. The build server can make API requests
directly to the public Fulcio instance and authenticate with its SPIFFE bundle.

I think this particular scenario introduces some trust issues. The organisation has to fully trust the public Fulcio instance to not leak the SPIFFE JWT token, since it can potentially be used to gain access to other systems of the organisation.

The SPIFFE docs also advise against using the JWT token if possible.

If X.509-SVID could be used instead, it would get rid of this particular issue.

Trying to run make produce an error due to swagger not being found.

$ make
swagger generate server -f openapi.yaml -q -r COPYRIGHT.txt -t pkg/generated --exclude-main -A fulcio_server --exclude-spec --flag-strategy=pflag -P github.com/coreos/go-oidc/v3/oidc.IDToken
make: swagger: Command not found
Makefile:10: recipe for target 'pkg/generated/models/%.go' failed
make: *** [pkg/generated/models/%.go] Error 127

I'm not sure how I'm supposed to install it, I'd like some guidance. Happy to improve the docs with a PR.
Thanks!

For some reason the errors we return to clients cause them to barf parsing JSON. I think @bobcallaway fixed this in rekor with typed error messages, but I can't remember how.

In sigstore/cosign#108, we're discussing support for yubikey and other PIV devices in our signing clients.

One feature of these devices is that they can produce a hardware-bound attestation certificate with a unique serial number from the manufacturer. It might be interesting to allow Fulcio's root CA to issue a longer-term device certificate that can bind these devices to the oidc account through some kind of registration process.

The actual signing keys on the devices can be reset/rotated more frequency, but it might be nice to somehow "lock" the device itself to the user with a multi-year, discoverable certificate on the tlog.

refactor client side code and make it easier to access from clients. currently there is a fair amount of code in root.go (such as OIDC code, key creation), these should be put into their own pkgs so other clients such as cosign / sigstore can utilize and we can have a common set of tests running in fulcio CI rather then replicated in each client repo.

as per #107 , implement a modular layer for CA / HSM integration.