There aren't any tests currently for request-cert, should add essential tests.

cc: @joshimhoff

Most of the api libraries used in the request-cert controller are outdated.

For an example: Core() is now CoreV1():

https://github.com/cockroachdb/k8s/blob/master/request-cert/k8s_certificates.go#L193

Upgrade them, this upgrade is also dependent on issue #26.

cc: @joshimhoff

This is my scenario:

1. Create namespace TEST
2. Bring up a new cockroach cluster under TEST
3. Approve the signing requests with kubectl certificates approve ....
4. Delete the namespace TEST
5. Repeat 1-4

The second time around however, the cluster wide approvals still exist but the secrets that stored the certificates in the namespace TEST are gone. So request-cert attempts new requests. This however causes the request-cert to fail "already exists" error. I think I've found the culprit and I'm wondering why the initial request isn't allowed to use previous requests.

request-cert allows you to pass an invalid username on its command line (./request-cert -user foo_bar or ./request-cert -user foo-bar). In both cases the CSR will be created and can be approved, however:

foo_bar:

When it comes time to save the foo_bar certificate secret K8S rejects it with the error:

2019/07/03 23:29:18 could not store secrets: Secret "default.client.foo_bar" is invalid: metadata.name: Invalid value: "default.client.foo_bar": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')

foo-bar:

Dashes are not allowed in cockroachdb usernames.

As discussed at https://youtu.be/JMCCeAb9eY4?t=1h41s, using the common name "root" for the root client inside a Kubernetes cluster is a bad idea, because it could potentially give the certificate root access to the Kubernetes cluster itself (given that we're using the Kubernetes cluster's CA). We should try to scope this down using some other common name if possible.

@mberhault

As of Kubernetes 1.22, certificates.k8s.io/v1beta1 is no longer being served.

The request-cert container/binary uses the v1beta1 API and will fail to create a CSR in newer Kubernetes clusters:

failed to get certificate: CertificateSigningRequest.Create(%s) failed: the server could not find the requested resource

To mitigate this issue, we'll have to upgrade to certificates.k8s.io/v1 and drop support for Kubernetes < 1.19.

It'd be easier to clean up all the resources created by us if we were to put labels on the CSRs and Secrets when we create them. Currently our docs have to instruct users to clean those up separately from everything else because they don't have the expected labels.

It should be easy enough to use the Kubernetes downward API to get the pod's labels from within the request-cert init container, then copy all of them verbatim into all CSRs and Secrets we create.

This would make a good starter task for anyone interested in playing around with kubernetes.

cc @mberhault @bobvawter

Rather than writing to an emptyDir volume mount, request-cert should support writing to a configured K8s secret. This would make it possible to project the secrets generated by request-cert into the same directory as other certificates, such as UI certificates.

When trying to go get request-cert the k8s dependencies aren't get-able due to their recent change in repo structure. There may be other better k8s issues to reference but kubernetes/kubernetes#24343 seems to be a main thread.

go get github.com/cockroachdb/k8s/request-cert
package k8s.io/client-go/pkg/api/v1: cannot find package "k8s.io/client-go/pkg/api/v1" in any of:
    /usr/local/go/src/k8s.io/client-go/pkg/api/v1 (from $GOROOT)
    $GOPATH/src/k8s.io/client-go/pkg/api/v1 (from $GOPATH)
package k8s.io/client-go/pkg/apis/certificates/v1beta1: cannot find package "k8s.io/client-go/pkg/apis/certificates/v1beta1" in any of:
    /usr/local/go/src/k8s.io/client-go/pkg/apis/certificates/v1beta1 (from $GOROOT)
    $GOPATH/src/k8s.io/client-go/pkg/apis/certificates/v1beta1 (from $GOPATH)

I get the following error when running the request-cert pod before the crdb statefulset:

2018/07/20 14:10:29 Looking up cert and key under secret test.client.root
W0720 14:10:29.516979       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2018/07/20 14:10:29 Secret test.client.root not found, sending CSR
Sending create request: test.client.root for 
2018/07/20 14:10:29 failed to get certificate: CertificateSigningRequest.Create(test.client.root) failed: certificatesigningrequests.certificates.k8s.io "test.client.root" already exists

Edit: I deleted the CSR and waited for a restart, now i am sitting at:

2018/07/20 14:35:01 Looking up cert and key under secret test.client.root
W0720 14:35:01.555728       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2018/07/20 14:35:01 Secret test.client.root not found, sending CSR
Sending create request: test.client.root for 
Request sent, waiting for approval. To approve, run 'kubectl certificate approve test.client.root'
CSR approved, but no certificate in response. Waiting some more
2018-07-20 14:35:56.390760318 +0000 UTC m=+54.838340077: waiting for 'kubectl certificate approve test.client.root'
2018-07-20 14:36:26.391007553 +0000 UTC m=+84.838587413: waiting for 'kubectl certificate approve test.client.root'
2018-07-20 14:36:56.391195726 +0000 UTC m=+114.838775452: waiting for 'kubectl certificate approve test.client.root'
2018-07-20 14:37:26.391489733 +0000 UTC m=+144.839069581: waiting for 'kubectl certificate approve test.client.root'
2018-07-20 14:37:56.391885125 +0000 UTC m=+174.839465012: waiting for 'kubectl certificate approve test.client.root'

The CSR is approved. So it waits indefinitely since there is no response to the waitCh. The status stays in approved and no certificate gets issued.

This project (request-cert) and it's dependencies aren't managed properly with a proper Go dependency manager (example, dep, glide go mod etc).

As go mod is becoming the standard these days, vendor all dependencies using go modules and commit the go.mod and go.sum files to the repo.

This will ensure that the project dependencies are tracked properly with every incremental change.

https://github.com/cockroachdb/k8s/tree/master/request-cert

cc: @joshimhoff

request-cert type node works but type client give me the following error

   initContainers:
      - name: init-certs
        image: cockroachdb/cockroach-k8s-request-cert:0.4
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        volumeMounts:
        - name: certs
          mountPath: /cockroach-certs
        - name: client
          mountPath: /cockroach-client
        command:
        - "/bin/ash"
        - "-xec"
        # - "/request-cert -namespace=${POD_NAMESPACE} -certs-dir=/cockroach-certs -type=node -addresses=localhost,127.0.0.1,$(hostname -f),$(hostname -f | cut -f 1-2 -d '.'),cockroach-lb,cockroachdb-lb.$(hostname -f | cut -f 3- -d '.') -symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt;"
        - "/request-cert -namespace=${POD_NAMESPACE} -certs-dir=/cockroach-client -type=client -user=root -symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"

% kubectl get csr
NAME                       AGE       REQUESTOR                                 CONDITION
default.client.root        18m       system:serviceaccount:default:cockroach   Approved,Issued
default.node.cockroach-0   51m       system:serviceaccount:default:cockroach   Approved,Issued
default.node.cockroach-1   48m       system:serviceaccount:default:cockroach   Approved,Issued
default.node.cockroach-2   48m       system:serviceaccount:default:cockroach   Approved,Issued

+ /request-cert '-namespace=default' '-certs-dir=/cockroach-client' '-type=client' '-user=root' '-symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
2018/10/29 04:06:51 Looking up cert and key under secret default.client.root
W1029 04:06:51.502966       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2018/10/29 04:06:51 Secret default.client.root not found, sending CSR
Sending create request: default.client.root for 
2018/10/29 04:06:51 failed to get certificate: CertificateSigningRequest.Create(default.client.root) failed: certificatesigningrequests.certificates.k8s.io "default.client.root" already exists

PS Not related to the problem but what's the best way to combine node and client cmd into one init cmd? Tried ; but that doesn't work for me.

In https://www.youtube.com/watch?v=JMCCeAb9eY4, Joe forcefully killed a VM that a cockroach pod was on by writing to /proc/sysrq-trigger. After the machine came back up, the cockroachdb pod's init container got into a crash loop because the symlink for ca.crt already existed. While I'm still not quite sure how the certs volume got into this state, there's no reason for us to fail in such situations.

@mberhault

The locality-checker container image is failing security audits due to vulnerabilities in earlier versions of Go. Update locality-checker to use the latest Go version.

I believe that for my java app running in kubernetes I need a pkcs8 formatted key. Is --also-generate-pkcs8-key argument available for this request certificate job?

https://www.cockroachlabs.com/docs/v19.1/create-security-certificates.html#general

I follow the instructions in README.md but I get error when trying to fetch dependencies via go get ./... command.

$ go get ./...
# k8s.io/client-go/transport
../../../k8s.io/client-go/transport/round_trippers.go:70:11: cannot convert klog.V(9) (type klog.Verbose) to type bool
../../../k8s.io/client-go/transport/round_trippers.go:72:11: cannot convert klog.V(8) (type klog.Verbose) to type bool
../../../k8s.io/client-go/transport/round_trippers.go:74:11: cannot convert klog.V(7) (type klog.Verbose) to type bool
../../../k8s.io/client-go/transport/round_trippers.go:76:11: cannot convert klog.V(6) (type klog.Verbose) to type bool

golang version is 1.13 and tried with 1.10, 1.11 and others, no success

We put the short-form and long-form names of each node into the node certs (e.g. cockroachdb-0.cockroachdb and cockroachdb-0.cockroachdb.default.svc.cluster.local) but only put the short-form name for the client-facing service: cockroachdb-public. We should also include the long-form name so that the service can be accessed from other namespaces.

@mberhault

I am currently writing a Kubernetes Operator for CRDB. While porting over the statefulset artefacts from the crdb repo it became clear that when running multiple CRDB clusters in the same namespace in K8s the root key (default.client.root) will be valid in all clusters, which is probably not what we want.

All other client keys with the same name will also work on all clusters inside K8S (granted the user exists.) . I propose changing the naming convention for keys to something like:

<namespace>.<cluster-name>.<role>.<name>
in stead of
<namespace>.<role>.<name> what we have today.

Can we have a latest tag please for development image: cockroachdb/cockroach-k8s-request-cert:latest like image: cockroachdb/cockroach:latest