Images and utilities to run cockroach on kubernetes
cockroachdb / k8s Goto Github PK
View Code? Open in Web Editor NEWImages and utilities to run cockroach on kubernetes
License: Apache License 2.0
Images and utilities to run cockroach on kubernetes
License: Apache License 2.0
There aren't any tests currently for request-cert, should add essential tests.
cc: @joshimhoff
Most of the api libraries used in the request-cert controller are outdated.
For an example: Core() is now CoreV1():
https://github.com/cockroachdb/k8s/blob/master/request-cert/k8s_certificates.go#L193
Upgrade them, this upgrade is also dependent on issue #26.
cc: @joshimhoff
This is my scenario:
kubectl certificates approve ...
.The second time around however, the cluster wide approvals still exist but the secrets that stored the certificates in the namespace TEST are gone. So request-cert attempts new requests. This however causes the request-cert to fail "already exists" error. I think I've found the culprit and I'm wondering why the initial request isn't allowed to use previous requests.
request-cert allows you to pass an invalid username on its command line (./request-cert -user foo_bar
or ./request-cert -user foo-bar
). In both cases the CSR will be created and can be approved, however:
foo_bar:
When it comes time to save the foo_bar certificate secret K8S rejects it with the error:
2019/07/03 23:29:18 could not store secrets: Secret "default.client.foo_bar" is invalid: metadata.name: Invalid value: "default.client.foo_bar": a DNS-1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*')
foo-bar:
Dashes are not allowed in cockroachdb usernames.
As discussed at https://youtu.be/JMCCeAb9eY4?t=1h41s, using the common name "root" for the root client inside a Kubernetes cluster is a bad idea, because it could potentially give the certificate root access to the Kubernetes cluster itself (given that we're using the Kubernetes cluster's CA). We should try to scope this down using some other common name if possible.
As of Kubernetes 1.22, certificates.k8s.io/v1beta1
is no longer being served.
The request-cert container/binary uses the v1beta1
API and will fail to create a CSR in newer Kubernetes clusters:
failed to get certificate: CertificateSigningRequest.Create(%s) failed: the server could not find the requested resource
To mitigate this issue, we'll have to upgrade to certificates.k8s.io/v1
and drop support for Kubernetes < 1.19.
It'd be easier to clean up all the resources created by us if we were to put labels on the CSRs and Secrets when we create them. Currently our docs have to instruct users to clean those up separately from everything else because they don't have the expected labels.
It should be easy enough to use the Kubernetes downward API to get the pod's labels from within the request-cert init container, then copy all of them verbatim into all CSRs and Secrets we create.
This would make a good starter task for anyone interested in playing around with kubernetes.
Rather than writing to an emptyDir
volume mount, request-cert should support writing to a configured K8s secret. This would make it possible to project the secrets generated by request-cert into the same directory as other certificates, such as UI certificates.
When trying to go get request-cert
the k8s dependencies aren't get-able due to their recent change in repo structure. There may be other better k8s issues to reference but kubernetes/kubernetes#24343 seems to be a main thread.
go get github.com/cockroachdb/k8s/request-cert
package k8s.io/client-go/pkg/api/v1: cannot find package "k8s.io/client-go/pkg/api/v1" in any of:
/usr/local/go/src/k8s.io/client-go/pkg/api/v1 (from $GOROOT)
$GOPATH/src/k8s.io/client-go/pkg/api/v1 (from $GOPATH)
package k8s.io/client-go/pkg/apis/certificates/v1beta1: cannot find package "k8s.io/client-go/pkg/apis/certificates/v1beta1" in any of:
/usr/local/go/src/k8s.io/client-go/pkg/apis/certificates/v1beta1 (from $GOROOT)
$GOPATH/src/k8s.io/client-go/pkg/apis/certificates/v1beta1 (from $GOPATH)
I get the following error when running the request-cert pod before the crdb statefulset:
2018/07/20 14:10:29 Looking up cert and key under secret test.client.root
W0720 14:10:29.516979 1 client_config.go:529] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2018/07/20 14:10:29 Secret test.client.root not found, sending CSR
Sending create request: test.client.root for
2018/07/20 14:10:29 failed to get certificate: CertificateSigningRequest.Create(test.client.root) failed: certificatesigningrequests.certificates.k8s.io "test.client.root" already exists
Edit: I deleted the CSR and waited for a restart, now i am sitting at:
2018/07/20 14:35:01 Looking up cert and key under secret test.client.root
W0720 14:35:01.555728 1 client_config.go:529] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2018/07/20 14:35:01 Secret test.client.root not found, sending CSR
Sending create request: test.client.root for
Request sent, waiting for approval. To approve, run 'kubectl certificate approve test.client.root'
CSR approved, but no certificate in response. Waiting some more
2018-07-20 14:35:56.390760318 +0000 UTC m=+54.838340077: waiting for 'kubectl certificate approve test.client.root'
2018-07-20 14:36:26.391007553 +0000 UTC m=+84.838587413: waiting for 'kubectl certificate approve test.client.root'
2018-07-20 14:36:56.391195726 +0000 UTC m=+114.838775452: waiting for 'kubectl certificate approve test.client.root'
2018-07-20 14:37:26.391489733 +0000 UTC m=+144.839069581: waiting for 'kubectl certificate approve test.client.root'
2018-07-20 14:37:56.391885125 +0000 UTC m=+174.839465012: waiting for 'kubectl certificate approve test.client.root'
The CSR is approved. So it waits indefinitely since there is no response to the waitCh. The status stays in approved and no certificate gets issued.
This project (request-cert) and it's dependencies aren't managed properly with a proper Go dependency manager (example, dep, glide go mod etc).
As go mod
is becoming the standard these days, vendor all dependencies using go modules and commit the go.mod
and go.sum
files to the repo.
This will ensure that the project dependencies are tracked properly with every incremental change.
https://github.com/cockroachdb/k8s/tree/master/request-cert
cc: @joshimhoff
request-cert type node works but type client give me the following error
initContainers:
- name: init-certs
image: cockroachdb/cockroach-k8s-request-cert:0.4
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: certs
mountPath: /cockroach-certs
- name: client
mountPath: /cockroach-client
command:
- "/bin/ash"
- "-xec"
# - "/request-cert -namespace=${POD_NAMESPACE} -certs-dir=/cockroach-certs -type=node -addresses=localhost,127.0.0.1,$(hostname -f),$(hostname -f | cut -f 1-2 -d '.'),cockroach-lb,cockroachdb-lb.$(hostname -f | cut -f 3- -d '.') -symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt;"
- "/request-cert -namespace=${POD_NAMESPACE} -certs-dir=/cockroach-client -type=client -user=root -symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
% kubectl get csr
NAME AGE REQUESTOR CONDITION
default.client.root 18m system:serviceaccount:default:cockroach Approved,Issued
default.node.cockroach-0 51m system:serviceaccount:default:cockroach Approved,Issued
default.node.cockroach-1 48m system:serviceaccount:default:cockroach Approved,Issued
default.node.cockroach-2 48m system:serviceaccount:default:cockroach Approved,Issued
+ /request-cert '-namespace=default' '-certs-dir=/cockroach-client' '-type=client' '-user=root' '-symlink-ca-from=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt'
2018/10/29 04:06:51 Looking up cert and key under secret default.client.root
W1029 04:06:51.502966 1 client_config.go:529] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
2018/10/29 04:06:51 Secret default.client.root not found, sending CSR
Sending create request: default.client.root for
2018/10/29 04:06:51 failed to get certificate: CertificateSigningRequest.Create(default.client.root) failed: certificatesigningrequests.certificates.k8s.io "default.client.root" already exists
PS Not related to the problem but what's the best way to combine node and client cmd into one init cmd? Tried ;
but that doesn't work for me.
In https://www.youtube.com/watch?v=JMCCeAb9eY4, Joe forcefully killed a VM that a cockroach pod was on by writing to /proc/sysrq-trigger
. After the machine came back up, the cockroachdb pod's init container got into a crash loop because the symlink for ca.crt already existed. While I'm still not quite sure how the certs volume got into this state, there's no reason for us to fail in such situations.
The locality-checker container image is failing security audits due to vulnerabilities in earlier versions of Go. Update locality-checker to use the latest Go version.
I believe that for my java app running in kubernetes I need a pkcs8 formatted key. Is --also-generate-pkcs8-key argument available for this request certificate job?
https://www.cockroachlabs.com/docs/v19.1/create-security-certificates.html#general
I follow the instructions in README.md but I get error when trying to fetch dependencies via go get ./...
command.
$ go get ./...
# k8s.io/client-go/transport
../../../k8s.io/client-go/transport/round_trippers.go:70:11: cannot convert klog.V(9) (type klog.Verbose) to type bool
../../../k8s.io/client-go/transport/round_trippers.go:72:11: cannot convert klog.V(8) (type klog.Verbose) to type bool
../../../k8s.io/client-go/transport/round_trippers.go:74:11: cannot convert klog.V(7) (type klog.Verbose) to type bool
../../../k8s.io/client-go/transport/round_trippers.go:76:11: cannot convert klog.V(6) (type klog.Verbose) to type bool
golang version is 1.13 and tried with 1.10, 1.11 and others, no success
We put the short-form and long-form names of each node into the node certs (e.g. cockroachdb-0.cockroachdb
and cockroachdb-0.cockroachdb.default.svc.cluster.local
) but only put the short-form name for the client-facing service: cockroachdb-public
. We should also include the long-form name so that the service can be accessed from other namespaces.
I am currently writing a Kubernetes Operator for CRDB. While porting over the statefulset artefacts from the crdb repo it became clear that when running multiple CRDB clusters in the same namespace in K8s the root key (default.client.root) will be valid in all clusters, which is probably not what we want.
All other client keys with the same name will also work on all clusters inside K8S (granted the user exists.) . I propose changing the naming convention for keys to something like:
<namespace>.<cluster-name>.<role>.<name>
in stead of
<namespace>.<role>.<name>
what we have today.
Can we have a latest tag please for development image: cockroachdb/cockroach-k8s-request-cert:latest
like image: cockroachdb/cockroach:latest
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.