Comments (10)
kfox1111
commented on September 28, 2024
1
I added both of them. However, I believe that people using clusterissuers would also face the same problem.
Trying to figure out how to fix it in general. If both are needed, we need to figure out a way to let the user specify a list of them. If only one is needed, a string is probably ok.
Also, I don't understand why the -ca is appended even in the overrides. If someone wants to use existing issuers this won't work well as the issuers will mistach.
Its a bug. we gotta figure out how to fix it without breaking too many things.
from helm-charts-hardened.
ranjit-se7en
commented on September 28, 2024
1
@kfox1111 - I can confirm that #361 is working as expected without the RBAC changes.
from helm-charts-hardened.
ranjit-se7en
commented on September 28, 2024
Adding more context on the problem reported in the spire slack channel.
We were trying to use the spire helm chart to install the server in K8s using cert-manager upstream. We have cert-manager running and using the AWSPCAClusterIssuer issuers in our clusters. When we try to use the AWSPCAClusterIssuer clusterissuers for the cert-manager upstream, the spire server complains that it couldn't find the issuer.
Below are the clusterissuers.
k get awspcaclusterissuer
NAME AGE
aws-acm-cluster-issuer 545d
aws-acm-cluster-issuer-ecdsa 545d
Upstream Config used in the helm chart installation
## @param upstreamAuthority.certManager.issuer_name Defaults to the release name, override if CA is provided outside of the chart
issuer_name: "aws-acm-cluster-issuer-ecdsa"
## @param upstreamAuthority.certManager.issuer_kind Defaults to "Issuer", override if CA is provided outside of the chart
issuer_kind: "AWSPCAClusterIssuer"
## @param upstreamAuthority.certManager.issuer_group Defaults to "cert-manager.io", override if CA is provided outside of the chart
issuer_group: "awspca.cert-manager.io"
## @param upstreamAuthority.certManager.namespace Specify to use a namespace other then the one the chart is installed into
namespace: ""Spire server pod logs captured using debug log level.
time="2024-05-17T03:38:57Z" level=debug msg="Preparing X509 CA" slot=B subsystem_name=ca_manager
time="2024-05-17T03:38:57Z" level=info msg="Waiting for certificaterequest to be signed" external=false name=spiffe-ca-f5vwn namespace=spire-server plugin_name=cert-manager plugin_type=UpstreamAuthority subsystem_name=catalog
time="2024-05-17T03:38:57Z" level=error msg="Created CertificateRequest has failed" external=false message="issuer could not be found" name=spiffe-ca-f5vwn namespace=spire-server plugin_name=cert-manager plugin_type=UpstreamAuthority reason=Failed subsystem_name=catalog
time="2024-05-17T03:38:57Z" level=debug msg="Optimistically cleaning-up stale CertificateRequests" external=false plugin_name=cert-manager plugin_type=UpstreamAuthority subsystem_name=catalog
time="2024-05-17T03:38:57Z" level=debug msg="Deleting stale CertificateRequest" external=false message="issuer could not be found" name=spiffe-ca-f5vwn namespace=spire-server plugin_name=cert-manager plugin_type=UpstreamAuthority reason=Failed subsystem_name=catalog type=Ready
time="2024-05-17T03:38:57Z" level=error msg="Unable to rotate X509 CA" error="rpc error: code = Internal desc = upstreamauthority(cert-manager): request has failed" subsystem_name=ca_managerWe know what the problem is now, since spire deletes the failed CertificateRequests, we captured it using a script and could see the below CertificateRequests. If you notice the issuer name you can see the name aws-acm-cluster-issuer-ecdsa-ca , we noticed it charts templates appends the -ca to the issuer name here. Not sure if this is intended?
{
"apiVersion": "v1",
"items": [
{
"apiVersion": "cert-manager.io/v1",
"kind": "CertificateRequest",
"metadata": {
"creationTimestamp": "2024-05-17T09:08:32Z",
"generateName": "spiffe-ca-",
"generation": 1,
"labels": {
"cert-manager.spiffe.io/trust-domain": "abcxz.net"
},
"name": "spiffe-ca-4vblx",
"namespace": "spire-server",
"resourceVersion": "884971325",
"uid": "af6c70c9-60b6-4d2d-8c44-c8c5bc23254d"
},
"spec": {
"duration": "24h0m0s",
"extra": {
"authentication.kubernetes.io/pod-name": [
"spire-server-0"
],
"authentication.kubernetes.io/pod-uid": [
"2247bb84-0fa5-4d8c-93ad-50e2e650495d"
]
},
"groups": [
"system:serviceaccounts",
"system:serviceaccounts:spire-server",
"system:authenticated"
],
"isCA": true,
"issuerRef": {
"group": "awspca.cert-manager.io",
"kind": "AWSPCAClusterIssuer",
"name": "aws-acm-cluster-issuer-ecdsa-ca"
},We also noticed that the spire-server-cm role doesn't have permissions for clusterissues and awspcaclusterissuers , it's only limited to certificaterequests and issuers here.
rules:
- apiGroups: ["cert-manager.io"]
resources:
- certificaterequests
- issuers
verbs:
- list
- get
- create
- delete
- patch
- updateSolution
We added the permissions for clusterissuers and awspcaclusterissuers in the spire-server-r, removed the -ca from the configmap, and deployed the chart which was successful 🎉 .
Logs
time="2024-05-18T03:52:07Z" level=info msg="Connected to SQL database" read_only=false subsystem_name=sql type=sqlite3 version=3.45.1
time="2024-05-18T03:52:07Z" level=info msg="Plugin loaded" external=false plugin_name=disk plugin_type=KeyManager subsystem_name=catalog
time="2024-05-18T03:52:07Z" level=info msg="Plugin loaded" external=false plugin_name=aws_iid plugin_type=NodeAttestor subsystem_name=catalog
time="2024-05-18T03:52:07Z" level=info msg="Plugin loaded" external=false plugin_name=k8sbundle plugin_type=Notifier subsystem_name=catalog
time="2024-05-18T03:52:07Z" level=info msg="Plugin loaded" external=false plugin_name=cert-manager plugin_type=UpstreamAuthority subsystem_name=catalog
time="2024-05-18T03:52:07Z" level=debug msg="Loading journal from datastore" subsystem_name=ca_manager
time="2024-05-18T03:52:07Z" level=info msg="There is not a CA journal record that matches any of the local X509 authority IDs" subsystem_name=ca_manager
time="2024-05-18T03:52:07Z" level=debug msg="Loading journal from disk" path=/run/spire/data/journal.pem subsystem_name=ca_manager
time="2024-05-18T03:52:07Z" level=info msg="Journal loaded" jwt_keys=0 subsystem_name=ca_manager x509_cas=0
time="2024-05-18T03:52:07Z" level=debug msg="Preparing X509 CA" slot=A subsystem_name=ca_manager
time="2024-05-18T03:52:07Z" level=info msg="Waiting for certificaterequest to be signed" external=false name=spiffe-ca-8zscl namespace=spire-server plugin_name=cert-manager plugin_type=UpstreamAuthority subsystem_name=catalog
time="2024-05-18T03:52:12Z" level=debug msg="Optimistically cleaning-up stale CertificateRequests" external=false plugin_name=cert-manager plugin_type=UpstreamAuthority subsystem_name=catalog
time="2024-05-18T03:52:12Z" level=debug msg="Deleting stale CertificateRequest" external=false message="certificate issued" name=spiffe-ca-8zscl namespace=spire-server plugin_name=cert-manager plugin_type=UpstreamAuthority reason=Issued subsystem_name=catalog type=Ready
time="2024-05-18T03:52:12Z" level=info msg="Creating a new CA journal entry" subsystem_name=ca_manager
time="2024-05-18T03:52:12Z" level=debug msg="Successfully stored CA journal entry in datastore" ca_journal_id=1 local_authority_id= subsystem_name=ca_manager
time="2024-05-18T03:52:12Z" level=info msg="X509 CA prepared" expiration="2024-05-19 03:52:07 +0000 UTC" issued_at="2024-05-18 03:52:07.15075208 +0000 UTC" local_authority_id=5d017bf915fedfa4490185d69b1aec37550ac048 self_signed=false slot=A subsystem_name=ca_manager
time="2024-05-18T03:52:12Z" level=info msg="X509 CA activated" expiration="2024-05-19 03:52:07 +0000 UTC" issued_at="2024-05-18 03:52:07.15075208 +0000 UTC" local_authority_id=5d017bf915fedfa4490185d69b1aec37550ac048 slot=A subsystem_name=ca_manager
time="2024-05-18T03:52:12Z" level=debug msg="Successfully stored CA journal entry in datastore" ca_journal_id=1 local_authority_id=5d017bf915fedfa4490185d69b1aec37550ac048 subsystem_name=ca_manager
time="2024-05-18T03:52:12Z" level=debug msg="Successfully rotated X.509 CA" subsystem_name=ca_manager trust_domain_id="spiffe://abczx.net" ttl=86394.630562809
time="2024-05-18T03:52:12Z" level=debug msg="Preparing JWT key" slot=A subsystem_name=ca_manager
time="2024-05-18T03:52:12Z" level=warning msg="UpstreamAuthority plugin does not support JWT-SVIDs. Workloads managed by this server may have trouble communicating with workloads outside this cluster when using JWT-SVIDs." plugin_name=cert-manager subsystem_name=ca_manager
time="2024-05-18T03:52:12Z" level=debug msg="Successfully stored CA journal entry in datastore" ca_journal_id=1 local_authority_id=5d017bf915fedfa4490185d69b1aec37550ac048 subsystem_name=ca_manager
time="2024-05-18T03:52:12Z" level=info msg="JWT key prepared" expiration="2024-05-19 03:52:12.369510742 +0000 UTC" issued_at="2024-05-18 03:52:12.369510742 +0000 UTC" local_authority_id=6VwWRwbeipMgIIcD2OYdO4mNGOX72OFt slot=A subsystem_name=ca_manager
time="2024-05-18T03:52:12Z" level=info msg="JWT key activated" expiration="2024-05-19 03:52:12.369510742 +0000 UTC" issued_at="2024-05-18 03:52:12.369510742 +0000 UTC" local_authority_id=6VwWRwbeipMgIIcD2OYdO4mNGOX72OFt slot=A subsystem_name=ca_manager
time="2024-05-18T03:52:12Z" level=debug msg="Successfully stored CA journal entry in datastore" ca_journal_id=1 local_authority_id=5d017bf915fedfa4490185d69b1aec37550ac048 subsystem_name=ca_manager
time="2024-05-18T03:52:12Z" level=debug msg="Rotating server SVID" subsystem_name=svid_rotator
time="2024-05-18T03:52:12Z" level=debug msg="Signed X509 SVID" expiration="2024-05-18T07:52:12Z" spiffe_id="spiffe://abczx.net/spire/server" subsystem_name=svid_rotator
time="2024-05-18T03:52:12Z" level=info msg="Building in-memory entry cache" subsystem_name=endpoints
time="2024-05-18T03:52:12Z" level=info msg="Completed building in-memory entry cache" subsystem_name=endpoints
time="2024-05-18T03:52:12Z" level=info msg="Logger service configured" launch_log_level=debug
time="2024-05-18T03:52:12Z" level=debug msg="Initializing API endpoints" subsystem_name=endpoints
time="2024-05-18T03:52:12Z" level=info msg="Starting Server APIs" address="[::]:8081" network=tcp subsystem_name=endpoints
time="2024-05-18T03:52:12Z" level=info msg="Starting Server APIs" address=/tmp/spire-server/private/api.sock network=unix subsystem_name=endpoints
time="2024-05-18T03:52:12Z" level=debug msg="Initializing health checkers" subsystem_name=health
time="2024-05-18T03:52:12Z" level=info msg="Serving health checks" address="0.0.0.0:8080" subsystem_name=health
time="2024-05-18T03:52:12Z" level=debug msg="Notifier handled event" event="bundle loaded" notifier=k8sbundle subsystem_name=ca_manager
from helm-charts-hardened.
kfox1111
commented on September 28, 2024
Some more context to the issue: spiffe/helm-charts#378
Looks like -ca is always being appended, rather then just if issuer is not overridden.
from helm-charts-hardened.
kfox1111
commented on September 28, 2024
clusterissuers and awspcaclusterissuers
Are both really needed for reading by spire rather then just awspcaclusterissuers? Which one did you configure it to point to? We may need to come up with another value for additional permissions.
from helm-charts-hardened.
ranjit-se7en
commented on September 28, 2024
I added both of them. However, I believe that people using clusterissuers would also face the same problem.
from helm-charts-hardened.
ranjit-se7en
commented on September 28, 2024
Also, I don't understand why the -ca is appended even in the overrides. If someone wants to use existing issuers this won't work well as the issuers will mistach.
from helm-charts-hardened.
ranjit-se7en
commented on September 28, 2024
The clusterissuers are needed for sure, if any custom issuers are used like awspcaclusterissuers we should have a provision to specify them(in helm) and accordingly update the RBAC. Here is a list of all issuers that cert-manager supports https://cert-manager.io/docs/configuration/issuers/
from helm-charts-hardened.
kfox1111
commented on September 28, 2024
I deployed cert manager in minikube, the aws plugin, made a dummy awspcaclusterissuer. then deployed with:
spire-server:
upstreamAuthority:
certManager:
enabled: true
issuer_name: example
issuer_kind: AWSPCAClusterIssuer
issuer_group: awspca.cert-manager.io/v1beta1
It creates the certrificate request:
NAME APPROVED DENIED READY ISSUER REQUESTOR AGE
spiffe-ca-nc4fw example-ca system:serviceaccount:default:spire-server 30s
So, I can reproduce the extra -ca added to the name... I don't have access to AWSPCA really though so cant test it further.
I looked through the code in spire though and cant find any references to it actually needing to get/list issuers though with regard to the upstream plugin.
How does it behave/fail if it doesn't have the extra types listed in the role?
from helm-charts-hardened.
kfox1111
commented on September 28, 2024
At least in the case of switching the kind to ClusterIssuer, I was able to use it without needing any changes to the role. The spire server doesn't seem to need it.
Could you please try pr 361 and see if that lone is enough to get things working for you?
from helm-charts-hardened.
Related Issues (20)
- Fix documentation SPIRE references
- Valid YAML checker HOT 2
- trust-domain added to a fqdn in ingress HOT 1
- AWS KMS is hardcoded to use key_metadata_file
- Multicluster kubeConfig cluster kubeConfigName setting documenation
- SVID is not valid: public key "X5ZOAszrYj0LnaHdUqRWLZcMtzpgcY9L" not found in trust domain {federated cluster} HOT 10
- incorrect versions in subcharts HOT 1
- move spire-controller-manager to a separate pod HOT 1
- Move closer to production configuration HOT 8
- Tornjak gate testing might be broken
- unit test duplicate object detection function
- nested tests failure HOT 1
- spiffe-oidc-discovery-provider - Allow disabling csi driver and specifying custom volume HOT 2
- install note addition for kubelet path
- How to configure spire-agent on cluster-A to talk to spire-server on root-spire-server HOT 5
- Can't set spiffe-oidc-discovery-provider pod labels that matches SpiffeID pod selector HOT 1
- Feature: Provide native support for `aws_iid` nodeAttestor plugin.
- Grafana Dashboard HOT 1
- federation bundleEndpoint upstream support
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from helm-charts-hardened.