Downtime after a caBundle until Secret propagation to pod about cert-controller HOT 12 OPEN

+1 on supporting multiple bundles.

from cert-controller.

What's the invariant that'd make restarting the pod pick up the new secret?

If anything, the same container will be just restarted by the container runtime on the same host (restartpolicy) and it will still see the same volume, no? And with that, it will miss the change event and keep using the same secret.

I think it would be a lot better to do something like cross signing, or expanding the CA list by keeping both old/new (prev/next) CA certs around.

from cert-controller.

+1 to having multiple bundles. Might be worth figuring out a way to gradually roll out the cert across processes too.

from cert-controller.

That's right, which is exactly why RestartOnSecretRefresh exists - it's probably faster to just kill the pod and let it restart

from cert-controller.

There's probably a better way but I'm not sure what it is and this worked well enough :)

from cert-controller.

I'm not sure. In my experience restarting the pod has always worked instantly. I feel like someone told me that once but it was some time ago...

Yup, serving up multiple certs would definitely be a cleaner way of doing this. Given that we set a default 10yr expiry period (IIRC), we were most concerned with the startup performance, where the original secret is effectively empty so there's zero chance of it working during the initial startup. Again, it wouldn't surprise me if cert-manager solved this in some much better way.

from cert-controller.

In our setup we'd much rather not use cert-manager (it comes with multiple components/CRDs).
Similarly RestartOnSecretRefresh also doesn't work because it's too violent (drops requests, and we run some HA webhooks where simultaneously restarting all of them is a recipe for disaster).

I think developing a patch around keeping both CAs in the caBundle may solve the current downtime problem while the Secret propagates and is picked up.

from cert-controller.

sgtm in all cases except for the initial startup. @maxsmythe , @ritazh , wdyt?

from cert-controller.

Hi Folks, there is a similar issue deislabs/ratify#821. The mTLS is required between Gatekeeper and external data provider. By default, cert-controller is used to generate and rotate Gatekeeper's webhook certificate. In our case, the user manually rotated the certificate. It seems Kubernetes took about 60-90 seconds to propagate changes to Secrets. During this period of delay, the request being sent to external data provider will fail.

from cert-controller.

@acpana This could be interesting work.

from cert-controller.

thanks for the tag max! I can have a look at this in my downtime from other projects. I will assign it to myself when I get to it. In the meantime, folks can feel free to jump on it if they have cycles.

from cert-controller.

I also raised this problem a while ago. See #13 .

from cert-controller.