cloudfoundry / java-buildpack-security-provider Goto Github PK

I need to configure a reactive HTTP client with an instance of KeyManager, such that calls can be made using the Instance Identity Credentials.

In a similar request, #2, it was recommended to use the default SSLContext, that gets picked up by most clients automatically. This does not seem to be the case for Reactor-Netty however, probably because it uses a different type of SSLContext (io.netty.handler.ssl vs javax.net.ssl).

I'm therefore trying to configure the Netty SSLContext by passing an Instance of KeyManager manually, but obtaining this instance proves very difficult. I have found a way to get an instance using reflections that works, but cannot be the intended way:

KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm());

Field factorySpiField = KeyManagerFactory.class.getDeclaredField("factorySpi");
factorySpiField.setAccessible(true);
KeyManagerFactorySpi kmfSpi = (KeyManagerFactorySpi) factorySpiField.get(kmf);

if (!kmfSpi.getClass().getName()
        .equals("org.cloudfoundry.security.CloudFoundryContainerKeyManagerFactory$SunX509"))
    throw new RuntimeException("KeyManagerFactorySpi is not of expected type for cloud foundry");

Method getContainerKeyManagerMethod = Class
        .forName("org.cloudfoundry.security.CloudFoundryContainerKeyManagerFactory")
        .getDeclaredMethod("getContainerKeyManager");
getContainerKeyManagerMethod.setAccessible(true);
X509ExtendedKeyManager keyManager = (X509ExtendedKeyManager) getContainerKeyManagerMethod.invoke(kmfSpi);

SslContextBuilder sslCtxBuilder = SslContextBuilder.forClient().keyManager(keyManager);

HttpClient httpClient = HttpClient.create();
httpClient = httpClient.secure(builder -> builder.sslContext(sslCtxBuilder));

What is the correct (or intended) method of obtaining this instance?

Pivotal uses GITBOT to synchronize Github issues and pull requests with Pivotal Tracker.
Please add your new repo to the GITBOT config-production.yml in the Gitbot configuration repo.
If you don't have access you can send an ask ticket to the CF admins. We prefer teams to submit their changes via a pull request.

Steps:

• Fork this repo: cfgitbot-config
• Add your project to config-production.yml file
• Submit a PR

If there are any questions, please reach out to [email protected].

We have noticed a situation where occasionally, a single instance of an app, will start raising bad_certificate exceptions when calling other apps in cloud foundry.

This error is due to a client certificate refusal by the go-router and only for apps that do not explicitly use mtls, meaning, the ones doing mtls with the go-router using the instance certificate.

We have thoroughly investigated this incident. Looking inside the app instance container shows a proper key/certificate pair, with proper validity. Restarting the app instance will eliminate the issue (at least for the following 24 hours), the errors don't start getting raised after the certificate expires, but after the certificate is renewed. Touching the certificate and/or key file in the mounted volume (through the diego-cell as root), will cause the filewatcher to be triggered and will also resolve the issue.

Our conclusion is that this is a racing-condition. The certificate file is always updated first and the key file follows. Most of the times, this update is almost instantaneous, but there are times where you can see a millisecond or two of difference. Since the watcher threads do not update the keystore in a synchronised block, we think this occurs because occasionally the first file watcher thread that is triggered, reads a mismatching private key file, and it finishes after the second file watching thread (which reads and updates the keystore with a matching key/certificate pair), updating the keystore with a mismatched key/certificate pair.

Such a situation will also match the observed behaviour (raised bad_certificate exceptions).

We have built our own custom security provider to try to log this behaviour (and other potential root causes), but (un)fortunately, all apps using our custom security provider always work. We do notice an occasional thread reading a mismatched key/certificate pair, but it's always the proper order, meaning, the thread reading the mismatched key/certificate pair always finishes first and the keystore is then properly updated with the second thread which will read a properly matched key/certificate pair.

This started occurring since 6 months ago, more or less. We run two foundations with 1600+ apps each and a total amount of instances above 3000 on each foundation. This can potentially happen each day to less than 1% of the instances. The main difference starting 6 months ago was using bionic stemcells and starting to use larger and newer vms for the diego-cells in azure (we also have aws foundations but they are not affected).

Since this began to become a real nuisance for us, we couldn't wait for our logging to show any hard proof of the racing condition, as it seems to be a typical heisenbug. The extra log entries and check operations might add just enough delay for the thread to always read a properly matched key/certificate pair.

I will add a pull request with a solution for this situation. While a simple synchronized block would already fix the issue, by guaranteeing that the second thread always finishes last, I prefer to instead update the keystore only if the private key matches the certificate, so that the theoretical case of a request being done in the microseconds that a key store with a mismatched key/certificate pair is updated doesn't occur.

Hey not sure if this project is in a stable state, but I am using PCF and PAS and want to check for the updated certificates and keys to our java microservices. I can't find any documentation on how to do it, but seems this library may do what is required. How do I get the new certificate created from this library?

The used bouncy castle version is 1.60 and is dated from June / 2018.