First, awesome idea for a project 👍 Describe the bug</stron

<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clip

Client Certificate not presented by Apache HTTP Client when loaded in SSL Context,about hakky54/sslcontext-kickstart

Comments (15)

Hakky54 commented on June 14, 2024 1

Could you give me a bit more info on how you are doing this so that I can understand it further? My understanding is that the PEM Tools are somehow not loading the certificate, so it silently doesn't end up in the truststore as one would expect, is that correct? Is the parsing issue just because of the "BEGIN TRUSTED CERTIFICIATE" text at the top of the file, or is it deeper than that?

Sure, I would love to. The PemUtils has two main functions. One for the key/identity material and the other for the certificate material. It has support for different kind of pem files. The different files can be recognised by their header.

As key material the following types are supported:

Encrypted key containing header -> -----BEGIN ENCRYPTED PRIVATE KEY-----
Unencrypted ec key containing header ->-----BEGIN EC PRIVATE KEY-----
Unencrypted rsa key containing header -> -----BEGIN RSA PRIVATE KEY-----

The key material should be accompanied with a certificate chain. The certificate chain pem file can be recognised with the following header: -----BEGIN CERTIFICATE-----

As trust material only pem files containing the header -----BEGIN CERTIFICATE----- are allowed/supported.

See here for all possible files which the library supports: Pem Files
And here for the unit test as a reference the usage: Unit test

My understanding is that the PEM Tools are somehow not loading the certificate, so it silently doesn't end up in the truststore as one would expect, is that correct?

So back to your question. The library was indeed ignoring the pem file which contained the header -----BEGIN TRUSTED CERTIFICATE----- because it was allowing only a file with the header: -----BEGIN CERTIFICATE-----. During the design of the library I initially wanted to validate if every certificate has been added correctly, but that can be very error prone. Because one can use a single file containing private key, certificate chain and certificate. Or a single file containing multiple certificates, but I might need to re think about this use case and provide a better flow handling.

Is the parsing issue just because of the "BEGIN TRUSTED CERTIFICIATE" text at the top of the file, or is it deeper than that?

It is not just the header, if you change the header to -----BEGIN CERTIFICATE----- it will still not work because it cannot parse the content easily. However I can use a library, Bouncy-Castle, to parse that for this use case, I tried as I mentioned before but the TrustManager within the SSLContext is not able to use that somehow. And I am not quite sure why, so it is a deeper issue. I will further investigate this although I am pretty sure this is something much deeper.

I added a certificate validation into the library, so at least users will get some useful information when something is not going as expected, see here for the details: 1ff7207

from sslcontext-kickstart.

tadhgpearson commented on June 14, 2024 1

$ curl --cacert ca.pem --key client.key --cert client.cer https://prod.idrix.eu/secure/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

from sslcontext-kickstart.

tadhgpearson commented on June 14, 2024 1

(Also, it's amazing how much stuff is suddenly broken when you disable that one root cert!)

from sslcontext-kickstart.

Hakky54 commented on June 14, 2024 1

Good to hear that you get the same results. Now we can conclude that curl somehow obtains the trusted certificates from the keychain of mac os x while checking if the server is trusted. It could not find any trusted certificate which matched the CA or any of the intermediate certificate of https://prod.idrix.eu/secure/ and therefor it is now failing. On the java side we had the same kind of behaviour as Java also couldn't find any matching trusted certificate for the certificate which is sent by the server. I didn't know curl did something clever like this, but that makes it a bit tougher to debug.

If you refactor your code while using either of the the following pem files:

dst-root-ca-x3.pem

In java:

X509ExtendedTrustManager trustManager = PemUtils.loadTrustMaterial("dst-root-ca-x3.pem");
X509ExtendedKeyManager keyManager = PemUtils.loadIdentityMaterial("client.cer", "client.key");
SSLFactory sslFactory = SSLFactory.builder()
        .withTrustMaterial(trustManager)
        .withIdentityMaterial(keyManager)
        .build();

Or with curl:

curl -v --cacert dst-root-ca-x3.pem --key client.key --cert client.cer https://prod.idrix.eu/secure/

prod.idrix.eu.pem

In java

X509ExtendedTrustManager trustManager = PemUtils.loadTrustMaterial("prod.idrix.eu.pem");
X509ExtendedKeyManager keyManager = PemUtils.loadIdentityMaterial("client.cer", "client.key");
SSLFactory sslFactory = SSLFactory.builder()
        .withTrustMaterial(trustManager)
        .withIdentityMaterial(keyManager)
        .build();

Or with curl

curl -v --cacert prod.idrix.eu.pem --key client.key --cert client.cer https://prod.idrix.eu/secure/

It should work with these cases.

By the way, thank you for your valuable input! I have refactored the library to support certificates with the header -----BEGIN TRUSTED CERTIFICATE----- It is already available in the latest version of the master branch but not released yet. I will publish it very soon. Please let me know if you have other questions or else I will close this issue.

from sslcontext-kickstart.

Hakky54 commented on June 14, 2024 1

My bad, I was running your code on Java 8 and therefor it was working for me. I tried it with Java 11 and it was indeed failing. The issue is something with the new default TLSv1.3 protocol with java 11. It might be more secure and therefor not allowing it, but I am not sure. But I need to dig into it. This answer on stackoverflow might be useful https://stackoverflow.com/a/57607812/6777695. For now you can just use TLSv1.2

If you refactor your code to the following snippet it should work:

public class SSLContextFactory {

    public static SSLFactory buildFactory(){
        X509ExtendedTrustManager trustManager = PemUtils.loadTrustMaterial( "dst-root.crt");
        X509ExtendedKeyManager keyManager = PemUtils.loadIdentityMaterial("client.cer", "client.key");
        return  SSLFactory.builder()
                .withDefaultTrustMaterial()
                .withTrustMaterial(trustManager)
                .withIdentityMaterial(keyManager)
                .withProtocols("TLSv1.2")
                .build();
    }

}

And your unit test:

class SSLContextFactoryTest {

    @Test
    void buildSSLContext() throws IOException, InterruptedException {
        SSLFactory sslFactory = SSLContextFactory.buildFactory();

        CloseableHttpClient client = HttpClients.custom()
                .setSSLSocketFactory(new SSLConnectionSocketFactory(
                        sslFactory.getSslContext(),
                        sslFactory.getSslParameters().getProtocols(),
                        sslFactory.getSslParameters().getCipherSuites(),
                        sslFactory.getHostnameVerifier()))
                .build();

        CloseableHttpResponse response = client.execute(new HttpGet("https://prod.idrix.eu/secure/"));
        int rsStatus = response.getStatusLine().getStatusCode();
        assertTrue(rsStatus >= 200 && rsStatus < 400);

        String responseBody = IOUtils.getContent(response.getEntity().getContent());
        assertFalse(responseBody.contains("No SSL client certificate presented"));
    }

}

from sslcontext-kickstart.

tadhgpearson commented on June 14, 2024

FYI - I also tried doing this with the SSL Socket Factory, but same result

from sslcontext-kickstart.

Hakky54 commented on June 14, 2024

Hi Tadhg,

Thank you for reporting this issue and proving an example project to reproduce the specific issue. I have analysed it and discovered couple of issues. Firstly I noticed that you use the ca.pem which is an openssl specific format. This certificate gets currently ignored by the library, because it is not supported yet. Today I tried to add the parsing function and it is able to parse it, however the Java TrustManager is somehow not able to handle this kind of certificate. I am getting the following exception when:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

This means that it could not find a matching certificate within the trust manager. This means that either the trustmanager is null, trustmanager does not contain trusted certificates or the trustmanager really does not have a matching certificate. While debugging I saw that the certificate is present, see below for a screenshot:

When I saw this I thought the ca certificate was not correct, to test that I tried the following curl:

curl --cacert ca.pem --key client.key --cert client.cer https://prod.idrix.eu/secure/

The above curl works with the certificates which you have provided in your repo, so the ca-pem file is a correct certificate file. So my feeling is that the standard java TrustManager implementation is not able to handle this kind of certificate out of the box. I extracted a fresh certificate from https://prod.idrix.eu/secure/ See below for the output file:

standard-ca.pem

If I use this certificate with the following snippet in your code it works:

X509ExtendedTrustManager trustManager = PemUtils.loadTrustMaterial("standard-ca.pem");
X509ExtendedKeyManager keyManager = PemUtils.loadIdentityMaterial("client.cer", "client.key");
SSLFactory sslFactory = SSLFactory.builder()
        .withTrustMaterial(trustManager)
        .withIdentityMaterial(keyManager)
        .build();

Although I could make it possible to parse the certificate however the trustmanager implementation of Java is not able to handle it. I would advise to not use a certificate containing the header: -----BEGIN TRUSTED CERTIFICATE----- and switch to a certificate containing the following header: -----BEGIN CERTIFICATE-----

from sslcontext-kickstart.

tadhgpearson commented on June 14, 2024

OK - that's insightful, thank you - I appreciate it. I'll try with the different certificate format

Today I tried to add the parsing function and it is able to parse it, however the Java TrustManager is somehow not able to handle this kind of certificate. I am getting the following exception when:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Funny, the reason I went to SSL Kickstart was because I was getting this exception when trying to load the certificate manually to the truststore :D
Could you give me a bit more info on how you are doing this so that I can understand it further? My understanding is that the PEM Tools are somehow not loading the certificate, so it silently doesn't end up in the truststore as one would expect, is that correct?

Is the parsing issue just because of the "BEGIN TRUSTED CERTIFICIATE" text at the top of the file, or is it deeper than that?

from sslcontext-kickstart.

tadhgpearson commented on June 14, 2024

OK - cool, thanks, I learnt something new. This is interesting stuff.

However, this is an aside from the bug I was initially trying to reproduce. (I'm working with a different set of underlying certs than the one I put in my test repo, and that cert chain does not have the "TRUSTED CERTIFICATE" format.)

If the only issue here were the parsing of that cert format, I would expect that I could solve it by converting it to the regular format. So I tried it by running

openssl x509 -in ca.pem -out ca.crt

on the "TRUSTED CERTIFICATE" in ca.pem, and then running the same code on the resulting cert without the openSSL specific extensions. See branch standard-ssl-cert on the same repository.

It still doesn't work, no error :/

from sslcontext-kickstart.

Hakky54 commented on June 14, 2024

If the only issue here were the parsing of that cert format, I would expect that I could solve it by converting it to the regular format. So I tried it by running

I also tried that, but forgot to mention it during my previous answer. So even though we can convert it to an appropriate format it still doesn't work. So I would assume something is missing within the certificate which is required for java...

Could you explain me how to created this ca.pem file? In that way I would be able to analyse this issue better.

from sslcontext-kickstart.

tadhgpearson commented on June 14, 2024

OK... weird!

This is what I did

openssl req -new -newkey rsa:1024 -nodes -out ca.csr -keyout ca.key
openssl x509 -trustout -signkey ca.key -days 365 -req -in ca.csr -out ca.pem
openssl genrsa -out client.key 1024
openssl req -new -key client.key -out client.csr
openssl x509 -req -days 365 -in client.csr -CA ca.pem -CAkey ca.key -set_serial 01 -out client.cer

based on this post from StackOverflow https://superuser.com/a/126165/1259703

from sslcontext-kickstart.

Hakky54 commented on June 14, 2024

I have discovered something but before jumping into conclusions I want to validate with you if your curl behaves the same as my curl.

First of all could you disable DST Root CA X3 certificate in your System Roots keychain?
See below for the location, you can double click to open the specific certificate.

Select the trust section and choose Never Trust option from When using this certificate section.

Apply the changes by closing this window. It will probably ask for your password to confirm your changes. After you have done this part, can you rerun your curl command curl --cacert ca.pem --key client.key --cert client.cer https://prod.idrix.eu/secure/ and tell me if it now passes or fails?

from sslcontext-kickstart.

tadhgpearson commented on June 14, 2024

OK... that makes sense, but I don't believe the issue is fixed

I pushed a new branch using your v6.1.1-SNAPSHOT code on master, and the DST root cert you pasted above, and changed the SSL Context builder like so

    X509ExtendedTrustManager trustedCerts = PemUtils.loadTrustMaterial( "dst-root.crt");
    X509ExtendedKeyManager key = PemUtils.loadIdentityMaterial("client.cer", "client.key");
    SSLFactory sslFactory = SSLFactory.builder().withDefaultTrustMaterial().withTrustMaterial(trustedCerts).withIdentityMaterial(key).build();

... but the client cert is not presented.

from sslcontext-kickstart.

tadhgpearson commented on June 14, 2024

Yep - works with TLS 1.2, wow... not obvious at all. Interesting that OpenSSL still defaults to old ciphers.

Thank you. Looking forward to the release of 6.1 where I can replace the big dirty class I wrote to load all the certs myself with this library :)

from sslcontext-kickstart.

Hakky54 commented on June 14, 2024

Glad to hear that you could confirm that it is working! Yes it is interesting to see the different behaviour of openssl with curl and java.

I just made a release with version 6.1.1, good luck with the simplifying 😊
Please let me know if you find any other things to improve

from sslcontext-kickstart.

Client Certificate not presented by Apache HTTP Client when loaded in SSL Context about sslcontext-kickstart HOT 15 CLOSED

Comments (15)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent