SSL/TLS: Certificate chain challenges

Update #1: I've updated the part "Get them into our webserver" by adding a link back to the first SSL/TLS Post where i describe how to correctly create the bundle files!

In my last post i've described what you need to do with the new site and intermediate certificates you got from your issuer in the paragraph "Get the certificate(s) in place".

Sometimes they can make your life harder when the chain is incorrect or you're losing legacy support when you use them. So here is my post about some challenges you may encounter when dealing with certificate chains and how to avoid them.

Different and missing intermediates

Recently i had to update a certificate chain running with GeoTrust TrueBusinessID Organization-validated certificates using 2048bit RSA / SHA-1 signed certificates.

Since SHA-1 signed certificates aren't secure anymore and it's highly advisable to use SHA-2 (SHA-256 or higher) cert's, i've updated the chain to 2048bit RSA / SHA-256 certificates - still GeoTrust TrueBusinessID OV cert's.

The first thing you'll notice is that the signing intermediate - the first intermediate which has issued your site certificate - is a different one. The SHA-1 signed cert had the "GeoTrust SSL CA - G2" whereas the SHA-256 signed was issues by "GeoTrust SSL CA - G3". So far so good.

But here is the first problem. Your apache2 or nginx bundle may only contain this single intermediate (The G3 intermediate) and would pass the Qualys SSL Labs test, since in recent browsers the G3 certificate is in the trusted store*.

But if you test your chain for example with older OpenSSL versions or with the Symantec SSL Toolbox they will tell you that your certificate is valid but the chain is incomplete. The problem is that older software or trusted stores do not contain the newer G3 intermediate.

*trusted store: a collection of root certificates your operating system or browser trusts in the first place so it can validate site certificates with them. On *nix systems they're mostly packed into a single bundle file called ca-certificates.pem.

Okay. But how to deal with it?

This is the tricky part if you've never done this before. We need to go step by step to find out the complete certificate chain back to the first root certificate even older trusted stores know (and trust in turn).

You can check every ssl certificate - site or intermediates - for various parameters like the issuer which we're looking for or other details like the encryption and signing algorithms.

In fact OpenSSL can do this for you with the following easy command:

openssl x509 -text -noout -in <certificate.pem/.crt>

If we try that with our G3 certificate the output is:

Certificate:
Data:
    Version: 3 (0x2)
    Serial Number: 146031 (0x23a6f)
Signature Algorithm: sha256WithRSAEncryption
    Issuer: C=US, O=GeoTrust Inc., CN=GeoTrust Global CA
    Validity
        Not Before: Nov  5 21:36:50 2013 GMT
        Not After : May 20 21:36:50 2022 GMT
    Subject: C=US, O=GeoTrust Inc., CN=GeoTrust SSL CA - G3
    Subject Public Key Info:
        Public Key Algorithm: rsaEncryption
            Public-Key: (2048 bit)
            Modulus:
                00:e3:be:7e:0a:86:a3:cf:6b:6d:3d:2b:a1:97:ad:
                49:24:4d:d7:77:b9:34:79:08:a5:9e:a2:9e:de:47:
                12:92:3d:7e:ea:19:86:b1:e8:4f:3d:5f:f7:d0:a7:
                77:9a:5b:1f:0a:03:b5:19:53:db:a5:21:94:69:63:
                9d:6a:4c:91:0c:10:47:be:11:fa:6c:86:25:b7:ab:
                04:68:42:38:09:65:f0:14:da:19:9e:fa:6b:0b:ab:
                62:ef:8d:a7:ef:63:70:23:a8:af:81:f3:d1:6e:88:
                67:53:ec:12:a4:29:75:8a:a7:f2:57:3d:a2:83:98:
                97:f2:0a:7d:d4:e7:43:6e:30:78:62:22:59:59:b8:
                71:27:45:aa:0f:66:c6:55:3f:fa:32:17:2b:31:8f:
                46:a0:fa:69:14:7c:9d:9f:5a:e2:eb:33:4e:10:a6:
                b3:ed:77:63:d8:c3:9e:f4:dd:df:79:9a:7a:d4:ee:
                de:dd:9a:cc:c3:b7:a9:5d:cc:11:3a:07:bb:6f:97:
                a4:01:23:47:95:1f:a3:77:fa:58:92:c6:c7:d0:bd:
                cf:93:18:42:b7:7e:f7:9e:65:ea:d5:3b:ca:ed:ac:
                c5:70:a1:fe:d4:10:9a:f0:12:04:44:ac:1a:5b:78:
                50:45:57:4c:6f:bd:80:cb:81:5c:2d:b3:bc:76:a1:
                1e:65
            Exponent: 65537 (0x10001)
    X509v3 extensions:
        X509v3 Authority Key Identifier: 
            keyid:C0:7A:98:68:8D:89:FB:AB:05:64:0C:11:7D:AA:7D:65:B8:CA:CC:4E

        X509v3 Subject Key Identifier: 
            D2:6F:F7:96:F4:85:3F:72:3C:30:7D:23:DA:85:78:9B:A3:7C:5A:7C
        X509v3 Basic Constraints: critical
            CA:TRUE, pathlen:0
        X509v3 Key Usage: critical
            Certificate Sign, CRL Sign
        X509v3 CRL Distribution Points: 

            Full Name:
              URI:http://g1.symcb.com/crls/gtglobal.crl

        Authority Information Access: 
            OCSP - URI:http://g2.symcb.com

        X509v3 Certificate Policies: 
            Policy: 2.16.840.1.113733.1.7.54
              CPS: http://www.geotrust.com/resources/cps

        X509v3 Subject Alternative Name: 
            DirName:/CN=SymantecPKI-1-539
Signature Algorithm: sha256WithRSAEncryption
     a0:d4:f7:2c:fb:74:0b:7f:64:f1:cd:43:6a:9f:62:53:1c:02:
     7c:98:90:a2:ee:4f:68:d4:20:1a:73:12:3e:77:b3:50:eb:72:
     bc:ee:88:be:7f:17:ea:77:8f:83:61:95:4f:84:a1:cb:32:4f:
     6c:21:be:d2:69:96:7d:63:bd:dc:2b:a8:1f:d0:13:84:70:fe:
     f6:35:95:89:f9:a6:77:b0:46:c8:bb:b7:13:f5:c9:60:69:d6:
     4c:fe:d2:8e:ef:d3:60:c1:80:80:e1:e7:fb:8b:6f:21:79:4a:
     e0:dc:a9:1b:c1:b7:fb:c3:49:59:5c:b5:77:07:44:d4:97:fc:
     49:00:89:6f:06:4e:01:70:19:ac:2f:11:c0:e2:e6:0f:2f:86:
     4b:8d:7b:c3:b9:a7:2e:f4:f1:ac:16:3e:39:49:51:9e:17:4b:
     4f:10:3a:5b:a5:a8:92:6f:fd:fa:d6:0b:03:4d:47:56:57:19:
     f3:cb:6b:f5:f3:d6:cf:b0:f5:f5:a3:11:d2:20:53:13:34:37:
     05:2c:43:5a:63:df:8d:40:d6:85:1e:51:e9:51:17:1e:03:56:
     c9:f1:30:ad:e7:9b:11:a2:b9:d0:31:81:9b:68:b1:d9:e8:f3:
     e6:94:7e:c7:ae:13:2f:87:ed:d0:25:b0:68:f9:de:08:5a:f3:
     29:cc:d4:92

The interesting part for us is the Issuer field. It reads C=US, O=GeoTrust Inc., CN=GeoTrust Global CA.

Okay. So to complete our apache2/nginx bundle file, we need to find this intermediate and add it accordingly.

The 3-headed hydra

When trying to find an intermediate or root certificate from a certificate issuer, the first step is to go to the issuers website and try to get the certificate from them. They have a web page which lists all of their root and intermediates namely called signing CA's. Some of them are as old as the sun and others are so new that they're still not used for signing.

Anyways. We're looking at GeoTrusts certificates page for a certificate with the CN (Common Name) "GeoTrust Global CA" and indeed we found one.

Great!...... really?

If you add this to your bundle openssl will tell you, that you're using a self-signed certificate, SSL Labs will complain about the attached anchor (the root CA) and Symantec will tell you the same again that your chain is incomplete.

What the f***!?

In fact there exists three GeoTrust Global CA signing CA's. We got one which is also recognized only by newer trusted stores and not yet the path we really want for legacy systems.

If you have a look at the SSL-Tools website you can see all three available certificates. For real legacy support we have to use one that was issued by the very first Root CA - the Equifax Secure Certificate Authority.

When doing so, even systems with a really old trusted store are able to verify the whole certificate chain up to our shiny new SHA-256 certificate because we can validate each certificate up to the ancient Equifax Secure Certificate Authority.

Getting the chain right

Okay. We will use the Global CA certificate with the fingerprint 7359­755c­6df9­a0ab­c306­0bce­3695­64c8­ec45­42a3 and attach it to our bundle.

Now the certificate chain is complete and looks like the following:

example.com -> GeoTrust SSL CA - G3 -> GeoTrust Global CA -> Equifax Secure Certificate Authority.

We did it! As a bonus here is how to bundle them for nginx and apache2.

Get them into our webserver

I've explained a lot for this step in the first part of the SSL/TLS series, in "Get the certificate(s) in place", so i'll only post you the parameters and what they contain.

Apache2

SSLCertificateFile /etc/ssl/certs/example_com_032015.crt
SSLCertificateChainFile /etc/ssl/certs/example_com_032015_bundle.crt
SSLCertificateKeyFile /etc/ssl/private/example_com_032015.key

The SSLCertificateFile only contains our site or leaf certificate example.com. The SSLCertificateChainFile contains all other intermediates except the very first root CA Equifax Secure Certificate Authority and the .key file is obvious.

nginx

This one is quite easy as you remember that all intermediates and the site certificate are bundled together into a single certificate file

ssl_certificate /etc/ssl/certs/example_com_032015_nginx.crt

What i did not mentioned before is that nginx also can separate the site and intermediate certificates in the ssl_trusted_certificate parameter which works similar to apache2. This is optional now but necessary when using OCSP Stapling or when you want to verify client certificates. OCSP stapling will be covered in a later post. For this parameter you need at least nginx 1.3.7 or higher.

Conclusion

SSL/TLS can be easy but also can be a real brain f*ck if you do it wrong. The worst thing which can happen is that customers complain that they cannot connect to your site or service anymore but you can. Those wrongly configured certiticate chains can be the cause and really hard to find as one in the first place so be careful with that!