Let's Encrypt DST X3 Root Certificate Expiration

Updated

Summary

The root certificate DST Root CA X3 used to cross-sign Let's Encrypt X1 root certificate expired on 2021-09-30 and RHEL7 or earlier systems that are not adjusted may see secure connections fail.

Background

The root Certificate Authority (CA) certificate with CN = DST Root CA X3 expired at 2021:09:30 14:01:15 GMT.

Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Sep 30 21:12:19 2000 GMT
            Not After : Sep 30 14:01:15 2021 GMT
        Subject: O = Digital Signature Trust Co., CN = DST Root CA X3
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier:
                C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10

Root certificate expiry is a normal, if infrequent, occurrence. In most cases connections would not be affected as any End-Entity certificates would also expire prior to the root expiry and would have to be replaced in the normal way and in doing so would acquire a new chain and root.

In this particular case the Let's Encrypt ISRG Root X1 root certificate is cross-signed by the DST Root CA X3 and both of these certificates are in the RHEL7 system certificate bundle. openssl in RHEL7 has an older method of validating a certificate chain that results in the cross-signing also being followed and after 2021:09:30 this will result in a validation failure due to the expiry.

Updated System Certificate Packages

We have released updated ca-certificates packages for RHEL6 ELS, RHEL7 and RHEL8.

These can be updated via the normal yum or dnf update commands.

There is no updated package for the non-ELS RHEL6. The client workaround below can be used to remove the expired certificate from the system bundle.

RHEL6 Problems

It should be noted that RHEL6 as a client can still have issues depending on the intermediate chain provided from the server. Notably, if the server provides the X1 cross-signed certificate as the last item in the chain then

  • RHEL6 clients with the DST X3 certificate in the system bundle will fail connections with - certificate has expired
  • RHEL6 clients where the DST X3 certificate is not in the system bundle will fail connections with - unable to get local issuer certificate

If the server is under your control then it may be possible to use the server workaround to adjust the provided chain to avoid the cross-sign root, but note that this may cause problems with other clients. See below for links regarding this issue.

Client Connection Workaround

To work around the openssl client problem on RHEL 6 first ensure your ca-certificates package is updated to the most recently available in your RHEL6 channels ca-certificates-2020.2.41-65.1.el6_10.noarch.rpm.

Then to remove the expired root CA from the system trust store,

  • Create an exclusion file: 

      # perl -e 'while(<>){last if $_ =~ m/DST Root CA X3/;}print $_;while(<>){last if length($_)==1;print $_}' </etc/pki/tls/certs/ca-bundle.crt > /etc/pki/ca-trust/source/blacklist/DST_Root_CA_X3.pem

  • Update the system trust store: 

      # update-ca-trust extract

Server Configuration Workaround

The following workaround is designed for cases where the client population has problems with completing the trust chain to a known anchor. Most end-user clients should validate correctly, the Let's Encrypt articles on Content from letsencrypt.org is not included.Certificate Compatibility and Content from letsencrypt.org is not included.Android Device Compatibility cover the details.

For machines that run services with Let's Encrypt certificates for clients that are unable to validate a provided chain that includes the cross-signed X1 certificate the problem can be avoided by adjusting the chain that is provided to clients to avoid including the cross-signed version. This then leaves the single intermediate R3 as the end of the provided chain and allows libraries to validate straight to the self-signed X1 anchor.

As of 2021-09-22 the standard chain provided when using certbot to obtain or renew a Let's Encrypt certificate includes the cross-signed X1 certificate pointing to the expiring X3

chain.pem - CERTIFICATE
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            91:2b:08:4a:cf:0c:18:a7:53:f6:d6:2e:25:a7:5f:5a
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
        Validity
            Not Before: Sep  4 00:00:00 2020 GMT
            Not After : Sep 15 16:00:00 2025 GMT
        Subject: C=US, O=Let's Encrypt, CN=R3
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Subject Key Identifier:
                14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6
            X509v3 Authority Key Identifier:
                keyid:79:B4:59:E6:7B:B6:E5:E4:01:73:80:08:88:C8:1A:58:F6:E9:9B:6E

            Authority Information Access:
                CA Issuers - URI:http://x1.i.lencr.org/

            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://x1.c.lencr.org/

            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1


chain.pem - CERTIFICATE
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            40:01:77:21:37:d4:e9:42:b8:ee:76:aa:3c:64:0a:b7
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
        Validity
            Not Before: Jan 20 19:14:03 2021 GMT
            Not After : Sep 30 18:14:03 2024 GMT
        Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            Authority Information Access:
                CA Issuers - URI:http://apps.identrust.com/roots/dstrootcax3.p7c

            X509v3 Authority Key Identifier:
                keyid:C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10

            X509v3 Certificate Policies:
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.root-x1.letsencrypt.org

            X509v3 CRL Distribution Points:

                Full Name:
                  URI:http://crl.identrust.com/DSTROOTCAX3CRL.crl

            X509v3 Subject Key Identifier:
                79:B4:59:E6:7B:B6:E5:E4:01:73:80:08:88:C8:1A:58:F6:E9:9B:6E

There are two ways of working around this.

Adjusting The Chain from certbot

If you are using certbot for management of these certificates you can edit the individual renewal configuration files under /etc/letsencrypt/renewal/ and add the following line to the [renewalparams] block

preferred_chain = ISRG Root X1

Then on the next renewal the chain will be adjusted the chain to not include the cross-signed root.

Manually editing the chain file

To manually edit the files that cause issues the chain.pem file needs to be updated to remove the 2nd certificate. And in the fullchain.pem file the last (3rd) certificate should be removed. This will allow clients to switch to their own trust chains to locate the pure X1 certificate rather than possibly using the X1/X3 and referring to the expired X3 and flagging the connection as insecure. If you are using any renew_hook scripts to deploy your certificates to alternate locations these will need to be re-run to acquire the adjusted chains.

SBR
Product(s)
Category
Components
Article Type