ADDTrust External Root Expiration May 2020

ADDTrust External CA Root Expiration in 2020

More Information: 

https://support.sectigo.com/articles/Knowledge/Sectigo-AddTrust-External-CA-Root-Expiring-May-30-2020

https://www.cmu.edu/iso/service/cert-auth/addtrust.html

Summary:

Sectigo's legacy AddTrust External CA Root certificate expires on May 30, 2020. Modern clients should largely be unaffected. However, a compilation of affected users is listed below.

Devices that received security updates after mid 2015 should have the modern USERTrust RSA Certification Authority root certificate (valid until Jan 2038) in their operating system or browser truststores and should be largely unaffected.

Legacy devices that have not received updates to support newer roots will also likely be missing other essential security updates and support for standards required by the modern Internet. We strongly encourage decommissioning these devices if their software cannot be upgraded. Non-upgraded, legacy devices should never be exposed to the Internet and special mitigations should be applied to isolate them from neighbor systems.

Additional Impacts:

  • Client software based on OpenSSL prior to version 1.1.1 appears to have broken certificate path validation logic and will require workarounds
  • OpenLDAP clients on some platforms appear to have broken certificate path validation logic and will require workarounds
  • Clients configured to explicitly trust the AddTrust root may need client reconfiguration to either:
    • rely on operating system or vendor managed truststores or
    • explicitly trust the USERTrust RSA Certification Authority root or the alternative legacy AAA Certificate Services root (valid until Dec 2028).

Who is affected?

Anyone who Administers:

  • Legacy clients that have not received security updates since before mid 2015 and connect to Secure Sockets Layer (SSL) or Transport Layer Security (TLS) encrypted services such as web and email
  • OpenSSL based client software
  • Linux or macOS OpenLDAP clients that connect to ldap.berkeley.edu

Clients configured to explicitly trust the AddTrust External CA Root instead of relying on an operating system or vendor managed truststore. For example:

  • Java applications that do not use the default truststore
  • Lightweight Directory Access Protocol over SSL (LDAPS)
  • Anyone who administers SSL or TLS encrypted services connected to by the above clients.

Details:

The majority of modern clients are unaffected by this expiry, browsers simply choose a chain directly to the SHA-2 root (COMODO or USERTrust) and the cross-cert back to AddTrust is simply ignored. When the AddTrust External CA Root expires, Trust Chain A will no longer be used by clients, instead they will chain up via Trust Chain B.

Certificate path validation is done client-side from leaf to root. Modern clients that receive Trust Chain A with the cross signed intermediate (see below) from servers should ignore it and instead follow Trust Chain B. This applies even after the root of Trust Chain A expires on May 30, 2020. However, some clients may have problems if one or more of the following conditions is true:

Condition 1
Condition 2
Condition 3

The client is too old and does not have the USERTrust RSA Certification Authority root in its operating system or vendor managed truststore.

The client does not properly process Trust Chain B and instead always tries to follow Trust Chain A.

The client is configured to explicitly trust the AddTrust External CA Root and ignores its operating system or vendor managed truststore.

Conditions 1 and 2 may be addressed by configuring the server to send Trust Chain C.

Condition 3 requires the client to be reconfigured to either: 1) use the operating system or vendor managed truststore or 2) explicitly trust the USERTrust RSA Certification Authority root or the alternative legacy AAA Certificate Services root.

Certificates issued by the UC Berkeley Authority from April 30, 2020 through InCommon (Sectigo) will include a link to download Trust Chain C.  We encourage you to use Trust Chain B unless you specifically need Trust Chain C for legacy device compatibility or to work around broken client issues.

Trust Chain Path A: Trust Chain Path B: Trust Chain Path C (not shown):
AddTrust External CA Root [Root]
USERTrust RSA Certification Authority (Intermediate) [Intermediate 2]
InCommon RSA Server CA  [Intermediate 1]
End Entity [Leaf Certificate]
USERTrust RSA Certification Authority (Root CA) [Root]
InCommon RSA Server CA  [Intermediate 1]
End Entity [Leaf Certificate]
AAA Certificate Services [Root] (Exp: Dec 31, 2028)
USERTrust RSA Certification Authority [Intermediate 2] (Exp: Dec 31, 2028)
InCommon RSA Server CA [Intermediate 1] (Exp: Oct 5, 2024)
End Entity [Leaf Certificate]

What you need to do:

(Big thank you to Carnegie Mellon for detailed testing and identification)

  1. Evaluate whether you have devices that meet Conditions 1, 2, or 3 by setting a client device clock to after June 1, 2020 and testing connectivity.
  2. Apply the recommended fix for clients and/or servers meeting one or more of the Conditions below.

Condition 1 - Legacy Devices - Known Examples

We strongly encourage decommissioning these legacy devices if their software cannot be upgraded. Non-upgraded, legacy devices should never be exposed to the Internet and special mitigations should be applied to isolate them from neighbor systems. Legacy compatibility may be extended by reconfiguring servers to send Trust Chain C (see above). Contact your server admins to discuss whether that is possible.

  • Apple Mac OS X 10.11 (El Capitan) or earlier
  • Apple iOS 9 or earlier.
  • Google Android 5.0 or earlier.
  • Microsoft Windows Vista & 7 if the Update Root Certificates Feature has been disabled since before June 2010.
  • Microsoft Windows XP if an Automatic Root Update has not been received since before June 2010.
  • Mozilla Firefox 35 or earlier.
  • Oracle Java 8u50 or earlier.
  • Embedded devices (especially copy machines) that have not installed a firmware update since before mid 2015.

Condition 2 - Broken Clients - Known Examples

Reconfigure the server to send Trust Chain B or Trust Chain C and reconfigure the client to use the operating system managed truststore. Click the link for additional configuration information.

  • OpenSSL based client software

Client software that use OpenSSL libraries prior to version 1.1.1 for certificate path validation appear to always validate the full Trust Chain A sent from the server even though modern roots were configured to validate Trust Chain B.

This  behavior was observed on Red Hat Enterprise Linux 6 (OpenSSL 1.0.1e-fips) and 7 (OpenSSL 1.0.2k-fips). Advancing the clock past June 1, 2020 and attempting connections to servers that sent Trust Chain A resulted in failed connections. In these tests, OpenSSL returned expired certificate errors even though Trust Chain B's root was available in the truststores. This behavior appears to be fixed in Red Hat Enterprise Linux 8 (OpenSSL 1.1.1c FIPS) and Ubuntu 14.04 (OpenSSL 1.1.1) as the same clock advancing tests resulted in successful connections and OpenSSL validating properly to Trust Chain B's root.

To work around this behavior, remove the expired AddTrust root from the client's operating system managed truststore or explicitly trust either the USERTrust RSA Certification Authority root or AAA Certificate Services root (depending on what the server sends). 

ldap.berkeley.edu is already configured to send Trust Chain B

Examples of client programs that use OpenSSL for certificate path validation and were tested include OpenLDAP and postfix.

  • OpenLDAP clients on Linux and macOS

OpenLDAP clients on Red Hat Enterprise Linux 6 and 7 appear to always validate the full Trust Chain A sent from the server even though modern roots were configured to validate Trust Chain B. Once the client device clock was advanced to June 1, 2020, LDAPS connections failed validation on the expired AddTrust root until our test server was reconfigured to send either Trust Chain B or Trust Chain C.

OpenLDAP clients only enable TLS when either TLS_CACERT or TLS_CACERTDIR is set in ldap.conf. Either set TLS_CACERT to be the operating system managed PEM file or set TLS_CACERTDIR to an empty directory to force fallback to the operating system managed truststore.

Test whether a successful LDAPS connection can be made with:
ldapsearch -H ldaps://ldap.berkeley.edu:636 -x '(uid=111111)'

A successful connection will show query results while failing to negotiate TLS will show:
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

Or display debug info including the names of the chain certificates with:
ldapsearch -d 1 -H ldaps://ldap.berkeley.edu:636 -x '(uid=111111)'

ldap.berkeley.edu is currently sending Trust Chain B. Please reconfigure your OpenLDAP clients to use the operating system managed truststore if you are experiencing query failures.

Condition 3 - Explicit CA Trust - Known Examples

Reconfigure the client to use the operating system or vendor managed truststore if possible. If not, reconfigure the client to explicitly trust either the USERTrust RSA Certification Authority root or AAA Certificate Services root (depending on what the server sends). Click the links for additional configuration information.

  • Java applications that do not use the default truststore at $JAVA_HOME/lib/security/cacerts

Reconfigure the Java application to use the default truststore that is included with the JDK/JRE for client SSL/TLS connections. See vendor documentation for more information.

  • Lightweight Directory Access Protocol over SSL (LDAPS) clients

See vendor documentation for more information.

ldap.berkeley.edu is currently sending Trust Chain B. Please reconfigure your OpenLDAP clients to use the operating system managed truststore if you are experiencing query failures.

  • Simple Mail Transfer Protocol Secure (SMTPS) and SMTP with STARTTLS
    • Postfix
      Set smtp_tls_CAfile to the operating system managed truststore (e.g. for RHEL7 /etc/pki/tls/certs/ca-bundle.crt)

    • Sendmail
      Set confCACERT_PATH to the operating system managed truststore (e.g. for RHEL7 /etc/pki/tls/certs)

Downloads:

Please download CA certificates from trusted sources: For the root certificate download the "SHA-2 Root: USERTrust RSA Certification Authority" (Expires Jan 2038) from SectigoFor the intermediate certificate download the "InCommon RSA Server CA [PEM]" (Expires October 5, 2024) from Internet2

An example using only the InCommon RSA, typical for general purpose web servers would look like this, saved as ca_bundle.crt:

-----BEGIN CERTIFICATE-----
MIIF+TCCA+GgAwIBAgIQRyDQ+oVGGn4XoWQCkYRjdDANBgkqhkiG9w0BAQwFADCB
iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl
cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV
BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTQx
MDA2MDAwMDAwWhcNMjQxMDA1MjM1OTU5WjB2MQswCQYDVQQGEwJVUzELMAkGA1UE
CBMCTUkxEjAQBgNVBAcTCUFubiBBcmJvcjESMBAGA1UEChMJSW50ZXJuZXQyMREw
DwYDVQQLEwhJbkNvbW1vbjEfMB0GA1UEAxMWSW5Db21tb24gUlNBIFNlcnZlciBD
QTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJwb8bsvf2MYFVFRVA+e
xU5NEFj6MJsXKZDmMwysE1N8VJG06thum4ltuzM+j9INpun5uukNDBqeso7JcC7v
HgV9lestjaKpTbOc5/MZNrun8XzmCB5hJ0R6lvSoNNviQsil2zfVtefkQnI/tBPP
iwckRR6MkYNGuQmm/BijBgLsNI0yZpUn6uGX6Ns1oytW61fo8BBZ321wDGZq0GTl
qKOYMa0dYtX6kuOaQ80tNfvZnjNbRX3EhigsZhLI2w8ZMA0/6fDqSl5AB8f2IHpT
eIFken5FahZv9JNYyWL7KSd9oX8hzudPR9aKVuDjZvjs3YncJowZaDuNi+L7RyML
fzcCAwEAAaOCAW4wggFqMB8GA1UdIwQYMBaAFFN5v1qqK0rPVIDh2JvAnfKyA2bL
MB0GA1UdDgQWBBQeBaN3j2yW4luHS6a0hqxxAAznODAOBgNVHQ8BAf8EBAMCAYYw
EgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUH
AwIwGwYDVR0gBBQwEjAGBgRVHSAAMAgGBmeBDAECAjBQBgNVHR8ESTBHMEWgQ6BB
hj9odHRwOi8vY3JsLnVzZXJ0cnVzdC5jb20vVVNFUlRydXN0UlNBQ2VydGlmaWNh
dGlvbkF1dGhvcml0eS5jcmwwdgYIKwYBBQUHAQEEajBoMD8GCCsGAQUFBzAChjNo
dHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vVVNFUlRydXN0UlNBQWRkVHJ1c3RDQS5j
cnQwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnVzZXJ0cnVzdC5jb20wDQYJKoZI
hvcNAQEMBQADggIBAC0RBjjW29dYaK+qOGcXjeIT16MUJNkGE+vrkS/fT2ctyNMU
11ZlUp5uH5gIjppIG8GLWZqjV5vbhvhZQPwZsHURKsISNrqOcooGTie3jVgU0W+0
+Wj8mN2knCVANt69F2YrA394gbGAdJ5fOrQmL2pIhDY0jqco74fzYefbZ/VS29fR
5jBxu4uj1P+5ZImem4Gbj1e4ZEzVBhmO55GFfBjRidj26h1oFBHZ7heDH1Bjzw72
hipu47Gkyfr2NEx3KoCGMLCj3Btx7ASn5Ji8FoU+hCazwOU1VX55mKPU1I2250Lo
RCASN18JyfsD5PVldJbtyrmz9gn/TKbRXTr80U2q5JhyvjhLf4lOJo/UzL5WCXED
Smyj4jWG3R7Z8TED9xNNCxGBMXnMete+3PvzdhssvbORDwBZByogQ9xL2LUZFI/i
eoQp0UM/L8zfP527vWjEzuDN5xwxMnhi+vCToh7J159o5ah29mP+aJnvujbXEnGa
nrNxHzu+AGOePV8hwrGGG7hOIcPDQwkuYwzN/xT29iLp/cqf9ZhEtkGcQcIImH3b
oJ8ifsCnSbu0GB9L06Yqh7lcyvKDTEADslIaeSEINxhO2Y1fmcYFX/Fqrrp1WnhH
OjplXuXE0OPa0utaKC25Aplgom88L2Z8mEWcyfoB7zKOfD759AN7JKZWCYwk
-----END CERTIFICATE-----

full chain example CA bundle in this case would look like this, saved as ca_bundle.crt:

-----BEGIN CERTIFICATE-----
MIIF3jCCA8agAwIBAgIQAf1tMPyjylGoG7xkDjUDLTANBgkqhkiG9w0BAQwFADCB
iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl
cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV
BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTAw
MjAxMDAwMDAwWhcNMzgwMTE4MjM1OTU5WjCBiDELMAkGA1UEBhMCVVMxEzARBgNV
BAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0plcnNleSBDaXR5MR4wHAYDVQQKExVU
aGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNVBAMTJVVTRVJUcnVzdCBSU0EgQ2Vy
dGlmaWNhdGlvbiBBdXRob3JpdHkwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK
AoICAQCAEmUXNg7D2wiz0KxXDXbtzSfTTK1Qg2HiqiBNCS1kCdzOiZ/MPans9s/B
3PHTsdZ7NygRK0faOca8Ohm0X6a9fZ2jY0K2dvKpOyuR+OJv0OwWIJAJPuLodMkY
tJHUYmTbf6MG8YgYapAiPLz+E/CHFHv25B+O1ORRxhFnRghRy4YUVD+8M/5+bJz/
Fp0YvVGONaanZshyZ9shZrHUm3gDwFA66Mzw3LyeTP6vBZY1H1dat//O+T23LLb2
VN3I5xI6Ta5MirdcmrS3ID3KfyI0rn47aGYBROcBTkZTmzNg95S+UzeQc0PzMsNT
79uq/nROacdrjGCT3sTHDN/hMq7MkztReJVni+49Vv4M0GkPGw/zJSZrM233bkf6
c0Plfg6lZrEpfDKEY1WJxA3Bk1QwGROs0303p+tdOmw1XNtB1xLaqUkL39iAigmT
Yo61Zs8liM2EuLE/pDkP2QKe6xJMlXzzawWpXhaDzLhn4ugTncxbgtNMs+1b/97l
c6wjOy0AvzVVdAlJ2ElYGn+SNuZRkg7zJn0cTRe8yexDJtC/QV9AqURE9JnnV4ee
UB9XVKg+/XRjL7FQZQnmWEIuQxpMtPAlR1n6BB6T1CZGSlCBst6+eLf8ZxXhyVeE
Hg9j1uliutZfVS7qXMYoCAQlObgOK6nyTJccBz8NUvXt7y+CDwIDAQABo0IwQDAd
BgNVHQ4EFgQUU3m/WqorSs9UgOHYm8Cd8rIDZsswDgYDVR0PAQH/BAQDAgEGMA8G
A1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQEMBQADggIBAFzUfA3P9wF9QZllDHPF
Up/L+M+ZBn8b2kMVn54CVVeWFPFSPCeHlCjtHzoBN6J2/FNQwISbxmtOuowhT6KO
VWKR82kV2LyI48SqC/3vqOlLVSoGIG1VeCkZ7l8wXEskEVX/JJpuXior7gtNn3/3
ATiUFJVDBwn7YKnuHKsSjKCaXqeYalltiz8I+8jRRa8YFWSQEg9zKC7F4iRO/Fjs
8PRF/iKz6y+O0tlFYQXBl2+odnKPi4w2r78NBc5xjeambx9spnFixdjQg3IM8WcR
iQycE0xyNN+81XHfqnHd4blsjDwSXWXavVcStkNr/+XeTWYRUc+ZruwXtuhxkYze
Sf7dNXGiFSeUHM9h4ya7b6NnJSFd5t0dCy5oGzuCr+yDZ4XUmFF0sbmZgIn/f3gZ
XHlKYC6SQK5MNyosycdiyA5d9zZbyuAlJQG03RoHnHcAP9Dc1ew91Pq7P8yF1m9/
qS3fuQL39ZeatTXaw2ewh0qpKJ4jjv9cJ2vhsE/zB+4ALtRZh8tSQZXq9EfX7mRB
VXyNWQKV3WKdwrnuWih0hKWbt5DHDAff9Yk2dDLWKMGwsAvgnEzDHNb842m1R0aB
L6KCq9NjRHDEjf8tM7qtj3u1cIiuPhnPQCjY/MiQu12ZIvVS5ljFH4gxQ+6IHdfG
jjxDah2nGN59PRbxYvnKkKj9
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIF+TCCA+GgAwIBAgIQRyDQ+oVGGn4XoWQCkYRjdDANBgkqhkiG9w0BAQwFADCB
iDELMAkGA1UEBhMCVVMxEzARBgNVBAgTCk5ldyBKZXJzZXkxFDASBgNVBAcTC0pl
cnNleSBDaXR5MR4wHAYDVQQKExVUaGUgVVNFUlRSVVNUIE5ldHdvcmsxLjAsBgNV
BAMTJVVTRVJUcnVzdCBSU0EgQ2VydGlmaWNhdGlvbiBBdXRob3JpdHkwHhcNMTQx
MDA2MDAwMDAwWhcNMjQxMDA1MjM1OTU5WjB2MQswCQYDVQQGEwJVUzELMAkGA1UE
CBMCTUkxEjAQBgNVBAcTCUFubiBBcmJvcjESMBAGA1UEChMJSW50ZXJuZXQyMREw
DwYDVQQLEwhJbkNvbW1vbjEfMB0GA1UEAxMWSW5Db21tb24gUlNBIFNlcnZlciBD
QTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJwb8bsvf2MYFVFRVA+e
xU5NEFj6MJsXKZDmMwysE1N8VJG06thum4ltuzM+j9INpun5uukNDBqeso7JcC7v
HgV9lestjaKpTbOc5/MZNrun8XzmCB5hJ0R6lvSoNNviQsil2zfVtefkQnI/tBPP
iwckRR6MkYNGuQmm/BijBgLsNI0yZpUn6uGX6Ns1oytW61fo8BBZ321wDGZq0GTl
qKOYMa0dYtX6kuOaQ80tNfvZnjNbRX3EhigsZhLI2w8ZMA0/6fDqSl5AB8f2IHpT
eIFken5FahZv9JNYyWL7KSd9oX8hzudPR9aKVuDjZvjs3YncJowZaDuNi+L7RyML
fzcCAwEAAaOCAW4wggFqMB8GA1UdIwQYMBaAFFN5v1qqK0rPVIDh2JvAnfKyA2bL
MB0GA1UdDgQWBBQeBaN3j2yW4luHS6a0hqxxAAznODAOBgNVHQ8BAf8EBAMCAYYw
EgYDVR0TAQH/BAgwBgEB/wIBADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUH
AwIwGwYDVR0gBBQwEjAGBgRVHSAAMAgGBmeBDAECAjBQBgNVHR8ESTBHMEWgQ6BB
hj9odHRwOi8vY3JsLnVzZXJ0cnVzdC5jb20vVVNFUlRydXN0UlNBQ2VydGlmaWNh
dGlvbkF1dGhvcml0eS5jcmwwdgYIKwYBBQUHAQEEajBoMD8GCCsGAQUFBzAChjNo
dHRwOi8vY3J0LnVzZXJ0cnVzdC5jb20vVVNFUlRydXN0UlNBQWRkVHJ1c3RDQS5j
cnQwJQYIKwYBBQUHMAGGGWh0dHA6Ly9vY3NwLnVzZXJ0cnVzdC5jb20wDQYJKoZI
hvcNAQEMBQADggIBAC0RBjjW29dYaK+qOGcXjeIT16MUJNkGE+vrkS/fT2ctyNMU
11ZlUp5uH5gIjppIG8GLWZqjV5vbhvhZQPwZsHURKsISNrqOcooGTie3jVgU0W+0
+Wj8mN2knCVANt69F2YrA394gbGAdJ5fOrQmL2pIhDY0jqco74fzYefbZ/VS29fR
5jBxu4uj1P+5ZImem4Gbj1e4ZEzVBhmO55GFfBjRidj26h1oFBHZ7heDH1Bjzw72
hipu47Gkyfr2NEx3KoCGMLCj3Btx7ASn5Ji8FoU+hCazwOU1VX55mKPU1I2250Lo
RCASN18JyfsD5PVldJbtyrmz9gn/TKbRXTr80U2q5JhyvjhLf4lOJo/UzL5WCXED
Smyj4jWG3R7Z8TED9xNNCxGBMXnMete+3PvzdhssvbORDwBZByogQ9xL2LUZFI/i
eoQp0UM/L8zfP527vWjEzuDN5xwxMnhi+vCToh7J159o5ah29mP+aJnvujbXEnGa
nrNxHzu+AGOePV8hwrGGG7hOIcPDQwkuYwzN/xT29iLp/cqf9ZhEtkGcQcIImH3b
oJ8ifsCnSbu0GB9L06Yqh7lcyvKDTEADslIaeSEINxhO2Y1fmcYFX/Fqrrp1WnhH
OjplXuXE0OPa0utaKC25Aplgom88L2Z8mEWcyfoB7zKOfD759AN7JKZWCYwk
-----END CERTIFICATE-----

To validate the contents of your custom CA bundle:

openssl crl2pkcs7 -nocrl -certfile ca_bundle.crt | openssl pkcs7 -print_certs -text -noout

To validate the server certificate agasint the CA bundle:

openssl verify -CAfile ca_bundle.crt my_host_cert.crt