Cluster operator authentication is degraded with certificate error in logs

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform
    • 4.X

Issue

  • RouterCertsDegraded: secret/v4-0-config-system-router-certs.spec.data[apps.example.com] -n openshift-authentication: certificate could not validate route hostname oauth-openshift.apps.example.com: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "MyCorp Issuing CA")

Resolution

The bundle set in the proxy/cluster/trustedCA should include ALL trusted certificates.

Please refer to the documentation Enabling the cluster-wide proxy which describe how to set create the bundle in a configmap.

Root Cause

Red Hat Openshift Container Platform (RHOCP) uses the proxy/cluster/trustedCA field to store the trusted certificates that it should rely on.
This bundle is merged to the default node trusted bundle to ensure that all desired certificate can be trusted.

So when using an intermediate certificate to trust domains, this intermediate certificate should also be added to the trust bundle in RHOCP, in addition to the related Root CA.

Chain of Trust:
Domain Certificate => Intermediate CA => Root CA
The arrows point to whom is trusting the requester.

When using an intermediate certificate, it's not possible to trust a domain directly from the Root CA: Domain Certificate => Root CA This will not work if the Domain Certificate has been created from an intermediate certificate

If the intermediate certificate MyCorp Issuing CA is used to trust the domain *.apps.example.com.
This intermediate certificate MyCorp Issuing CA will be trusted by the company Root CA MyCorp root CA

Looking at the error:

RouterCertsDegraded: secret/v4-0-config-system-router-certs.spec.data[apps.example.com] -n openshift-authentication: certificate could not validate route hostname oauth-openshift.apps.example.com: **x509: certificate signed by unknown authority** (possibly because of "**crypto/rsa: verification error**" while trying to verify candidate authority certificate "**MyCorp Issuing CA**")

The message describes that the domain apps.example.com has been signed by unknown authority, as there is an error while trying to verify candidate authority certificate "MyCorp Issuing CA"

This means that the intermediate certificate has not been added to the trustedCA bundle.

Diagnostic Steps

Check the Operator error

$ oc get co authentication -o json | jq -r '.status.conditions[] | select(.type == "Degraded")'
{
  "lastTransitionTime": "2021-06-10T18:40:36Z",
  "message": "RouterCertsDegraded: secret/v4-0-config-system-router-certs.spec.data[apps.exmaple.com] -n openshift-authentication: certificate could not validate route hostname oauth-openshift.apps.exmaple.com: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"MyCorp Issuing CA\")",
  "reason": "RouterCerts_InvalidServerCertRouterCerts",
  "status": "True",
  "type": "Degraded"
}

Check if the console uses an intermediate certificate

$ curl -vvv https://oauth-openshift.apps.exmaple.com/healthz
[…]
$ openssl s_client -showcerts -connect oauth-openshift.apps.example.com:443
CONNECTED(00000005)
depth=2 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Global Root CA
verify return:1
depth=1 C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1
verify return:1
depth=0 C = US, ST = California, L = Los Angeles, O = Internet Corporation for Assigned Names and Numbers, CN = oauth-openshift.apps.example.org
verify return:1
---
Certificate chain
 0 s:/C=US/ST=California/L=Los Angeles/O=MyOrg/CN=oauth-openshift.apps.example.org
   i:/C=US/O=MyOrg/CN=MyCorp Issuing CA
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
 1 s:/C=US/O=MyOrg/CN=MyCorp Issuing CA
   i:/C=US/O=MyOrg/CN=MyCorp root CA
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
 2 s:/C=US/O=MyOrg/CN=MyCorp root CA
   i:/C=US/O=MyOrg/CN=MyCorp root CA
-----BEGIN CERTIFICATE-----
[...]
-----END CERTIFICATE-----
---
Server certificate
subject=/C=US/ST=California/L=Los Angeles/O=MyOrg/CN=oauth-openshift.apps.example.org
issuer=/C=US/O=MyOrg/CN=MyCorp Issuing CA

The openssl command will return the certificate chain used to access the URI.
If the chain used more than 2 certificates, this means that you should have all intermediate certificates added to the trusted bundle as same as the root CA.

Check the proxy/cluster/trustedCA

Checking the default used certificate in the ingress-operator

$ oc get proxy/cluster -o json | jq -r .spec
{
  "trustedCA": {
    "name": "custom-ca-cm"
  }
}

Check which certificates are stored in the bundle

If the Root CA is stored in a configmap, check the content of the bundle

$ oc get configmap custom-ca-cm -n openshift-config -o json  | jq -r '.data."ca-bundle.crt"' | openssl crl2pkcs7 -nocrl -certfile  /dev/stdin |openssl pkcs7 -print_certs -text -noout | grep -A4 Issuer:

This will display all certificates included in the bundle.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.