RouterCertsDegraded with x509 certificate signed by unknown authority in OpenShift 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Ingress custom certificate

Issue

  • After replacing the default ingress certificate with a custom one, the authentication Cluster Operator becomes degraded with the error below:

    RouterCertsDegraded: secret/v4-0-config-system-router-certs.spec.data[apps.example.com] -n openshift-authentication: certificate could not validate route hostname oauth-openshift.apps.example.com: x509: certificate signed by unknown authority

    Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"<UID>",                                                             
APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded changed from False to True ("RouterCertsDegraded: secret/v4-0-config-system-router-certs.spec.data[apps.example.com] -n openshift-authentication: certificate could not validate route hostname oauth-openshift.apps.example.com: x509: certificate signed by unknown authority")
[...]

Resolution

Any of the certificate requirements are not met, and create a new certificate that meets the requirements is required.

Note: if it is not possible to create a new certificate at this moment (for example, if a different department is in charge of that), it is possible to temporarily revert to the default ingress certificate generated by the Ingress operator: How to revert to the default ingress certificate after replacing it with a custom certificate in Red Hat OpenShift Container Plattform 4?.

Use cert-manager Operator for Red Hat OpenShift

Starting from cert-manager Operator for Red Hat OpenShift 1.13.0 (supported in OpenShift 4.13 and newer), it is now possible to manage certificates for the Ingress Controller: Creating certificates for the Ingress Controller.

Re-create the ingress certificate manually


Follow the steps from [Replacing the default ingress certificate](https://docs.openshift.com/container-platform/4.14/security/certificates/replacing-default-ingress-certificate.html), and in the step "*Create a secret that contains the wildcard certificate chain and key*" check that:

  • The root CA needs to be a trusted root CA.

  • The certificate must include the subjectAltName extension showing *.apps.[clustername].[domain].

  • The certificate must include the wildcard certificate and the root CA (with the entire chain if there are intermediates: How to check the order of certificates for certificate chain):

    wildcard certificate (*.apps.[cluster_name].[domain])
intermediate CA (if available)
root CA

  • The certificate doesn't contain as end of line Control+M or other characters.

  • The custom-ca configmap in openshift-config namespace includes the root CA certificate used to sign the wildcard certificate.

Root Cause

The root CA from the new custom certificate provided is not recognized by the system.

Diagnostic Steps

The authentication Cluster Operator is in a Degraded state with a RouterCertsDegraded message:

$ oc get co authentication
NAME             VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE
authentication   4.2.0     True        False         True       11d

$ oc get co authentication -o yaml
[...]
      RouterCertsDegraded: secret/v4-0-config-system-router-certs.spec.data[apps.example.com] -n openshift-authentication: certificate could not validate route hostname oauth-openshift.apps.example.com: x509: certificate signed by unknown authority
[...]

From an oauth-openshift pod there are logs like:

$ oc logs <oauth-openshift pod> -n openshift-authentication
[...]
Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-authentication-operator", Name:"authentication-operator", UID:"<UID>",                                                             
APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'OperatorStatusChanged' Status for clusteroperator/authentication changed: Degraded changed from False to True ("RouterCertsDegraded: secret/v4-0-config-system-router-certs.spec.data[apps.example.com] -n openshift-authentication: certificate could not validate route hostname oauth-openshift.apps.example.com: x509: certificate signed by unknown authority")
[...]

Check that the custom certificate configured in the ingresscontroller contains the CA chain (refer to How to check the order of certificates for certificate chain):

$ oc get ingresscontroller.operator -n openshift-ingress-operator -o yaml | grep -A1 "defaultCertificate"
    defaultCertificate:
      name: [certificate_secret_name]

$ oc extract secret/[certificate_secret_name] -n openshift-ingress --to /tmp/
/tmp/tls.crt
/tmp/tls.key
$ openssl crl2pkcs7 -nocrl -certfile /tmp/tls.crt | openssl pkcs7 -print_certs -noout

Check that the Root CA is included in the configmap configured as trustedCA in the proxy resource:

$ oc get proxy cluster -o yaml | grep -A1 "trustedCA"
  trustedCA:
    name: [custom-ca_name]


$ oc extract cm/[custom-ca_name] -n openshift-config --to /tmp/
/tmp/ca-bundle.crt
$ openssl crl2pkcs7 -nocrl -certfile /tmp/ca-bundle.crt | openssl pkcs7 -print_certs -noout
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.