Consolidated Troubleshooting Article for OpenShift Container Platform (OCP) 4: Certificates

Updated

Index

Certificates

SSL/TLS certificate is a digital object that allows systems to verify the identity & subsequently establish an encrypted network connection to another system using the Secure Sockets Layer/Transport Layer Security (SSL/TLS) protocol.
  • CA: Certificate Authority. It’s an entity that issues certificates on its name. Users send Certificate Signing Requests (CSRs) created using their own private key and once signed they get a public certificate. Private key is never expected to leave the user’s private storage.
  • CA Bundle: CA bundle is a file that contains CA certificates.
  • Certificate Chains: End-entity certificate along with the chain of CAs where each CA certificate is signed by a "higher-in-the-chain" CA.
  • Certificate: A public key + attributes describing how the entity owning its matching private key is supposed to use the private key.
  • Signing Certificate: A certificate that is used to sign other certificates.
  • Serving Certificate: A certificate that is used to secure communications implementing TLS (used by K8s services).
  • Client Certificate: A certificate that is used to secure communications using mTLS (used by K8s workloads).

This page is not included, but the link has been rewritten to point to the nearest parent document.Types of Certificates

CertificatesManagement
Control plane certificatesRotated automatically and managed by the system
Node certificatesRotated automatically and managed by the system
Custom certificates for the API serverUser-provided certificates are managed by the user
Proxy certificatesUser-provided certificates are managed by the user
Service CA certificatesRotated automatically and managed by the system
Bootstrap certificatesRotated automatically and managed by the system
etcd certificatesRotated automatically and managed by the system
OpenShift Lifecycle Manager(OLM) certificatesRotated automatically and managed by the system
Aggregated API client certificatesRotated automatically and managed by the system
Custom certificates for default ingressUser-provided certificates are managed by the user
Ingress certificatesIngress certificates are managed by the user

Configuration

Configuring This page is not included, but the link has been rewritten to point to the nearest parent document.Custom Ingress Certificate

- Create a config map that includes only the root CA certificate used to sign the wildcard certificate:

    ```
    $ oc create configmap custom-ca \
    --from-file=ca-bundle.crt=</path/to/example-ca.crt> \
    -n openshift-config
    ```


- Update the `cluster-wide` proxy configuration with the newly created config map:

    ```
    $ oc patch proxy/cluster \
    --type=merge \
    --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'
    ```


- Create a secret that contains the `wildcard certificate` chain and key:

    ```
    $ oc create secret tls <secret> \
    --cert=</path/to/cert-chain.crt> \
    --key=</path/to/cert.key> \
    -n openshift-ingress
    ```


- Update the Ingress Controller configuration with the newly created secret:

    ```
    $ oc patch ingresscontroller.operator default \
    --type=merge -p \
    '{"spec":{"defaultCertificate": {"name": "<secret>"}}}' \
    -n openshift-ingress-operator
    ```

Configuring This page is not included, but the link has been rewritten to point to the nearest parent document.Custom API-Server Certificates

- Create a secret that contains the certificate chain and private key in the `openshift-config` namespace.

    ```
    $ oc create secret tls <secret> \
    --cert=</path/to/cert-chain.crt> \
    --key=</path/to/cert.key> \
    -n openshift-config
    ```


- Update the `API server` to reference the created secret.

    ```
    $ oc patch apiserver cluster \
    --type=merge -p \
    '{"spec":{"servingCerts": {"namedCertificates":
    [{"names": ["<FQDN>"],
    "servingCertificate": {"name": "<secret>"}}]}}}'
    ```

Configuring Custom certificates on Console Route

- Edit the cluster Ingress configuration:

    ```
    $ oc edit ingress.config.openshift.io cluster
    ```


- Set the custom hostname, serving certificate and key and save the changes-

    ```
    apiVersion: config.openshift.io/v1
    kind: Ingress
    metadata:
      name: cluster
    spec:
      componentRoutes:
        - name: console
          namespace: openshift-console
          hostname: <custom_hostname> 
          servingCertKeyPairSecret:
            name: <secret_name>
    ```

Trust Custom Root CA cluster-wide through Proxy configuration in Red Hat OpenShift Platform 4.x

- Create a config map that includes the root CA certificate used to sign the wildcard certificate:

    ```
    $ oc create configmap custom-ca \
             --from-file=ca-bundle.crt=</path/to/example-ca.crt> \
             -n openshift-config
    ```


- Update the cluster-wide proxy configuration with the newly created config map:

    ```
    $ oc patch proxy/cluster \
             --type=merge \
             --patch='{"spec":{"trustedCA":{"name":"custom-ca"}}}'
    ```

Trust External Registry RootCA for builds and deployments

- Create a ConfigMap in the openshift-config namespace containing the trusted certificates for the registries that use self-signed certificates. For each CA file, ensure the key in the ConfigMap is the hostname of the registry in the hostname[..port] format:

    ```
    $ oc create configmap registry-cas -n openshift-config --from-file=myregistry.corp.com..5000=/etc/docker/certs.d/myregistry.corp.com:5000/ca.crt --from-file=otherregistry.com=/etc/docker/certs.d/otherregistry.com/ca.crt
    ```


- Update the cluster image configuration:

    ```
    $ oc patch image.config.openshift.io/cluster --patch '{"spec":{"additionalTrustedCA":{"name":"registry-cas"}}}' --type=merge
    ```

Configure other components to trust custom Root CA from cluster-wide proxy

- [Injecting Custom Root CA in OLM Operators](https://docs.openshift.com/container-platform/4.14/operators/admin/olm-configuring-proxy-support.html#olm-inject-custom-ca_olm-configuring-proxy-support)
- [Injecting Custom Root CA in Application/Deployment](https://docs.openshift.com/container-platform/4.14/networking/configuring-a-custom-pki.html#certificate-injection-using-operators_configuring-a-custom-pki)
- [Injecting Custom Root CA in RHCOS trustbundle through MachineConfig object](https://docs.openshift.com/container-platform/4.14/security/certificate_types_descriptions/proxy-certificates.html#customization)
- [Injecting Custom Root CA in External DNS Operator](https://docs.openshift.com/container-platform/4.14/networking/external_dns_operator/nw-configuring-cluster-wide-egress-proxy.html)
- [Injecting Custom Root CA in cert-manager Operator](https://docs.openshift.com/container-platform/4.14/security/cert_manager_operator/cert-manager-operator-proxy.html#cert-manager-proxy-support_cert-manager-operator-proxy)
- [Injecting Custom Root CA in AWS Load Balancer Operator](https://docs.openshift.com/container-platform/4.14/networking/aws_load_balancer_operator/configure-egress-proxy-aws-load-balancer-operator.html)

Verifying Certificate Validity

  • Use the following one-liner to list all OpenShift certificate expiry dates:

    $ echo -e "NAMESPACE\tNAME\tEXPIRY" && oc get secrets -A -o go-template='{{range .items}}{{if eq .type "kubernetes.io/tls"}}{{.metadata.namespace}}{{" "}}{{.metadata.name}}{{" "}}{{index .data "tls.crt"}}{{"\n"}}{{end}}{{end}}' | while read namespace name cert; do echo -en "$namespace\t$name\t"; echo $cert | base64 -d | openssl x509 -noout -enddate; done | column -t

  • How to check the internal certificates expiry in OpenShift Container Platform 4?

Certificate Renewal

Note: In recent OpenShift 4 versions, the cluster can automatically recover from expired control plane certificates. However, it could be needed to manually approve the pending CSRs. Refer to the This page is not included, but the link has been rewritten to point to the nearest parent document.documentation for additional information.

Approve pending CSR's

  • Get the list of current CSRs and check the details of a CSR and approve valid node-bootstrapper and kubelet serving CSR's:
$ oc get csr
$ oc describe csr <csr_name> 
$ oc adm certificate approve <csr_name>

Regenerating cluster certificates

How to renew Master node Certificate in OpenShift 4

Renew Etcd certificates

Renew or Recreate Kubelet certificates in OpenShift 4.x

Replacing mirror registry for Red Hat OpenShift SSL/TLS certificates

Frequently used Knowledge-base Articles

SBR
Product(s)
Category
Components
Tags
Article Type