OpenShift 4.18 upgrade blocked by x509: invalid certificate policies

Environment

• Red Hat Openshift Container Platform (RHOCP)
  • 4.18
• OpenShift OVN-Kubernetes
• Golang 1.24

Issue

• The OpenShift upgrade stalls during OVN-Kubernetes Operator reconciliation.

• The operator fails to roll out the target release and remains pinned to the previously deployed version.

• Control plane components enter a degraded state, with partial loss of functionality.

• The network-node-identity daemonset fails to initialize due to a CA validation error (x509: invalid certificate policies), preventing successful webhook registration and blocking operator progression:

  Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: error creating pool from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: x509: invalid certificate policies
  

Resolution

Note: As a preventive measure, it is strongly recommended to validate and, if needed, update the API CA certificate prior to initiating the upgrade to OpenShift 4.18 or newer releases.

To prevent this upgrade blockage, the API custom CA certificate must be replaced with a corrected version that complies with RFC 5280 and does not contain invalid or malformed Certificate Policies. If needed, coordinate with the certification authority to regenerate the CA certificate with a valid policy configuration (or without Certificate Policies if they are not strictly required).

For replacing the API certificate, please refer to adding API server certificates .

After the corrected API CA is consumed by the network-node-identity daemonset, the OVN Operator is able to initialize successfully and the upgrade proceeds as expected.

Root Cause

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

The upgrade became blocked because the cluster’s API trust CA (ca.crt) contained an invalid/malformed X.509 Certificate Policies extension (not an expiry issue). When the network-node-identity daemonset (built with Golang 1.24 or newer) started, it attempted to build a root CA pool from the ServiceAccount CA path /var/run/secrets/kubernetes.io/serviceaccount/ca.crt and failed with x509: invalid certificate policies.
That failure prevented network-node-identity from initializing (and its webhook/service behavior), which in turn blocks OVN Operator reconciliation during the upgrade. The operator remains pinned to the previous deployed version, leaving the upgrade stuck and the control plane in a degraded state.

As explained in component returns x509: invalid certificate policies in RHOCP 4, Golang 1.24 introduced a verification to enforce a rule from Content from datatracker.ietf.org is not included.RFC 5280. Starting with the OVN-Kubernetes version for OpenShift 4.18, Golang 1.24 is used.

Diagnostic Steps

• Check the error at the daemonset pods:

  $ oc get pods -o name -n openshift-network-node-identity | sed 's#pod/#oc logs -c webhook #g' | sh  | grep "invalid certificate policies"
  2026-02-05T02:07:11.511866135Z E0205 02:07:11.511847       1 config.go:562] Expected to load root CA config from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt, but got err: error creating pool from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: x509: invalid certificate policies
  

• Check the Certificate Policies:

  • Getting the API certificate from the URL (store the individual certificates in different files and verify if the certificates has x509 policies):

    $ true | openssl s_client -showcerts -connect <api-int.domain:6443>
    

    $ openssl x509 -in <file> -ext certificatePolicies -noout
    X509v3 Certificate Policies:
    [...]
    

  • Via the API certificate:

      $ oc get apiserver cluster -o json | jq -r '.spec.servingCerts.namedCertificates[].servingCertificate.name'
      [custom-api-cert_name]
      
      $ python3 -c "import sys, base64; print(base64.b64decode(sys.argv[1]).decode('utf8'))" $(oc get secret [custom-api-cert_name] -n openshift-config -o json | jq -r '.data."tls.crt"') | openssl x509 -ext certificatePolicies -noout
    X509v3 Certificate Policies:
    [...]
    

    • Verify:
      • Missing policyIdentifier.
      • Duplicate or malformed Certificate Policies OID.
      • Broken or non-UTF8 CPS qualifiers.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.