error: x509 certificate signed by unknown authority when logging in OpenShift 4 using the installation kubeconfig file

Environment

• Red Hat OpenShift Container Platform
  • 4

Issue

• When logging as a user using the kubeconfig from the installation (auth/kubeconfig) an error is shown: error: x509 certificate signed by unknown authority error..
• Below warning is seen while trying to logging in to the cluster.
  WARNING: Using insecure TLS client config. Setting this option is not supported!
• Using a separate kubeconfig allows logging but then the new kubeconfig doesnt contain the information for passwordless login as system:admin user.

Resolution

• Extract the CA of the ingress and save it to a file:
  note: This does not contain the right API certificate on latest versions, please continue with the openssl method instead

  $ oc rsh -n openshift-authentication <oauth-openshift-pod> cat /run/secrets/kubernetes.io/serviceaccount/ca.crt > ingress-ca.crt
  

• If you have no access to the oc command, you can also retrieve the Certificate Authority certificates using openssl:

  $ URI=oauth-openshift.<ingress_cluster_domain>
  $ PORT=443
  $ echo | openssl s_client -showcerts -connect ${URI}:${PORT} 2>/dev/null | awk '/-----BEGIN CERTIFICATE-----/,/-----END CERTIFICATE-----/' > ingress-ca.crt
  

• Then login using that specific file as certificate file:

  $ oc login -u username -p password https://api.example.local:6443 --certificate-authority=ingress-ca.crt
  

  • If it works properly, then the CA can be added as trusted in the system bundle as described in this KCS.

Root Cause

• The CA which issues the API URL (kube-apiserver-localhost-signer) is included in the kubeconfig
• The CA which issues the Authentication endpoint (ingress-ca) is not included in the kubeconfig and therefore must be trusted in the system and/or provided with --certificate-authority=parameter in the login command
• Doing a login operation falls in the authentication workflow and the ingress certificate must be considered in the connection flow
• Using --insecure-skip-tls-verify=true doesn't help because it will affect only to the URL provided in the login (api.ocp4.example.local:6443) and not to the authentication URL (oauth-openshift.apps.ocp4.example.local).

Diagnostic Steps

$ export KUBECONFIG=/path/to/installation/auth/kubeconfig
$ oc whoami
admin
$ oc login -u username -p password https://api.ocp4.example.local:6443 --loglevel 5
...
I1015 10:39:54.831593   49324 round_trippers.go:438] HEAD https://oauth-openshift.apps.ocp4.example.local  in 35 milliseconds
I1015 10:39:54.831619   49324 round_trippers.go:444] Response Headers:
I1015 10:39:54.831626   49324 request_token.go:439] falling back to kubeconfig CA due to possible x509 error: x509: certificate signed by unknown authority
...

Other considerations

• The method described above adds the cluster in the kubeconfig file as a new one but it collides with the existing one (both are for the same API url), therefore the passwordless login as system:admin must be done by switching the context (and not with oc login):

$ oc whoami
username
$ oc config use-context admin
Switched to context "admin".
$ oc whoami
system:admin

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.