Identity Issues with 3.4.0 GA

Solution Verified - Updated

Environment

  • OpenShift Container Platform 3.4.0

Issue

  • Upgrading to 3.4.0 causes existing user logins to stop working.
  • User identities created after a new 3.4.0 install are stored in the wrong location.

Resolution

Initial installs/upgrades should not be effected by this issue if you are pulling in packages shipped/provided with Errata: RHBA-2017:0187, see Release Notes on the 3.4.0.40 Bug Fix Update and 3.4.0 Known Issue for more information.

If you installed, or updated a cluster prior to when this errata shipped you will need to run a tool, to correct your cluster, however this tool has not yet been released!

With Errata RHBA-2017:0267, the product provides a script (found in /usr/share/openshift/migration/fix-3.4-paths.sh) to help fix broken clusters.

Please update to Errata RHBA-2017:0267 and run the denoted, script installed the masters (the script needs to be run from 1 master only once - provided there are no errors).

To run this script you first want to dry-run the script to see if changes are needed: 

# ./fix-3.4-paths.sh https://ETCD_IP:ETCD_PORT

To apply changes (needed to correct clusters in a bad state) run the following: 

# ./fix-3.4-paths.sh -a https://ETCD_IP:ETCD_PORT

If you have questions or need help running the script, please This content is not included.open a support case

Root Cause

Previously, upgrading from OpenShift Container Platform 3.3 to 3.4 caused all user identities to disappear, though they were still present in etcd, and OAuth-based users could no longer log in. New 3.4 installations were also affected. This was caused by an unintentional change in the etcd prefix for user identities; egressnetworkpolicies were similarly affected.

Fixes for this issue plan to restore the previous etcd prefix for user identities and egressnetworkpolicies, and as a result users can log in again successfully to clusters that were upgraded from 3.3 to 3.4.

Diagnostic Steps

  • Using Knowledge Solution 2542841 you can use etcdctl to see if identities reside in the correct locations: 

    # etcdctl --endpoints https://${etcd_endpoint} --cert-file ${cert_file} --key-file ${key_file} --ca-file ${ca_file} ls /openshift.io/identities
# etcdctl --endpoints https://${etcd_endpoint} --cert-file ${cert_file} --key-file ${key_file} --ca-file ${ca_file} ls /openshift.io/useridentities
    • If users are seen in the identities section, you have a 3.4 system that does not have its users in the right location. As a result you will need to correct the cluster.

  • Using Knowledge Solution 2542841 you can use etcdctl to see if your egressnetwork information is also in the wrong location. 

    # etcdctl --endpoints https://${etcd_endpoint} --cert-file ${cert_file} --key-file ${key_file} --ca-file ${ca_file} ls /openshift.io/registry/ | grep egressnetwork
    • You either want to see no output from this command, or egressnetworkpolicies shown in the output. If you see, egressnetworkpolicy you will need to correct the cluster.
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.