RHOCP 4 AWS cloud-credentials access key is expired.

Environment

• Red Hat OpenShift Container Platform (RHOCP) 4

Issue

• The AWS service account used when installing the cluster is expired how can the cloud credentials be updated?
• OpenShift cluster operator, cloud-credential, is in a degraded state.
• cloud-credential-operator error log error syncing credentials: error determining whether a credentials update is needed: unable to read info for username.
• The oc adm upgrade command reports the error: the cluster operator cloud-credential is degraded.
• How can the names of IAM service accounts be changed for Naming Compliance Conventions policies?

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

• If the cluster is already installed, please follow the outlined steps.

  1. Make sure the aws account has the This content is not included.required aws permissions.

  2. Create or obtain an Content from docs.aws.amazon.com is not included.access key via AWS.

  3. Update aws-creds secret in kube-system namespace, below is an example using the default profile configured ~/.aws/credentials.

     $ PROFILE=default 
     
     $ aws --profile ${PROFILE} sts get-caller-identity # Quick credential test
     
     $ AWS_ACCESS_KEY_ID=$(aws configure get ${PROFILE}.aws_access_key_id) 
     $ AWS_SECRET_ACCESS_KEY=$(aws configure get ${PROFILE}.aws_secret_access_key)
     
     $ oc create secret generic aws-creds -n kube-system --from-literal="aws_access_key_id=${AWS_ACCESS_KEY_ID}" --from-literal="aws_secret_access_key=${AWS_SECRET_ACCESS_KEY}" --dry-run -o yaml | oc replace -f -   
     

• To set the IAM account name for an as-yet-uninstalled cluster, refer to the This content is not included.documentation.

Root Cause

• When the AWS service account in use reaches its expiration, it necessitates replacement. The same procedure can be employed to replace the IAM account for various other reasons, such as when there is a need to update credentials or address naming compliance issues.

Diagnostic Steps

1. OpenShift cluster operator cloud-credential is in a degraded state.

$ oc get co cloud-credential
NAME               VERSION   AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
cloud-credential   4.14.45   True        True          True       1d13h 

2. cloud-credential-operator log the error message syncing credentials: error determining whether a credentials update is needed: unable to read info for username.

  $ oc logs -n openshift-cloud-credential-operator $(oc get pod -n openshift-cloud-credential-operator -o name)

  ...
  time="2023-12-16T08:31:46Z" level=error msg="error syncing credentials: error determining whether a credentials update is needed: unable to read info for username 

3. IAM access key is expired. It's possible to check for it using the AWS Management Console or the AWS Command Line Interface (CLI).

   Here are the steps for both methods:

   • Using AWS CLI:
     Ensure to have the AWS CLI installed and configured with the necessary permissions.
     Run the following command replacing the strings default and <IAM-USERNAME> with the actual aws profile and IAM username:

   $ aws --profile default iam list-access-keys --user-name <IAM-USERNAME>
   

   Review the output:
   
   Look for the "Status" field. If it is `Inactive`, it means the access key is no longer valid.
   

   • Using AWS Management Console:
     1. Open the AWS Management Console.
     2. Navigate to IAM: in the top search bar, type IAM and click on the IAM service.
     3. Select the User: in the left navigation pane, choose Users.
     4. Choose the User: find and click on the user to check the access key.
     5. Review Access Keys: in the Security credentials tab, find the Access keys section. The Status field can be found here. If it is Inactive, it means the access key is no longer valid.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.