fence_aws, AWS resource agents, and AWS CLI commands time out when IMDSv2 is required

Solution Unverified - Updated 13 Jun 2024

Environment

Red Hat Enterprise Linux 7 (with the High Availability Add-on)
Red Hat Enterprise Linux 8 (with the High Availability Add-on)
Amazon Web Services (AWS) EC2 Instances as Cluster Members

Issue

After Content from docs.aws.amazon.com is not included.disabling IMDSv1 and requiring IMDSv2, AWS resource agents and commands likefence_aws and aws (from awscli) time out unless credentials are specified.

Resolution

Red Hat Enterprise Linux 7

Red Hat Enterprise Linux 8

Red Hat Enterprise Linux 8.1

The issue (bugzilla bug: 2052623) has been resolved with the errata RHBA-2022:1588 with the following package(s): fence-agents-4.2.1-30.el8_1.8 or later for RHEL 8.1.z

Red Hat Enterprise Linux 8.2

Upgrade to fence-agents-aws-4.2.1-41.el8_2.9 or later. This issue was tracked in This content is not included.Bug 2052625 - fence_aws fails to find credentials when only IMDSv2 is enabled [rhel-8.2.0.z].

Red Hat Enterprise Linux 8.3

There are no plans to fix this issue on RHEL 8.3.

Red Hat Enterprise Linux 8.4

Upgrade to fence-agents-aws-4.2.1-65.el8_4.3 or later. This issue was tracked in This content is not included.Bug 2052626 - fence_aws fails to find credentials when only IMDSv2 is enabled [rhel-8.4.0.z].

Red Hat Enterprise Linux 8.5

Upgrade to fence-agents-aws-4.2.1-75.el8_5.2 or later. This issue was tracked in This content is not included.Bug 2052627 - fence_aws fails to find credentials when only IMDSv2 is enabled [rhel-8.5.0.z].

Red Hat Enterprise Linux 8.6

The issue (bugzilla bug: 2048857) has been resolved with the errata RHBA-2022:1757 with the following package(s): fence-agents-4.2.1-89.el8 or later.

Workaround

Other credential mechanisms

Provide credentials in any of Boto3's Content from boto3.amazonaws.com is not included.credential lookup locations. (This likely includes specifying a profile with an IAM role configured, as discussed in the Content from boto3.amazonaws.com is not included.Assume role provider section, though this has not been tested).

Enabling IMDSv1

Alternatively, you can Content from docs.aws.amazon.com is not included.re-enable IMDSv1 until you can install the patch.

Root Cause

Boto is the Content from aws.amazon.com is not included.AWS SDK for Python and consists primarily of the Boto3 and Botocore components. Boto does most of the work in fence_aws and in the aws command from awscli.

Requests require credentials, and Boto3 looks for credentials in a Content from boto3.amazonaws.com is not included.series of locations in a particular order. The final location, if credentials aren't found anywhere else, is the instance metadata service. Boto3 checks the instance metadata service to determine whether an EC2 instance has an IAM role configured. If so, it tries to use that role for authorization.

AWS offers two versions of their instance metadata service (IMDS). IMDSv1 uses a request/response method, while Content from docs.aws.amazon.com is not included.IMDSv2 uses a session-oriented, token-based method.

It's possible to disable IMDSv1 for an EC2 instance, Content from docs.aws.amazon.com is not included.requiring IMDSv2 to be used for any instance metadata access. This is often done with the goal of enhanced security.

Older versions of Botocore don't support IMDSv2's token-based method. Support had to be added to Botocore after IMDSv2 was released.

The version of Botocore that Red Hat ships at the time of this writing for RHEL 8 via the python3-botocore package -- and prior to the fix, for RHEL 7 bundled inside the python-s3transfer package -- does not contain the necessary code to support IMDSv2. Therefore, lookups that require IMDSv2 will fail.

The result is that fence_aws and aws commands time out if IMDSv2 is required and an EC2 instance IAM role is used in place of credentials. In other words, if IMDSv1 has been disabled for an EC2 instance and credentials (or a profile for use with the Content from boto3.amazonaws.com is not included.assume role provider) are not specified, then Botocore hangs when it tries to contact the IMDS to check for an instance IAM role.

The patch for this issue bundles an updated version of Botocore into the fence-agents-aws package on RHEL 8 (or the python-s3transfer package on RHEL 7). This updated version supports IMDSv2 and can be used with instance IAM role authorization.

Diagnostic Steps

The steps below assume that IMDSv2 is required for the EC2 instance where the commands are run, and that no credentials are specified in any of Boto3's Content from boto3.amazonaws.com is not included.credential locations.

Similar issues may be observed with the aws command (from awscli) or with AWS resource agents (e.g., aws-vpc-move-ip).

Run the fence_aws -o list command with debugging enabled. The command times out without listing any instances.

 # fence_aws -r us-west-2 -o list --boto3_debug 1 -vvvvvv
 2022-02-02 07:08:49,647 DEBUG: Boto debug level is 1 and sending debug info to /var/log/fence_aws_boto3.log

Check the fence_aws_boto3.log. Find that botocore failed to find credentials in every location until iam-role, and then it exceeded the maximum number of attempts to access the instance metadata service.

 2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: env
 2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: assume-role
 2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: shared-credentials-file
 2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: custom-process
 2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: config-file
 2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: ec2-credentials-file
 2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: boto-config
 2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: container-role
 2022-02-02 07:08:49,662 botocore.credentials DEBUG    Looking for credentials via: iam-role
 2022-02-02 07:08:49,669 botocore.utils DEBUG    Max number of attempts exceeded (1) when attempting to retrieve data from metadata service.

Content from docs.aws.amazon.com is not included.Enable IMDSv1 and perform the above steps again. Contrast the successful logs in fence_aws_boto3.log (below) with the previous failure.

 2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: env
 2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: assume-role
 2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: shared-credentials-file
 2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: custom-process
 2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: config-file
 2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: ec2-credentials-file
 2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: boto-config
 2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: container-role
 2022-02-02 07:07:22,861 botocore.credentials DEBUG    Looking for credentials via: iam-role
 2022-02-02 07:07:22,871 botocore.credentials DEBUG    Found credentials from IAM Role: awstest-ec2-rhel-ha

SBR

Clusterha

Product(s)

Components

cluster

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.