Fix for CVE-2023-2088 on OSP13

Solution Verified - Updated

Environment

OpenStack deployments using iSCSI or FC transport protocols for the volumes are affected by this issue, other protocols such as RBD/Ceph, NFS, and NVMe-oF are not affected.

Not all storage systems using iSCSI and FC will be affected, as it depends on the Cinder driver and Storage array specific behavior. For example, it doesn't affect iSCSI Cinder drivers that use a per-volume target instead of a per-host target (what we call shared-target), and there are also storage arrays that send a Power-on or device reset Unit Attention event that triggers actions on the Linux kernel side that prevent the issue from happening (this signal has been seen in a HPE 3PAR FC system).

Issue

An unauthorized access to a volume could occur when an iSCSI using shared targets or FC connection from a host is severed due to a volume being unmapped on the storage system and the device is later reused for another volume on the same host.

This data leak can be triggered by two different situations as reported in CVE-2023-2088

Accidental case
If there is a problem with network connectivity during a normal detach operation, OpenStack may fail to clean the connection up properly. Instead of force-detaching the compute node device, Nova ignores the error, since the instance has already been deleted. Due to this incomplete operation OpenStack may end up selecting the wrong multipath device when connecting another volume to an instance.

Intentional case
A regular user can create an instance with a volume, and then delete the volume attachment directly in Cinder, which neglects to notify Nova. The compute node SCSI plumbing (over iSCSI/FC) will continue trying to connect to the original host/port/LUN, not knowing the attachment has been deleted. If a subsequent volume attachment re-uses the host/port/LUN for a different instance and volume, the original instance will gain access to it once the SCSI plumbing reconnects.

Resolution

The only solution to protect an OSP13 deployment from the intentional attack is to use a policy to restrict operations that normal users can run.

We recommend customers engage with Red Hat to apply this solution, as these are rough instructions to resolve this CVE.

Note: Glance is only affected by this bug when using Cinder as a Glance Backend and with the standard OSP configuration the quick fix proposed here works, but it won’t if it’s changed to use the user token instead of a dedicated project to store the images.

These are the instructions:

  1. Install the files from this article on the director node, and give execution privilege to the shell scripts:

    • cve-service-configuration.yml
    • ensure-service-roles.sh
    • generate-cve-policy-overrides.sh

  2. Execute the ensure-service-roles.sh script

  3. Execute the generate-cve-policy-overrides.sh, which generates a new cve-policy-overrides.yaml file.
    NOTE: Pay particular attention to the note the script displays regarding situations where any existing policy overrides need to be merged into the cve-policy-overrides.yaml file.

  4. Review the cve-service-configuration.yml, and make any required updates necessary to configure all overcloud roles where cinder-volume, glance-api and nova services are running. For example, the cve-service-configuration.yml file assumes cinder, glance and nova control plane services are running on "Controller" nodes, and the nova-compute service is running on "Compute" nodes.

  5. Re-execute the overcloud deployment command, with two additional environment files appended.

    (undercloud) $ openstack overcloud deploy --templates …\
-e <existing deployment environment files> \
-e cve-service-configuration.yml  \
-e cve-policy-overrides.yaml

For those wanting a better understanding of what the above steps do, here is a brief description of what is being automated via Director:

  • Set the service role for the glance user

  • Set the service role for the nova user

  • Configure Nova, on "Compute" and "Controller" nodes, to send the service token, which requires adding lines similar to this to the nova configuration file:

        [service_user]
    send_service_user_token = True

    # Copy following options from [keystone_authtoken] section
    project_domain_name = Default
    project_name = service
    user_domain_name = Default
    password = abc123
    username = nova
    auth_url = http://192.168.121.66/identity
    auth_type = password

  • Deploy the Cinder policy rules to only allow services to do attachment deletion, connection termination, and detach operations and to prevent anyone from using force_detach (to prevent human operators from unintentionally shooting themselves on the foot). The automated steps converting these policy rules to JSON format and fill in the < nova_service_uuid >

    "is_service": "role:service or service_user_id:<nova_service_uuid>"
"volume:attachment_delete": "rule:admin_or_owner and rule:is_service"
"volume_extension:volume_actions:terminate_connection": "rule:admin_or_owner and rule:is_service"
"volume_extension:volume_actions:detach": "rule:admin_or_owner and rule:is_service"
"volume_extension:volume_admin_actions:force_detach": "!"

Code changes are necessary to prevent the accidental case in Computes and Controller nodes, please refer to the Red Hat CVE page to see when these are available on the CDN.

Root Cause

The underlying issue is the way SCSI based transport protocols work on Linux and how devices under /dev/ are not automatically removed when the connection to the storage array LUN is severed or the volumes are unexported/unmapped.

Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.