Simultaneous operations on instances sharing multi-attach volume fail with "Invalid volume error"

Solution Verified - Updated

Environment

  • Red Hat OpenStack Platform 16.2
  • Red Hat OpenStack Platform 17.1
  • Cinder storage backend with multi-attach capability
  • Multiple instances sharing a single multi-attach volume

Issue

  • Heat stack creation fails with multiple instances sharing a single multi-attach volume.

  • Some of the instances become ERROR status after performing openstack server resize on multiple instances sharing a single multi-attach volume at once.

Resolution

The following operations could cause the similar issue. Please create a case for Red Hat technical support: DB surgery is likely needed and it should be performed by Red Hat's associates.

  • openstack server resize (the original issue)
  • openstack server migrate
  • openstack server migrate --live
  • openstack server evacuate
  • openstack server unshelve

The following operations also may fail because of the volume status.
Please try the "nova reset-state" command. If it is not fixed, It is need to open a case for Red Hat technical support to resolve this.

  • openstack server add volume
  • openstack server remove volume
  • cinder backup
  • openstack server reboot --hard

Root Cause

Nova and Cinder operations may make implicit state transition on the related volume.
These commands check the volume state before execution, and fail if the expected state is not met.
Simultaneous execution of such commands on instances sharing a single volume can lead to conflicts due to volume state changes, resulting in failures

This issue can occur not only with nova operations but also with combinations of nova and cinder operations.

Operations on a single volume or instances attached to the same multi-attach volume should be performed one at a time.
When managing these resources in a Heat stack, please add dependencies to prevent simultaneous management.

Diagnostic Steps

  • Heat stack creation fails with the following error message:
CREATE_FAILED BadRequest: resources.xxxxxx: Invalid volume: Invalid input
received: Invalid volume: Volume xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx status must
be available or in-use or downloading to reserve, but the current status is
reserved. (HTTP 400) (Request-ID: req-xxx
  • Resizing an instance may fail and result in an ERROR state with the following error message:
$ openstack server show vm-resize-test
...
| fault | {'code': 400, 'created': '2024-06-
04T10:05:00Z', 'message': 'Invalid input received: Invalid volume: Volume
xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx status must be available or in-use or
downloading to reserve, but the current status is attaching. (HTTP 400) (RequestID: req-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx)'} |
SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.