Cannot delete a cinder volume although the instance is deleted in Red Hat OpenStack Platform

Solution Verified - Updated

Environment

Red Hat OpenStack Platform

Issue

A cinder volume cannot be deleted. It lists itself as attached to an instance, but the instance does not exist.

[root@ctr1 cinder(keystone_admin)]# cinder list
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
|                  ID                  |   Status  | Display Name | Size | Volume Type | Bootable |             Attached to              |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+
| 4327bc98-026d-4566-a9b9-df02ff5df554 | available |              |  20  |     None    |   true   | a43554df-967c-4db5-afa6-812e56048d95 |
+--------------------------------------+-----------+--------------+------+-------------+----------+--------------------------------------+

[root@ctr1 cinder(keystone_admin)]# cinder  delete 4327bc98-026d-4566-a9b9-df02ff5df554
Delete for volume 4327bc98-026d-4566-a9b9-df02ff5df554 failed: Volume 4327bc98-026d-4566-a9b9-df02ff5df554 is still attached, detach volume first. (HTTP 400) (Request-ID: req-19cd1916-7583-497b-b8ec-4139dfd11dc8)
ERROR: Unable to delete any of the specified volumes.
[root@ctr1 cinder(keystone_admin)]# cinder  force-delete 4327bc98-026d-4566-a9b9-df02ff5df554
Delete for volume 4327bc98-026d-4566-a9b9-df02ff5df554 failed: Volume 4327bc98-026d-4566-a9b9-df02ff5df554 is still attached, detach volume first. (HTTP 400) (Request-ID: req-4773a999-0376-46ac-a364-b48b48c1d01b)
ERROR: Unable to force delete any of the specified volumes.
[root@ctr1 cinder(keystone_admin)]# cinder  reset-state 4327bc98-026d-4566-a9b9-df02ff5df554
[root@ctr1 cinder(keystone_admin)]# cinder  delete 4327bc98-026d-4566-a9b9-df02ff5df554
Delete for volume 4327bc98-026d-4566-a9b9-df02ff5df554 failed: Volume 4327bc98-026d-4566-a9b9-df02ff5df554 is still attached, detach volume first. (HTTP 400) (Request-ID: req-ee62a45d-4c69-4ee7-b2ce-3a52efaf994f)
ERROR: Unable to delete any of the specified volumes.
[root@ctr1 cinder(keystone_admin)]# cinder  force-delete 4327bc98-026d-4566-a9b9-df02ff5df554
Delete for volume 4327bc98-026d-4566-a9b9-df02ff5df554 failed: Volume 4327bc98-026d-4566-a9b9-df02ff5df554 is still attached, detach volume first. (HTTP 400) (Request-ID: req-bdaa24c3-5da8-40fb-aa50-523a51030498)
ERROR: Unable to force delete any of the specified volumes.
[root@ctr1 cinder(keystone_admin)]# nova list
+----+------+--------+------------+-------------+----------+
| ID | Name | Status | Task State | Power State | Networks |
+----+------+--------+------------+-------------+----------+
+----+------+--------+------------+-------------+----------+

With the openstack client, the error message looks like this:

(overcloud) [stack@director ~]$ openstack volume delete <vol>
Failed to delete volume with name or ID '<vol>': Invalid volume: Volume status must be available or error or error_restoring or error_extending or error_managing and must not be migrating, attached, belong to a group, have snapshots or be disassociated from snapshots after volume transfer. (HTTP 400) (Request-ID: <req>)

Resolution

It is planned to implement a force detach feature in future OpenStack versions as explained in the following blueprint: Content from blueprints.launchpad.net is not included.Content from blueprints.launchpad.net is not included.https://blueprints.launchpad.net/nova/+spec/add-force-detach-to-nova.

Note that cinder volumes can be set as available and detached by changing their state. This "Explicitly updates the entity state in the Cinder database. Being a database change only, this has no impact on the true state of the entity and may not match the actual state. This can render a entity unusable in the case of changing to the 'available' state."

(overcloud) [stack@ucdr-a-001 ~]$ cinder help reset-state
usage: cinder reset-state [--type <type>] [--state <state>]
                          [--attach-status <attach-status>]
                          [--reset-migration-status]
                          <entity> [<entity> ...]

Explicitly updates the entity state in the Cinder database. Being a database
change only, this has no impact on the true state of the entity and may not
match the actual state. This can render a entity unusable in the case of
changing to the 'available' state.

Positional arguments:
  <entity>              Name or ID of entity to update.

Optional arguments:
  --type <type>         Type of entity to update. Available resources are:
                        'volume', 'snapshot', 'backup', 'group' (since 3.20)
                        and 'group-snapshot' (since 3.19), Default=volume.
  --state <state>       The state to assign to the entity. NOTE: This command
                        simply changes the state of the entity in the database
                        with no regard to actual status, exercise caution when
                        using. Default=None, that means the state is
                        unchanged.
  --attach-status <attach-status>
                        This is only used for a volume entity. The attach
                        status to assign to the volume in the database, with
                        no regard to the actual status. Valid values are
                        "attached" and "detached". Default=None, that means
                        the status is unchanged.
  --reset-migration-status
                        This is only used for a volume entity. Clears the
                        migration status of the volume in the DataBase that
                        indicates the volume is source or destination of
                        volume migration, with no regard to the actual status.

On newer version of OpenStack (Since OpenStack Stein - OSP15), This can be done using the openstack client openstack volume set --detached <VOLUME>

Root Cause

This issue can result from the instance being deleted, or the instance failing to spawn after the cinder volume was created/attached via the nova-boot command. It can also be the result of manual database manipulation.

A bug existed in early versions of Red Hat OpenStack Platform that could lead into this situation. A fix for this issue has been released for Red Hat Enterprise Linux OpenStack Platform 6.0. Please see the errata release notes at RHEA-2015-0151.

SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.