What happens when a snapshot is deleted in Red Hat Virtualization?

Environment

• Red Hat Virtualization (RHV) 4.3
• Red Hat Virtualization (RHV) 4.4
• Red Hat Virtualization (RHV) 4.4 SP1

Issue

• Why does RHV require writing to the storage when deleting a snapshot?
• Why does RHV take so long to delete snapshots?
• Why is so much free space required in a Storage Domain in order to delete a snapshot ?
• Why does a snapshot deletion fail due to lack of free space in a Storage Domain ?

Resolution

Background Information;

• When a VM is created in RHV, any disk that is then created in a Storage Domain can either be in RAW (pre-allocated) or QCOW2 (thin-provisioned) format.
• At that time each disk will contain a single image. This is called the base image.
• This image will either be a volume in a file-based Storage Domain, e.g. NFS, or a Logical Volume (LV) in a block-based Storage Domain, e.g. iSCSI or FibreChannel.
• When a snapshot of a VM is taken, a new image is added to the image chain for each disk, thus a new volume is created in the appropriate Storage Domain.
• This new image created and each subsequent image generated by a snapshot will be in QCOW2 format.
• As of RHV 4.1, Offline Snapshot Deletion is performed in a similar manner to Live Merge, and so additional volumes no longer need to be created.

Live Merge (online snapshot deletion);

• As of RHEV 3.5, snapshot deletion can be performed while the VM is running, as long as the host is on RHEL or RHEV-H 7.1 or later.

• The merging of data is performed by libvirt and qemu-kvm and the result is passed back to VDSM, as opposed to Offline Snapshot Deletion, which is driven step-by-step by VDSM.

• VDSM is still responsible for the volume removal step after the data has been merged.

• There are no interim or additional volumes created during this sequence.

Detailed Live Merge Information;

• There are two volumes involved in any snapshot deletion. One contains the snapshot, the other is the child volume. Data blocks that exist in the child are merged back into the parent (the volume containing the snapshot). When this is complete, the child volume is removed.

• For example, a volume chain consists of 4 volumes, where A is the base volume of the disk, and D is the leaf, or active volume of the disk;

  A (base) - B - C - D (leaf)
  

  If the snapshot in B is removed, the data in C is merged back into B. When complete, C will be unlinked from the volume chain and removed, and B will still exist. B will now contain the data from C, as well as what was originally in B, and will now be associated with the snapshot that was in C previously. The result will be;

  A (base) - B - D (leaf)
  

• In the above example, B is the base volume of the merge and C is the top volume of the merge. The base volume of a merge is not the same as the base volume of a volume chain, although it can be of course. If the base volume of the merge is qcow, which it would be in this example, then if it is smaller than the top volume of the merge, it will get extended to the size of the top volume plus some additional space.

• The above is called an internal layer merge, because it only involved internal (non-leaf) volumes. The data in both volumes is read-only, so there is no new data being written to either. If a failure occurs, the snapshot deletion can be retried, regardless of where and when it failed, as long as the volume chain is consistent. The volume chain exists in several places, so all of these need to be consistent. To verify this, please contact Red Hat Technical Support for assistance.

• If, in the example above, the snapshot in C had been removed, then D would have been merged back into C. However, D is the leaf/active volume, i.e. the one currently being written to. This is called an active layer merge. When this occurs, C and D are mirrored for writes, so that all new data will be written to both volumes. Therefore, just the previously-written blocks in D need to be merged back into C. Once complete, a "pivot" is performed, which unmirrors the writes and makes C the active/writable volume, and unlinks D from the volume chain. Then, D should be removed. So, now the volume chain would be;

  A (base) - B - C (leaf)
  

• If a failure occurs, such that perhaps D did not get removed, we have to be careful. We have to know which is the current active volume. As long as the VM is up, we can tell which volumes are being used by the VM. However, if the VM has been shutdown, then we have to look at the vdsm logs. If the vdsm logs have rolled over by the time we get a sosreport, we don't have any evidence of what happened, so we can only make an assumption as to what happened.

• In this case too, if C were smaller than D prior to the merge, C would have been extended.

Additional Considerations;

• The Wipe After Delete option provides a means of writing zeroes to a disk's images before they are removed. This will also take some time, obviously depending upon the size of the image(s). As such, this will result in the snapshot deletion taking a longer time than if this option were not set.

• The longer the image chain, the longer the time required to perform certain functions, particularly with block-based storage domains, where LVM is used. In such a case, every Logical Volume in the image chain for each disk has to be activated via lvchange before the disk can be worked on.

• Additional information can be found in our Technical Reference Guide

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.