Recommended Sizes and Technical Limitations of RHEV Storage Domains

Solution Verified - Updated

Environment

  • Red Hat Enterprise Virtualization (RHEV) 3.x

  • Block Storage Devices - iSCSI/FC Storage Domains

  • Specific note regarding Red Hat Virtualization (RHV) 4.1

Issue

  • How do I plan iSCSI or FibreChannel storage in RHEV for the best performance?
  • Is there a maximum number of VMs allowed in a storage domain?
  • Is there any limitation on the number of Storage Domains per Data Center?

Resolution

Note regarding RHV 4.1:
In RHV 4.1, the limit has been raised to 1300 logical volumes per block-based storage domain. Other than that, the information below, which was written for RHEV 3.x, is still pertinent. In the examples below, the RHEV 3.x limit of 300 is used.

Important: Updating to RHEV Manager 3.5 or later will now warn to the user when the number of block storage domains exceed a certain number of logical volumes defined in the RHEV configuration (default: 300). Once this limit is exceeded, each action that results in the creation of a new logical volume on the domain will add an audit log warning.

An iSCSI or Fibre Channel (FC) Storage Domain (SD) in RHEV is represented by a Volume Group (VG) in LVM that manages block storage devices.

There is a significant performance degradation noticed with VGs having 400+ Logical Volumes while serving as a RHEV Storage Domain. The size of the Storage Domain itself does not matter, as long as the number of Logical Volumes is kept low. The optimal value would be no more than 300-350 Logical Volumes.

The recommendation is to split your available storage into separate Storage Domains, with each not exceeding 350 Logical Volumes.

What is a Logical Volume (LV) in RHEV terms? An LV is a virtual disk of a virtual machine (VM). And for every single snapshot for virtual disk, a new LV would be created.

Do non-persistent VMs (aka stateless VMs) in a pool add to the total number of LVs? Yes, when powered on, each stateless virtual machine creates 1 snapshot per virtual disk inherited from the template. A pool of these stateless VMs will also include 1 LV for the parent template image. The total would be: disks in template + [ (disks in template) * (VMs in the pool) ] + [ (disks in template) * (powered on VMs) ]. For planning purpose we can simplify the formula this way: Total LVs = disks in template + [ 2 * (disks in template) * (VMs in the pool) ].

Note: This information is also available in the This content is not included.Technical Reference Guide.

Example Scenarios

  1. 100 VMs with single virtual disk attached and if each VM is going to have 2-3 snapshots during the life time of the VM [Note: this would be about 300 LVs in total], one Storage Domain should be sufficient.
  2. 500 VMs in the same scenario as #1: It would be better to split the available storage into several separate Storage Domains which can coexist under same Data Center.

Note: If an enterprise-grade NAS is available on a high speed network, NFS can potentially provide better Storage Domain performance than either iSCSI or FC.

On another hand, it is important to maintain the amount of Storage Domains per Data Center low as well, since RHEV constantly scans and check all its storage health, and scanning multiple storage domains will also reduce the performance and may cause SPM instability. There is no recommended value on the number of storage domains, it should be tuned based on the environment. But recommendation is to start from very few and increase if needed. Or, better consider splitting into a separate Data Center.

Minimizing Downtime

  • Use "Live Storage Migration" to move disks over to the new Storage Domain. In general, this approach is good, but bear in mind that Live Storage Migration takes a "live snapshot" prior to moving the disk. Prior to RHV 4.0, snapshots did not have the ability to be live deleted. This caused the destination Storage Domain to end up having one extra LV per migrated disk. However, as of RHV 4.0, these snapshots are automatically deleted.

    For example, if one has a storage domain with 500 LVs and they plan to move, by using Live Storage Migration, 200 LVs to a new storage domain, it is recommended to actually prepare at least two new storage domains and then move 100 LVs to each. This example would result in the following storage domains:

    • Storage_domain_1 with 300 LVs
    • Storage_domain_2 with 200 LVs
    • Storage_domain_3 with 200 LVs

  • RHEV admins should schedule a planned maintenance window where the VMs are powered down so that the extra snapshots that were generated during the Live Storage Migration can be deleted. This task - or part of it - can be carried out during a periodically occurring maintenance window. The rationale behind cleaning up the snapshots is due to the fact that excessive snapshots can have an impact on performance.

Note regarding offline snapshot deletion prior to RHV 4.1:

  • When a snapshot is deleted, if it's the first or only snapshot of a VM, then a new Logical Volume will be temporarily created for each of the VM's disks in the same Storage Domain. This will require additional space approximately equivalent to the size of the base image(s).
  • These additional LVs will be deleted upon completion, along with the volumes containing the snapshot. The end result being that the VM will consist of one less LV per disk. Therefore, the additional space will essentially be reclaimed when the snapshot deletion has completed.
  • If the base image is qcow2 then upon completion it will be roughly 10% larger than it originally was.
  • The point here being that additional Logical Volumes and free space are required in order to delete a snapshot, albeit temporarily. This becomes more of an issue the larger the base image(s), the more disks per VM and if multiple snapshot deletions are performed at the same time.
  • This is not an issue with live snapshot deletion (Live Merge), or with offline snapshot deletion (Cold Merge) in RHV 4.1 , as neither cause additional volumes to be created.

Root Cause

Multiple LVs increase LVM metadata, especially when sparse disks are in use. Which, in turn, increases the time to read this metadata. Since RHEV needs to read LVM metadata for each disk creation/cloning/deletion operation, it affects vdsmd performance during those operations.

As of RHEV 3.5 and later, when a block storage domain exceeds a certain number of logical volumes defined in a configuration value, each action that results in the creation of a new logical volume on the domain will add an audit log warning that the number of logical volumes on that domain has exceeded the defined number. The number of logical volumes is defined in the configuration value AlertOnNumberOfLVs and it's default value is 300.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.