LVM2 does not have metadata balancing support across physical volumes

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux AS release 4, 5, 6
  • LVM with a large number of physical volumes

Issue

  • LVM commands such as pvs, vgs, lvs are taking a long time to complete on a system with a large number of physical volumes.
  • I would like to remove LVM metadata from some physical volumes to speed up LVM commands.

Resolution

  • Red Hat Enterprise Linux 5.6 / 6.0 (any lvm2 version > 2.02.69) includes metadata balancing across physical volumes.  See the "vgmetadatacopies" option of vgchange, vgcreate, and lvm.conf man pages,  for further information.
  • For other versions of LVM, a manual procedure may be employed to remove metadata copies from physical volumes.

Root Cause

  • By default, LVM creates one copy of metadata on each physical volume.  In systems with a large number of physical volumes, this may cause lvm commands to be slow as all physical volumes must be read for metadata to construct the LVM configuration on the machine.
  • There is a trade-off between data resilience (many copies of metadata implies loss of a PV will not lose the LVM configuration) and performance (too many copies of metadata will slow down commands).
  • For systems with a large number (100s or 1000s) of physical volumes, it is a best practice NOT to have all pvs with metadata.

Diagnostic Steps

  • Other common causes of slow LVM command performance should be ruled out.  For common causes of slow LVM command performance, see https://access.redhat.com/kb/docs/DOC-61336
  • For lvm2 versions > 2.02.69 (i.e. RHEL5.6 or above, or RHEL6.0 or above), see the metadata rebalancing option, "vgmetadatacopies", detailed in the man pages.
  • For all other versions of lvm2, a manual process may be used to remove metadata from selected physical volumes.  Contact your support representative for this process.
  • To determine the impact of a large number of physical volumes on execution time of an LVM command, a procedure similar to the following may be used.
    • The following test procedure shows how using a limited number of metadatacopies on physical volumes can impact the performance of the 'vgs' command.  In the following example, there are 350 physical volumes in the volume group.  The "--config ' devices { filter = ..." option has the effect of limiting the command to only the 5 physical volumes specified.  When only 5 physical volumes are used instead of the full 350, execution time improves from 12.5 seconds to 2.5 seconds: 
      # time vgs vgtest
  VG     #PV #LV #SN Attr   VSize VFree
  vgtest 350   0   0 wz--n- 2.05G 2.05G

      real     0m12.522s
      
user     0m0.884s
      
sys     0m0.472s
      

      grep -v # /etc/lvm/lvm.conf | grep filter
      

      filter = [ "a|/dev/mapper/mpath|", "r|.*|" ]

      

      ls /dev/mapper/mpath* | tail -5
      

      /dev/mapper/mpath9p5
      
/dev/mapper/mpath9p6
      
/dev/mapper/mpath9p7
      
/dev/mapper/mpath9p8
      
/dev/mapper/mpath9p9
      

      time vgs --config 'devices { filter = ["a|/dev/mapper/mpath9p[56789]|","r/.*/"] } ' vgtest
      

      Couldn't find device with uuid eEGX8J-cp0e-5jYu-aSmW-XPrX-g47F-aYbcxu.
      
...
      
VG     #PV #LV #SN Attr   VSize VFree
      
vgtest 350   0   0 wz-pn- 2.05G 2.05G
      

      real     0m2.454s
      
user     0m0.712s
      
sys     0m1.224s

    • The following example shows the use of vgmetadatacopies option, to improve the performance of the above vgs command.  Below, the number of metadatacopies is limited to 5, similar to the previous test, and an execution time of 3 seconds is seen.  Next, increasing vgmetadatacopies to 25, a slightly larger execution time of 4 seconds is seen.  With this example, we can see the tradeoff between the better data resilience in the 50 copies case, and the better execution time with 5 copies. 
      # vgchange --vgmetadatacopies 5 vgtest
  Volume group "vgtest" successfully changed
# time vgs vgtest
  VG     #PV #LV #SN Attr   VSize VFree
  vgtest 350   0   0 wz--n- 2.05G 2.05G

      real     0m2.959s
      
user     0m0.284s
      
sys     0m0.216s
      

      vgchange --vgmetadatacopies 50 vgtest
      

      Volume group "vgtest" successfully changed
      

      time vgs vgtest
      

      VG     #PV #LV #SN Attr   VSize VFree
      
vgtest 350   0   0 wz--n- 2.05G 2.05G
      

      real     0m4.245s
      
user     0m0.384s
      
sys     0m0.300s

Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.