How do I debug an LVM problem?

Environment

• Red Hat Enterprise Linux 4
• Red Hat Enterprise Linux 5
• Red Hat Enterprise Linux 6

Issue

• How do I debug an LVM problem?

Resolution

Common Problems:

• "found duplicate PV" / LVM filter configuration: Why do I see 'found duplicate pv' warnings when using LVM with multipath storage in RHEL?
• Simple LVM filter configuration: Is there an easy way to set a proper LVM filter?
• LVM and multipath configuration: How do I setup multipath on a system that already has LVM configured?
• LVM command hangs or takes a long time: LVM command hangs or takes a long time to complete in RHEL
• "Incorrect metadata area header checksum": lvm metadata error
• Recover single PV metadata: How do I recover a physical volume's metadata?
• Restore a volume group: How do I restore a volume group in Red Hat Enterprise Linux if one of the physical volumes that constitutes the volume group has failed?
• "File-based locking initialisation failed": Running an LVM command returns "File-based locking initialisation failed" or "Locking type 1 initialisation failed"
• I/O errors on running lvm commands: Why do I get I/O errors when I run lvm commands on my system configured with multipath in RHEL?

Troubleshooting Tips

• If a specific command is failing, enable verbose mode by adding "-vvvv" to the command, and redirect it to a file with "&>".  This will provide debugging information similar to strace, but more specific to LVM.

    # pvs -vvvv &> pvs.txt
    # head pvs.txt 
    #lvmcmdline.c:915         Processing: pvs -vvvv
    #lvmcmdline.c:918         O_DIRECT will be used
    #config/config.c:950       Setting global/locking_type to 1
    #locking/locking.c:223       File-based locking selected.
    #config/config.c:927       Setting global/locking_dir to /var/lock/lvm
    #config/config.c:955       report/aligned not found in config: defaulting to 1
    #config/config.c:955       report/buffered not found in config: defaulting to 1
    #config/config.c:955       report/headings not found in config: defaulting to 1
    #config/config.c:933       report/separator not found in config: defaulting to  
    #config/config.c:955       report/prefixes not found in config: defaulting to 0

• Within a running system, or the chroot /mnt/sysimage environment of Rescue Mode; use # lvmdump to gather more information. Depending on the type of problem, you may need to run # lvmdump with different parameters, or use an alternative script. This option is not available from a non-chroot rescue environment.

• "lvmdump -am".  Use this if metadata corruption is suspected, or the problem is poor performance of LVM commands (For example pvs, vgs, lvs, vgscan, pvscan, etc).  Using "-m" flag will attempt to 'dd' a copy of metadata, but requires 'pvs' to run properly.  NOTE: If pvs fails or is the problem, the '-m' flag may not work, and instead, you may need to use a script based on /etc/lvm/cache/.cache, to 'dd' the first portion of each device (for RHEL4/5, the first 192KB, for RHEL6, the first 1MB), and store in a tarball (See attached lvm-save-metadata-device-cache.sh script).
• "lvmdump -c". Use '-c' if clvmd is running, to gather cluster data as well.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.