Some LVM commands falsely report missing PVs even though the corresponding disks are present, have correct metadata and they are not marked with the "MISSING" flag.

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 8

Issue

  • Some types of LVM commands report one or more PVs as missing. As an example, lvs reports the missing PVs, while pvs does not.

  • These PVs exist on the system, they are not flagged as MISSING in LVM metadata and their on-disk metadata is correct. A sample of an lvs command reporting the missing PVs is:

      # lvs
    WARNING: Couldn't find device with uuid kkkkkk-kkkk-kkkk-kkkk-kkkk-kkkk-kkkkkk.
    WARNING: VG vgtest is missing PV kkkkkk-kkkk-kkkk-kkkk-kkkk-kkkk-kkkkkk (last written to /dev/sdk1).
    WARNING: Couldn't find all devices for LV vgtest/pool_meta_rimage_1 while checking used and assumed devices.
    WARNING: Couldn't find all devices for LV vgtest/pool_meta_rmeta_1 while checking used and assumed devices.
    WARNING: Couldn't find all devices for LV vgtest/pool_rimage_1 while checking used and assumed devices.
  ...

Resolution

  • Delete the hints file (/run/lvm/hints)
  • Run an lvm command (e.g. pvs) to recreate it.

Root Cause

  • Some LVM commands are using the hints file, as an optimization, in order to limit the number of devices they scan. If this file becomes out of date and misses a PV, then this PV will be reported missing by commands that use the hints file.

  • This is a different issue from a PV being marked with the "MISSING" flag

Diagnostic Steps

  • Review the verbose output of a command reporting the missing PV. As an example:

      lvs -vvvv

    Observe that the command reads the hints file, which does not list the PV later reported to be missing. From that point on, lvm does not try to scan the device and later reports it as missing.

      17:58:47.118717 lvs[4355] label/hints.c:684  Reading hint file
  17:58:47.118755 lvs[4355] config/config.c:1475  devices/global_filter not found in config: defaulting to global_filter = [ "a|.*|" ]
  17:58:47.118774 lvs[4355] config/config.c:1475  devices/filter not found in config: defaulting to filter = [ "a|.*|" ]
  17:58:47.118790 lvs[4355] label/hints.c:807  add hint /dev/sdq1 <UUID of sdq1> 65:1 vg:vgtest
  17:58:47.118803 lvs[4355] label/hints.c:807  add hint /dev/sda2 <UUID of sda2> 8:2 vg:r8vg
  17:58:47.118815 lvs[4355] label/hints.c:807  add hint /dev/sdr1 <UUID of sdr1> 65:17 vg:vgtest
  17:58:47.118827 lvs[4355] label/hints.c:807  add hint /dev/sdj1 <UUID of sdj1> 8:145 vg:vgtest
  17:58:47.118838 lvs[4355] label/hints.c:807  add hint /dev/sdl1 <UUID of sdl1> 8:177 vg:vgtest
  17:58:47.118850 lvs[4355] label/hints.c:807  add hint /dev/sdm1 <UUID of sdm1> 8:193 vg:vgtest
  17:58:47.118861 lvs[4355] label/hints.c:807  add hint /dev/sdn1 <UUID of sdn1> 8:209 vg:vgtest
  17:58:47.118873 lvs[4355] label/hints.c:807  add hint /dev/sdo1 <UUID of sdo1> 8:225 vg:vgtest
  17:58:47.118885 lvs[4355] label/hints.c:807  add hint /dev/sdp1 <UUID of sdp1> 8:241 vg:vgtest
  ...
  17:58:47.119521 lvs[4355] label/hints.c:846  accept hints found 9
  17:58:47.119540 lvs[4355] label/hints.c:1389  get_hints: applied using 9 other 27
  ...
  17:58:47.119582 lvs[4355] label/label.c:589  open /dev/sdq1 ro di 0 fd 5
  17:58:47.119714 lvs[4355] label/label.c:589  open /dev/sda2 ro di 1 fd 6
  17:58:47.119813 lvs[4355] label/label.c:589  open /dev/sdr1 ro di 2 fd 7
  17:58:47.120224 lvs[4355] label/label.c:589  open /dev/sdj1 ro di 3 fd 8
  17:58:47.120292 lvs[4355] label/label.c:589  open /dev/sdl1 ro di 4 fd 9
  17:58:47.120350 lvs[4355] label/label.c:589  open /dev/sdm1 ro di 5 fd 10
  17:58:47.120403 lvs[4355] label/label.c:589  open /dev/sdn1 ro di 6 fd 11
  17:58:47.120443 lvs[4355] label/label.c:589  open /dev/sdo1 ro di 7 fd 12
  17:58:47.120482 lvs[4355] label/label.c:589  open /dev/sdp1 ro di 8 fd 13
  17:58:47.120517 lvs[4355] label/label.c:728  Scanning submitted 9 reads
  ...
  17:58:47.134271 lvs[4355] metadata/metadata.c:5122  WARNING: VG vgtest is missing PV kkkkkk-kkkk-kkkk-kkkk-kkkk-kkkk-kkkkkk (last written to /dev/sdk1).

    (in the output above, note how sdk1 was not listed in the hints file and was not scanned)

  • Review the contents of the hints file:

      cat /run/lvm/hints

    Observe if the missing PVs are included in the file. If not, then LVM commands that use the hints file will be skipping the PVs that are not included in it and will be reporting them as missing.

SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.