kernel panic occurs after installing and booting a new kernel if lvm filters are applied

Environment

• Red Hat Enterprise Linux 4, 5, 6

Issue

• After updating the Red Hat Enterprise Linux 5 system to 2.6.18-92.1.13.e15 kernel and rebooting the system, system panicked with the following errors.

  Unable to access resume device (/dev/VolGroup00/LogVol01)
  mount: could not find filesystem '/dev/root'
  setuproot: moving /dev failed: No such file or directory
  switchroot: mount failed: No such file or directory
  Kernel panic - not syncing: Attempted to kill init!
  

• The same issue occurs after migration to a different server e.g. a P2V migration

Resolution

• Edit /etc/lvm/lvm.conf and replace following entry

  filter = [ "r/disk/", "r/sd.*/", "a/.*/" ]
  

  with

  filter = [ "a/dev/mapper/.*/", "a/dev/sda2$/", "r/.*/" ]
  

  This filter will enable all /dev/mpath/* devices and /dev/sda2 device where the / was mounted.

• Clear /etc/lvm/cache/.cache.

• Re-create initrd as described in article How do I rebuild the initial ramdisk image in Red Hat Enterprise Linux?

• Run vgscan command

  # vgscan
  

• Reinstall the latest kernel rpm.

  # rpm -e kernel-2.6.18-128.1.10.el5
  # yum install kernel
  

Rebuilding the initrd would also work.

Root Cause

Following entry in  /etc/lvm/lvm.conf  will exclude all /dev/disk/* and /dev/sd* devices.

filter = [ "r/disk/", "r/sd./", "a/./" ]

When a new kernel is installed after this, the initrd of the new kernel will not be able to see the devices.

Diagnostic Steps

In the case with a P2V migration - if the original Physical server has filters e.g.

filter = [ "a/|/dev/cciss/.*|/", "a/|/dev/mpath/.*/|, "r/.*/" ]

Need to be changed to:

filter = [ "a/.*/" ]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.