System unable to boot post leapp upgrade from RHEL8 to RHEL 9 failing with `Failed to start Load Kernel Modules`

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9

Issue

  • After leapp upgrade from RHEL8 to RHEL9, system did not boot and was stuck at stage

    [FAILED] Failed to start Load Kernel Modules.
See 'systemctl status systemd-modules-load.service' for details. 
[FAILED] Failed to start Rule-based Manager for Device Events and Files.
See 'systemctl status systemd-udevd.service' for details. 
A start job for /dev/disk/by-uuid/69d8a426-8782-47cb-a14e-12446ab4a4d3 (3min 5s / no limit)

  • Unable to boot and everytime system gets stuck at either /dev/disk/by-uuid/ or Rule-based Manager service job running. 

    A start job for Rule-based Manager for Device Events and Files (3min 5s / no limit)
and
A start job for /dev/disk/by-uuid/69d8a426-8782-47cb-a14e-12446ab4a4d3 (3min 10s / no limit)

  • Entries in /etc/nsswitch.conf caused system boot failure. 

    # cat /etc/nsswitch.conf 
passwd:     sss files systemd cyberark
group:      cyberark sss files systemd
initgroups: cyberark files

Resolution

  • Boot system in rescue mode using How to boot a system into rescue mode.

  • Change the sequence for database used for password, group and initgroups or any other entry in /etc/nsswitch.conf.

    # vi /etc/nsswitch.conf
passwd:     sss files systemd cyberark
group:      sss files systemd cyberark
initgroups: files cyberark

  • Rebuild the initramfs image. 

    # dracut /boot/initramfs-kernel-version.img kernel-version
e.g.
# dracut /boot/initramfs-5.14.0-284.64.1.el9_2.x86_64.img 5.14.0-284.64.1.el9_2.x86_64

  • Boot the system normally and check if system boots fine.

Root Cause

  • The cyberark database being preferred over files database i.e. /etc/group and other files.

  • Entries in /etc/nsswitch.conf caused system boot failure. 

    # cat /etc/nsswitch.conf 
passwd:     sss files systemd cyberark
group:      cyberark sss files systemd
initgroups: cyberark files
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.