System is failing to boot as 'sd' names for devices are changing causing mount issues

Environment

• Red Hat Enterprise Linux all versions
• Red Hat Enterprise Linux CoreOS (RHCOS) all versions

Issue

• There are several sd devices seen from internal and external storage devices. The system is failing to boot at times as sd names for devices are changing.
• During the boot process, the system displays the below message and enters emergency mode:

[ TIME ] Timed out waiting for device dev-sde1.device. Unable to enter rescue mode.

• How to make the sd name for boot device(s) persistent across reboot?

Resolution

Note In case the system is failing to boot, to make the below changes to /etc/fstab it is necessary to either boot into the emergency mode and log in with the root password, or enter a rescue environment (How to boot a system into rescue mode).

• Mount the devices that are used for booting with their UUIDs. For example, change:

/dev/sda1 /boot ext4 defaults 1 2 

• In /etc/fstab file to:

UUID=1234bdbf-92fa-4964-af17-e956787624f8 /boot   ext4    defaults        1 2

• If the boot devices are LVM devices, then the below entry should do: (e.g., /home, /boot is not recommended on LVM device)

/dev/VG/homelv  /home    ext4    defaults        1 2

• If partitions like /home and other problematic partitions which are failing to mount at boot time due to the changing sd name, are not on LV, the above method seen for /boot could be used:

UUID=1234abcd-92fa-4964-af17-e95456782123 /home   ext4    defaults    1 2

• To check the UUID, use the command blkid (must be run as root):

# blkid
/dev/sda1: UUID="1234bdbf-92fa-4964-af17-e956787624f8" BLOCK_SIZE="512" TYPE="ext4" PARTUUID="1c0bff91-01"
/dev/mapper/VG-homelv: UUID="1234abcd-92fa-4964-af17-e95456782123" BLOCK_SIZE="512" TYPE="ext4"

Alternatively

• Use the names appearing in /dev/disk/by-id/ to mount :

    # ls -l /dev/disk/by-id/scsi-SATA_HITACHI_HTS7275_J3311111GWPSLA-part1
    lrwxrwxrwx. 1 root root 10 Mar 30 00:25 /dev/disk/by-id/scsi-SATA_HITACHI_HTS7275_J3311111GWPSLA-part1 -> ../../sda1
    
    # blkid
    /dev/sda1: UUID="1234bdbf-92fa-4964-af17-e956787624f8" TYPE="ext4"

    => /etc/fstab entry:
    /dev/disk/by-id/scsi-SATA_HITACHI_HTS7275_J3311111GWPSLA-part1 /boot   ext4    defaults        1 2

• The above-mentioned names are persistent across reboots, which will ensure that the system is using the correct device to boot from.

Root Cause

• sd device names appearing for scsi devices are not persistent across reboot. If the device to mount is something like /dev/sda1, and with several sd devices in the system, system might fail to boot as the boot device might take some other sd name. For example /dev/sdb.

• RHEL Documentation:
  * Within each of the following, sdX name space is defined as non-persistent as the names are assigned in "as discovered" order which can and will change across boots.

        The major and minor number range and associated sd names are allocated for each 
        device when it is detected. This means that the association between the major and 
        minor number range and associated sd names can change if the order of device 
        detection changes. 
  

  • RHEL9: "Overview of persistent naming attributes
  • RHEL 8: "Overview of persistent naming attributes
  • RHEL 7: "Persistent Naming"
  • RHEL 6: "Persistent Naming"
  • RHEL 5: "Persistent Naming"

• "Storage device names, such as /dev/sdX, are inconsistent between boots

• "What persistent device names are provided by the kernel in Red Hat Enterprise Linux 7, 8 and 9?"

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.