How to make disk letter /dev/sdX more likely to be consistent at each reboot

Solution Unverified - Updated

Environment

  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6
Caution!   This is not applicable to RHEL 9 due to the addition of PROBE_PREFER_ASYNCHRONOUS which added full asynchronous lun discovery/probing. See Content from www.spinics.net is not included."Unreliable disk detection order in 5.x" for a brief upstream discussion on this. Especially its conclusion "...even with {scsi_mod.scan=} synchronous scanning, device names are not reliable and there are no guarantees that one particular device will always have the same name".

Caution!   In RHEL 8 and earlier, a loss or addition of a device, adding a USB device, reservation conflicts, replacing and moving a storage controller as well as other issues can result in the sdX assigned names changing versus expectations even when scsi_mod.scan=sync is specified on the grub boot line. The preferred and recommended method of accessing devices is either through a persistent disk name in /dev/disk/by-id and similar or by using custom udev rules to create your own persistent disk names (although you cannot use sdX name space to do so).

Issue

  • At each boot, the disk letter will change at some rate, e.g. the same hard drive can be detected as /dev/sda at one reboot, and /dev/sdc at the next reboot.
  • How to make sure the same drive is identified to /dev/sdX at each reboot?

Resolution

Root Cause

Changing of non-persistent names, such as sda, is neither an error nor regression but is baked into how the linux kernel discovers disks. Often a customer notes that non-persistent names have been consistent across boots on most of their nodes except a few others, or were consistent up until a kernel update. Note that the number and type of scsi hosts and luns on each bus can and will affect discovery timing. If it appears to have been consistent, then its basically luck or a small storage configuration. Also note that each major kernel version attempts to speed up boot time and this means speeding up device discovery. There were major changes within RHEL 8 to increase scsi host discovery in parallel and increase scsi lun discovery in parallel was added within RHEL 9. Each such change can result in the non-persistent device names being even more non-persistent.

By default, Linux detects and discovers devices in parallel and asynchronously from each other. Typically discover gets initialized in an as found order (ordinal fashion), but drive letters are assigned in first response order. So in some cases it will be original (lun number/adapter order), while others not. This is not a flaw but is the way Linux is designed. Access by device name, such as sda, is not designed to be permanently coupled to any physical disk location or ordering. Instead, accessing disks should be by self-identifier of either the data container (disk) or the data itself inside the container (aka PV/LV/filesystem etc.). Disks self-identify by World Wide IDentifier (WWID), serial number or similar, and partitions, PV, LV, filesystems nominally by some type of UUID (for example a GUID for a GPT partition table entry).

For example see "Persistent Names" and "Assigning persistent custom names" along with "What persistent device names are provided by the kernel in Red Hat Enterprise Linux 8?".

By default, Linux detects and discovers devices in parallel and asynchronously from each other. Typically discover gets initialized in an as found order (ordinal fashion), but drive letters are assigned in first response order. So in some cases it will be original (lun number/adapter order), while others not. This is not a flaw but is the way linux is designed. Access by device name is not designed to be permanently coupled to any physical disk location or ordering. Instead, accessing disks should be by self-identifier of either the data container (disk) or the data itself (aka PV/LV/filesystem etc.). Disks self-identify by World Wide IDentifier (WWID) or similar, and partitions, PV, LV, filesystems by, typically, some type of UUID (for example a GUID for a GPT partition table entry).

If the device scan method is set to "sync", then it would serialize the SCSI device scan. This would make the device name assignment more consistent, but please note that even with "sync" scan method we could not guarantee the persistence of "/dev/sdx" name assigned to a give device.

From sysfs space we can see the default setting is async:

$ cat /sys/module/scsi_mod/parameters/scan
async

From kernel_doc package's scsi/scsi-parameters.txt file there are three different settings described:

        scsi_mod.scan=  async scans scsi busses in kernel threads 
                         sync scans scsi busses as they are discovered (serially)
                         none ignores scsi busses, expecting user space to do the scanning

The "scsi_mod.scan=sync" kernel parameter is available for use in both RHEL 6, 7, 8 and 9. If the "sync" scan method is used, then there could be additional delays observed during device scan and related discovery process during the system boot. This delay increases with the number of devices connected to system. If the system is having large number of SCSI devices connected (e.g. 100 or more) then use of this parameter could significantly increase the time required in completing boot process. Also, This parameter would only affect the method used for SCSI device scanning, but it does not have any implications on the IO performance on devices. So, if the system is having less number of devices, then you could still apply the above kernel option.

NOTE: In RHEL9, additional asynchronous lun discovery threads were added to further speed up device discovery at boot time.

Below is a partial snip of functions from above file which shows that it also verifies the SCSI hosts' scan method by checking "shost->async_scan" bit:

drivers/scsi/scsi_scan.c
1072 static int scsi_probe_and_add_lun(struct scsi_target *starget,
1073 uint lun, int *bflagsp,
1074 struct scsi_device **sdevp,
1075 enum scsi_scan_mode rescan,
1076 void *hostdata)
1077 {
...
1184 res = scsi_add_lun(sdev, result, &bflags, shost->async_scan);	<<----------
...

1582 struct scsi_device *__scsi_add_device(struct Scsi_Host *shost, uint channel,
1583 uint id, uint lun, void *hostdata)
1584 {
...
1598 if (!shost->async_scan)
1599 scsi_complete_async_scans();
...

1716 void scsi_scan_target(struct device *parent, unsigned int channel,
1717 unsigned int id, unsigned int lun, enum scsi_scan_mode rescan)
1718 {
...
1729 if (!shost->async_scan)					<<----------
1730 scsi_complete_async_scans();
...

1772 int scsi_scan_host_selected(struct Scsi_Host *shost, unsigned int channel,
1773 unsigned int id, unsigned int lun,
1774 enum scsi_scan_mode rescan)
1775 {
...
1786 if (!shost->async_scan)				<<----------
1787 scsi_complete_async_scans();
...

As seen in above snip, the async scan method not only affects the way kernel threads are spawned to scan the HBA, but it also has effect on device discovery on individual HBA as well. The SCSI host specific shost->async_scan bit is checked in each of above functions which are used during device discovery process. This should also explain the reason for difference in scsi device discovery while using async and sync methods.
We could use the “sync” method to make the device naming more consistent as compared to async scan mode. This parameter only forces the specific scan method used during device discovery, but it has no effect on IO performance on the devices.

Diagnostic Steps

The "scsi_mod" module in RHEL 7 kernel uses "async" method to scan the SCSI devices. This could be verified with the current value for below parameter in scsi_mod module:

$ cat /sys/module/scsi_mod/parameters/scan
async

If we check the value for above module parameter for RHEL 6.x kernel, then it could be seen that in RHEL 6 also SCSI device scan method is set to "async" only:

$ uname -r
2.6.32-642.6.2.el6.x86_64
$ cat /sys/module/scsi_mod/parameters/scan
async

Both RHEL 6 and 7 kernels are currently using "async" (asynchronous) scan method to scan the SCSI devices connected to system. But, due to the number of enhancements, improvements added to RHEL 7.x kernels to improve the time required in SCSI device discovery and processing, it may get a slightly different result with async SCSI device scan with RHEL 6 and 7 kernels.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.