How are custom persistent names assigned for SCSI devices using udev in Red Hat Enterprise Linux 7 and later kernels?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 7, 8, and 9

Issue

  • The udev rule /etc/udev/rules.d/20-names.rules from Red Hat Enterprise Linux 6 does not work on Red Hat Enterprise Linux 7 and later kernels
  • How can static names be assigned for SCSI devices using udev in Red Hat Enterprise Linux?
  • How can I assign static /dev/sdN names to device to prevent them from changing across boots?
  • How can static names be assigned for SCSI devices using udev in Red Hat Enterprise Linux 7 and later kernels?
  • How can I assign static names for SCSI devices using udev in Red Hat Enterprise Linux 7 and later kernels?
  • The /dev/sdx name associated with storage devices changes between boots
  • The system had assigned /dev/sdn device name to the a disk, but after a reboot this name was changed to /dev/sdk, is this expected behaviour?
  • I am unable to rename the '/dev/sdX' device on the server.

Resolution

Caution!  NO DO NOT ATTEMPT TO USE /DEV/SDN NAMES EXPECTING THEM TO BE PERSISTENT!
  • The use of one of the provided persistent names or your own custom persistent name -- based on device self-identification -- is required in things like grub and /etc/fstab for supported and trouble free operations.
  • The /dev/sdN name space is one of severeal non-persistent name spaces that represent the access paths to storage devices.
    • Red Hat Enterprise Linux 9: "Disadvantages of non-persistent naming attributes" section in the "Managing file systems"
    • Red Hat Enterprise Linux 8: "Disadvantages of non-persistent naming attributes" section in the "Managing file systems
    • Red Hat Enterprise Linux 7: "Persistent Naming" section in "Storage Administration Guide" for details.
    • Red Hat Enterprise Linux 6: "Persistent Naming" section in "Storage Administration Guide" for details.
    • Red Hat Enterprise Linux 5: "Persistent Naming" section in "Online Storage Configuration Guide" for details.
      • "The major and minor number range and associated sd names can change if the order of device detection changes. Although this is unusual with some hardware configurations (for example, with an internal SCSI controller and disks that have their SCSI target ID assigned by their physical location within a chassis), it can nevertheless occur."
      • It is therefore "undesirable to use the major and minor number range or the associated sd names when referring to devices, such as in the /etc/fstab file. There is the possibility that the wrong device will be mounted and data corruption could result."
    • You cannot assign nor change sdN names as these are owned and assigned by the kernel.
    • The design of the linux kernel architecture is focused on using self-identification of data, disks, partitions, and filesystems (targets) for accessing and mounting of same. This is done via the use of unique names provided directly from the storage device (WWIDs) or by the data there on (UUIDs). The correct method of accessing a target is via the unique names provided by the target itself rather than trying to depend on sdN device name ordering which is not designed, and thus not guaranteed, to be the same across boots.
  • Attempting to use or expecting a /dev/sdN to represent access to a specific storage device across boots (be persistent) has never been supported even though commonly used -- and in some cases did seem to be more or less persistent in some cases.
    • The /dev/sdN names can and will change across boots. It has always been just a matter of time and catching the hardware in a particular state.
    • With each major release, RHEL 5 ... RHEL 9, the kernel has introduced more and more asynchronous discovery code paths into device discovery. These additions are to speed up boot time. Each such addition makes the non-persistent /dev/sdN name space that much more conspicuously non-persistent across boots.
      • With the release of RHEL 9, additional async discovery probing of devices has been introduced from upstream. This helps accelerate faster boots with configurations with large numbers of luns. A side affect is that the non-persistent sdN name space is even more so. That is sdN names are even more likely to change across boots in RHEL 9.

The udev rules in this article will only work on Red Hat Enterprise Linux 7 and later kernels. The udev rule syntax has changed slightly between the versions of udev on Red Hat Enterprise Linux 7 from prior versions of RHEL.

  • create a file in /etc/udev/rules.d to hold your udev rule, for example /etc/udev/rules.d/20-names.rules
  • create the rules needed to identify the target disk(s) you want to create unique names for. Each udev rule consists of three parts
    • (1)ACTIONs (2)FILTERs (3)CHANGEs
      • (1)ACTIONs, or when to apply the rule. The most common actions to trigger on are add and/or change.
      • (2)FILTERs, or how to identify the disk you are interested in to apply changes to. There are several common methods of filtering disks, including by subsystem and wwid (or other unique identifier). Some filters are fairly broad in that the changes to be applied are to be done across a number of disks, like changing the owner or access permisions on the disk. However, its more common that the filters end up uniquely targetting an individual disk, and for that some unique self-identifier like serial number or wwid is commonly used.
      • (3)CHANGEs, or what we want to do with the device. The most common change is to create a unique and persistent device name that then can be used by applications. But other changes such as device ownership and permissions are also sometimes needed -- for example, for Oracle asm disks.

Lets look at a specific example.


|<--- ACTIONs ---->|  |<-------------------- FILTERs ------------------->|    |<--------- CHANGEs ---------------->|
ACTION=="add|change", KERNEL=="sd*", ENV{ID_SCSI_SERIAL}=="<serial-number>",   SYMLINK+="<my-persistent-disk-name>%n"
ACTION=="add|change", KERNEL=="sd*", ENV{ID_SCSI_SERIAL}=="50014380212E90E0", SYMLINK+="data.A%n"
  • Notes:
    • The %n is the partition number, so if sda was the disk identified by the FILTERs within above, then sda, sda1, sda2 will have data.A, data.A1, data.A2, etc. also created.
    • Do not attempt to make non-persistent kernel enumerated names such as sd* dm* st* nt* eth* etc., persistent by using the NAME (or SYMLINK) udev method. That use is not supported and can cause issues.. The SYMLINK value my-persistent-disk-name should be unique, and not something already used by the kernel.

This rule will identify an sdN disk with the serial number specified and create a unique /dev/my-persistent-disk-name entry in devfs. Then applications use /dev/my-persistent-disk-name instead of attempting to use sdN which by definition is non-persistent.

To find a unique self-identifer (one provided by the disk itself), use the udevadm info command. For example:


[root@host]# udevadm info --query=all /dev/sda | egrep -i "serial|wwn"
S: disk/by-id/wwn-0x600508b1001ca98d5d765bea5a0dd3fa
E: DEVLINKS=/dev/disk/by-path/pci-0000:04:00.0-scsi-0:1:0:0 /dev/disk/by-id/scsi-0HP_LOGICAL_VOLUME_00000000 /dev/disk/by-id/scsi-3600508b1001ca98d5d765bea5a0dd3fa /dev/disk/by-id/scsi-SHP_LOGICAL_VOLUME_50014380212E90E0 /dev/disk/by-id/wwn-0x600508b1001ca98d5d765bea5a0dd3fa
E: ID_SCSI_SERIAL=50014380212E90E0
E: ID_SERIAL=3600508b1001ca98d5d765bea5a0dd3fa
E: ID_SERIAL_SHORT=600508b1001ca98d5d765bea5a0dd3fa
E: ID_WWN=0x600508b1001ca98d
E: ID_WWN_VENDOR_EXTENSION=0x5d765bea5a0dd3fa
E: ID_WWN_WITH_EXTENSION=0x600508b1001ca98d5d765bea5a0dd3fa
E: SCSI_IDENT_SERIAL=50014380212E90E0

The "E:" indicates these are ENV{name}="value" types and in the example above, I chose to use ENV{ID_SCSI_SERIAL} as that is unique in my configuration across disks. I could have chosen the World Wide Name (wwid) ENV{ID_WWN} or any of the other named identifiers for this disk.

An alternative to using the ENV{name} clause for filtering, is to use ATTR{name}=="value" or ATTRS{name}=="value" filtering. These named pairs can be found by again using the udevadm command:


[root@host]# udevadm info --path=/sys/class/block/sda --attribute-walk | grep -i ww
    ATTRS{wwid}=="naa.600508b1001ca98d5d765bea5a0dd3fa"

Note that this ATTRS{name} has the same value as ENV{ID_WWN_WITH_EXTENSION} but with the "naa." pre-pended to the wwid value to indicate it is in one of the NAA IEEE formats.
To verify the rule:


[root@host]# udevadm test /block/<sdX> 2>&1 | grep <my-persistent-disk-name>

To apply the rules once verified: reload and trigger them.


[root@host]# udevadm control --reload-rules
[root@host]# udevadm trigger --type=devices --action=change

NOTE: To monitor the rules, open a monitor while you do the above. In a separate terminal run the udev monitor, control-C to exit.
[root@host]# udevadm monitor

Additional References

Footnotes

RHEL9 :: PROBE_PREFER_ASYNCHRONOUS

Our sd enumeration design and documentation, while improved, has not been altered. (Its still non-persistent.)

Some customers have noted that the RHEL 9 kernel seems discover sdN devices in more a more non-persistent manner than previous major releases. We are consistently working to improve our product and what has been noticed (while seemingly counter-intuative) is another example of example of our code evolution. The change in behavior with RHEL 9 is the addition and use of the PROBE_PREFER_ASYNCHRONOUS flag for sd probing. This helps speed up discovery of devices in larger configurations. It does not change, but emphasizes, that the /dev/sdN name space is non-persistent as noted in our documentation at least since RHEL 7.

static struct scsi_driver sd_template = {
        .gendrv = {
                .name           = "sd",
                .owner          = THIS_MODULE,
                .probe          = sd_probe,
                .probe_type     = PROBE_PREFER_ASYNCHRONOUS,
                .remove         = sd_remove,
                .shutdown       = sd_shutdown,
                .pm             = &sd_pm_ops,
        }, 
        .rescan                 = sd_rescan,
        .init_command           = sd_init_command,
        .uninit_command         = sd_uninit_command,
        .done                   = sd_done,
        .eh_action              = sd_eh_action,
        .eh_reset               = sd_eh_reset,
}

In both RHEL 8 and RHEL 9, the index value which determines the sd name is allocated from the sd_probe function. In RHEL 8 (and older kernels), this will run synchronously with the scsi scan, only part of sd setup is done asynchronously with the sd_probe_async function completing the probe. With RHEL 9, the entire probe is now the sd_probe function and all of it is done asynchronously from the flag telling the device layer to do probing for sd asynchronously. RHEL 9 sd probing has no ordering constraints.

RHEL9.3 :: sd_mod.probe=sync

Present in RHEL 9.2.z kernel-5.14.0-284.48.1.el9_2 and RHEL 9.3 kernel-5.14.0-362.8.1.el9_3, as a feature, sd_mod.probe=sync can be added to the grub line as an opt-out on the async probe. The feature does have the potential to slow down boot on systems with a high number of scsi devices and is not guaranteed to order the scsi devices exactly as seen in prior kernel versions. The behavior and boot times should be similar to the RHEL8 kernel.

NOTICE  
  • The upstream community has rejected this feature so it will be RHEL only feature and is available in RHEL10 as well.

Root Cause

In Red Hat Enterprise Linux 7 and later kernels, the scsi_id command resides in /usr/lib/udev and SUBSYSTEM is scanned rather than BUS.

The udev rule in this article will only work on Red Hat Enterprise Linux 7 and later kernels. The udev rule syntax has changed slightly between the versions of udev, within the major releases. For more information on previous major releases, please refer to the following:
How can static names be assigned for SCSI devices using udev in Red Hat Enterprise Linux 5?
How can static names be assigned for SCSI devices using udev in Red Hat Enterprise Linux 6?

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.