How do I change the order of storage devices during boot in RHEL 5 and 6?
Environment
- Red Hat Enterprise Linux (RHEL) 5
- Red Hat Enterprise Linux (RHEL) 6
Issue
- Originally the system has one local disk /dev/sda, then connected with a new external storage (a storage array or a fiber-channel or a SAS disk). After the system was booted up, the internal disk on the server were changed to /dev/sdb while the external disks on the fiber storage were now /dev/sda.
- How to control storage device ordering during boot. For instance, having the local storage be mapped to sd names before the SAN storage. An example of this would
/dev/sdabeing local and/dev/sdbbeing external. - control storage order discovery between local RAID and HBA Qlogic
- control scsi device ordering during boot
- Sometimes when we reboot a server which take part in cluster (and only this server currently), the boot hang because HBA Qlogic card is seen as scsi0 instead of local RAID and so no boot partition is available.
- I have a problem with the server booting from SAN instead of local storage. How can I delay activiation of SAN storage until after the server has booted up properly?
- How to add persistent device names for local scsi device in Red Hat Enterprise Linux 5 / 6? We are seeing an issue on one of our RHEL6 server where we have device mapper multipath configured for SAN storage. SAN device names change on every reboot, causing local devices to acquire different names too? How can we prevent this?
- Root disk appears to be
/dev/sderather than expected/dev/sda - Why does SAN device take
/dev/sdadevice name?
Resolution
NOTE: The design of the linux kernel architecture is focused on using self-identification of data, disks, partitions, and filesystems (targets) for accessing and mounting of same. This is done via the use of unique names provided by the target (see Diagnostics section for more details). The correct method of accessing a target is via the unique names provided by the target itself rather than trying to depend on device name ordering which is not designed, and thus not guarenteed, to be the same across boots. There are some ways to workaround the architecture of using self-identification and instead trying to relying upon device names as outlined below. But these workarounds are contrary to the design of linux and will not always work in all cases (see Root Cause section for more details).
1. Specify the module load ordering in the bootloader, in /boot/grub/grub.conf.
-
The following arguments simply needed to be appended to the kernel line.
rdloaddriver=ata_piix,hpsa,qla2xxx,lpfc,megaraid_sas,sata_sil24- Note:
- all the respective drivers need to be within the initial ramdisk for the rdloaddriver grub line to work
- drivers will be loaded in left to right order; ata_piix first, hpsa, qla2xxx, lpfc, megraid_sas and then sata_sil24 last
- the driver list above is for illustration purposes only, change to the appropriate drivers needed for your configuration
- Note:
2. Specify a module load order via modprobe
-
For example: if you have storage attached via drivers ata_piix, hpsa, qla2xxx, lpfc, megaraid_sas, and sata_sil24 and want them loaded in that order then adding the lines below to
/etc/modprobe.d/storage-driver-order.confwould create that ordering precedence:install hpsa { /sbin/modprobe ata_piix ; } ; /sbin/modprobe --first-time --ignore-install hpsa install qla2xxx { /sbin/modprobe hpsa ; } ; /sbin/modprobe --first-time --ignore-install qla2xxx install lpfc { /sbin/modprobe qla2xxx ; } ; /sbin/modprobe --first-time --ignore-install lpfc install megaraid_sas { /sbin/modprobe lpfc ; } ; /sbin/modprobe --first-time --ignore-install megaraid_sas install sata_sil24 { /sbin/modprobe megaraid_sas ; } ; /sbin/modprobe --first-time --ignore-install sata_sil24 -
Next it is important to rebuild your initramfs, which can be done by following the article below How do I rebuild the initial ramdisk image in Red Hat Enterprise Linux?.
-
Note:
storage-driver-order.confis a chosen name and can be changed to suit your needs.- verify that all the respective drivers are being loaded during initrd image build for RHEL5, if one or more drivers don't show up add
alias scsi_hostadapterN driver-namelines to the/etc/modprobe.conffile and rebuild the initrd image again.
3. For RHEL 6 there's another alternative: Exclude modules from the initial ramdisk.
For example: If you don't need SAN-based LUNs to boot the system (i.e. the root filesystem is local) you could configure dracut (the utility that maintains initramfs) to omit the HBA modules. This would make local drives get the lower device names (sda, etc). This approach is well explained in: How to remove a particular module from RHEL 6 kernel initrd ?.
- For a more generic version of this instructions, refer How do I create initramfs using dracut on Red Hat Enterprise Linux 6? .
Root Cause
The reason storage disks (/dev/sgN) names, or tape or other device names assigned by the kernel, can change across boots is driver initialization and device discovery take slightly different amounts of time within different reboots thus complete in different time order. The time order depends on both the the amount of parallelism with the kernel version, the time the HBA card takes to complete initialization, as well as storage latency times during device discovery.
The sdN names are assigned in as discovered (time) order, so the first device discovered gets the name of /dev/sda, the second /dev/sdb, and so on. If the time order changes across boots, then the order in which sdN names are assigned can change. The sdN name space is owned by the kernel and using things like uname rules to rename devices as different sdN names is not supported.
While the linux architecture promotes and encourages use of self-declared identifiers of the target device or data, many legacy third party applications or command scripts depend on device name order remaining the same across boots. This causes a conflict between the linux kernel design and the application/script design. The above workaround(s) can help so that such legacy applications can keep on working.
The as discovered order is fairly easy to manipulate within RHEL 5 which was has mostly serial device discovery code paths. It is slightly harder to manipulate in RHEL 6 due to parallelism introduced into kernel device discovery functionality to help speed-up boot time. There is even more parallelism added to the device discovery code within RHEL 7 (again to help speed-up boot time) which makes the above workarounds even less viable, although the modprobe method can still be used in a modified form. It is just that within RHEL 7, the workarounds can be less effective over all sets of boots and configurations.
A different set of workarounds are available within RHEL 7:
There is no means of controlling the parallel discovery architecture within the linux kernel. It is contrary to the design goals of faster boot times. At best, system administrators can try workarounds to finesse the discovery order in support of legacy third party software.
The preferred method is to use the provided self-identification from the device or data rather than /dev/sdN device names.
For example, within grub boot lines the following identify the root device using an identifier that will not change across system reboots:
root=/dev/mapper/vg1-root << lvm volume by name
root=UUID=f2dc94c1-6124-4368-b215-f04ab171e024 << filesystem by UUID
root=PARTUUID=7d3bb988-02d0-4c6c-a864-14f4acfd0398 << partition by GUID
root=/dev/mapper/sysdiskp1 << multipath alias device name (partition)
root=/dev/disk/by-id/wwn-50014ee058de2dc0-part3 << partition by WWID (RHEL6 only)
None of the above will change irrespective of device discovery order. As long as what is identified is singularly unique, the identifier can be used. For example if that PARTUUID references a disk partition that is used as an LVM PV, then that partition UUID is not unique enough. That is, the LVM PV could be combined with other PV into an LVM VG and that volume group split into multiple LMV LV. In that case you'd need to use the filesystem UUID (2nd line/variant).
The same thing applies to /etc/fstab for mounting devices. The specified reference should not include or be reliant upon sdN names.
/dev/mapper/vg1-root / ext4 defaults 1 1 <== LVM LV assigned name
UUID=0c12f366-3dfa-4ea9-923c-5a57994ceef8 /boot ext3 defaults 1 2 <== ext3 filesystem UUID
There are couple methods of binding device self-identification information to other, user specified, device identifiers. The two nominal methods use:
- multipath alias naming capability, or
- custom udev rules.
Multipath
In some cases you can bind a user friendly name like "mydata" to one of these self declared identifiers. For example within multipath.conf:
(RHEL6)
multipaths {
:
multipath {
wwid 3600605b0020060501e58aabf175ee9b4 <== wwid
alias mydata <== will appear as /dev/mapper/mydata -> ../dm-N
}
:
}
See the appropriate multipath user guide for your kernel version on creating/binding user friendly names to a multipath device.
UDEV Rules
Or through the use of udev rules, for example a USB drive is plugged in, but it is one of several that can be present at any given time. A desire is to bind a user friendly name when it is plugged into the system in order to access the device in a consistent manner:
(RHEL6)
# udevadm info --query=all -n /dev/sdc
:
E: ID_SERIAL=WDC_WD32_00AAJS-00L7A0_DCA261376668-0:0
E: ID_SERIAL_SHORT=DCA261376668
:
#
# echo 'ACTION=="add", KERNEL=="sd*", ENV{ID_SERIAL_SHORT}=="DCA261376668", SYMLINK+="mydata%n"' >> /etc/udev/rules.d/75-usb.rules
#
With the above rule in place, when the drive with the matching ID_SERIAL_SHORT is plugged into the system, then the following devices are created allowing this USB device, when plugged in, to be accessed by the device name of /dev/mydata:
# ls -l /dev/mydata*
lrwxrwxrwx. 1 root root 3 Jan 29 11:41 /dev/mydata -> sdc
lrwxrwxrwx. 1 root root 4 Jan 29 11:41 /dev/mydata1 -> sdc1
lrwxrwxrwx. 1 root root 4 Jan 29 11:41 /dev/mydata2 -> sdc2
See the appropriate udev instructions for your kernel version on creating custom device rules.
Diagnostic Steps
The identifiers that should be used to locate a storage target -- whether that target is a disk, partition, lvm volume, or filesystem -- are the identification names provided by the device or data (partition, lvm volume or filesystem) itself rather than by an identifying construct assigned by the kernel. These self-identifying names are unique, if not universally, then at least within the current system. These self declared identifiers should be used rather than depending on the arbitrary name assigned to a device during discovery by the linux kernel.
Typically these self-identifying names fall into two general classes:
- WWN (world wide names), such as device WWIDs (World Wide IDentifiers) as retrieved from page 0x83 of SCSI INQUIRY commands, and
- UUIDs (Universally Unique IDentifiers) associated with data such filesystems, or GPT partitioned disks and its sub-partitions.
Note that identifiers associated with GPT partitioning are really GUIDs, which is a Microsoft implementation of the UUID standard ISO/IEC 9834-8:2014. So while all UUIDs are GUIDs, not all GUIDs are UUIDs. GUIDs are a superset of UUIDs. But in a practical sense, both are 128-bit UUIDs.
Note that LVM PV/VG/LV UUIDs are strings and not 128-bit (binary) UUIDs, but are still universally unique identifiers -- just in a different ascii string format.
Lets take an example of a disk that is partitioned by GPT into 5 partitions, one of which -- partition #5 -- is an LVM PV, and show the identifier hierarchy from disk through the LVM PV to multiple LVM LVs and terminating in the filesystem UUID.
Identifier Target Notes
[1] 0x5000c5003d223669 /dev/sda WWID of disk
[2] 26ADCA41-66A3-4126-9F07-F7B2D1C7F740 /dev/sda GUID of gpt partition table (which in turn identifies the disk)
[3] 7D3BB988-02D0-4C6C-A864-14F4ACFD0398 /dev/sda5 GUID of gpt partition entry #5, in this case the data within sda5 is a LVM PV
[4] 8OefjX-Xqwm-01a1-5Ma9-wDKL-Scge-5D2iHn PV LVM PV UUID string that identifies this specific physical volume
[5] b0TOUT-KodU-W1OI-ZENU-xSrJ-OHc1-mAYxVO VG 'vgsys' LVM VG 'vgsys' UUID string that identifies this specific volume group (container for a bunch of PVs)
[6] d9Mac8-0xam-o6sz-cyx5-Catl-FZem-UT8u3L /dev/mapper/vgsys-home LVM LV 'home' UUID string that identifies this specific logical volume (a chunk of VG assigned to logical "disk")
10d2Lq-eMku-da1I-7ZQP-oyGt-6e3d-Vmr2eO /dev/mapper/vgsys-root LVM LV 'root' UUID string " "
iu69uf-m6kE-eTdW-e9X6-W6zF-KOwu-nffBJ5 /dev/mapper/vgsys-tmp LVM LV 'tmp' UUID string " "
[7] 4ec273cf-29fc-4d7a-b766-011327f00645 /dev/mapper/vgsys-home Filesystem UUID for this xfs filesystem
ed840d93-29d7-4a8c-bfee-3eacb0eb6fc3 /dev/mapper/vgsys-root Filesystem UUID for this ext4 filesystem
f8678976-2c87-444e-851b-6acfb269d2a8 /dev/mapper/vgsys-tmp Filesystem UUID for this ext4 filesystem
When a fileystem is present, then the filesystem UUID is the preferred method of accessing the data associated with a /dev/<device>.
What are examples of commands that can be used to see the above identifiers?
[1] # sg_inq -p 0x83 /dev/sda | egrep "NAA 5|NAA 6" -A 3 <== must support page 83 Device Identification page
[2] # fdisk -l /dev/sda | egrep "label|identifier" <== must be gpt partition type
[3] # blkid /dev/sda5 <== look for PARTUUID
# sgdisk -i 5 /dev/sda | grep "Partition unique GUID" <== if command is available
[4] # pvs -v | egrep "DevSize|sda5"
[5] # vgs -v | egrep "VSize|vgsys"
[6] # lvs -v | egrep "LSize|home"
# lvs -v | egrep "LSize|root"
# lvs -v | egrep "LSize|tmp"
[7] # blkid /dev/mapper/vgsys-home
# blkid /dev/mapper/vgsys-root
# blkid /dev/mapper/vgsys-tmp
- or if available:
# lsblk +o UUID | egrep "NAME|vgsys-home"
# lsblk +o UUID | egrep "NAME|vgsys-boot"
# lsblk +o UUID | egrep "NAME|vgsys-tmp"
Example output of the above commands:
[1] # sg_inq -p 0x83 /dev/sda | egrep "NAA 5|NAA 6" -A 3 <== must support page 83 Device Identification page
# sg_inq -p 0x83 /dev/sda
VPD INQUIRY: Device Identification page <== has multiple identifiers, some vendor specific, some universal
:
Designation descriptor number 3, descriptor length: 12
designator_type: NAA, code_set: Binary <== Network Address Authority/binary
associated with the addressed logical unit
NAA 5, IEEE Company_id: 0xc50 <== which NAA identifier format type
Vendor Specific Identifier: 0x3d223669
[0x5000c5003d223669] <== WWID; which consists of format-type + company-id + vendor-id
[2] # fdisk -l /dev/sda | egrep "label|identifier" <== must be gpt partition type
# fdisk -l /dev/sda | egrep "label|identifier"
Disk label type: gpt <== if present, then...
Disk identifier: 26ADCA41-66A3-4126-9F07-F7B2D1C7F740 <== GUID of whole disk [NOTE 1]
[3] # blkid /dev/sda5 <== look for PARTUUID
# sgdisk -i 5 /dev/sda | grep "Partition unique GUID" <== if command is available
# blkid /dev/sda5
/dev/sda5: UUID="8OefjX-Xqwm-01a1-5Ma9-wDKL-Scge-5D2iHn" TYPE="LVM2_member" PARTUUID="7d3bb988-02d0-4c6c-a864-14f4acfd0398"
PV UUID GPT Partition GUID
# sgdisk -i 5 /dev/sda | grep "Partition unique GUID"
Partition unique GUID: 7D3BB988-02D0-4C6C-A864-14F4ACFD0398 [Note 2]
[4] # pvs -v | egrep "DevSize|sda5"
# pvs -v | egrep "DevSize|sda5"
PV VG Fmt Attr PSize PFree DevSize PV UUID
/dev/sda5 vgsys lvm2 a-- 2.29t 0 2.29t 8OefjX-Xqwm-01a1-5Ma9-wDKL-Scge-5D2iHn
[5] # vgs -v | egrep "VSize|vgsys"
# vgs -v | egrep "VSize|vgsys"
VG Attr Ext #PV #LV #SN VSize VFree VG UUID VProfile
vg_rhel7 wz--n- 4.00m 1 3 0 2.29t 0 b0TOUT-KodU-W1OI-ZENU-xSrJ-OHc1-mAYxVO
[6] # lvs -v | egrep "LSize|home"
# lvs -v | egrep "LSize|root"
# lvs -v | egrep "LSize|tmp"
# lvs -v | egrep "LSize|vgsys"
LV VG #Seg Attr LSize Maj Min KMaj KMin Pool ... LV UUID LProfile
home vgsys 1 -wi-ao---- 878.91g -1 -1 253 2 d9Mac8-0xam-o6sz-cyx5-Catl-FZem-UT8u3L
root vgsys 1 -wi-ao---- 488.28g -1 -1 253 0 10d2Lq-eMku-da1I-7ZQP-oyGt-6e3d-Vmr2eO
tmp vgsys 1 -wi-ao---- 976.56g -1 -1 253 1 iu69uf-m6kE-eTdW-e9X6-W6zF-KOwu-nffBJ5
[7] # blkid /dev/mapper/vgsys-home
# blkid /dev/mapper/vgsys-root
# blkid /dev/mapper/vgsys-tmp
- or if available:
# lsblk +o UUID | egrep "NAME|vgsys-home"
# lsblk +o UUID | egrep "NAME|vgsys-boot"
# lsblk +o UUID | egrep "NAME|vgsys-tmp"
# ls -1c /dev/mapper/vgsys* | xargs -I {} blkid {}
/dev/mapper/vgsys-root: LABEL="ROOT" UUID="ed840d93-29d7-4a8c-bfee-3eacb0eb6fc3" TYPE="ext4"
/dev/mapper/vgsys-home: UUID="4ec273cf-29fc-4d7a-b766-011327f00645" TYPE="xfs"
/dev/mapper/vgsys-tmp: LABEL="TEMP" UUID="f8678976-2c87-444e-851b-6acfb269d2a8" TYPE="ext
# lsblk -o +UUID | egrep "NAME|vgsys|sda5"
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT UUID
└─sda5 8:5 0 2.3T 0 part 8OefjX-Xqwm-01a1-5Ma9-wDKL-Scge-5D2iHn <== PV UUID
├─vgsys-root 253:0 0 488.3G 0 lvm / ed840d93-29d7-4a8c-bfee-3eacb0eb6fc3 <== filesystem UUID
├─vgsys-tmp 253:1 0 976.6G 0 lvm /tmp f8678976-2c87-444e-851b-6acfb269d2a8 <== " "
└─vgsys-home 253:2 0 878.9G 0 lvm /home 4ec273cf-29fc-4d7a-b766-011327f00645 <== " "
The above device WWID is present within the /dev/disk/by-* special device files:
(RHEL6)
# ls -l /dev/disk/by-*/* | grep sda | grep -v sda[1-9]
lrwxrwxrwx. 1 root root 9 Jan 29 09:59 /dev/disk/by-id/scsi-35000c5003d223669 -> ../../sda
lrwxrwxrwx. 1 root root 9 Jan 29 09:59 /dev/disk/by-id/wwn-0x5000c5003d223669 -> ../../sda <== WWN (WWID)
lrwxrwxrwx. 1 root root 9 Jan 29 09:59 /dev/disk/by-path/pci-0000:00:1f.2-ata-1.1 -> ../../sdaThe sda5 partition also shows up within the /dev/disk/by-* special device files:
(RHEL6)
# ls -l /dev/disk/by-*/* | grep sda5
lrwxrwxrwx. 1 root root 10 Jan 29 09:59 /dev/disk/by-id/scsi-35000c5003d223669-part5 -> ../../sda5
lrwxrwxrwx. 1 root root 10 Jan 29 09:59 /dev/disk/by-id/lvm-pv-uuid-8OefjX-Xqwm-01a1-5Ma9-wDKL-Scge-5D2iHn -> ../../sda5
lrwxrwxrwx. 1 root root 10 Jan 29 09:59 /dev/disk/by-id/wwn-0x5000c5003d223669-part5 -> ../../sda5
lrwxrwxrwx. 1 root root 10 Jan 29 09:59 /dev/disk/by-partuuid/7d3bb988-02d0-4c6c-a864-14f4acfd0398 -> ../../sda5
lrwxrwxrwx. 1 root root 10 Jan 29 09:59 /dev/disk/by-path/pci-0000:00:1f.2-ata-1.1-part5 -> ../../sda5The above device special file names can be used in places where a device name is needed in RHEL6 (the links are created too late within RHEL5 to be referenced within the boot process, for example). While a disk was used within the examples above, the same applies to other types of devices such as tapes.
NOTES
- Note 1
-
When the partition type is MBR (msdos), then the device identifier displayed is from the MBR which is only 32-bits in length. The 128-bit UUID is displayed only if the partition table is gpt. Also note, that GUID partition tables nominally should create a single partition within the MBR partition table with the code of 'ee' (GPT). This is a protective partition so that legacy command utilities that don't understand GUID partition tables still will show the device in-use having a partition table covering the whole disk (well, at least up to 2TiB limit for MBR partition tables for 512b/512b or 512e disks). An example of an MBR partition table when a GUID partition table is not present only shows the expected 32-bit length identifier:
# fdisk -l /dev/sda | egrep "label|identifier" Disk identifier: 0xb8000000 # parted /dev/sda print | grep "Partition Table" Partition Table: msdos
-
- Note 2
- The "Partition unique GUID" is the globally unique identifier for this partition.
- A "Partition GUID code" is also displayed by the sgdisk command. The "Partition GUID code" is a UUID field within the GPT that replaces the 1 byte partition type code within MBR partition tables. The GUID values that populate this field are fixed and define the type of data within the partition. For example GUID 'E6D6D379-F507-44C2-A23C-238F2A3DF928' is defined as specifying 'Linux LVM' is contained within the partition.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.