Disk device (/dev/sda - /dev/sdb - /dev/sdc - /dev/sdd) changes after each node rebooting and impact pods using Local Storage Operator

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat CoreOS (RHCOS)
  • Local Storage Operator (LSO)

Issue

  • Nodes disk devices (/dev/sda - /dev/sdb - /dev/sdc - /dev/sdd) alternate after a reboot in a way that affects pods using persistent storage by using local volumes with Local Storage Operator.

Related Articles:
rook-ceph-osd-X Pod Stuck in CrashLoopBackOff/init after Node Reboot/OCP Upgrade monclient(hunting) - OpenShift Data Foundation

Resolution

  • It's recommended to use the unique LocalVolume resource by-id, for example: /dev/disk/by-id/<wwid> for devicePaths instead of /dev/<disk-name>.

  • Verify with the ls -l /dev/disk/by-id command inside a node to see the WWID. Change the /dev/sd(major number)(minor number) as needed.

    ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx. 1 root root  9 Jul 31 14:37 scsi-36000c294b13c4d0ba678222b80acd2c7 -> ../../sda
lrwxrwxrwx. 1 root root 10 Jul 31 14:37 scsi-36000c294b13c4d0ba678222b80acd2c7-part1 -> ../../sda1
lrwxrwxrwx. 1 root root 10 Jul 31 14:37 scsi-36000c294b13c4d0ba678222b80acd2c7-part2 -> ../../sda2
lrwxrwxrwx. 1 root root 10 Jul 31 14:37 scsi-36000c294b13c4d0ba678222b80acd2c7-part3 -> ../../sda3
lrwxrwxrwx. 1 root root 10 Jul 31 14:37 scsi-36000c294b13c4d0ba678222b80acd2c7-part4 -> ../../sda4
lrwxrwxrwx. 1 root root  9 Jul 31 14:38 scsi-36000c2955f12437151ab37b1cbbab1ef -> ../../sdc
lrwxrwxrwx. 1 root root  9 Jul 31 14:38 scsi-36000c29bae6b3edc77ce80e938fbcec7 -> ../../sdb
lrwxrwxrwx. 1 root root  9 Jul 31 14:37 scsi-SVMware_Virtual_disk_6000c294b13c4d0ba678222b80acd2c7 -> ../../sda
lrwxrwxrwx. 1 root root 10 Jul 31 14:37 scsi-SVMware_Virtual_disk_6000c294b13c4d0ba678222b80acd2c7-part1 -> ../../sda1
lrwxrwxrwx. 1 root root 10 Jul 31 14:37 scsi-SVMware_Virtual_disk_6000c294b13c4d0ba678222b80acd2c7-part2 -> ../../sda2
lrwxrwxrwx. 1 root root 10 Jul 31 14:37 scsi-SVMware_Virtual_disk_6000c294b13c4d0ba678222b80acd2c7-part3 -> ../../sda3
lrwxrwxrwx. 1 root root 10 Jul 31 14:37 scsi-SVMware_Virtual_disk_6000c294b13c4d0ba678222b80acd2c7-part4 ->   ../../sda4
lrwxrwxrwx. 1 root root  9 Jul 31 14:38 scsi-SVMware_Virtual_disk_6000c2955f12437151ab37b1cbbab1ef -> ../../sdc
lrwxrwxrwx. 1 root root  9 Jul 31 14:38 scsi-SVMware_Virtual_disk_6000c29bae6b3edc77ce80e938fbcec7 -> ../../sdb
lrwxrwxrwx. 1 root root  9 Jul 31 14:37 wwn-0x6000c294b13c4d0ba678222b80acd2c7 -> ../../sda
lrwxrwxrwx. 1 root root 10 Jul 31 14:37 wwn-0x6000c294b13c4d0ba678222b80acd2c7-part1 -> ../../sda1
lrwxrwxrwx. 1 root root 10 Jul 31 14:37 wwn-0x6000c294b13c4d0ba678222b80acd2c7-part2 -> ../../sda2
lrwxrwxrwx. 1 root root 10 Jul 31 14:37 wwn-0x6000c294b13c4d0ba678222b80acd2c7-part3 -> ../../sda3
lrwxrwxrwx. 1 root root 10 Jul 31 14:37 wwn-0x6000c294b13c4d0ba678222b80acd2c7-part4 -> ../../sda4
lrwxrwxrwx. 1 root root  9 Jul 31 14:38 wwn-0x6000c2955f12437151ab37b1cbbab1ef -> ../../sdc
lrwxrwxrwx. 1 root root  9 Jul 31 14:38 wwn-0x6000c29bae6b3edc77ce80e938fbcec7 -> ../../sdb

  • Change the devicePaths with the disk wwid:

    spec:
...
  storageClassDevices:
    - storageClassName: "local-sc" 
      volumeMode: Filesystem 
      fsType: xfs 
      devicePaths: 
        - /dev/disk/by-id/wwn-0x6000c29bae6b3edc77ce80e938fbcec7  #<-- Change from any /dev/<disk-name> to /dev/disk/by-id/<wwid>
...
...

  • For more information, see: Provisioning local volumes by using the Local Storage Operator

Root Cause

The OS assigns the names /dev/sd(major number)(minor number). Traditionally, non-persistent names in the form of /dev/sd(major number)(minor number) are used on Linux to refer to storage devices. The major and minor number range and associated sd names are allocated for each device when it is detected. This means that the association between the major and minor number range and associated sd names can change if the order of device detection changes.

For more information see: Overview of persistent naming attributes

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.