Ceph: Solid State Disks (SSDs) are detected as rotational disks (HDDs).

Environment

Red Hat Enterprise Linux (RHEL) 8
Red Hat Enterprise Linux (RHEL) 9
Red Hat Ceph Storage (RHCS) 4+

Issue

Solid State Disks (SSDs) are detected as rotational disks (HDDs).

Example:

# ceph osd metadata 0
{
[...]
    "bluefs_db_rotational": "1",    <----
    "bluefs_db_size": "32212254720",
    "bluefs_db_type": "hdd",
[...]
}

On the nodes hosting the OSD, check how the OSD sees the disks

# grep -Hv "zz" /sys/block/sd*/queue/rotational
/sys/block/sda/queue/rotational:1
/sys/block/sdb/queue/rotational:1
/sys/block/sdc/queue/rotational:1

See also KCS Solution #7008833, How does the kernel determine if a SCSI disk is HDD (rotational) or SSD (non-rotational)?

Resolution

To resolve this there are two possible approaches:

1. The first one is to adjust the storage controller configuration.
   Some controllers pass on wrong flags depending on the current mode. E.g. if the controller is set to RAID, test if the flags are passed correctly when being switched to JBOD mode. Just be aware that performance might differ a lot across modes, as not all controllers do utilize battery-backed cache in every mode. Hence a verification of the mode with typical disk benchmark tools is recommended after the change.

2. The second variant is to override the rotational flag on the operating system level.

   • Update the running kernel by writing to the /sys directory with the following commands:

     # cat /sys/block/sdd/queue/rotational 
     1
     # echo 0 > /sys/block/sdd/queue/rotational 
     # cat /sys/block/sdd/queue/rotational 
     0
     

     Note:

     • If there is already an existing LVM structure on top of the device, the rotational flag has to be changed for the respective dm-X device as well to get the proper value recognized for OSD.

       # echo 0 > /sys/block/dm-1/queue/rotational 
       

     • If the LVM structure is about to be created, it will inherit the current value of rotational from what is set for the device.

   • To make this change permanent, use the udev rule.
     As an example, the below steps would set the rotational flag to 0 (zero) for device /dev/sdb.

     • Identify the serial number of the device:

       # udevadm info --query=all /dev/sdb | grep  "ID_SCSI_SERIAL"
       E: ID_SCSI_SERIAL=452dab4b-b03f-46c5-a92b-5f677d102938
       

     • Create the udev rule for the device. Make sure that ID_SCSI_SERIAL is updated correctly.

       # cat /etc/udev/rules.d/99-force-ssd-rotational.rules
       ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="sd*", ENV{ID_SCSI_SERIAL}=="452dab4b-b03f-46c5-a92b-5f677d102938", ATTR{queue/rotational}="0"
       

     • When the drives are not consecutive, use multiple lines.

     • Reload and trigger the udev rule:

       # udevadm control --reload-rules
       # udevadm trigger --type=devices --action=change
       

     • Check the device:

       # cat /sys/block/sdb/queue/rotational
       0
       

Root Cause

The value in /sys/block/<device>/queue/rotational is wrong because the driver is passing the wrong information to the kernel from the storage controller.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.