Override device rotational flag in OCS/ODF environments

Updated

Preface

This is meant as a temporary solution until the underlying platform (VMWare,Nutanix,Hyper-V,etc.) is able to properly present the drives to OpenShift Data Foundation as a flash drive. The use of HDDs in internal OpenShift Data Foundation is not supported per Supported configurations for Red Hat OpenShift Data Foundation 4.X

Mark the devices only if you are certain that they are flash devices. Incorrectly marking devices as flash devices regardless if they are flash devices may cause performance issues or data corruption.

NOTE1. It is the Storage responsible to present the rotational flag to the kernel, see How does the kernel determine if a SCSI disk is HDD (rotational) or SSD (non-rotational)? . This document just provide a way to change the rotational value to force the device to be treated as an SSD.

NOTE2. About the requirement of the rotational flag set to 0 (non-rotational) , see KCS Can I use HDD with ODF?

Device type customization

If the device is reported as a rotational drive your configuration must be customized to force the rotational flag for the devices provisioned. This can be achieved by adding a udev rule file to the OCS/ODF nodes of your cluster via a machine configuration setup.

  1. Name of the file: /etc/udev/rules.d/99-ibm.rules
  2. Content of the file template
ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="{device_prefix}{device_filter}", ATTR{queue/rotational}="0"

Note on the variable field of the above template:

  • {device_prefix} matches your environment as found in /dev ( e.g. sd, vd, dm)
  • {device_filter} matches your environment as found in /dev for {device_prefix} (e.g. [a-z], [0-9])

Here is an example for an environment where the physical devices are exposed to the nodes as /dev/sd[a-z] and a device mapper configuration is performed on top of the dual attached devices as /dev/dm[0-9]. In this example we override the rotational flag (0 = ssd) for all devices.

ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="sd[a-z]",  ATTR{queue/rotational}="0"
ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="dm-[0-9]", ATTR{queue/rotational}="0"

After create the udev rule you will need to run the commands below to apply the new rule:

udevadm control -R
udevadm trigger

Your Red Hat OpenShift Platform environment is not deployed yet

  • Add the file above to your worker node ignition file
  • If you are not familiar with the procedure to customize your ignition files please visit the following page on ignition file customization.

Your cluster Red Hat OpenShift Platform is already deployed

  • Use the MachineConfig mechanism to add the above udev rule to your worker nodes.
  • Once you apply the changes all nodes will be rebooted to apply the changes.

Machine configuration example

In the following example we have 3 worker nodes, each with the following:

  • One boot drive (/dev/xvda)
  • One flash drive (/dev/nvme0n1)
  • One device mapper device (/dev/dm-0[a][b])

We will use the machine configuration feature to force the rotational flag of each drive or device to one as an example. The udev rule we want to install on each node is illustrated below:

ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="xvd[a-z]",  ATTR{queue/rotational}="0"
ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="dm-[0-9]*", ATTR{queue/rotational}="0"

The udev rule can also be configured using disk vendor ID to only target devices provided by the storage provider. Simply replace the KERNEL=="xvd[a-z]" in the rule above with ATTR{idVendor}=="{nnnn}" if represented in "/sys/block//" or using ENV{ID_VENDOR}=="<name>".

The exact vendor identification can be inspected using the following command:

# udevadm info /dev/{device_name} | grep ID_VENDOR=
E: ID_VENDOR=VMware_

Encode the udev rule using base64.

$ cat << EOF | base64 -w0
> ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="nvme[0-2]n[0-1]",  ATTR{queue/rotational}="0"
> ACTION=="add|change", SUBSYSTEM=="block", KERNEL=="xvd[a-z]",  ATTR{queue/rotational}="0"
> EOF
QUNUSU9OPT0iYWRkfGNoYW5nZSIsIFNVQlNZU1RFTT09ImJsb2NrIiwgS0VSTkVMPT0ibnZtZVswLTJdblswLTFdIiwgIEFUVFJ7cXVldWUvcm90YXRpb25hbH09IjAiCkFDVElPTj09ImFkZHxjaGFuZ2UiLCBTVUJTWVNURU09PSJibG9jayIsIEtFUk5FTD09Inh2ZFthLXpdIiwgIEFUVFJ7cXVldWUvcm90YXRpb25hbH09IjAiCg==

Create machine configuration file

$ cat ./99-worker-udev-configuration.yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  labels:
    machineconfiguration.openshift.io/role: worker
  name: worker-udev-configuration
spec:
  config:
    ignition:
      config: {}
      security:
        tls: {}
      timeouts: {}
      version: 2.2.0
    networkd: {}
    passwd: {}
    storage:
      files:
      - contents:
          source: data:text/plain;charset=utf-8;base64,QUNUSU9OPT0iYWRkfGNoYW5nZSIsIFNVQlNZU1RFTT09ImJsb2NrIiwgS0VSTkVMPT0ibnZtZVswLTJdblswLTFdIiwgIEFUVFJ7cXVldWUvcm90YXRpb25hbH09IjAiCkFDVElPTj09ImFkZHxjaGFuZ2UiLCBTVUJTWVNURU09PSJibG9jayIsIEtFUk5FTD09Inh2ZFthLXpdIiwgIEFUVFJ7cXVldWUvcm90YXRpb25hbH09IjAiCg==
          verification: {}
        filesystem: root
        mode: 420
        path: /etc/udev/rules.d/99-ibm.rules
  osImageURL: ""

Note: Replace the text in bold italic above with the base64 encoded version of your udev rule.

Apply machine configuration to worker nodes

$ oc apply -f ./99-worker-udev-configuration.yaml
machineconfig.machineconfiguration.openshift.io/worker-udev-configuration created

Monitor Machine Config Update Monitoring

$ oc get mcp -o custom-columns=NAME:.metadata.name,COUNT:.status.machineCount,UPDATED:.status.updatedMachineCount,READY:.status.readyMachineCount
NAME     COUNT   UPDATED   READY
master   3       3         3
worker   3       0         0

When the configuration has been successfully applied the UPDATED column for the worker node should read True.

$ oc get mcp -o custom-columns=NAME:.metadata.name,COUNT:.status.machineCount,UPDATED:.status.updatedMachineCount,READY:.status.readyMachineCount -w
NAME     COUNT   UPDATED   READY
master   3       3         3
worker   3       0         0
worker   3       1         1
worker   3       1         1
worker   3       2         2
worker   3       2         2
worker   3       3         3

Verify device rotational flag

Connect to each OCS/ODF worker node to verify the device rotational flag for each OSD device has been set appropriately.

# oc debug node/{ocs_node}  -- chroot /host cat /sys/block/{osd_device}/queue/rotational
[... Truncated …]
To use host binaries, run `chroot /host`
0      <- 0 means reported as flash (SSD). 1 means rotational (HDD).
[... Truncated …]
SBR
Product(s)
Category
Article Type