Tuned rules don't get applied to new devices
Environment
Red Hat Enterprise Linux 8
Red Hat Enterprise Linux 7
Issue
- Tuned is not setting all values, specifically 'readahead'
Resolution
A solution for this issue has been released via the following errata which have the fix for this issue:
- RHEL 8: RHBA-2019:3633 -
tuned-2.12.0-3.el8or later - RHEL 7: RHBA-2019:2201 -
tuned-2.11.0-5.el7or later
If you still see the problem after the package has been upgraded, contact Red Hat Support.
Root Cause
The issue was that when udev sends the 'remove' event it does not send any device attributes with it so _device_is_supported check must be omitted in the code for the remove event to be received properly by tuned.
Diagnostic Steps
Before attaching a new device, enable 'tuned' in debug mode with the following command:
# tuned -D -l /tmp/tuned_tests.log
Reproduce this issue is easy.
Initial status:
# lsblk -t
NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE RA WSAME
sda 0 512 0 512 512 1 deadline 128 8192 2G
sdb 0 512 0 512 512 1 deadline 128 8192 2G
sr0 0 512 0 512 512 1 deadline 128 128 0B
vda 0 512 0 512 512 1 mq-deadline 256 8192 0B
├─vda1 0 512 0 512 512 1 mq-deadline 256 8192 0B
└─vda2 0 512 0 512 512 1 mq-deadline 256 8192 0B
├─rhel-root 0 512 0 512 512 1 128 8192 0B
└─rhel-swap 0 512 0 512 512 1 128 8192 0B
vdb 0 512 0 512 512 1 mq-deadline 256 8192 0B
vdc 0 512 0 512 512 1 mq-deadline 256 8192 0B
vdd 0 512 0 512 512 1 mq-deadline 256 8192 0B
└─vdd1 0 512 0 512 512 1 mq-deadline 256 8192 0B
└─rhel-root 0 512 0 512 512 1 128 8192 0B
After removing the four test devices (sda, sdb, vdb, vdc):
# lsblk -t
NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE RA WSAME
sr0 0 512 0 512 512 1 deadline 128 128 0B
vda 0 512 0 512 512 1 mq-deadline 256 8192 0B
├─vda1 0 512 0 512 512 1 mq-deadline 256 8192 0B
└─vda2 0 512 0 512 512 1 mq-deadline 256 8192 0B
├─rhel-root 0 512 0 512 512 1 128 8192 0B
└─rhel-swap 0 512 0 512 512 1 128 8192 0B
vdd 0 512 0 512 512 1 mq-deadline 256 8192 0B
└─vdd1 0 512 0 512 512 1 mq-deadline 256 8192 0B
└─rhel-root 0 512 0 512 512 1 128 8192 0B
We add five devices (sda, sdb, vdb, vdc, vde), but only the fifth one, vde, gets the expected `readahead=8192 and other values from tuned profile applied:
# lsblk -t
NAME ALIGNMENT MIN-IO OPT-IO PHY-SEC LOG-SEC ROTA SCHED RQ-SIZE RA WSAME
sda 0 512 0 512 512 1 deadline 128 128 2G
sdb 0 512 0 512 512 1 deadline 128 128 2G
sr0 0 512 0 512 512 1 deadline 128 128 0B
vda 0 512 0 512 512 1 mq-deadline 256 8192 0B
├─vda1 0 512 0 512 512 1 mq-deadline 256 8192 0B
└─vda2 0 512 0 512 512 1 mq-deadline 256 8192 0B
├─rhel-root 0 512 0 512 512 1 128 8192 0B
└─rhel-swap 0 512 0 512 512 1 128 8192 0B
vdb 0 512 0 512 512 1 mq-deadline 256 128 0B
vdc 0 512 0 512 512 1 mq-deadline 256 128 0B
vdd 0 512 0 512 512 1 mq-deadline 256 8192 0B
└─vdd1 0 512 0 512 512 1 mq-deadline 256 8192 0B
└─rhel-root 0 512 0 512 512 1 128 8192 0B
vde 0 512 0 512 512 1 mq-deadline 256 8192 0B ← This is the new one.
The content of 'tuned.log' in debug mode:
[...]
2018-06-11 08:07:23,270 INFO tuned.plugins.hotplug: device 'vdb' added
2018-06-11 08:07:25,764 DEBUG tuned.daemon.daemon: updating monitors
2018-06-11 08:07:25,765 DEBUG tuned.units.manager: updating monitor <tuned.monitors.monitor_disk.DiskMonitor object at 0x7f35c34211d0>
2018-06-11 08:07:25,765 DEBUG tuned.daemon.daemon: performing tunings
2018-06-11 08:07:35,778 DEBUG tuned.daemon.daemon: updating monitors
2018-06-11 08:07:35,778 DEBUG tuned.units.manager: updating monitor <tuned.monitors.monitor_disk.DiskMonitor object at 0x7f35c34211d0>
2018-06-11 08:07:35,779 DEBUG tuned.daemon.daemon: performing tunings
2018-06-11 08:07:36,803 INFO tuned.plugins.hotplug: device 'sda' added
2018-06-11 08:07:45,792 DEBUG tuned.daemon.daemon: updating monitors
2018-06-11 08:07:45,793 DEBUG tuned.units.manager: updating monitor <tuned.monitors.monitor_disk.DiskMonitor object at 0x7f35c34211d0>
2018-06-11 08:07:45,793 DEBUG tuned.daemon.daemon: performing tunings
2018-06-11 08:07:46,772 INFO tuned.plugins.hotplug: device 'vdc' added
2018-06-11 08:07:55,368 INFO tuned.plugins.hotplug: device 'sdb' added
2018-06-11 08:07:55,806 DEBUG tuned.daemon.daemon: updating monitors
2018-06-11 08:07:55,807 DEBUG tuned.units.manager: updating monitor <tuned.monitors.monitor_disk.DiskMonitor object at 0x7f35c34211d0>
2018-06-11 08:07:55,807 DEBUG tuned.daemon.daemon: performing tunings
2018-06-11 08:08:05,820 DEBUG tuned.daemon.daemon: updating monitors
2018-06-11 08:08:05,820 DEBUG tuned.units.manager: updating monitor <tuned.monitors.monitor_disk.DiskMonitor object at 0x7f35c34211d0>
2018-06-11 08:08:05,821 DEBUG tuned.daemon.daemon: performing tunings
2018-06-11 08:08:05,971 INFO tuned.plugins.hotplug: device 'vde' added
2018-06-11 08:08:05,972 INFO tuned.plugins.hotplug: instance disk: adding new device vde
2018-06-11 08:08:05,972 DEBUG tuned.utils.commands: Read data from file: '<closed file u'/sys/block/vde/queue/read_ahead_kb', mode 'r' at 0x7f35c411bae0>' > '128
'
2018-06-11 08:08:05,972 DEBUG tuned.utils.commands: Writing to file: '/sys/block/vde/queue/read_ahead_kb' < '8192'
2018-06-11 08:08:05,973 DEBUG tuned.utils.commands: Executing ['hdparm', '-B', u'/dev/vde'].
2018-06-11 08:08:15,834 DEBUG tuned.daemon.daemon: updating monitors
2018-06-11 08:08:15,834 DEBUG tuned.units.manager: updating monitor <tuned.monitors.monitor_disk.DiskMonitor object at 0x7f35c34211d0>
2018-06-11 08:08:15,835 DEBUG tuned.daemon.daemon: performing tunings
[...]
The line that appears with a new device:
2018-06-11 08:08:05,972 INFO tuned.plugins.hotplug: instance disk: adding new device vde
The sequence is:
(1) "tuned.plugins.hotplug: device 'vde' added" ---> (2) "tuned.plugins.hotplug: instance disk: adding new device vde"
And as mentioned just above the line marked as (2) does not appear when the values are not applied. The code is in tuned/plugins/hotplug.py:
def _hardware_events_callback(self, event, device):
if event == "add":
log.info("device '%s' added" % device.sys_name) ← ← ← ← (1)
self._add_device(device)
elif event == "remove":
log.info("device '%s' removed" % device.sys_name)
self._remove_device(device)
def _add_device(self, device):
device_name = device.sys_name
if device_name in (self._assigned_devices | self._free_devices): ← ← This is where we decide to apply the rules or not
return
for instance_name, instance in self._instances.items():
if len(self._get_matching_devices(instance, [device_name])) == 1:
log.info("instance %s: adding new device %s" % (instance_name, device_name)) ← ← (2)
self._assigned_devices.add(device_name)
instance.devices.add(device_name)
self._call_device_script(instance, instance.script_pre, "apply", [device_name])
self._added_device_apply_tuning(instance, device_name)
self._call_device_script(instance, instance.script_post, "apply", [device_name])
break
So the existing devices are already in either _assigned_devices or in _free_devices:
if device_name in (self._assigned_devices | self._free_devices)
This is where we should be removing the device from the list:
def _remove_device(self, device):
device_name = device.sys_name
if device_name not in (self._assigned_devices | self._free_devices):
return
for instance in self._instances.values():
if device_name in instance.devices:
self._call_device_script(instance, instance.script_post, "unapply", [device_name])
self._removed_device_unapply_tuning(instance, device_name)
self._call_device_script(instance, instance.script_pre, "unapply", [device_name])
instance.devices.remove(device_name)
instance.active = len(instance.devices) > 0
self._assigned_devices.remove(device_name)
break
else:
self._free_devices.remove(device_name)
And this removal happens when we receive a udev event for 'remove':
def _hardware_events_callback(self, event, device):
if event == "add":
log.info("device '%s' added" % device.sys_name)
self._add_device(device)
elif event == "remove":
log.info("device '%s' removed" % device.sys_name)
self._remove_device(device)
Testing 'remove' and 'add' uevent:
# echo "remove" > /sys/block/sda/uevent
2018-06-11 09:27:43,448 INFO tuned.plugins.hotplug: device 'sda' removed
2018-06-11 09:27:43,448 DEBUG tuned.utils.commands: Writing to file: '/sys/block/sda/queue/read_ahead_kb' < '128'
# echo "add" > /sys/block/sda/uevent
2018-06-11 09:29:00,927 DEBUG tuned.utils.commands: Writing to file: '/sys/block/sda/queue/read_ahead_kb' < '8192'
2018-06-11 09:29:00,931 DEBUG tuned.utils.commands: Executing ['hdparm', '-B', u'/dev/sda'].
On removal we have this in the output of the command 'udevadm monitor':
KERNEL[5622.611054] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0/2:0:0:0/bsg/2:0:0:0 (bsg)
KERNEL[5622.611460] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0/2:0:0:0/scsi_generic/sg1 (scsi_generic)
KERNEL[5622.611471] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0/2:0:0:0/scsi_device/2:0:0:0 (scsi_device)
KERNEL[5622.611478] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0/2:0:0:0/scsi_disk/2:0:0:0 (scsi_disk)
KERNEL[5622.611538] remove /devices/virtual/bdi/8:0 (bdi)
KERNEL[5622.611580] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0/2:0:0:0/block/sda (block)
KERNEL[5622.614160] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0/2:0:0:0 (scsi)
UDEV [5622.614790] remove /devices/virtual/bdi/8:0 (bdi)
KERNEL[5622.617180] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0 (scsi)
UDEV [5622.617805] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0/2:0:0:0/scsi_disk/2:0:0:0 (scsi_disk)
UDEV [5622.618328] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0/2:0:0:0/scsi_device/2:0:0:0 (scsi_device)
UDEV [5622.618751] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0/2:0:0:0/block/sda (block)
UDEV [5622.620970] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0/2:0:0:0/bsg/2:0:0:0 (bsg)
UDEV [5622.764687] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0/2:0:0:0/scsi_generic/sg1 (scsi_generic)
UDEV [5622.765310] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0/2:0:0:0 (scsi)
UDEV [5622.765554] remove /devices/pci0000:00/0000:00:0c.0/virtio7/host2/target2:0:0 (scsi)
SBR
Product(s)
Components
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.