Virtual Machine loses access to its disks after hot-plugging some extra disks.

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 9.2
  • Red Hat OpenStack Platform (RHOSP) 17.1
  • Red Hat OpenShift Container Platform (RHOCP) 4.13, 4.14
    • OpenShift Virtualization

Issue

  • OpenShift worker nodes running on OpenStack are hanging and getting corrupted.
  • Hosted Cluster (Hosted Control Planes) nodes regularly hang when running on OpenShift Virtualization
  • The OS Disk of the Virtual Machine stops working after hotplug of some other extra disk:
[  272.879151] sd 0:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[  283.104067] sd 0:0:0:0: [sda] tag#112 abort
[  283.109673] sd 0:0:0:0: Warning! Received an indication that the LUN assignments on this target have changed. The Linux SCSI layer does not automatical
[  283.122063] sd 0:0:0:0: device reset
[  293.344050] sd 0:0:0:0: [sda] tag#112 abort
[  293.348559] sd 0:0:0:0: Device offlined - not ready after error recovery
[  293.354367] sd 0:0:0:0: Device offlined - not ready after error recovery
[  293.360096] scsi_io_completion_action: 2 callbacks suppressed
[  293.360099] sd 0:0:0:0: [sda] tag#113 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK cmd_age=20s
[  293.372435] sd 0:0:0:0: [sda] tag#113 CDB: Write(10) 2a 00 00 a3 04 58 00 00 38 00
[  293.378486] print_req_error: 10 callbacks suppressed
[  293.378487] blk_update_request: I/O error, dev sda, sector 10683480 op 0x1:(WRITE) flags 0x100000 phys_seg 7 prio class 0
[  293.391906] sd 0:0:0:0: rejecting I/O to offline device
[  293.402309] blk_update_request: I/O error, dev sda, sector 5850816 op 0x1:(WRITE) flags 0x100000 phys_seg 42 prio class 0
[  293.420761] blk_update_request: I/O error, dev sda, sector 10683440 op 0x1:(WRITE) flags 0x100000 phys_seg 1 prio class 0
[  293.439903] sda3: writeback error on inode 16912535, offset 4096, sector 10683448
[  293.439910] sda3: writeback error on inode 16912535, offset 8192, sector 10683536
[  293.448109] sda3: writeback error on inode 8900865, offset 221184, sector 5850824
[  293.455655] sda3: writeback error on inode 8900865, offset 225280, sector 5851168
[  293.497802] blk_update_request: I/O error, dev sda, sector 10607666 op 0x1:(WRITE) flags 0x9800 phys_seg 1 prio class 0
[  293.515957] XFS (sda3): log I/O error -5
[  293.519904] XFS (sda3): Log I/O error (0x2) detected at xlog_ioend_work+0x74/0x80 [xfs] (fs/xfs/xfs_log.c:1284). Shutting down filesystem
[  293.530178] XFS (sda3): Please unmount the filesystem and rectify the problem(s)

Resolution

  • For standalone RHEL KVM, upgrading to RHEL 9.3 fixes the problem, and it is also fixed in 9.2.z by RHBA-2024:0849.
  • Use virtio-blk as a workaround, if possible.
  • RHOSP version 17.1.3+ contains this fix.
  • OpenShift Virtualization versions 4.13.9+ and 4.14.4+ contain this fix.

Root Cause

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.