Node using iSCSI LUNs has reservation conflicts during VMware virtual machine snapshot

Solution Unverified - Updated

Environment

  • Red Hat Enterprise Linux Server 7 (with the High Availability Add-on)
  • Red Hat Enterprise Linux Server 8 (with the High Availability Add-on)

Issue

  • Nodes fence each other repeatedly after snapshot backups start.
  • GFS2 withdrawals during snapshots
  • Cluster node has SCSI reservation conflicts during VMware virtual machine snapshot.
  • Virtual guest with iSCSI initiator hangs during snapshot.

Resolution

Per VMware's documentation, "VMware does not support snapshots of raw disks, RDM physical mode disks, or guest operating systems that use an iSCSI initiator in the guest." In light of this documented limitation, VMware guests should utilize a different backup mechanism for VMs that use iSCSI storage. Affected nodes should be excluded from virtual machine snapshot jobs.

Root Cause

Per VMware's Content from docs.vmware.com is not included.Snapshot Limitations document, "guest operating systems that use an iSCSI initiator in the guest" are unsupported.

Diagnostic Steps

  1. Check whether the system is using an iSCSI initiator.

     # systemctl is-active iscsi
 active
 # systemctl is-active iscsid
 active
 # iscsiadm -m session
 tcp: [1] 192.168.122.52:3260,1 iqn.1994-05.com.redhat:71fc25b8ad3 (non-flash)

  2. Find the time the snapshot began.

    alt_text

  3. Correlate snapshot start time with SCSI reservation conflict start time in /var/log/messages or journalctl.

     Feb  2 18:26:05 node-a kernel: sd 3:0:0:2: reservation conflict
 Feb  2 18:26:05 node-a kernel: scsi_io_completion: 67 callbacks suppressed
 Feb  2 18:26:05 node-a kernel: sd 3:0:0:2: [sdh] FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_OK
 Feb  2 18:26:05 node-a kernel: sd 3:0:0:2: [sdh] CDB: Write(16) 8a 00 00 00 00 00 00 01 d6 c0 00 00 00 08 00 00
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.