I/O hangs with 3PAR storage with SCSI error in "CDB: Unmap/Read...Sense Key: Illegal Request" in RHEL 6.3 and earlier

Environment

• Red Hat Enterprise Linux (RHEL) 6 Update 3 or earlier
  • kernel prior to release 2.6.32-279.el6
• 3PAR storage
• SCSI errors with Sense information matching the following:

sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 1:0:0:0: [sdb] CDB: Unmap/Read sub-channel: 42 00 00 00 00 00 00 00 18 00
sd 1:0:0:0: [sdb] Sense Key : Illegal Request [current]
sd 1:0:0:0: [sdb] Add. Sense: Incompatible volume type

Issue

• mkfs.ext4 never completes on 3Par storage
• mkfs.ext4 hangs, with repeating error messages in the logs
• booting hangs, with SCSI errors and sense keys indicating Unmap was an illegal request:

sd 1:0:0:0: [sdb] Done: SUCCESS
sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
sd 1:0:0:0: [sdb] CDB: Unmap/Read sub-channel: 42 00 00 00 00 00 00 00 18 00
sd 1:0:0:0: [sdb] Sense Key : Illegal Request [current]
sd 1:0:0:0: [sdb] Add. Sense: Incompatible volume type
sd 1:0:0:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
sd 1:0:0:0: [sdb] Sense Key : Illegal Request [current]
sd 1:0:0:0: [sdb] Add. Sense: Incompatible volume type
sd 1:0:0:0: [sdb] CDB: Unmap/Read sub-channel: 42 00 00 00 00 00 00 00 18 00
end_request: I/O error, dev sdb, sector 32768
device-mapper: multipath: Failing path 8:16.

• Paths are failing and reinstating over and over on 3Par storage

Resolution

• Update to kernel-2.6.32-279.el6 or later to resolve the path failures.

• If mkfs.ext4 is triggering this problem, as a workaround, add the option "-E nodiscard" to prevent the use of UNMAP commands.

• It is not yet known how to correct the problem of the kernel detecting the storage supports UNMAP when in reality it does not

Root Cause

3Par storage is for some reason reporting that it supports UNMAP (Discard, or Trim) commands, but then is rejecting those commands once received. In versions of the kernel prior to RHEL 6 Update 3, there is an issue in which it will treat errors such as these as path failures, which can cause the error to propagate to all paths constantly, resulting in a continuous loop of submitting I/O, queuing it once all paths fail, resubmitting it after paths return, queuing it after paths fail, etc. Later versions of the kernel will treat this as a target failure and pass the I/O error back up to the application, causing it to handle it rather than for it to fail continuously.

Red Hat is investigating the reasons for the kernel detecting the storage supporting UNMAP when it does not.

Diagnostic Steps

• Check to see if the paths are being presented as thinly provisioned by 3Par

# sg_readcap -l /dev/sdX   <-where X is the sd umerated path of mpio
Read Capacity results:
   Protection: prot_en=0, p_type=0, p_i_exponent=0
   Thin provisioning: tpe=1, tprz=1
   Last logical block address=587202559 (0x22ffffff), Number of logical blocks=587202560
   Logical block length=512 bytes
   Logical blocks per physical block exponent=0
   Lowest aligned logical block address=0

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.