Warning: block/blk-core.c:2656 blk_update_request
Environment
- Red Hat Enterprise Linux for ARM 64 7.4 aarch64
- Red Hat Enterprise Linux for Power 9 7.4 ppc64le
- kernel-4.11.0-44.el7a
Issue
- Warning seen in logs:
Nov 23 03:52:10 localhost kernel: WARNING: CPU: 17 PID: 0 at block/blk-core.c:2656 blk_update_request+0x5a8/0x5b0
Nov 23 03:52:10 localhost kernel: Modules linked in: binfmt_misc vfat fat xfs fcoe libfcoe libfc scsi_transport_fc ast i2c_algo_bit ttm drm_kms_helper at24 nvmem_core drm fb_sys_fops syscopy
area ses sysfillrect enclosure sysimgblt scsi_transport_sas sg ofpart i2c_opal powernv_flash shpchp ipmi_powernv opal_prd ipmi_devintf i2c_core mtd uio_pdrv_genirq ipmi_msghandler uio ext4 j
bd2 mbcache loop nls_utf8 isofs sr_mod cdrom uas usb_storage sd_mod i40e aacraid ptp pps_core sunrpc xts lrw gf128mul dm_crypt dm_round_robin dm_multipath dm_snapshot dm_bufio dm_mirror dm_r
egion_hash dm_log dm_zero dm_mod linear raid10 raid456 async_raid6_recov async_memcpy libcrc32c async_pq async_xor xor async_tx raid6_pq raid1 raid0 iscsi_tcp libiscsi_tcp libiscsi scsi_tran
sport_iscsi squashfs cramfs ctr(E)
Nov 23 03:52:10 localhost kernel: CPU: 17 PID: 0 Comm: swapper/17 Tainted: G E ------------ 4.11.0-44.el7a.ppc64le #1
Nov 23 03:52:10 localhost kernel: task: c00000078d625200 task.stack: c00000078d6ec000
Nov 23 03:52:10 localhost kernel: NIP: c0000000005f3b08 LR: c0000000005f3828 CTR: c0000000005fb9b0
Nov 23 03:52:10 localhost kernel: REGS: c00000000ff73890 TRAP: 0700 Tainted: G E ------------ (4.11.0-44.el7a.ppc64le)
Nov 23 03:52:10 localhost kernel: MSR: 9000000000029033 <SF,HV,EE,ME,IR,DR,RI,LE>
Nov 23 03:52:10 localhost kernel: CR: 48042882 XER: 00000000
Nov 23 03:52:10 localhost kernel: CFAR: c0000000005f38f4 SOFTE: 1
GPR00: c0000000005f3828 c00000000ff73b10 c000000001403100 c00000076f821b00
GPR04: 0000000000000000 0000000000000000 c000000001373100 0000000000000001
GPR08: 0000000000047087 0000000000000000 c00000076f821b00 c008000008b79460
GPR12: 0000000000004000 c00000000fb4a180 0000000000200042 00000000fffffdaf
GPR16: c00000000ff70000 0000000000000000 c000000000fe4080 c000000001433b00
GPR20: c000000000fe4080 000000000000000a 0000000000000005 0000000000000bb8
GPR24: 0000000008100002 0000000000000000 0000000000000000 0000000000000000
GPR28: 0000000000000000 c00000076f821b00 0000000000000000 c00000076f80d800
Nov 23 03:52:10 localhost kernel: NIP [c0000000005f3b08] blk_update_request+0x5a8/0x5b0
Nov 23 03:52:10 localhost kernel: LR [c0000000005f3828] blk_update_request+0x2c8/0x5b0
Nov 23 03:52:10 localhost kernel: Call Trace:
Nov 23 03:52:10 localhost kernel: [c00000000ff73b10] [c0000000005f3828] blk_update_request+0x2c8/0x5b0 (unreliable)
Nov 23 03:52:10 localhost kernel: [c00000000ff73ba0] [c00000000083b2ec] scsi_end_request+0x6c/0x2c0
Nov 23 03:52:10 localhost kernel: [c00000000ff73c00] [c00000000083e004] scsi_io_completion+0x204/0x9d0
Nov 23 03:52:10 localhost kernel: [c00000000ff73cc0] [c00000000082ed4c] scsi_finish_command+0x10c/0x1b0
Nov 23 03:52:10 localhost kernel: [c00000000ff73d40] [c00000000083b948] scsi_softirq_done+0x198/0x200
Nov 23 03:52:10 localhost kernel: [c00000000ff73dc0] [c000000000604d20] blk_done_softirq+0xd0/0x100
Nov 23 03:52:10 localhost kernel: [c00000000ff73e00] [c00000000012f9b4] __do_softirq+0x164/0x420
Nov 23 03:52:10 localhost kernel: [c00000000ff73ef0] [c0000000001301b4] irq_exit+0x1e4/0x1f0
Nov 23 03:52:10 localhost kernel: [c00000000ff73f20] [c0000000000167e0] __do_irq+0xa0/0x200
Nov 23 03:52:10 localhost kernel: [c00000000ff73f90] [c00000000002d510] call_do_irq+0x14/0x24
Nov 23 03:52:10 localhost kernel: [c00000078d6efaa0] [c0000000000169e0] do_IRQ+0xa0/0x120
Nov 23 03:52:10 localhost kernel: [c00000078d6efb00] [c000000000009a74] h_virt_irq_common+0x114/0x120
Nov 23 03:52:10 localhost kernel: --- interrupt: ea1 at arch_local_irq_restore+0x68/0x130
LR = cpuidle_enter_state+0x114/0x450
Nov 23 03:52:10 localhost kernel: [c00000078d6efdf0] [c000000000992a3c] cpuidle_enter_state+0xec/0x450 (unreliable)
Nov 23 03:52:10 localhost kernel: [c00000078d6efe50] [c0000000001a6c60] do_idle+0x350/0x3e0
Nov 23 03:52:10 localhost kernel: [c00000078d6efef0] [c0000000001a6f58] cpu_startup_entry+0x38/0x40
Nov 23 03:52:10 localhost kernel: [c00000078d6eff20] [c000000000053074] start_secondary+0x314/0x370
Nov 23 03:52:10 localhost kernel: [c00000078d6eff90] [c00000000000a96c] start_secondary_prolog+0x10/0x14
Nov 23 03:52:10 localhost kernel: Instruction dump:
Nov 23 03:52:10 localhost kernel: 61290040 b13d0018 4bfffd3c 3cc2ff9d 38c64620 4bfffb68 3cc2fff7 88e666f1
Nov 23 03:52:10 localhost kernel: 2f870000 40fefdfc 38e00001 98e666f1 <0fe00000> 4bfffdec 3c4c00e1 3842f5f0
- Also, 'Write Same' failures seen:
Nov 23 03:52:10 localhost kernel: sd 0:2:14:0: [sda] tag#8 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Nov 23 03:52:10 localhost kernel: sd 0:2:14:0: [sda] tag#8 Sense Key : Illegal Request [current]
Nov 23 03:52:10 localhost kernel: sd 0:2:14:0: [sda] tag#8 Add. Sense: Invalid field in cdb
Nov 23 03:52:10 localhost kernel: sd 0:2:14:0: [sda] tag#8 CDB: Write same(16) 93 08 00 00 00 00 03 a0 27 f9 00 40 00 07 00 00
Nov 23 03:52:10 localhost kernel: blk_update_request: critical target error, dev sda, sector 60827641
Nov 23 03:52:10 localhost kernel: sd 0:2:14:0: [sda] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Nov 23 03:52:10 localhost kernel: sd 0:2:14:0: [sda] tag#7 Sense Key : Illegal Request [current]
Nov 23 03:52:10 localhost kernel: sd 0:2:14:0: [sda] tag#7 Add. Sense: Invalid field in cdb
Nov 23 03:52:10 localhost kernel: sd 0:2:14:0: [sda] tag#7 CDB: Write same(16) 93 08 00 00 00 00 03 20 27 fa 00 7f ff ff 00 00
Nov 23 03:52:10 localhost kernel: blk_update_request: critical target error, dev sda, sector 52439034
Nov 23 03:52:10 localhost kernel: sd 0:2:14:0: [sda] tag#6 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Nov 23 03:52:10 localhost kernel: sd 0:2:14:0: [sda] tag#6 Sense Key : Illegal Request [current]
Nov 23 03:52:10 localhost kernel: sd 0:2:14:0: [sda] tag#6 Add. Sense: Invalid field in cdb
Nov 23 03:52:10 localhost kernel: sd 0:2:14:0: [sda] tag#6 CDB: Write same(16) 93 08 00 00 00 00 02 a0 27 fb 00 7f ff ff 00 00
Resolution
- Please reach out to your hardware vendor for hardware and firmware updates/diagnostics. These errors represent a partial request completion being returned from storage due to an illegal request of the 'Write Same' type.
- Further information on the 'critical target` and 'Illegal Request' errors can be found here:
- In the event it is expected that the storage fail the 'Write Same' commands, the following articles can be used to disable sending these types of requests.
Root Cause
Content from patchwork.kernel.org is not included.block: improve handling of the magic discard payload introduced better handling when dealing with discard (write same) payloads. These special requests are flagged as RQF_SPECIAL_PAYLOAD. If we end up in the blk_update_request() function it means that a partial completion has taken place. At the time of this patch, it was believed that this would not happen with the magic discard payloads. However, this was incorrect, proven by the warning posted.
Once this was discovered, further patches were written to handle this case if a partial completion is returned from a RQF_SPECIAL_PAYLOAD request. This was done via Content from patchwork.kernel.org is not included.block: handle partial completions for special payload requests.
SBR
Components
Category
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.