System crashes when removing large files on 3PAR disk that is mounted with discard option.

Environment

• Red Hat Enterprise Linux 6
• HP Proliant BL460 Gen8
• kernel 2.6.32-279.19.1.el6

Issue

• When hung_task_panic is enabled, system crashes while removing large files on 3PAR disk that is mounted with discard option.

INFO: task rm:24315 blocked for more than 300 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this
message.
rm		D 0000000000000001     0 24315	13602 0x00000004
 ffff880818d11d08 0000000000000082 0000000000000000 ffff880813ce4dc8
 ffff880817ec4ae0 ffff88102593dc00 0000000000001000 ffff880822baaac0
 ffff88082a817ab8 ffff880818d11fd8 000000000000fb88 ffff88082a817ab8
Call Trace:
 [<ffffffffa013604a>] start_this_handle+0x25a/0x480 [jbd2]
 [<ffffffff81090857>] ? bit_waitqueue+0x17/0xd0
 [<ffffffff81090990>] ? autoremove_wake_function+0x0/0x40
 [<ffffffffa0136455>] jbd2_journal_start+0xb5/0x100 [jbd2]
 [<ffffffffa017dbd8>] ext4_journal_start_sb+0x58/0x90 [ext4]
 [<ffffffffa016cb7d>] ext4_unlink+0x9d/0x2b0 [ext4]
 [<ffffffff81183d5f>] vfs_unlink+0x9f/0xe0
 [<ffffffff81182aaa>] ? lookup_hash+0x3a/0x50
 [<ffffffff81186293>] do_unlinkat+0x183/0x1c0
 [<ffffffff811b5d0e>] ? dnotify_flush+0x7e/0x140
 [<ffffffff8117389d>] ? filp_close+0x5d/0x90
 [<ffffffff81186432>] sys_unlinkat+0x22/0x40
 [<ffffffff8100b072>] system_call_fastpath+0x16/0x1b
Kernel panic - not syncing: hung_task: blocked tasks

• DISCARDs cause a continuous stream of errors if filesystem is mounted with "-o discard" and also cause mkfs to fail compeletely with the "-E discard" option.

Resolution

Upgrade to kernel-2.6.32-431.el6, released with RHSA-2013-1645.

Root Cause

Storage that reports a discard_granularity that is not a power of two will cause the kernel to improperly issue discard requests to the underlying storage. This results in I/O errors associated with the failed discard requests.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.