Kernel reporting "Buffer I/O error on dev xxx"

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 10
  • Red Hat Enterprise Linux 9
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 6
  • SCSI and NVMe disks

Issue

  • Kernel reporting "Buffer I/O error on dev xxx".
Feb  4 11:17:56 test kernel: end_request: I/O error, dev sdc, sector 48
Feb  4 11:17:56 test kernel: Buffer I/O error on device sdc, logical block 6
Feb  4 11:17:56 test kernel: Buffer I/O error on device sdc, logical block 7
Feb  4 11:17:56 test kernel: Buffer I/O error on device sdc, logical block 8
Feb  4 11:17:56 test kernel: Buffer I/O error on device sdc, logical block 9

Feb  4 12:52:52 test kernel: Buffer I/O error on device sdc, logical block 268435440
Feb  4 12:52:52 test kernel: end_request: I/O error, dev sde, sector 0
Feb  4 12:52:52 test kernel: end_request: I/O error, dev sdc, sector 2147483640
Feb  4 12:52:52 test kernel: end_request: I/O error, dev sde, sector 0
Feb  4 12:52:53 test kernel: end_request: I/O error, dev sdc, sector 2147483640


Nov 11 19:23:49 hostname kernel: blk_update_request: I/O error, dev nvme2n1, sector 3125627392
Nov 11 19:23:49 hostname kernel: blk_update_request: I/O error, dev nvme2n1, sector 3125627392
Nov 11 19:23:49 hostname kernel: Buffer I/O error on dev nvme2n1, logical block 390703424, async page read
Nov 11 19:23:49 hostname kernel: blk_update_request: I/O error, dev nvme2n1, sector 3125627392
Nov 11 19:23:49 hostname kernel: Buffer I/O error on dev nvme2n1, logical block 390703424, async page read

Nov 11 19:23:50 hostname kernel: blk_update_request: I/O error, dev nvme2n1, sector 0
Nov 11 19:23:50 hostname kernel: blk_update_request: I/O error, dev nvme2n1, sector 0
Nov 11 19:23:50 hostname kernel: Buffer I/O error on dev nvme2n1, logical block 0, async page read
Nov 11 19:23:50 hostname kernel: blk_update_request: I/O error, dev nvme2n1, sector 0
Nov 11 19:23:50 hostname kernel: Buffer I/O error on dev nvme2n1, logical block 0, async page read
Nov 11 19:23:50 hostname kernel: blk_update_request: I/O error, dev nvme2n1, sector 0
Nov 11 19:23:50 hostname kernel: Buffer I/O error on dev nvme2n1, logical block 0, async page read
Nov 11 19:23:50 hostname kernel: blk_update_request: I/O error, dev nvme2n1, sector 0
Nov 11 19:23:50 hostname kernel: Buffer I/O error on dev nvme2n1, logical block 0, async page read

Resolution

  • Engage Hardware Vendor to run hardware diagnostic tools.

  • Use the disk checking tool badblocks to scan the specified hard disk block by block. For example, to scan /dev/hda issue the commands:

      # mount | grep hda                  # find all mounted partitions of hda
  # umount /dev/hda1                  # unmount the partitions (may be more then one)
  # badblocks -n -vv /dev/hda

  • Where -n is use non-destructive read-write mode. By default only a non-destructive read-only test is done.

  • If the messages similar to the examples found below appear in /var/log/messages or to the console following the running of badblocks it is recommended to backup any data on the affected devices and replace the device: 

      Feb  4 13:50:40 test kernel: hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
  Feb  4 13:50:40 test kernel: hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=74367249, sector=74367232
  Feb  4 13:50:40 test kernel: ide: failed opcode was: unknown
  Feb  4 13:50:40 test kernel: end_request: I/O error, dev hda, sector 74367232
  Feb  4 13:50:42 test kernel: hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }
  Feb  4 13:50:42 test kernel: hda: dma_intr: error=0x40 { UncorrectableError }, LBAsect=74367249, sector=74367240
  Feb  4 13:50:42 test kernel: ide: failed opcode was: unknown
  Feb  4 13:50:42 test kernel: end_request: I/O error, dev hda, sector 74367240
  Feb  4 13:50:44 test kernel: hda: dma_intr: status=0x51 { DriveReady SeekComplete Error }

Additional Notes

  • If the output of badblocks is going to be fed to the e2fsck or mke2fs programs, it is important that the block size is properly specified, since the block numbers which are generated are very dependent on the block size in use by the filesystem. For this reason, it is strongly recommended that users not run badblocks directly, but rather use the -c option of the e2fsck and mke2fs programs.
  • Never use the -w option on a device containing an existing file system. This option erases data! If write-mode testing needs to be performed on an existing file system, use the -n option instead. It is slower, but it will preserve the data.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.