Kernel messages "bio too big device dm-10 (512 > 256)" on RHEL 5 and 6

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 5.6
  • Red Hat Enterprise Linux (RHEL) 6
  • SAN storage connected to HP EVA6400 storage array

Issue

  • kernel log "bio too big device dm-10 (512 > 256)" are appearing in the messages file.
  • task blocked for more than 120 seconds error messages start to appear in /var/log/messages

Resolution

Remove any rc.local override entries shown below, and allow the system to use the default of 512kB for all storage devices following a reboot.

# sed -i "/max_sectors_kb/d" /etc/rc.d/rc.local
# reboot

This loop will restore the default values of 512kB to a running system.

# for DEVICE in {a..z}; do
  echo 512 > /sys/block/sd$DEVICE/queue/max_sectors_kb
done

Additional cleanup may be required if LVM is used on the reconfigured block devices. After the re-initalization of the LUN's that went offline, the I/O limit is set to 512. After running pvcreate, vgcfgrestore, and vcghange -ay, the underlying physical devices have the default max_sectors_kb size rather than the altered 128kb.

Root Cause

A site-specific override was added to the /etc/rc.d/rc.local file. This reconfiguration was causing multipathd to show errors when doing an online reconfiguration of disks. Removing those lines from /etc/rc.d/rc.local and rebooting, or changing the values as shown above on a running system, will stop the block I/O errors and read/write problems.

Diagnostic Steps

Observe kernel output from /var/log/messages:

# grep "bio too big" /var/log/messages
Aug 30 10:31:05 host5 kernel: bio too big device dm-13 (272 > 256)
Aug 30 10:31:05 host5 kernel: bio too big device dm-12 (512 > 256)
Aug 30 10:31:05 host5 kernel: bio too big device dm-13 (424 > 256)

Identify how the max_sectors_kb value was changed from the default 512kB to something smaller:

# grep "max_sectors_kb" /etc/rc.d/rc.local
echo 128 > /sys/block/sda/queue/max_sectors_kb
echo 128 > /sys/block/sdb/queue/max_sectors_kb
echo 128 > /sys/block/sdc/queue/max_sectors_kb
Product(s)
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.