Getting error "kernel: blk_rq_check_limits: over max size limit" when changing the max_sectors_kb value on multipath devices or their underlying individual paths

Environment

• Red Hat Enterprise Linux 8
• Red Hat Enterprise Linux 7
• Red Hat Enterprise Linux 6
• Red Hat Enterprise Linux 5
• DM-Multipath

Issue

• We are receiving the following errors when accessing LUNS.
  This only happens when setting /sys/block/dm-\*/slaves/sd\*/queue/max_sectors_kb to 128.

    Jun 14 14:54:55 server kernel: blk_rq_check_limits: over max size limit.
    Jun 14 14:54:55 server kernel: blk_rq_check_limits: over max size limit.
  

  This does not occur when we leave the max_sectors_kb at 512.
  the error is causing path failures and paths fail over.

• With kernel-2.6.32-642.el6 and later, there is a slightly different error logged:

    Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
    Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
  

Resolution

• There are two ways the value of max_sectors_kb can be changed without triggering this issue:

1. Multipath will determine a valid value for its max_sectors_kb when a multipath device is created.
   If the max_sectors_kb value for the paths are changed before the multipath device is created, it will be
   created with a workable value.

2. If changing values after multipath is running to reduce the max_sectors_kb value, the value on multipath's
   queue should be changed first, then the paths. If increasing the value, change the value on the paths first,
   then multipath's value. See "How to set custom 'max_sectors_kb' option for devices under multipathd control? "
   for additional information, including the safe upper limit for max_sectors_kb.

Root Cause

• Reducing the max_sectors_kb value of a multipath path or increasing the max_sectors_kb value of the multipath device can break the stacking between multipath's request queue and the path's request queue.

• In Red Hat Enterprise Linux 6, multipath's queue builds requests and sends the assembled request down to a path's queue. If the values of max_sectors_kb are out of sync, it can result in multipath building requests larger than the path's queue will accept.
  When multipath then sends the too big request, it's rejected and fails with the blk_rq_check_limits: over max size limit message.

Diagnostic Steps

• The message blk_rq_check_limits: over max size limit is an indication a block limit has been violated. This usually happens when limits have been changed or mis-calculated with stacked devices (e.g. multipath, md).

• Getting copies of /sys/block/\*/queue/max_sectors_kb should show the discrepancy between the lower block devices (e.g. sd*) and the block devices built on top.

• If the issue is encountered, the below script can be run which outputs file /tmp/data.tgz. For a more speedy resolution, provide to Red Hat Support a sosreport from the system and the resulting data.tgz file.

      #!/bin/bash
  
      for dev in $(ls /sys/block | egrep 'dm-|md|sd|nvme')
      do
          for val in $(ls /sys/block/$dev/queue/ | grep -v iosched)
          do
              echo /dev/$dev: $val: `cat /sys/block/$dev/queue/$val` &>> /tmp/queue_vals.out
          done
      done
  
      lvs -ao +devices &> /tmp/lvs.out
      pvs -a -v &> /tmp/pvs.out
      dmsetup info -C &> /tmp/dmsetup_info.out
      dmsetup table &> /tmp/dmsetup_table.out
      dmsetup status &> /tmp/dmsetup_status.out
  
      cat /proc/mdstat &> /tmp/mdstat.out
      ls -laR /dev &> /tmp/ls_dev.out
      cat /proc/partitions &> /tmp/partitions.out
      lsblk &> /tmp/lsblk.out
      dmesg &> /tmp/dmesg.out
  
      tar -zcf /tmp/data.tgz /tmp/*.out
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.