Getting 'blk_cloned_rq_check_limits: over max size limit' errors after updating the systems to kernel 2.6.32-642.el6 or later

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 6.8, 7.3
    o kernel-2.6.32-642.el6 or later
    o kernel-3.10.0-327.el7 or later
  • IBM GPFS version earlier than 4.1.1.8, 4.2.1.

Issue

  • After updating the system to kernel-2.6.32-642.el6 or later, there are following errors getting logged and this is causing dm-multipath path failures:

      Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
  Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
  Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
  Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
  Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
  Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
  Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
  Jun 14 09:33:38 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
  Jun 14 09:33:38 testhost kernel: device-mapper: multipath: Failing path 70:64.
  Jun 14 09:33:38 testhost kernel: device-mapper: multipath: Failing path 71:80.
  Jun 14 09:33:38 testhost multipathd: 70:64: mark as failed
  Jun 14 09:33:38 testhost multipathd: nsd_crs_04: remaining active paths: 1
  Jun 14 09:33:38 testhost multipathd: 71:80: mark as failed
  [...]

Resolution

  • IBM GPFS filesystem for use with RHEL 6.8 from version 4.1.1.8 and 4.2.1 erroneously reports this as fixed..
    This has now been confirmed by IBM in fact not to be the case. Currently IBM GPFS team members are still working on this issue.
    This issue remains an issue for GPFS filesystem deployments and will require the workaround documented in Resolution section.

    Given the stacking nature of DM multipath (where DM multipath layers ontop of all the individual underlying SCSI devices) the proper place to modify the max_sectors_kb parameter is at the underlying SCSI devices.

Workaround 1:

Workaround 2:

(without using the updated device-mapper-multipath rpms as described in Workaround 1)

  • Use following udev rule to increase max_sectors_kb value for underlying sub paths to the SAN devices, this should help us to make sure that request_queue limits are not violated:

    1. Create a new udev rule file /etc/udev/rules.d/99-custom.rules with the following content:

           ACTION!="add|change", GOTO="rule_end"
     ENV{ID_VENDOR}=="NETAPP*",  RUN+="/bin/sh -c 'echo 4096 > /sys%p/queue/max_sectors_kb'"
     LABEL="rule_end"

      Note: Replace NETAPP in above line with name of the vendor of SAN devices in your setup/environment.

    2. If the system is not configured to boot from SAN, then this step can be skipped.

    For boot from SAN systems, please also rebuild the initial ram disk image with following commands so that initramfs image will have above udev rule file to set `max_sectors_kb` limit during early stage in boot process when multipath devices are created:

 First take a backup of current initramfs image:

          $ cp  -av  /boot/initramfs-$(uname -r).img  /boot/initramfs-$(uname -r).img.bak

 Rebuild the initramfs image with udev rule file included:

          $ dracut -f -v -i /etc/udev/rules.d/99-custom.rules /etc/udev/rules.d/99-custom.rules
    1. Once above udev rule is added, please try to reboot the system and this error messages should no longer occur.

Root Cause

  • The error messages blk_cloned_rq_check_limits: over max size limit are logged from the following code block in blk_cloned_rq_check_limits:

      block/blk-core.c
  1870 static int blk_cloned_rq_check_limits(struct request_queue *q,
  1871                                       struct request *rq)
  1872 {
  1873         if (blk_rq_sectors(rq) > blk_queue_get_max_sectors(q, rq->cmd_flags)) {
  1874                 printk(KERN_ERR "%s: over max size limit.\n", __func__);
  1875                 return -EIO;		<----------
  1876         }
  [...]

  • The check in above snip calls blk_rq_sectors to verify number of sectors in cloned request and compare it with max_discard_sectors or max_sectors limits for the request_queue. If it finds that number of sectors in cloned request are more than max_discard_sectors or max_sectors limits for the request_queue, then we print error and return IO error (-EIO) The check marked in above blk_cloned_rq_check_limits function was added in RHEL 6.8, 7.3 kernels through following commits:

          Upstream commit bf4e6b4e757488dee1b6a581f49c7ac34cd217f8
  	block: Always check queue limits for cloned requests

  • Since the cloned request was violating the request_queue limits, kernel had reported blk_cloned_rq_check_limits: over max size limit errors. This subsequently resulted in path failure in multipath.

    Above patch was not present in the kernel versions older than 2.6.32-642.el6, 3.10.0-327.el7, due to which the same request queue limit violations were not resulting IO failures with old kernels.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.