multipath devices failing with error - 'blk_cloned_rq_check_limits: over max size limit.'

Environment

• Red Hat Enterprise Linux 6.9

Issue

• Every day multipathd loses many paths with blk_cloned_rq_check_limits errors as shown below:

    Jan 11 13:15:09 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
    Jan 11 13:15:09 testhost kernel: device-mapper: multipath: Failing path 130:0.
    Jan 11 13:15:09 testhost multipathd: 130:0: mark as failed
    Jan 11 13:15:09 testhost multipathd: mpathag: remaining active paths: 3
    Jan 11 13:15:09 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
    Jan 11 13:15:09 testhost kernel: device-mapper: multipath: Failing path 129:160.
    Jan 11 13:15:09 testhost multipathd: 129:160: mark as failed
    Jan 11 13:15:09 testhost multipathd: mpathy: remaining active paths: 3
    Jan 11 13:15:09 testhost kernel: blk_cloned_rq_check_limits: over max size limit.
  

  [...]

• We have already configured 4096 max_sectors_kb value in /etc/multipath.conf file, added a udev rule as well to tune the max_sectors_kb value as shown in "How to set custom 'max_sectors_kb' option for devices under multipathd control?" but still the same issue persists.

Resolution

• Remove/comment out the commands in custom cronjobs/scritps which were causing a change in max_sectors_kb value for disk devices:

    [sosreport]$ cat /etc/cron.daily/script.sh
    [...]
    for i in $(find /sys/block/ -name "sd*" -o -name "em*") ; do
    	if [ "$(cat ${i}/queue/max_sectors_kb)" != "128" ]; then	<---
    		echo 128 > ${i}/queue/max_sectors_kb			<---
    	fi								<---
    [...]
  

• After commenting out above lines, please reload the crond service to make the changes effective:

    $ /etc/init.d/crond reload
  

Root Cause

• The max_sectors_kb value for disk devices was getting changed from following line in custom cronjob script /etc/cron.daily/script.sh

    [sosreport]$ cat /etc/cron.daily/script.sh
    [...]
    for i in $(find /sys/block/ -name "sd*" -o -name "em*") ; do
    	if [ "$(cat ${i}/queue/max_sectors_kb)" != "128" ]; then	<---
    		echo 128 > ${i}/queue/max_sectors_kb			<---
    	fi								<---
    [...]
  

• Above custom script script.sh was was running on daily basis:

    $ less /var/log/cron
    ...
    run-parts(/etc/cron.daily)[43339]: starting script.sh
    run-parts(/etc/cron.daily)[70497]: finished script.sh
    run-parts(/etc/cron.daily)[52173]: starting script.sh
    run-parts(/etc/cron.daily)[80047]: finished script.sh
    ...
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.