The 7.3 kernel with the HPSA driver may advertise a higher value for a virtual device with many back-end devices

Environment

7.3 kernel and server with HPSA serving virtual devices with higher back-end device counts will see this larger size assigned on boot for the SCSI HPSA virtual device that the server accesses.

Issue

7.3 kernel with HPSA driver sees higher max_sectors_kb sizes assigned for the virtual device as number of back-end devices is increased.
This will result in coalescing of I/O so larger I/O is submitted to the device resulting in higher latency per I/O for transaction based workloads.

Resolution

Check the device size using

 cat /sys/block/sdxx/queue/max_sectors_kb

If you want to preserve the same behaviour as seen by the older kernels such as 7.2 use:

 echo 512 > /sys/block/sdxx/queue/max_sectors_kb

For all devices that now start with a higher default max_sectors_kb value.

Root Cause

The actual reason as to how the HPSA advertises the larger value is still being investigated but the root cause is that the size is simply defaulting to larger default settings.

In one customer's environment we see the following:

4 disks - 512
8 disks - 1024
12 disks - 1536
So it appears to add 128 per disk added for back-end devices

Diagnostic Steps

Diagnostics
If you experience higher latency in buffered I/O to a file system due to the page_cache flushing happening less often
because of the the larger I/O size, tune the max_sectors_kb down to a lower value using:

 echo 512 > /sys/block/sdxx/queue/max_sectors_kb

Example

7.3 kernel

$ cat sdb/queue/max_sectors_kb 
2816

 # DISK STATISTICS (/sec)
 #                   <---------reads---------------><---------writes--------------><--------averages--------> Pct
 #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size  Wait  RWSize  QLen  Wait SvcTim Util
 17:31:10 sdb              0      0    0    0     0  1779004 451139  820 2170   130    2169   136   130      0   79
 17:31:20 sdb              0      0    0    0     0  1672347 417749  771 2170   128    2170   134   128      0   73
 17:31:30 sdb              0      0    0    0     0  1845596 455125  851 2170   133    2169   136   133      0   82
 17:31:40 sdb              0      0    0    0     0  2260209 569596 1041 2170   131    2170   137   131      0  100

7.2 kernel

$ cat sdb/queue/max_sectors_kb 
512

 # DISK STATISTICS (/sec)
 #                   <---------reads---------------><---------writes--------------><--------averages--------> Pct
 #Time     Name       KBytes Merged  IOs Size  Wait  KBytes Merged  IOs Size  Wait  RWSize  QLen  Wait SvcTim Util
 17:48:10 sdb              0      0    0    0     0  1920768 478139 3752  512    23     512   119    23      0   72
 17:48:20 sdb              0      0    0    0     0  1712793 425097 3345  512    20     512   103    20      0   66
 17:48:30 sdb              0      0    0    0     0  1656422 408927 3235  512    30     512   135    30      0   71
 17:48:40 sdb              0      0    0    0     0  2000896 497999 3908  512    21     512   108    21      0   78

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.