How to change the queue depth on VMware virtual guest system running RHEL 6.3?

Environment

• Red Hat Enterprise Linux 6.3
• VMware guest
• VMware PVSCSI storage adapter
• vmw_pvscsi.ko driver

Issue

. How to change the queue depth on VMware virtual guest system running RHEL 6.3? does it require any downtime or it could be done online?

Resolution

The VMware virtual disks on RHEL 6 are managed by vmw_pvscsi kernel module. This driver uses default queue depth of 64:

    $ cat /sys/module/vmw_pvscsi/parameters/cmd_per_lun
    64

Queue depth value used by vmw_pvscsi driver could be changed using one of the following two methods:

Method I

1. Edit /boot/grub/grub.conf file and add following options to kernel line:

    vmw_pvscsi.cmd_per_lun=X vmw_pvscsi.ring_pages=Y
   

   The value for ring_pages parameter is calculated as cmd_per_lun / 8. e.g. to increase the queue depth (cmd_per_lun) from 64 to 128, the ring_pages parameter would also be required to be increased to 16. These options could be added to kernel line in /boot/grub/grub.conf file as follows:

    title Red Hat Enterprise Linux Server (2.6.32-279.11.1.el6.x86_64)
    root (hd0,0)
    kernel /vmlinuz-2.6.32-279.11.1.el6.x86_64 ro root=/dev/mapper/VolGroup00-LogVol00 rd_LVM_LV=VolGroup00/LogVol00 rd_LVM_LV=VolGroup00/LogVol01 vmw_pvscsi.cmd_per_lun=128 vmw_pvscsi.ring_pages=16
    initrd /initramfs-2.6.32-279.11.1.el6.x86_64.img
   

2. Reboot the system after adding above kernel options. To verify the new qdepth values following commands could be used:

    $ cat /sys/module/vmw_pvscsi/parameters/cmd_per_lun
    $ cat /sys/module/vmw_pvscsi/parameters/ring_pages
    
    $ cat /sys/block/sdXX/device/queue_depth
   

   Please replace sdXX in above command with name of the VMware Virtual disk device.

Method II

1. The queue depth used by vmw_pvscsi driver could also be modified by creating a new configuration file within /etc/modprobe.d directory with following contents:

    $ cat /etc/modprobe.d/vmw_pvscsi_qdepth.conf
    options vmw_pvscsi cmd_per_lun=128 ring_pages=16
   

2. After creating a custom file /etc/modprobe.d/vmw_pvscsi_qdepth.conf with above options, please rebuild the initial ram disk image using the steps described in following article and reboot the system with newly created initrd image:
   How do I rebuild the initial ramdisk image in Red Hat Enterprise Linux?

Note: For VMware PVSCSI HBAs, currently it is not possible to dynamically change the queue depth value, it is required to unload and load the module with new value for queue depth or reboot the system.

Root Cause

• The queue depth value indicates the number of I/O requests (SCSI commands) that can be queued at one time on a target storage controller. Each I/O request from the host's initiator HBA to the storage controller's target adapter consumes a queue entry. Typically, a higher queue depth equates to better performance. However, if the storage controller's maximum queue depth is reached, then that storage controller would start rejecting the incoming commands by returning a QFULL response to the host HBAs. If a large number of hosts are accessing a storage controller, plan carefully to avoid QFULL conditions, which significantly degrade system performance and can lead to errors on some systems.

• The default queue depth value would be suitable for most of the deployments. But if there is any performance issue being observed while using default queue depth values, then we could try to tune the queue depth value to elevate the performance. This would need a detailed analysis of IO statistics to determine an optimal queue depth value to resolve the IO performance issues. But as described above, a tuning to queue depth value would be required only if there is a IO performance issue being observed on the system. Also, the SAN/Storage vendor could provide recommendations for the maximum queue depth to be used for specific SAN devices.

• The parameter cmd_per_lun available in vmw_pvscsi.ko kernel module could be used to tune the queue depth value at driver level, this parameter is also dependent upon another option - ring_pages. When there is any change made to cmd_per_lun variable, then the value of ring_pages options is required to be set accordingly. For specific queue depth value, ring_pages could be calculated as follows:

    ring_pages = cmd_per_lun / `8`
  

  for queue depth of 64, ring_pages = 64 / 8 = 8
  for queue depth of 128, ring_pages = 128 / 8 = 16
  for queue depth of 255, ring_pages = 255 / 8 = 32

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.