Red Hat Enterprise Linux reports lun<nnn> has a LUN larger than allowed by the host adapter

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL)
  • Emulex LightPulse Fibre Channel Host Adapter (lpfc driver)
  • QLogic Corp Fibre Channel to PCI Express HBA (qla driver)
  • Cisco Systems Inc VIC FCoE Host Adapter (fnic driver)

Issue

  • The following scsi errors are reported; where <nnn> is greater than lpfc module parameter lpfc_max_luns.

      kernel: scsi: host X channel Y id Z lun<nnn> has a LUN larger than allowed by the host adapter

  • This issue can be also seen if you have a LUN ID that is greater than 255 even though you have a much fewer number of LUNs than 255.

  • After hba rescan on physical server we are unable to add new LUN. We see this error in the system logs:

      scsi: host 0 channel 0 id 0 lun557 has a LUN larger than allowed by the host adapter
  scsi: host 1 channel 0 id 0 lun557 has a LUN larger than allowed by the host adapter

  • Unable to install RHEL 6.1 on LUNs exported from SAN when LUN id value is less than FC driver's supported maximum lun id value.

      ERR kernel:lpfc 0000:0f:00.0: 1:1303 Link Up Event x1 received Data: x1 xf7 x20 x0 x0 x0 0
  NOTICE kernel:scsi 3:0:0:0: RAID              EMC      SYMMETRIX        5874 PQ: 0 ANSI: 4
  WARNING kernel:scsi: host 3 channel 0 id 0 lun16384 has a LUN larger than allowed by the host adapter

Resolution

Overview : There will typically be four steps to resolving this issue:

  1. Increase the configured number of luns supported by the driver (this document's focus) if possible/supported by the specific driver,
    • create or edit the /etc/modprobe.d/<module-name>.conf file and update the appropriate driver parameter, in the following example the maximum number of luns per hba is being set to 2048. This value only takes affect at driver load time.
      • lpfc: lpfc_max_luns=2048 ; number should be between 255 and 65535, default is 255
      • qla2xxx: ql2xmaxlun=2048 ; default value is 65535
      • fnic: <not-applicable> ; the fnic driver has no settable option for maximum number of luns supported
  2. Check and increase the kernel's max_report_luns parmeter (if needed) to match the increase within the driver.
    • This is not required for RHEL 8 or later versions which tries with an initial list of size 512 and if that is too small automatically allocates a list size that can accommodate the full list available from storage and retries the REPORT LUNS command. As such the scsi_mod.max_report_luns parameter is no longer present in RHEL8 and later kernels.
  3. Rebuild the system's initramfs file to pick up the changed configuration information.
  4. Reboot the system in order to apply the new paramter values within initramfs (or initrd in the case of RHEL5).

See also "What is the theoretical maximum number of sd* devices supported? " for additional information on the total number luns the kernel can support.

 

**Details**: The following provides more information on each of the above steps.
In the following examples a value of 2048 is used, change that value to whatever you require for an increased LUN value.
See **Notes** section following this section for common pitfalls and snags to be aware of.
1. Increase the configured number of luns supported by the driver requires that the max luns supported by the driver be increased. Usually there is a maximum numbers supported by the driver and a current configured limit. The configured limit should be increased to cover the required number of luns. For example, lets say the new system configuration has 1800+ LUNs configured per storage target. Then the driver's configuration needs to be changed to accomodate this number of luns. For the sake of the example, we're going to increase the configuration to support 2048 luns. The the following configuration parameter(s) need to be created in a /etc/modprobe.d/<module>.conf configuration file (or for RHEL5, /etc/modprobe.conf file). * lpfc: `options lpfc lpfc_max_luns=2048` ; number should be between 255 and 65535 * qla2xxx: `options qla2xxx ql2xmaxlun=2048` (RHEL 5.8 and later only) * *Prior to 5.8, the maximum luns was hard coded to 65535 and could not be changed.* * As of 5.8 and all later releases, the `ql2xmaxlun` parameter was set by default to 65535, however you may need to update the value if it was previously configured to a lower value within a modprobe configuration file. * fnic: *There is no driver configuration option currently available to modify the behaviour of the fnic driver in this regard. However, max luns per target can be increased within the UCS web management console.* * A possible workaround for this issue is to reconfigure the storage configuration by removing the LUN from the Storage Group and adding it back in at a lower LUN ID that is within the supported range of the fnic driver. 2. Increase the configured number of luns supported, in general, by the kernel's IO stack, if needed. * See *["How do I setup multiple LUNs on Red Hat Enterprise Linux?](https://access.redhat.com/solutions/3292)"* for how to modify the kernel's `max_report_luns` configuration parameter. For RHEL 6 or 7 the following options parameter is needed on the kernel boot line: scsi_mod.max_report_luns=2048 3. Rebuild the system's `initramfs` file to pick up the changed configuration information, and reboot. * See *["How to rebuild the initial ramdisk image in Red Hat Enterprise Linux"](https://access.redhat.com/site/solutions/1958)* for full information. However, for and example with RHEL 6 or 7: # cp /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).bak.$(date +%m-%d-%H%M%S).img # dracut -f -v * After above changes, it is necessary to **reboot** system in order apply the configuration changes. 4. Reboot the system. # shutdown -r now

 

**Notes**: Additional notes and information. 1. Increase the configured number of luns supported by the driver * If in doubt of whether a specific driver supports changing the number of supported luns or whether it has such a parameter or how to correctly spell the parameter name, then performing a modinfo on the driver .ko module can reveal the desired information in either of those cases. In the following case of the `fnic` driver we see that no such parameter exists for changing the default number of luns supported by the driver: $ modinfo fnic | grep parm parm: fnic_log_level:bit mask of fnic logging levels (int) parm: fnic_trace_max_pages:Total allocated memory pages for fnic trace buffer (uint) parm: fnic_fc_trace_max_pages:Total allocated memory pages for fc trace buffer (uint) parm: fnic_max_qdepth:Queue depth to report for each LUN (uint) * Nominally, all the driver options must be on a single line and the option only appear once. There should only ever be one `options ` line. Otherwise additional option lines, other than the last (or in some cases the first) one, may be missed and ignored. All options for a given module must be on the same line to avoid this issue. This also goes for each option, and options should only appear once. If an option exists and needs a new value, just update the value within the existing options parameter rather than duplicating it by adding another one. * Good example: **all options on one line, each option only appears once** # cat /etc/modprobe.d/lpfc.conf options lpfc lpfc_nodev_tmo=10 lpfc_lun_queue_depth=32 lpfc_max_luns=2048 * Bad examples: # cat /etc/modprobe.d/lpfc.conf options lpfc lpfc_nodev_tmo=10 options lpfc lpfc_lun_queue_depth=32 options lpfc lpfc_max_luns=2048 << multiple option lines for same driver module # cat /etc/modprobe.d/lpfc.conf options lpfc lpfc_nodev_tmo=10 lpfc_max_luns=768 lpfc_max_luns=2048 lpfc_max_luns=2048 << option appears twice on -- unclear which one will be used in all cases - first or last? * A quick command line to change things on RHEL 6 or 7: $ ls -1c /etc/modprobe.d/*.conf 2> /dev/null | xargs -I {} bash -c 'echo -n "File {} : " ; grep -H options {} | wc -l' | grep -v ": 0" File /etc/modprobe.d/lpfc.conf : 3 File /etc/modprobe.d/mlx4.conf : 6 * If you need to apply a new maximum luns at installation time then the options change can be added/applied to the boot line. Again the caveat of only appearing once applies so if the option is already present, just update the value. For example, two common examples of changing lpfc or qla2xxx maximum lun values on the boot line would required the following added to the kernel boot line: * lpfc: ` lpfc.lpfc_max_luns=2048` * qla2xxx: `qla2xxx.ql2xmaxlun=2048` * Note that the driver limit is nominally the maximum number of luns the driver can track per storage target, not total. So for a given HBA which is scsi host 1 it may have 1:0:0:0-limit, 1:0:1:0-limit, 1:0:2:0-limit; where storage targets (H:B:**T*:L) are different. 2. Increase the configured number of luns supported, in general, by the kernel's IO stack, if needed. * If this step isn't done, then while eliminating the original logged event of "....." by increasing the driver's maximum support lun, you can typically end up with the following new event below: kernel: scsi: On host 3 channel 0 id 1 only 511 (max_scsi_report_luns) of 1122 luns reported, * So when increasing a driver's maximum supported lun, you have to check the current maximum luns supported within the kernel and increase that as necessary also. $ cat /sys/module/scsi_mod/parameters/max_report_luns 511 * On RHEL7 this is set to a high default value (16383), so on RHEL7 -- unless its been previously modified by the addition of `scsi_mod.max_report_luns=nnn`, changing this won't be required. 3. Rebuild the system's `initramfs` file to pick up the changed configuration information. 4. Reboot the system in order to apply the new paramter values within initramfs (or initrd in the case of RHEL5). ## Root Cause * Storage has returned a LUN id value that exceeds the current maximum LUN id value supported by the driver. Either * the driver's maximum supported LUN id value needs to be increased, as in the case of lpfc, or * the LUN id value provided by storage must be changed via reconfiguration within the storage box. * The default maximum LUNs for `lpfc` is 256 (starting at 0) LUNs. Having a LUN number larger than default (255) or `lpfc_max_luns` will generate error. * If your system doesn't have many LUNs and your system fails to detect it because its LUN ID is greater than 256, it is because the SCSI scanning logic uses it when a SCSI-2 device (something that doesn't support REPORT_LUNS) is detected. In these cases, the midlayer does a sequential scan and uses max_lun as it's top end of the loop. So in that respect, it is the maximum LUN ID that would be detect and used on that hba. But, if the device is SCSI-3 or if there is device-info structures (white list) that says a device supports it - the midlayer will use REPORT_LUNS and will use the LUN values from the response - which can be in excess of max_lun. There are other cases, such as SCSI-1 which inherently limit scanning of a target to 8 luns, and other device-info flags that do some other things. In a couple of these, max_lun is considered a cap on the lun number (but lun #'s are < 8 in these cases). * Related is the way the midlayer stores a lun number - it has not converted to keeping a full 64bits for lun number - it instead uses a 32bit number - and compresses the 64bits into a 32-bit value assuming level 1 or level 2 addressing only. Thus, when lpfc receives a lun number from the midlayer device struct (which may have come from report_luns), it calls a midlayer routine to expand it to 8bytes when it stuffs it in the FCP_CMD payload. lpfc makes no restriction on the content of the LUN number. ## Diagnostic Steps * Review the current value. $ cat /sys/class/scsi_host/hostX/lpfc_max_luns 255 * For systems where are present Qlogic HBA cards in `/var/log/dmesg` will be visible below error messages Vendor: EMC Model: SYMMETRIX Rev: 5874 Type: RAID ANSI SCSI revision: 04 qla2xxx 0000:06:00.0: scsi(1:0:3:0): Enabled tagged queuing, queue depth 128. scsi: host 1 channel 0 id 3 lun16384 has a LUN larger than allowed by the host adapter scsi: host 1 channel 0 id 3 lun16385 has a LUN larger than allowed by the host adapter **NOTE:** lun16384 in the above messages actually are due to the bug described in [Red Hat Enterprise Linux reports lunXXX has a LUN larger than allowed by the host adapter, where XXX is a very large number](https://access.redhat.com/site/solutions/67157) . This bug can sometimes be worked around by expanding both the kernel and driver max lun values. In the above messages, lun16384 is really lun number 0, and lun16385 is really lun number 1.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.