[Troubleshooting] How do I turn on additional lpfc driver extended logging and what logging is available?
Issue
- How do I gather more information from the lpfc driver?
- How do I turn on verbose error logging within the lpfc driver?
- How do I enable additional extended logging within the emulex driver ?
- How do I increase the logging verbosity for my Emulex HBA, using the lpfc driver?
- I want to enable debugging in lpfc driver
- What lpfc logging options are available?
Environment
- Red Hat Enterprise Linux 9
- Red Hat Enterprise Linux 8
- Red Hat Enterprise Linux 7
- Red Hat Enterprise Linux 6
- Red Hat Enterprise Linux 5
- Red Hat Enterprise Linux 4
Resolution
How do I to set lpfc_log_verbose?- dynamically during run-time, per host[1][2][3]:
Enable : # echo '<value>' > /sys/class/scsi_host/host<N>/lpfc_log_verbose Disable : # echo 0 > /sys/class/scsi_host/host<N>/lpfc_log_verbose View : # cat /sys/class/scsi_host/host<N>/lpfc_log_verboseEnable (all) : # for f in /sys/class/scsi_host/host*/lpfc_log_verbose ; do echo '<value>' > $f ; done Disable (all) : # for f in /sys/class/scsi_host/host*/lpfc_log_verbose ; do echo 0 > $f ; done View (all) : # ls -1c /sys/class/scsi_host/host*/lpfc_log_verbose | xargs -I {} grep -Hv "zZzZ" {}
- if needed at boot time[1][4][5][6]:
RHEL 4,5: in /etc/modprobe.conf, update the lpfc options line, add the line if necessary: options lpfc lpfc_log_verbose=<value> {<other options>} RHEL 6/7/8/9 : in /etc/modprobe.d/lpfc.conf, update the lpfc options line. If the file doesn't exist, create it. options lpfc lpfc_log_verbose=<value> {<other options>}Recommendation: Increase the kernel's event log buffer length. The event log is a fifo and early boot
events can be lost if the fifo size is too small. The default size is just 64k bytes.
Increase by adding the following option to the grub boot line:
log_buf_len=16M
[1]: <value> must be a decimal value in RHEL 4 and 5.
[2]: <N> is host number {0...n}
[3]: being dynamically set, this change won't persist across reboots
[4]: emulex documentation indicates that parameters set via modprobe.conf cannot be overridden
[5]: requires rebuild of initrd image file and reboot to take effect
[6]: boot time setting is applied to all scsi host instances
In RHEL6, 7, 8 and 9, the default lpfc parameter value as set at boot time can be found in:
$ cat /sys/module/lpfc/parameters/lpfc_log_verbose
<Value>
The lpfc driver uses a bit mask to enable additional logging of some specific events and/or additional information within the driver.The value must be a decimal value when echoed into the lpfc_log_verbose parameter. The kernel code uses a sscanf(buf, "%d", &val) to convert the incoming string to a number (within RHEL4/5, RHEL6 uses "%i" for this parameter which will decode decimal, octal and hexidecimal). When a hex string value of "0xffff", for example is used, the sscanf("%d") will only decode the leading '0' (zero) and use that. When in doubt use a decimal value.
A typical logged event would look like this within the source code. In this case, this is a LOG_FCP event and will only be seen if the LOG_FCP bit is set within the lpfc_log_verbose flag.
lpfc_printf_log(phba, KERN_WARNING, LOG_FCP,
"%d:0729 FCP cmd x%x failed <%d/%d> status: "
"x%x result: x%x Data: x%x x%x\n",
phba->brd_no, cmd->cmnd[0], cmd->device->id,
cmd->device->lun, lpfc_cmd->status,
lpfc_cmd->result, pIocbOut->iocb.ulpContext,
lpfc_cmd->cur_iocbq.iocb.ulpIoTag);
The following bit values are defined under Red Hat Enterprise Linux 4, 5, 6, 7, 8 and 9:
RHEL4 RHEL5 RHEL6 RHEL7 RHEL8 RHEL9 NAME Value Description
x x x x x x LOG_ELS 0x1 ELS events
x x x x x x LOG_DISCOVERY 0x2 Link discovery events
x x x x x x LOG_MBOX 0x4 Mailbox events
x x x x x x LOG_INIT 0x8 Initialization events
x x x x x x LOG_LINK_EVENT 0x10 Link events
x x LOG_IP 0x20 IP traffic history
x LOG_NODE_VERBOSE 0x20 Node Verbose/table lookups
x x x x x x LOG_FCP 0x40 FCP traffic history
x x x x x x LOG_NODE 0x80 Node table events
x x x x x x LOG_TEMP 0x100 Temperature sensor events
x x x LOG_BG 0x200 BlockGuard events
x x x x x x LOG_MISC 0x400 Miscellaneous events
x x x x x x LOG_SLI 0x800 SLI events
x x x x x LOG_FCP_ERROR 0x1000 Only log FCP errors
x LOG_CHK_COND 0x1000 FCP Check condition flag
x x x x x x LOG_LIBDFC 0x2000 LIBDFC events
x x LOG_ALL_MSG 0xffff LOG all messages
x x x x LOG_VPORT 0x4000 NPIV events
x x x LOG_SECURITY 0x8000 Security events
x LOG_LDS_EVENT 0x8000 Link Degrade Signaling events
x x x x LOG_EVENT 0x10000 CT, TEMP, DUMP, logging
x x x x LOG_FIP 0x20000 FIP events
x x x x LOG_FCP_UNDER 0x40000 FCP underrun errors
x x x LOG_SCSI_CMD 0x80000 ALL SCSI commands
x x LOG_NVME 0x00100000 NVME general events.
x x LOG_NVME_DISC 0x00200000 NVME Discovery/Connect events.
x x LOG_NVME_ABTS 0x00400000 NVME ABTS events.
x x LOG_NVME_IOERR 0x00800000 NVME IO Error events.
x x LOG_RSVD1 0x01000000 /* Reserved */
x x LOG_RSVD1 0x02000000 /* Reserved */
x x LOG_CGN_MGMT 0x04000000 /* Congestion Mgmt events */
x x LOG_TRACE_EVENT 0x80000000 /* Dmp the DBG log on this err */
x x LOG_ALL_MSG 0xffffffff Log all messages
x x LOG_ALL_MSG 0x7fffffff Log all messages
Extended logging parameters LOG_NVME, LOG_NVME_DISC, LOG_NVME_ABTS and LOG_NVME_IOERR are available in RHEL7 in kernel kernel-3.10.0-862.el7 and onwards.
For example, setting extended logging to a <value> of 4097 (0x1001) would result in both ELS events being logged and FCP errors. Other commonly used masks include:
- 4115 (0x1013) : ELS, Link discovery, Link events, FCP errors; common events useful for general troubleshooting
- 4243 (0x1093) : the above plus Node events; useful for when LIP or rescanning of storage is manually being invoked
Warning:Turning on all extended event logging can lead to soft lockups of the system in that the amount of extra event logging created causes more io to /var/log/messages. If the messages file is on a disk accessed via lpfc driver, then the logging of large amounts of additional events result in those io getting logging events of their own, etc. This type of soft system lockup is dependent upon other ongoing io activity on the system. Low to moderate io loads on the system usually won't cause an issue. For testing purposes you can create a bash script that turns on full logging, sleeps for a short while and then turns off all logging. Turning on full logging is not a recommended practice.
Diagnostic Steps
The sysfs filesystem is mounted and available as /sys. You must first identify the `scsi_host` which represents the HBA for which you wish to modify the driver parameters. All scsi_hosts bound to the lpfc driver can be viewed with the following command:# ls -d /sys/bus/pci/drivers/lpfc/*/host
Assuming you are interested in HBA scsi_host 7, you can list the driver parameters for this particular HBA as:
# ls -l /sys/class/scsi_host/host7/lpfc*
The above is for RHEL5 and RHEL6, in RHEL4: /proc/scsi/lpfc/... contains the information {not confirmed}.