[Troubleshooting] Changing the kernel's SCSI logging level
Issue
- How do I enable additional SCSI logging ?
- How do I turn on SCSI extended debug messages ?
- What are the field/flag definitions of scsi_logging_level ?
Environment
- Red Hat Enterprise Linux 4, 5, 6, 7, 8, 9
Resolution
Additional scsi logging messages can be enabled by writing to /proc/sys/dev/scsi/logging_level either via use of the echo or preferrably the sysctl command. The kernel parameter consists of ten packed fields, each 3 bits in length. Each field can have a value of 0 to 7. The higher the field's value, the more verbose the logging of messages associated with that field type.NOTE:
Turning on high levels of extended logging and/or multiple types of extended logging can slow down system performance, especially during boot and shutdown. Unless needed, avoid turning on scsi logging during boot up.
Run-Time
- enable : sysctl -q -w dev.scsi.logging_level=N, where N specifies which fields to enable and at what level of verbosity.
- disable: sysctl -q -w dev.scsi.logging_level=0, or echo 0 > /proc/sys/dev/scsi/logging_level
Boot Time
- enable (RHEL4,5):
- modify /etc/modprobe.conf with
options scsi_mod scsi_logging_level=N
- rebuild the initrd image file to pick up the new /etc/modprobe.conf file. See "How do I rebuild the initial ramdisk image in Red Hat Enterprise Linux" for specific steps.
- edit the boot line and remove 'quiet' if present, add either 'debug' or 'loglevel=10' -- they both do the same thing. Also add 'log_buf_len=8M' to increase the kernel's log fifo size since the additional logging will be creating a lot more messages.
- reboot and capture the messages output
- unless needed, turn off debugging after the system is booted up or especially before shutting down. During shutdown it can often seem like the system is hung as it outputs a ton of messages.
- modify /etc/modprobe.conf with
- enable (RHEL6,7):
Note:There are different mechanisms for editing the boot line on RHEL6 versus RHEL7 - on RHEL6:
- edit file /boot/grub/menu.lst to add or change parameters on the kernel line
- on RHEL7: (grub2)
- edit file /etc/default/grub and add or change parameters in the GRUB_CMDLINE_LINUX string
- changes to /etc/default/grub require rebuilding the grub.cfg file as follows:
- on BIOS-based machines, issue the following command as root:
- ~]# grub2-mkconfig -o /boot/grub2/grub.cfg
- on UEFI-based machines, issue the following command as root:
- ~]# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg
- See This content is not included. "Customizing GRUB 2" for more information on grub2
- edit the boot line and
- add scsi_mod.scsi_logging_level=N
- remove `quiet` if present and add either debug or loglevel=10 -- they both do the same thing.
- add log_buf_len=8M to increase the kernel's log fifo size since the additional logging will be creating a lot more messages
- reboot and capture the messages output
- unless needed, turn off debugging using run-time methods after the system is booted up or especially before shutting down. During shutdown, if scsi debug events are left enabled, it can often seem like the system is hung as it outputs a lot of messages which can add additional time to the shutdown process.
- disable: sysctl -q -w dev.scsi.logging_level=0
What the above flags do is to up the scsi extended logging level to include lots of info upon errors and timeouts and during scan processing plus upon any io completing off the high level queue.
At one point, the /usr/share/doc/kernel-doc-*/Documentation/kernel-parameters.txt file included the following description of scsi logging level:
scsi_logging_level= [SCSI] a bit mask of logging levels
See drivers/scsi/scsi_logging.h for bits. Also
settable via sysctl at dev.scsi.logging_level
(/proc/sys/dev/scsi/logging_level).
This text is no longer included kernel-parameters.txt file. There is a scsi_logging_level shell script available within the sg3_utils source rpm -- it isn't installed, but can be extracted from the sg3_utils*src.rpm file. This script is very useful for setting up mask values for scsi logging. For example, from the 1.25 source rpm:
sg3_utils-1.25/ sg3_utils-1.25/configure.ac sg3_utils-1.25/scripts/ sg3_utils-1.25/scripts/scsi_ready sg3_utils-1.25/scripts/scsi_start sg3_utils-1.25/scripts/scsi_readcap sg3_utils-1.25/scripts/scsi_logging_level <<< :
/* * This defines the scsi logging feature. It is a means by which the user * can select how much information they get about various goings on, and it * can be really useful for fault tracing. The logging word is divided into * 8 nibbles, each of which describes a loglevel. The division of things is * somewhat arbitrary, and the division of the word could be changed if it * were really needed for any reason. The numbers below are the only place * where these are specified. For a first go-around, 3 bits is more than * enough, since this gives 8 levels of logging (really 7, since 0 is always * off). */ #define SCSI_LOG_ERROR_SHIFT 0 #define SCSI_LOG_TIMEOUT_SHIFT 3 #define SCSI_LOG_SCAN_SHIFT 6 #define SCSI_LOG_MLQUEUE_SHIFT 9 #define SCSI_LOG_MLCOMPLETE_SHIFT 12 #define SCSI_LOG_LLQUEUE_SHIFT 15 #define SCSI_LOG_LLCOMPLETE_SHIFT 18 #define SCSI_LOG_HLQUEUE_SHIFT 21 #define SCSI_LOG_HLCOMPLETE_SHIFT 24 #define SCSI_LOG_IOCTL_SHIFT 27 #define SCSI_LOG_ERROR_BITS 3 additional logging associated with errors and recovery #define SCSI_LOG_TIMEOUT_BITS 3 additional logging associated with command timeouts #define SCSI_LOG_SCAN_BITS 3 additional logging associated with device scans and discovery #define SCSI_LOG_MLQUEUE_BITS 3 additional logging associated with mid-level command queueing #define SCSI_LOG_MLCOMPLETE_BITS 3 additional logging associated with mid-level command completions #define SCSI_LOG_LLQUEUE_BITS 3 additional logging associated with low-level command queueing #define SCSI_LOG_LLCOMPLETE_BITS 3 additional logging associated with low-level command completions #define SCSI_LOG_HLQUEUE_BITS 3 additional logging associated with hi-level command queueing #define SCSI_LOG_HLCOMPLETE_BITS 3 additional logging associated with hi-level command completions #define SCSI_LOG_IOCTL_BITS 3 additional logging associated with ioctl (typ: non-data commands) extern unsigned int scsi_logging_level;
As an example, to turn on maximum logging (level 7) for timeouts and level 3 for errors and ioctls, we would use the octal value 3000000073 (0x1800003B). This can be set via the hexadecimal or decimal values.
# sysctl -q -w dev.scsi.logging_level=0x1800003B -or- # sysctl -q -w dev.scsi.logging_level=402653243
Conversely, if sysctl -q dev.scsi.logging_level was used to query the current set value and was returned 402653425 (0x180000F1 or octal 3000000361), then currently the field masks have ioctl=3, scan=3, timeout=6, and error=1 values set.
Another example might be sysctl -q -w dev.scsi.logging_level=4099, this is 0x1003 (10003 octal), so error logging level 3 and mlcomplete level 1.