RHEL6: Speeding up kernel crash / hang analysis with the kernel log

Introduction

One of the most severe errors which may happen in a Red Hat Enterprise Linux system is a kernel crash or hang. When this occurs, often a very large file is produced containing a memory dump of the entire system. This file is called a vmcore, and may be very large (many gigabytes). Analysis of the kernel crash or hang often requires this large file be uploaded to Red Hat for analysis. Unfortunately transferring and analyzing such a large file may be impossible for some environments, or at the very least, take a long time.

Speeding up analysis with the kernel log

If the above situation applies to you, you can often speed up the vmcore analysis by obtaining the kernel log buffer and uploading this to Red Hat Support and/or the This content is not included.Kernel Oops Analyzer. The kernel log buffer is a much smaller text file (less than one megabyte) than the full vmcore. In some instances, the kernel log may contain enough information to fully identify a known issue, and no vmcore upload or analysis will be necessary. At the very least, the kernel log usually contains enough information to begin analysis on a kernel crash. It is recommended to begin upload of the vmcore at the same time the kernel log is provided to Red Hat. See How can I provide large files to Red Hat Support? for instructions on uploading the full vmcore.

Using vmcore-dmesg.txt from later versions of kexec-tools

Starting with Red Hat Enterprise Linux 6.4 and kexec-tools-2.0.0-258.el6, the kdump process will dump the kernel log to a file called vmcore-dmesg.txt before creating the vmcore file. Before uploading to Red Hat, it is recommended you rename this file to include the case number of your support ticket. For example:

# ls /var/crash/127.0.0.1-2012-11-21-09\:49\:25/
vmcore  vmcore-dmesg.txt
# cp /var/crash/127.0.0.1-2012-11-21-09\:49\:25/vmcore-dmesg.txt /tmp/00123456-vmcore-dmesg.txt

Using makedumpfile to obtain the kernel log

For other versions of Red Hat Enterprise Linux 6, or for cases where vmcore-dmesg.txt is not generated, you can use the following makedumpfile command to obtain the kernel log from an existing vmcore. NOTE: The below command should be run on the same, or similar machine that generated the vmcore (NOTE: The makedumpfile command is part of the kexec-tools package):

# makedumpfile --dump-dmesg [path-to-vmcore] [kernel-log-file] 

• Here is an example, where the case number 00123456 is added to the output filename. Once the command is complete, upload /tmp/00123456-vmcore-dmesg.txt for analysis.

# makedumpfile --dump-dmesg /var/crash/127.0.0.1-2013-06-14-16\:26\:07/vmcore /tmp/00123456-vmcore-dmesg.txt

The dmesg log is saved to /tmp/00123456-vmcore-dmesg.txt.

makedumpfile Completed.

• NOTE: If the above command fails, it may indicate the vmcore is corrupt to the point of not containing any useful information.

Using Kernel Oops Analyzer on the kernel log

Once the kernel log file is obtained, this file can be submitted to the This content is not included.Kernel Oops Analyzer tool. When you input a text or a file including one or more kernel oops messages, the search tool will find best match knowledge documents related to the issue. Read about the tool in the This content is not included.Kernel Oops Blog and leave feedback on the This content is not included.Kernel Oops App Info page. In some instances, submitting the kernel log to the oops analyzer is enough to find a match for a known issue, and no further analysis on a vmcore will be required. When in doubt, contact Red Hat Support for full analysis of the vmcore.