How can I use crash to send Red Hat some vmcore pre-analysis information before or while uploading the vmcore image?

Solution Verified - Updated 10 Jun 2025

Environment

Red Hat Enterprise Linux 4
Red Hat Enterprise Linux 5
Red Hat Enterprise Linux 6
Red Hat Enterprise Linux 7
Red Hat Enterprise Linux 8
Red Hat Enterprise Linux 9
Red Hat Enterprise Linux 10
A "vmcore" panic memory image created from a Red Hat Enterprise Linux system.

Issue

A system has paniced or crashed and a vmcore image has been generated. How can I analyze this vmcore file?
What steps can I take to assist in Red Hat while a vmcore is being uploaded for analysis.
How can I use crash to analyze a kernel crash or system hang?

Resolution

This article assumes that the Red Hat crash utility and the kernel-debuginfo package are installed (See Comments for more information). It also assumes that one of the kernel crash dump facilities has been used to generate the vmcore file.

When the system crashes with a mechanism to create the vmcore was successful, the crash utility can be used to analyse the information prior to uploading the vmcore to Red Hat Support. The following crash commands have useful output that may reduce the time to resolution.

    crash> set hex
    crash> sys > sys.out
    crash> bt -a > bt-a.out
    crash> mod > mod.out
    crash> log > log.out
    crash> ps > ps.out
    crash> foreach UN bt > bt-un.out
    crash> kmem -i > kmem-i.out
    crash> net -a > net-a.out
    crash> dev -d > dev-d.out

sys displays the system data

        crash> sys
         KERNEL: /usr/lib/debug/lib/modules/2.6.9-22.EL/vmlinux
         DUMPFILE: /home/eteo/crash/127.0.0.1-2007-04-30-21:38/vmcore
         CPUS: 1
         DATE: Mon Apr 30 21:38:40 2007
         UPTIME: 00:04:04
        LOAD AVERAGE: 0.36, 0.23, 0.08
         TASKS: 36
         NODENAME: localhost.localdomain
         RELEASE: 2.6.9-22.EL
         VERSION: #1 Mon Sep 19 18:20:28 EDT 2005
         MACHINE: i686 (1862 Mhz)
         MEMORY: 1 GB
         PANIC: "Oops: 0002 [#1]" (check log for details)

The output of the sys command has information about the system (e.g. kernel release, kernel version, number of CPUs, amount of memory, etc), the time of vmcore taken, the operating period, and the panic (e.g. Oops type, panic task/PID/command, etc).

The bt -a command displays the stack trace of the active task on each CPU in the system.

        crash> bt
        PID: 2857 TASK: f7b677f0 CPU: 0 COMMAND: "bash"
         #0 [f7191e04] start_disk_dump at f89d7bb3
         #1 [f7191e18] die at c010682e
         #2 [f7191e48] do_page_fault at c011ab00
        ...
         #9 [f7191fc0] system_call at c030f918
         EAX: 00000004 EBX: 00000001 ECX: b7de7000 EDX: 00000002
         DS: 007b ESI: 00000002 ES: 007b EDI: b7de7000
         SS: 007b ESP: bfe01650 EBP: bfe01670
         CS: 0073 EIP: 003297a2 ERR: 00000004 EFLAGS: 00000246

mod displays information on the kernel modules and symbols at the time of the vmcore creation.

        crash> mod
         MODULE NAME SIZE OBJECT FILE
         c8019000 soundcore 2788 (not loaded)
         c801b000 soundlow 336 (not loaded)
         c801d000 sound 59864 (not loaded)
         c802d000 ad1848 15728 (not loaded)
         c8032000 uart401 6000 (not loaded)
         c8035000 cs4232 2472 (not loaded)
         c8043000 opl3 11048 (not loaded)
         c8047000 3c59x 18152 (not loaded)
         c804d000 sunrpc 53796 (not loaded)
         c805c000 lockd 31528 (not loaded)
         c8065000 nfsd 151896 (not loaded)
         c8092000 nfs 29752 (not loaded)

The log command dumps the kernel log buffer contents in chronological order. This can show panic or oops and critical system log messages prior to the panic. The instructions below is a method of creating a single file (crash_data.log) which would be useful as a data point for support to start diagnosing the problem. .

        # crash /usr/lib/debug/lib/modules/`uname -r`/vmlinux /var/crash/<crash dir>/vmcore
        set hex
        sys > crash_data.log
        bt -a >> crash_data.log
        mod >> crash_data.log
        ps >> crash_data.log
        foreach UN bt >> crash_data.log
        kmem -i >> crash_data.log
        net -a >> crash_data.log
        dev -d >> crash_data.log
        log >> crash_data.log
        quit

Attach the crash_data.log file to the support case a file attachment, it is usually too large to use as a comment.

Comments

For more information on the kernel-debuginfo package, see How to download debuginfo packages like kernel-debuginfo.
On Red Hat Enterprise Linux crash can be installed from standard places like Red Hat Network (package name crash, usually part of the Development tools package group).
More information about the crash utility including its source code and a whitepaper on its usage can be found here: Content from crash-utility.github.io is not included.Content from crash-utility.github.io is not included.https://crash-utility.github.io/.
If the pre-analysis information is insufficient, we may request that you upload the vmcore file to This content is not included.ftp://dropbox.redhat.com/incoming/ . See kbase article How can I provide large files (such as a vmcore, big log files, or a large sysreport) to Red Hat Support? on how you can upload the vmcore file.

SBR

Product(s)

Red Hat Enterprise Linux

Components

crash
dump

Category

Learn more

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.