How to extract ELF cores from 'gcore' generated qemu core files for use with the 'crash' utility

Environment

• Red Hat Enterprise Linux (RHEL) 7
• qemu-kvm-[rhev] on RHEL 7 KVM based hosts.
• gcore on RHEL 7 KVM based hosts.

Note: In the RHEL 7.4 time-frame, with an upgrade to libvirt-3.2.0, the default setting for dump-guest-core has been changed from on to off. This means that when a process core file is generated by/from qemu-kvm-[rhev], it will not include any guest memory pages.

Issue

• Core dumps of running programs can be obtained for qemu-kvm guest processes as follows:

# ps -ef | grep <VM name>  # To obtain a virtual guest's PID.
# gcore <PID>

• What steps are required in order to extract ELF cores from resultant gcore output files for use with the crash utility?

• As of RHEL 7.4 with libvirt-3.2.0, the default setting for dump-guest-core is off.

Resolution

• The following process can be followed in order to extract a crash compatible virtual memory core from a gcore generated core file:

1. The dump-guest-memory.py gdb script is installed as part of the qemu-kvm-[rhev] package installation on the host system. This is located under /usr/share/qemu-kvm/dump-guest-memory.py.

2. Ensure that the qemu process that has been dumped had the command line option -machine dump-guest-core=on. In RHEL 6 and 7 prior to the 7.4 time-frame, the default is on. However, in the RHEL 7.4 time-frame, the default changed to dump-guest-core=off. If you see this, please contact Red Hat Technical Support for assistance.

3. Install the same qemu-kvm-[rhev] and qemu-kvm-[rhev]-debuginfo RPM versions on the debugging host as were installed on the qemu-kvm based hypervisor upon which the gcore <pid> command was run when the gcore was captured. Ensure that the host is of the same release and architecture also.

4. Initiate gdb passing the following arguments:

# gdb <qemu-kvm binary path used by original host> <gcore_file>

For example:

# gdb /usr/libexec/qemu-kvm qemu-kvm-core

5. Once gdb has initiated, source the dump-guest-memory.py script referenced in point (1) above:

(gdb) source /usr/share/qemu-kvm/dump-guest-memory.py

6. Ensuring that there is enough available space within the current working directory (at least as much as the original gcore file should suffice in most cases), extract the ELF core as follows from the (gdb) prompt (assuming here that the gcore file also resides in the current working directory). This gdb command re-implements the dump-guest-memory QMP command in Python, using the representation of guest memory as captured in the qemu coredump:

(gdb) dump-guest-memory <output_file>

For example:

(gdb) dump-guest-memory elf_core
guest RAM blocks:
target_start     target_end       host_addr        message count
---------------- ---------------- ---------------- ------- -----
0000000000000000 00000000000a0000 00007f0004000000 added       1
00000000000c0000 00000000000ca000 00007f00040c0000 added       2
00000000000ca000 00000000000cd000 00007f00040ca000 joined      2
00000000000cd000 00000000000ec000 00007f00040cd000 joined      2
00000000000ec000 00000000000f0000 00007f00040ec000 joined      2
00000000000f0000 0000000000100000 00007f00040f0000 joined      2
0000000000100000 00000000c0000000 00007f0004100000 joined      2
00000000fc000000 00000000fd000000 00007f021c200000 added       3
00000000fffc0000 0000000100000000 00007f021d600000 added       4
0000000100000000 0000000240000000 00007f00c4000000 added       5
dumping range at 00007f0004000000 for length 00000000000a0000
dumping range at 00007f00040c0000 for length 00000000bff40000
dumping range at 00007f021c200000 for length 0000000001000000
dumping range at 00007f021d600000 for length 0000000000040000
dumping range at 00007f00c4000000 for length 0000000140000000

NOTE: In more recent releases the format requires an ARCH argument, e.g.;

(gdb) dump-guest-memory elf_core
No valid arch type specified.
Currently supported types:
aarch64-be, aarch64-le, X86_64, 386, s390, ppc64-be, ppc64-le

(gdb) dump-guest-memory elf_core X86_64
guest RAM blocks:
target_start     target_end       host_addr        message count
---------------- ---------------- ---------------- ------- -----
0000000000000000 00000000000a0000 00007f0004000000 added       1
........ etc., etc., ........

7. The resultant file can then be loaded into the crash utility, provided that you reference the vmlinux extracted from the kernel-debuginfo file of the same version as the kernel running in the guest when the gcore was captured (as per standard crash procedure):

# crash <output_file> <vmlinux_file>

• The following additional article provides some detail on extracting vmlinux files from kernel-debuginfo RPMs, How to create a basic centralized crash analysis system to analyze vmcore locally .

NOTE:
For RHEL 7.4 systems with libvirt-3.2.0, the following is required in order for a complete dump of the guest's memory to be obtained;

• Upgrade gdb to gdb-7.6.1-100.el7_4.1.x86_64 or later, per This content is not included.BZ 1524312.

• Run gcore -a, e.g.

# ps -ef | grep <VM name>  # To obtain a virtual guest's PID.
# gcore -a <PID>

• Then follow the sequence above.

Root Cause

• virsh dump --verbose --memory-only <VM name> <dump file> saves cores using save-vm to extract a virtual memory core whereas gcore <PID> collects a core relating to a whole process. Subsequently, further extraction is required against gcore output files in order to prepare an ELF virtual memory core which is usable with the crash utility.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.