VMWare snapshot converted to a .core file fails to open with the crash utility

Environment

• Guest OS on VMWare
• Observed on RHEL 7.5 kernels and above

Issue

• VMWare snapshot converted to a .core file using "vmss2core-Linux64" utility fails to open even after a successful conversion and shows the following error message :

log: zero-size memory allocation! (called from 4e23f6)

• Manual retrace leads to the error shown below :

# crash vmss.core /usr/lib/debug/lib/modules/3.10.0-862.11.6.el7.x86_64/vmlinux 

crash 7.2.3
Copyright (C) 2002-2017  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
.
</snip>
.
WARNING: cannot read linux_banner string
crash: /usr/lib/debug/lib/modules/3.10.0-862.11.6.el7.x86_64/vmlinux and vmss.core do not match!

Usage:

  crash [OPTION]... NAMELIST MEMORY-IMAGE[@ADDRESS]    (dumpfile form)
  crash [OPTION]... [NAMELIST]                     (live system form)

Enter "crash -h" for details.

• .core opened in the --minimal mode leads to the same error as well.

Resolution

• A VMWare snapshot can be opened with the crash utility without converting to a .core file as follows

  • Copy both .vmss (or .vmsn) and .vmem files to a single folder.
  • The files should have the exact same name and their respective extensions should NOT be changed.
  • Run the crash command on a .vmsn or .vmss as shown below. Using it, the crash utility will automatically look for the .vmem file if needed.

# crash <NAME>.vmsn  /cores/retrace/repos/kernel/x86_64/usr/lib/debug/lib/modules/<kernel version>/vmlinux

• The support to debug KASLR enabled kernels was added in crash versions "7.2.3-8" or above, which can be seen from the This content is not included.Changelog.

Workaround

• Use NMI to force crash a hung guest OS.
• Boot the guest OS with nokaslr kernel command-line parameter. This involves rebuilding the grub and rebooting the server. However, doing so will defy the purpose of KASLR.

Root Cause

• Starting 7.5, RHEL kernels feature KASLR (Kernel Address Space Linear Randomization) enabled by default. KASLR is a security feature that enables the kernel to relocate itself to a random location on each boot, making writing exploits that depend on local resources significantly harder.

• The side effect of this feature is debugging tools like crash may encounter some trouble trying to open vmcores from KASLR-enabled kernels.

• For more information about this, please refer to this article.

Diagnostic Steps

• Boot into any guest running, with kernel greater than RHEL 7.5
• Collect a snapshot from VMWare and try to open it with the crash utility.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.