How should the crashkernel parameter be configured for using kdump on RHEL7?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 7
  • x86_64 architecture
  • kdump

Issue

  • When configuring the crashkernel parameter the kdump service fails to start:
$ systemctl start kdump
Job for kdump.service failed. See 'systemctl status kdump.service' and 'journalctl -xn' for details.
$ systemctl status kdump.service
kdump.service - Crash recovery kernel arming
   Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled)
   Active: failed (Result: exit-code) since Wed 2014-06-11 10:34:48 CEST; 3min 10s ago
  Process: 2259 ExecStart=/usr/bin/kdumpctl start (code=exited, status=1/FAILURE)
 Main PID: 2259 (code=exited, status=1/FAILURE)

Jun 11 10:34:47 rhel7 systemd[1]: Starting Crash recovery kernel arming...
Jun 11 10:34:48 rhel7 kdumpctl[2259]: No memory reserved for crash kernel.
Jun 11 10:34:48 rhel7 kdumpctl[2259]: Starting kdump: [FAILED]
Jun 11 10:34:48 rhel7 systemd[1]: kdump.service: main process exited, code=exited, status=1/FAILURE
Jun 11 10:34:48 rhel7 systemd[1]: Failed to start Crash recovery kernel arming.
Jun 11 10:34:48 rhel7 systemd[1]: Unit kdump.service entered failed state.
  • Unable to start/restart kdump service
  • What is the correct crashkernel parameter for kdump to work?
  • "crashkernel reservation failed - memory is in use" errors when kernel panics

Resolution

The kdump procedure

The received warning means the kdump operation might fail and the crashdump parameter should be configured correctly. This is the procedure of kdumping:

  1. The normal kernel is booted with crashkernel=... as a kernel option, reserving some memory for the kdump kernel. The memory reserved by the crashkernel parameter is not available to the normal kernel during regular operation.  It is reserved for later use by the kdump kernel.
  2. The system panics.
  3. The kdump kernel is booted using kexec, it used the memory area that was reserved w/ the crashkernel parameter.
  4. The normal kernel's memory is captured into a vmcore.

Note: Not reserving enough memory for the kdump kernel can lead to the kdump operation failing.

Configuring crashkernel on RHEL7.0 (and later) kernels

crashkernel is configured in the GRUB_CMDLINE_LINUX line in /etc/default/grub:

GRUB_TIMEOUT=5
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel00/root rd.lvm.lv=rhel00/swap
 rhgb quiet"
GRUB_DISABLE_RECOVERY="true"

After modifying /etc/default/grub, regenerate the GRUB2 configuration using the edited default file. If your system uses BIOS firmware, execute the following command: 

# grub2-mkconfig -o /boot/grub2/grub.cfg

On a system with UEFI firmware, execute the following instead:

# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

Starting with RHEL7 kernels crashkernel=auto should usually be used. The kernel will automatically reserve an appropriate amount of memory for the kdump kernel.

Keep in mind that it is an algorithmically calculated memory reservation and may not meet the needs of all systems (Especially for configurations with many IO cards and loaded drivers). Ensure that memory reserved by crashkernel=auto is sufficient for the target machine by testing kdump. If it is insufficient, reserve more memory using syntax crashkernel= XM where X is amount of memory to be reserved in megabytes.

The amount of memory reserved for the kdump kernel can be estimated with the following scheme:

base memory to be reserved = 160MB  
an additional 2 bits added for every 4 KB of physical RAM present in the system. 
So  for example if a system has 1TB of memory 224 MB is the minimum (160 + 64 MB).

Note: It is recommended to verify that kdump is working on all systems after installation of all applications. The memory reserved by crashkernel=auto takes only typical RHEL configurations into account. If 3rd party modules are used more memory may need to be reserved. If a testdump fails, it is a good strategy to verify if it works with crashkernel=768M@0M. It is recommended that until a test dump works without failure that kdump not be considered configured properly.

Note: RHEL7 with crashkernel=auto will only reserve memory on systems with 2GB or more physical memory. If the system has less than 2GB of memory the memory must be reserved by explicitly requesting the reservation size, for example: crashkernel=128M.

Note: Some environments still require manual configuration of the crashkernel option, for example if dumps to very large local filesystems are performed. Please refer to kdump fails with large ext4 file system because fsck.ext4 gets OOM-killed for details.

After adding the crashkernel parameter the system must be rebooted for the crashkernel memory to be reserved for use by kdump.

Where to find generated vmcores

When the hypervisor comes back up, the vmcore can be found by default under /var/crash/:

[root@host ~]# egrep '^path' /etc/kdump.conf
path /var/crash

[root@host ~]# ls /var/crash/127.0.0.1-2018-05-16-13\:15\:12/ -alh
total 102M
drwxr-xr-x. 2 root root   44 May 16 17:15 .
drwxr-xr-x. 3 root root   43 May 16 17:15 ..
-rw-------. 1 root root 102M May 16 17:15 vmcore
-rw-r--r--. 1 root root 252K May 16 17:15 vmcore-dmesg.txt

further information

Root Cause

A number of improvements related to crashkernel=auto and memory requirements of kdump have been made in RHEL7.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.