How should the crashkernel parameter be configured for using kdump on Red Hat Enterprise Linux 8 ?

Environment

• Red Hat Enterprise Linux (RHEL) 8
• all supported architectures
• kdump

Issue

• When configuring the crashkernel parameter the kdump service fails to start:

# systemctl start kdump
Job for kdump.service failed because the control process exited with error code.
See "systemctl status kdump.service" and "journalctl -xe" for details.
# systemctl status kdump
● kdump.service - Crash recovery kernel arming
   Loaded: loaded (/usr/lib/systemd/system/kdump.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2018-11-16 09:07:04 CET; 10s ago
  Process: 1164 ExecStart=/usr/bin/kdumpctl start (code=exited, status=1/FAILURE)
 Main PID: 1164 (code=exited, status=1/FAILURE)

11月 16 09:07:03 rhel8a.example.net systemd[1]: Starting Crash recovery kernel arming...
11月 16 09:07:04 rhel8a.example.net kdumpctl[1164]: Could not find a free area of memory of 0x1edb000 bytes...
11月 16 09:07:04 rhel8a.example.net kdumpctl[1164]: locate_hole failed
11月 16 09:07:04 rhel8a.example.net kdumpctl[1164]: kexec: failed to load kdump kernel
11月 16 09:07:04 rhel8a.example.net kdumpctl[1164]: Starting kdump: [FAILED]
11月 16 09:07:04 rhel8a.example.net systemd[1]: kdump.service: Main process exited, code=exited, status=1/FAILURE
11月 16 09:07:04 rhel8a.example.net systemd[1]: kdump.service: Failed with result 'exit-code'.
11月 16 09:07:04 rhel8a.example.net systemd[1]: Failed to start Crash recovery kernel arming.
# 

• What is the correct crashkernel parameter for kdump to work?
• "crashkernel reservation failed - memory is in use" errors when kernel panics

Resolution

The kdump procedure

The received warning means the kdump operation might fail and the crashdump parameter should be configured correctly. This is the procedure of kdumping:

1. The normal kernel is booted with crashkernel=... as a kernel option, reserving some memory for the kdump kernel. The memory reserved by the crashkernel parameter is not available to the normal kernel during regular operation.  It is reserved for later use by the kdump kernel.
2. The system panics.
3. The kdump kernel is booted using kexec, it used the memory area that was reserved w/ the crashkernel parameter.
4. The normal kernel's memory is captured into a vmcore.

Note: Not reserving enough memory for the kdump kernel can lead to the kdump operation failing.

Configuring crashkernel on Red Hat Enterprise Linux 8 kernels

• Use grub2-editenv or grubby to update crashkernel parameter, and reboot to apply the new configuration. For more details, kindly refer the following KB and Doc:
  How to manually modify the boot parameter in grub before the system boots
  Chapter 40. Configuring kernel command-line parameters Red Hat Enterprise Linux 8

• With Red Hat Enterprise Linux 8 kernels, crashkernel=auto should usually be used. The kernel will automatically reserve an appropriate amount of memory for the kdump kernel.

• Keep in mind that it is an algorithmically calculated memory reservation and may not meet the needs of all systems (especially for configurations with many IO cards and loaded kernel drivers). Ensure that memory reserved by crashkernel=auto is sufficient for the target machine by testing kdump. If it is insufficient, reserve more memory using syntax crashkernel=XM where X is amount of memory to be reserved in megabytes.

• The amount of memory reserved for the kdump kernel can be estimated with the following scheme:

For example, on a x86_64 system with 128GB of memory, crashkernel=auto will reserve 256MB for the kdump kernel. On a 32GB arm64 system, crashkernel=auto will reserve 512MB for the kdump kernel.

Note: It is recommended to verify that kdump is working on all systems after installation of all applications. The memory reserved by crashkernel=auto takes only typical Red Hat Enterprise Linux configurations into account. If Third party modules are used more memory may need to be reserved. If a testdump fails, it is a good strategy to verify if it works with crashkernel=256M or crashkernel=512M. It is recommended to only consider kdump as configured properly, once a test dump was successfully obtained.

Note: Red Hat Enterprise Linux 8 with crashkernel=auto will only work above a particular memory threshold. For e.g on x86_64 systems, crashkernel=auto will reserve memory on systems with 1GB or more physical memory. If the system has less than 1GB of memory, the memory must be reserved by explicitly requesting the reservation size: crashkernel=160M. Similarly, for s390x arch, crashkernel=auto will only work on systems with memory 1GB and above. Below that, one has to manually assign the value of crashkernel as crashkernel=160M which is the base value.

Note: Above table reflects the codebase of the last released RHEL8 minor release. Patches in rhel8.2 and rhel8.3 have changed these allocations. The values for older minor releases are in kbase5907911.

• After adding the crashkernel parameter the system must be rebooted for the crashkernel memory to be reserved for use by kdump. The command free will show a lower amount of memory available for the system. dmesg also shows details.

Where to find generated vmcores

When the system becomes available again, the vmcore can be found by default under /var/crash/:

# egrep '^path' /etc/kdump.conf
path /var/crash

# ls /var/crash/127.0.0.1-2018-05-16-13\:15\:12/ -alh
total 102M
drwxr-xr-x. 2 root root   44 May 16 17:15 .
drwxr-xr-x. 3 root root   43 May 16 17:15 ..
-rw-------. 1 root root 102M May 16 17:15 vmcore
-rw-r--r--. 1 root root 252K May 16 17:15 vmcore-dmesg.txt

Further information

• How should the crashkernel parameter be configured for using kdump on Red Hat Enterprise Linux?
• How should the crashkernel parameter be configured for using kdump on Red Hat Enterprise Linux 7?
• RHEL8 product documentation: Supported kdump configurations and targets

Root Cause

A number of improvements related to crashkernel=auto and memory requirements of kdump have been made in Red Hat Enterprise Linux 8.

In Red Hat Enterprise Linux 7, on top of a base memory amount, "an additional 2 bits added for every 4 KB of physical RAM" were reserved. This scheme was dropped in Red Hat Enterprise Linux 8 since makedumpfile has changed to use cyclic mode by default and makedumpfile does not need the extra memory in cyclic mode.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.