How should the crashkernel parameter be configured for using kdump on Red Hat Enterprise Linux 9 ?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 9
  • all supported architectures
  • kdump

Issue

  • When configuring the crashkernel parameter the kdump service fails to start:
# systemctl status kdump.service 
○ kdump.service - Crash recovery kernel arming
     Loaded: loaded (/usr/lib/systemd/system/kdump.service; enabled; vendor preset: enabled)
     Active: inactive (dead)
  Condition: start condition failed at Wed 2022-01-26 23:28:22 IST; 8s ago
             └─ ConditionKernelCommandLine=crashkernel was not met

Jan 26 23:26:16 localhost systemd[1]: Condition check resulted in Crash recovery kernel arming being skipped.
Jan 26 23:28:22 localhost.localdomain systemd[1]: Condition check resulted in Crash recovery kernel arming being skipped.
  • What is the correct crashkernel parameter for kdump to work?
  • "crashkernel reservation failed - memory is in use" errors when kernel panics

Resolution

The kdump procedure

The received warning means the kdump operation might fail and the crashdump parameter should be configured correctly. This is the procedure of kdumping:

  1. The normal kernel is booted with crashkernel=... as a kernel option, reserving some memory for the kdump kernel. The memory reserved by the crashkernel parameter is not available to the normal kernel during regular operation. It is reserved for later use by the kdump kernel.
  2. The system panics.
  3. The kdump kernel is booted using kexec, it used the memory area that was reserved w/ the crashkernel parameter.
  4. The normal kernel's memory is captured into a vmcore.

Note: Not reserving enough memory for the kdump kernel can lead to the kdump operation failing.

Configuring crashkernel on Red Hat Enterprise Linux 9 kernels

There are two ways through which we can set the crashkernel parameter:

1. Using kdumpctl reset-crashkernel command:

# kdumpctl reset-crashkernel

After running the above command, we can check using grubby command whether crashkernel parameter has been set or not:

# grubby --info DEFAULT
index=0
kernel="/boot/vmlinuz-5.14.0-39.el9.x86_64"
args="ro resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M"
root="/dev/mapper/rhel-root"
initrd="/boot/initramfs-5.14.0-39.el9.x86_64.img"
title="Red Hat Enterprise Linux (5.14.0-39.el9.x86_64) 9.0 (Plow)"
id="6b56f578b88e45aeaf1502729b134106-5.14.0-39.el9.x86_64"

Since, it's kernel command line parameter, reboot is required to take it's effect,

# reboot

To check whether crashkernel parameter has taken effect we can use below two methods,

  • From /proc/cmdline,
# cat /proc/cmdline 
BOOT_IMAGE=(hd0,msdos1)/vmlinuz-5.14.0-39.el9.x86_64 root=/dev/mapper/rhel-root ro resume=/dev/mapper/rhel-swap rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb quiet crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M
  • Using showmem or estimate sub-command,
# kdumpctl showmem
kdump: Reserved 192MB memory for crash kernel

# kdumpctl estimate
Reserved crashkernel:    192M
Recommended crashkernel: 192M

Kernel image size:   48M
Kernel modules size: 7M
Initramfs size:      31M
Runtime reservation: 64M
Large modules:
    xfs: 1982464
    kvm: 1056768

2. Change the crashkernel by using grubby or by modifying /etc/default/grub. Refer to How to modify the kernel command-line in Red Hat Enterprise Linux 9.

NOTE: Starting with RHEL9 kernels, crashkernel=auto is not supported. To automatically reserve an appropriate amount of memory for the kdump kernel use crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M value. For more information refer:crashkernel=auto parameter is deprecated in RHEL9.
NOTE: Note: The classic syntax like crashkernel=size[@offset] is also available, though it is recommended to use crashkernel=1G-4G:192M,4G-64G:256M,64G-:512M. If the classic syntax sets, please set auto_reset_crashkernel no to avoid overwriting the value when updating kexec-tools. Otherwise this value will be overwritten to default value: See the KCS

Where to find generated vmcores

When the system becomes available again, the vmcore can be found by default under /var/crash/:

# egrep '^path' /etc/kdump.conf
path /var/crash

# ls /var/crash/127.0.0.1-2022-01-30-21\:26\:38/ -alh
total 79M
drwxr-xr-x. 2 root root  67 Jan 30 21:26 .
drwxr-xr-x. 3 root root  43 Jan 30 21:26 ..
-rw-------. 1 root root 62K Jan 30 21:26 kexec-dmesg.log
-rw-------. 1 root root 78M Jan 30 21:26 vmcore
-rw-------. 1 root root 54K Jan 30 21:26 vmcore-dmesg.txt

Further information


[How should the crashkernel parameter be configured for using kdump on Red Hat Enterprise Linux?](https://access.redhat.com/solutions/1186753)
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.