How to configure kdump over remote target path (NFS) on Red Hat CoreOS

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP) 4

Issue

  • Unable to capture vmcore over NFS on Red Hat CoreOS, seeing below error on console:
mount /kdumproot bad option 
or
mount.nfs Protocol not supported
or
Bad kdump network destination: xxx.xxx.xxx.xxx

Resolution

  • Mention the below parameters in /etc/kdump.conf to add necessary nfs modules and binaries to kdump initramfs.
extra_bins /sbin/mount.nfs 
extra_modules nfs nfsv3 nfs_layout_nfsv41_files blocklayoutdriver nfs_layout_flexfiles nfs_layout_nfsv41_files
  • Restart kdump service for cluster using SDN network.
$ systemctl restart kdump

Note: For clusters using OVN network, a reboot is needed

Root Cause

  • Necessary nfs kernel modules and system binaries supporting nfs mount functionality were not part of kdump initramfs which would cause the nfs mount to fail.
  • The nfs module is being omitted during the creation of kdump initramfs as nfs on root is not supported. This is done as a preventive measure on coreos. The below file plays a role while creating initramfs.
$  cat /lib/dracut/dracut.conf.d/coreos-omits.conf  
# We don't support root on NFS, so we don't need it in the initramfs. It also
# conflicts with /var mount support in ignition because NFS tries to mount stuff
# https://github.com/dracutdevs/dracut/blob/1856ae95c873a6fe855b3dccd0144f1a96b9e71c/modules.d/95nfs/nfs-start-rpc.sh#L7
omit_dracutmodules+=" nfs lvm iscsi dmraid "
  • The kdump initramfs on a working setup should have the below nfs kernel modules and system binaries(mount.nfs):
# lsinitrd /var/lib/kdump/initramfs-$(uname -r)kdump.img | grep -i nfs
Arguments:  --add 'kdumpbase' --quiet --hostonly --hostonly-cmdline --hostonly-i18n --hostonly-mode 'strict' --hostonly-nics '' -o 'plymouth resume ifcfg earlykdump' --mount 'x.x.x.x:/data /kdumproot nfs defaults' --squash-compressor 'zstd' --add-drivers ' nfs nfsv3 nfs_layout_nfsv41_files blocklayoutdriver nfs_layout_flexfiles nfs_layout_nfsv41_files' --no-hostonly-default-device -f
drwxr-xr-x root/root               129 2024-08-27 13:54 squashfs-root/usr/lib/modules/5.14.0-427.28.1.el9_4.x86_64/kernel/fs/nfs
drwxr-xr-x root/root                46 2024-08-27 13:54 squashfs-root/usr/lib/modules/5.14.0-427.28.1.el9_4.x86_64/kernel/fs/nfs/blocklayout
-rw-r--r-- root/root             27720 1970-01-01 00:00 squashfs-root/usr/lib/modules/5.14.0-427.28.1.el9_4.x86_64/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko.xz
drwxr-xr-x root/root                52 2024-08-27 13:54 squashfs-root/usr/lib/modules/5.14.0-427.28.1.el9_4.x86_64/kernel/fs/nfs/filelayout
-rw-r--r-- root/root             21072 1970-01-01 00:00 squashfs-root/usr/lib/modules/5.14.0-427.28.1.el9_4.x86_64/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko.xz
drwxr-xr-x root/root                49 2024-08-27 13:54 squashfs-root/usr/lib/modules/5.14.0-427.28.1.el9_4.x86_64/kernel/fs/nfs/flexfilelayout
-rw-r--r-- root/root             35052 1970-01-01 00:00 squashfs-root/usr/lib/modules/5.14.0-427.28.1.el9_4.x86_64/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko.xz
-rw-r--r-- root/root            188084 1970-01-01 00:00 squashfs-root/usr/lib/modules/5.14.0-427.28.1.el9_4.x86_64/kernel/fs/nfs/nfs.ko.xz
-rw-r--r-- root/root             30264 1970-01-01 00:00 squashfs-root/usr/lib/modules/5.14.0-427.28.1.el9_4.x86_64/kernel/fs/nfs/nfsv3.ko.xz
-rw-r--r-- root/root            310944 1970-01-01 00:00 squashfs-root/usr/lib/modules/5.14.0-427.28.1.el9_4.x86_64/kernel/fs/nfs/nfsv4.ko.xz
drwxr-xr-x root/root                55 2024-08-27 13:54 squashfs-root/usr/lib/modules/5.14.0-427.28.1.el9_4.x86_64/kernel/fs/nfs_common
-rw-r--r-- root/root              3660 1970-01-01 00:00 squashfs-root/usr/lib/modules/5.14.0-427.28.1.el9_4.x86_64/kernel/fs/nfs_common/grace.ko.xz
-rw-r--r-- root/root              4908 1970-01-01 00:00 squashfs-root/usr/lib/modules/5.14.0-427.28.1.el9_4.x86_64/kernel/fs/nfs_common/nfs_acl.ko.xz
-rwxr-xr-x root/root            100816 1970-01-01 00:00 squashfs-root/usr/sbin/mount.nfs

Diagnostic Steps

  • Check if all necessary nfs kernel modules and mount.nfs binary is part of kdump initramfs.
# lsinitrd /var/lib/kdump/initramfs-$(uname -r)kdump.img  | grep -i nfs 
Arguments:  --add 'kdumpbase' --quiet --hostonly --hostonly-cmdline --hostonly-i18n --hostonly-mode 'strict' --hostonly-nics '' -o 'plymouth resume ifcfg earlykdump' --mount 'x.x.x.x:/data/ /kdumproot nfs defaults' --squash-compressor 'zstd' --no-hostonly-default-device --add-drivers 'nvme' -f
drwxr-xr-x root/root                32 2024-04-03 20:13 squashfs-root/usr/lib/modules/5.14.0-284.48.1.el9_2.x86_64/kernel/fs/nfs
-rw-r--r-- root/root            213616 1970-01-01 00:00 squashfs-root/usr/lib/modules/5.14.0-284.48.1.el9_2.x86_64/kernel/fs/nfs/nfs.ko.xz
drwxr-xr-x root/root                34 2024-04-03 20:13 squashfs-root/usr/lib/modules/5.14.0-284.48.1.el9_2.x86_64/kernel/fs/nfs_common
-rw-r--r-- root/root              4936 1970-01-01 00:00 squashfs-root/usr/lib/modules/5.14.0-284.48.1.el9_2.x86_64/kernel/fs/nfs_common/grace.ko.xz
  • Check for below errors in serial console logs:
mount /kdumproot bad option 
or
mount.nfs Protocol not supported
or
Bad kdump network destination: xxx.xxx.xxx.xxx
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.