Why does kdump fails to dump vmcore on OCP node with cipher_null-ecb type disk encryption?

Environment

• RHOCP 4.12

Issue

• Why does kdump fails to dump vmcore on OCP node with cipher_null-ecb type disk encryption?

[Mon Apr  8 12:05:16 2024]  sda: sda1 sda2 sda3 sda4
[Mon Apr  8 12:05:16 2024] sd 2:0:0:0: [sda] Attached SCSI disk
[Mon Apr  8 12:05:18 2024] SGI XFS with ACLs, security attributes, quota, no debug enabled
[Mon Apr  8 12:05:18 2024] XFS (dm-0): Mounting V5 Filesystem
[Mon Apr  8 12:05:18 2024] XFS (dm-0): Starting recovery (logdev: internal)
[Mon Apr  8 12:05:18 2024] XFS (dm-0): Ending recovery (logdev: internal)
[Mon Apr  8 12:05:18 2024] systemd[1]: Condition check resulted in /sysroot being skipped.
[Mon Apr  8 12:05:18 2024] systemd[1]: Started CoreOS: Mount (subsequent) /sysroot.
[Mon Apr  8 12:08:08 2024] dracut-initqueue[466]: Warning: dracut-initqueue timeout - starting timeout scripts
[Mon Apr  8 12:08:09 2024] dracut-initqueue[466]: Warning: dracut-initqueue timeout - starting timeout scripts
[Mon Apr  8 12:08:10 2024] dracut-initqueue[466]: Warning: dracut-initqueue timeout - starting timeout scripts
...

Resolution

• Add rhcos.root=crypt_rootfs and rd.luks=0 in KDUMP_COMMANDLINE_APPEND in the file `/etc/sysconfig/kdump.

• • Additionally, also append rhcos.root=crypt_rootfs in KDUMP_COMMANDLINE_REMOVE section. This is done due to a known This content is not included.bug where kdump uses incorrect argument.

• After modifying both the lines will look like below

KDUMP_COMMANDLINE_REMOVE="hugepages hugepagesz slub_debug quiet log_buf_len swiotlb rhcos.root=crypt_rootfs"
KDUMP_COMMANDLINE_APPEND="irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr novmcoredd hest_disable rhcos.root=crypt_rootfs rd.luks=0"

• NOTE - This workaround is only for coreos installation with cipher_null-ecb cipher used. This is not tested for coreos disk encryption which decrypts using tpm or tang.

Root Cause

• CoreOS sets up a dummy disk encryption and a service called CoreOS LUKS Opener is responsible to decrypting the disk during system bootup by referring presence of rhcos.root=crypt_rootfs in kernel commandline.
• However kdump fails to add that in kdump commandline and it only adds rhcos.. Refer below which shows snippet from /var/log/kdump.log

+ 2024-03-04 12:28:30 /usr/bin/kdumpctl@698: /sbin/kexec -s -d -p '--command-line=BOOT_IMAGE=(hd0,gpt1)/ostree/rhcos-d51cf071f47a741e53075d859adf5e8ccc8585404c3363cd683f5731011c7280/vmlinuz-4.18.0-372.89.1.el8_6.x86_64 rhcos. random.trust_cpu=on console=tty0 console=ttyS0,115200n8 ignition.platform.id=metal rd.luks.options=discard ostree=/ostree/boot.0/rhcos/d51cf071f47a741e53075d859adf5e8ccc8585404c3363cd683f5731011c7280/0 irqpoll nr_cpus=1 reset_devices cgroup_disable=memory mce=off numa=off udev.children-max=2 panic=10 rootflags=nofail acpi_no_memhotplug transparent_hugepage=never nokaslr novmcoredd hest_disable disable_cpu_apicid=0' --initrd=/var/lib/kdump/initramfs-4.18.0-372.89.1.el8_6.x86_64kdump.img /boot/ostree/rhcos-d51cf071f47a741e53075d859adf5e8ccc8585404c3363cd683f5731011c7280/vmlinuz-4.18.0-372.89.1.el8_6.x86_64

Diagnostic Steps

• Check the output of luksDump to verify if keyslot is using cipher_null-ecb

$ cryptsetup luksDump /dev/sda4
LUKS header information
Version:       	2
Epoch:         	5
Metadata area: 	16384 [bytes]
Keyslots area: 	16744448 [bytes]
UUID:          	00000000-0000-4000-a000-000000000002
Label:         	crypt_rootfs
Subsystem:     	(no subsystem)
Flags:       	(no flags)

Data segments:
  0: crypt
	offset: 16777216 [bytes]
	length: (whole device)
	cipher: cipher_null-ecb   <----
	sector: 512 [bytes]

Keyslots:
  0: luks2
	Key:        256 bits
	Priority:   normal
	Cipher:     cipher_null-ecb   <----
	Cipher key: 256 bits
	PBKDF:      argon2i
	Time cost:  4
	Memory:     524288
	Threads:    1
	Salt:       d0 fb 04 f9 2d f8 7e 74 d4 6e fc 4f bc 15 fe 3f 
	            a3 ed 31 1e 77 d2 03 c8 da 78 43 5c 99 7a 6f 88 
	AF stripes: 4000
	AF hash:    sha256
	Area offset:32768 [bytes]
	Area length:131072 [bytes]
	Digest ID:  0
Tokens:
  9: coreos
	Keyslot:  0
Digests:
  0: pbkdf2
	Hash:       sha256
	Iterations: 238312
	Salt:       48 c2 ba e6 46 d7 74 1e 1e f9 b6 1d e9 e2 af 40 
	            03 a1 0b 87 af 42 38 7d 12 77 25 4f 9d d6 45 4e 
	Digest:     05 7c 95 49 56 f7 08 c4 16 23 b0 96 f6 ed a0 3e 
	            13 3f bc 78 64 bf eb f2 b6 c2 96 0b 97 75 f3 2c 

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.