LUKS devices bound to TPM2 are not decrypted anymore after boot
Environment
- Red Hat Enterprise Linux 8 and later
- LUKS used for root devices
- Clevis TPM2 decryption
Issue
-
After installing a system through the network and kickstart with a %post script to automatically unlock the LUKS devices through TPM2, the LUKS devices do not get automatically unlocked
-
After installing a system using the DVD and kickstart with a %post script to automatically unlock the LUKS devices through TPM2, the LUKS devices do not get automatically unlocked
-
Booting a system which has its LUKS devices bound to TPM2 doesn't get its devices unlocked automatically anymore even though this was working in the past
Resolution
Assuming the PCR 7 register is already used to unlock the LUKS devices, the issue may happen:
-
on already installed systems in fhe following cases:
1.1. after the
shim-x64package has been updated and system rebooted1.2. after the Secure Boot state has been changed
-
on systems right after installation in the following cases:
2.1. when network installation has been used and the
BOOTX64.efiboot loader specified during UEFI PXE or HTTP phase is too old compared to the file shipped by theshim-x64package that will be installed on the system2.2. when the RHEL 8.x DVD has been used but the repository used to install the system are not the ones shipped with the DVD
In case other PCR registers are used to unlock the LUKS devices, the issue may happen after firmware update or due to other conditions, since the other registers are very unstable.
Case where the issue happens on the first boot after installing the system through the network and a kickstart
Reinstall the system after making sure you follow the guide lines below.
-
Make sure that when installing through the network, the
BOOTX64.efifile matching the OS release you are installing is used during UEFI PXE or HTTP phase.Don't use the
grubx64.efiboot loader in any case, even if the system is booting in non-Secure Boot.Examples:
-
If RHEL8.4 is to be installed and packages from the DVD level will be used, use
BOOTX64.efifile found on the RHEL 8.4 DVD as boot loader in the DHCP configuration. -
If packages from the CDN repository are to be installed on the system, download the
shim-x64package and use theBOOTX64.efifile shipped by the package (you may userpm2cpioto extract the package content).
-
-
Make sure that PCR 7 is used when binding the LUKS devices to TPM2, as shown in the example below
%post clevis luks bind -f -k - -d /dev/vda3 tpm2 '{"key":"ecc", "pcr_ids":"7"}' <<< "temppass" %endNote: here above we use a ECC key instead of RSA because ECC is far way faster (something like 12 times faster).
-
Make sure to set some complicated pass phrase for the LUKS devices, in case TPM2 unlocking doesn't work, otherwise recovery may become impossible at all.
Case where the issue happens after a reboot post installation of security fixes that updated the shim-x64 package
The recovery procedure depends whether you have a fail safe pass phrase to unlock or not:
>> If you have a pass phrase set for the LUKS devices, perform the steps below
The operations have to be performed on all LUKS devices (in the example below there is only one LUKS device: /dev/vda3).
-
Unlock manually the devices using the pass phrase
-
Collect the slot that holds the Clevis token
# clevis luks list -d /dev/vda3 0: tpm2 '{"hash":"sha256","key":"ecc","pcr_bank":"sha1","pcr_ids":"7"}'From above, we can see that the slot is use is slot 0.
-
Regenerate the binding
# clevis luks regen -d /dev/vda3 -s 0 Regenerating binding (device /dev/vda3, slot 0): Pin: tpm2, Config: '{"hash":"sha256","key":"ecc","pcr_bank":"sha1","pcr_ids":"7"}' Do you want to proceed? [ynYN] Y Enter existing LUKS password: <YOUR PASS PHRASE> Warning: Value 512 is outside of the allowed entropy range, adjusting it. Binding regenerated successfully
>> If you don't have a pass phrase set for the LUKS devices, perform the recovery steps below
-
Boot the system onto the DVD into Troubleshooting mode
Troubleshooting --> Rescue a Red Hat Enterprise Linux system -
Drop to the shell without trying to mount the root device, by selecting 3) Skip to shell
1) Continue 2) Read-only mount 3) Skip to shell 4) Quit (Reboot) Please make a selection from the above: 3 -
Mount the UEFI file system as
/boot/efiand collect the date of the boot loader fileIn the example below, the UEFI partition is hosted on
/dev/vda1.sh-4.4# mkdir /boot/efi && mount /dev/vda1 /boot/efi sh-4.4# ls -ld /boot/efi/EFI/redhat /boot/efi/EFI/redhat/shimx64.efi drwxr-xr-x. 3 root root 4096 Jul 9 07:40 /boot/efi/EFI/redhat -rwxr-xr-x. 1 root root 924888 Apr 7 15:17 /boot/efi/EFI/redhat/shimx64.efiIn the example above, we can see that
shimx64.efiis dated Apr 7 15:17 and the top directory/boot/efi/EFI/redhatwas modified on Jul 9 07:40 which hints that some update of theshim-x64packages was likely performed on Jul 9. -
On a Linux system, browse This content is not included.Red Hat's Package Browser to download the two latest releases of the
shim-x64package the extract the packages to check the dates of the files being shippedWe assume here that
shim-x64-15.4-2.el8_1is the latest release (as tagged on the Package Browser page) andshim-x64-15-16.el8is the previous one:4.1. Extract the packages
mydesktop $ cd Downloads mydesktop $ LATEST=shim-x64-15.4-2.el8_1 mydesktop $ PREVIOUS=shim-x64-15-16.el8 mydesktop $ for pkg in $LATEST $PREVIOUS; do (mkdir $pkg && cd $pkg && rpm2cpio ../$pkg.x86_64.rpm | cpio -icdmu); done4.2. Check the dates
mydesktop $ for pkg in $LATEST $PREVIOUS; do ls -l $pkg/boot/efi/EFI/redhat/shimx64.efi; done -rwx------. 1 rmetrich rmetrich 924888 Apr 7 17:17 shim-x64-15.4-2.el8_1/boot/efi/EFI/redhat/shimx64.efi -rwx------. 1 rmetrich rmetrich 1244488 Sep 22 2020 shim-x64-15-16.el8/boot/efi/EFI/redhat/shimx64.efiIn the example above, we can see that
shim-x64-15.4-2.el8_1matches the date found on the system being rescued (note that the timezone may differ).
We can hence expect thatshim-x64was updated toshim-x64-15.4-2.el8_1and the previous package we need to recover the system is the previous one:shim-x64-15-16.el8. -
Set up the network and collect the
shim-x64stored on the Linux systemNote: in the rescue environment, there is no
sshddaemon, so you need to collect the package usingftporscpcommand.
The network has to be set up usingip link / ip addrcommands in case of static networking ordhclient <dev>in case of DHCP.sh-4.4# dhclient enp1s0 sh-4.4# scp rmetrich@mydesktop:Downloads/shim-x64-15-16.el8.x86_64.rpm /tmp/ -
Extract the package to overwrite the files in
/boot/efiand verify the date of the boot loader filesh-4.4# cd / && rpm2cpio /tmp/shim-x64-15-16.el8.x86_64.rpm | cpio -icdmu sh-4.4# ls -ld /boot/efi/EFI/redhat/shimx64.efi -rwxr-xr-x. 1 root root 1244488 Sep 22 2020 /boot/efi/EFI/redhat/shimx64.efiIn the example above, we overwrote the file with the previous version, dated Sep 22 2020.
-
Exit from the rescue and reboot on the hard disk
sh-4.4# exitIf the system doesn't boot with LUKS automatic unlocking, redo the steps with a more ancient
shim-x64package.Otherwise perform the final steps below to set up a fail safe pass phrase on the LUKS devices.
The operations have to be performed on all LUKS devices (in the example below
/dev/vda3is the LUKS device). -
Collect the slot that holds the Clevis token
# clevis luks list -d /dev/vda3 0: tpm2 '{"hash":"sha256","key":"ecc","pcr_bank":"sha1","pcr_ids":"7"}'From above, we can see that the slot is use is slot 0.
-
Collect the Clevis pass phrase stored in the above slot
# echo $(clevis luks pass -d /dev/vda3 -s 0) <CLEVISPASSPHRASE> -
Add a fail safe pass phrase on the LUKS device
# cryptsetup luksAddKey /dev/vda3 Enter any existing passphrase: <CLEVISPASSPHRASE> Enter new passphrase for key slot: <SOME FAIL SAFE PASS PHRASE> Verify passphrase: <SOME FAIL SAFE PASS PHRASE> -
Reinstall the
shim-x64package to make sure it's now the newer available package# yum -y reinstall shim-x64 -
Reboot the system and follow the instructions again
This time you will meet the first case: If you have a pass phrase set for the LUKS devices, unlock manually the devices then regenerate the bindings.
Root Cause
It may happen that the PCR registers used to bind the LUKS devices against changed value.
Most PCR registers are not stable and should not be used, that's why Red Hat recommends only using PCR 7.
PCR 7 is quite stable but may change in the near future due to SBAT vulnerability fix which will like change the SbatLevelRT UEFI variable value.
PCR 7 also changes value when updating from RHEL 8.x.0 "DVD level" to more recent update of 8.x, due to updating the shim-x64 package, which was performed to introduce the SbatLevelRT UEFI variable.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.