LUKS devices bound to TPM2 are not decrypted anymore after boot

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 8 and later
    • LUKS used for root devices
    • Clevis TPM2 decryption

Issue

  • After installing a system through the network and kickstart with a %post script to automatically unlock the LUKS devices through TPM2, the LUKS devices do not get automatically unlocked

  • After installing a system using the DVD and kickstart with a %post script to automatically unlock the LUKS devices through TPM2, the LUKS devices do not get automatically unlocked

  • Booting a system which has its LUKS devices bound to TPM2 doesn't get its devices unlocked automatically anymore even though this was working in the past

Resolution

Assuming the PCR 7 register is already used to unlock the LUKS devices, the issue may happen:

  1. on already installed systems in fhe following cases:

    1.1. after the shim-x64 package has been updated and system rebooted

    1.2. after the Secure Boot state has been changed

  2. on systems right after installation in the following cases:

    2.1. when network installation has been used and the BOOTX64.efi boot loader specified during UEFI PXE or HTTP phase is too old compared to the file shipped by the shim-x64 package that will be installed on the system

    2.2. when the RHEL 8.x DVD has been used but the repository used to install the system are not the ones shipped with the DVD

In case other PCR registers are used to unlock the LUKS devices, the issue may happen after firmware update or due to other conditions, since the other registers are very unstable.


Case where the issue happens on the first boot after installing the system through the network and a kickstart

Reinstall the system after making sure you follow the guide lines below.

  1. Make sure that when installing through the network, the BOOTX64.efi file matching the OS release you are installing is used during UEFI PXE or HTTP phase.

    Don't use the grubx64.efi boot loader in any case, even if the system is booting in non-Secure Boot.

    Examples:

    • If RHEL8.4 is to be installed and packages from the DVD level will be used, use BOOTX64.efi file found on the RHEL 8.4 DVD as boot loader in the DHCP configuration.

    • If packages from the CDN repository are to be installed on the system, download the shim-x64 package and use the BOOTX64.efi file shipped by the package (you may use rpm2cpio to extract the package content).

  2. Make sure that PCR 7 is used when binding the LUKS devices to TPM2, as shown in the example below

    %post
    clevis luks bind -f -k - -d /dev/vda3 tpm2 '{"key":"ecc", "pcr_ids":"7"}' <<< "temppass"
    %end
    

    Note: here above we use a ECC key instead of RSA because ECC is far way faster (something like 12 times faster).

  3. Make sure to set some complicated pass phrase for the LUKS devices, in case TPM2 unlocking doesn't work, otherwise recovery may become impossible at all.


Case where the issue happens after a reboot post installation of security fixes that updated the shim-x64 package

The recovery procedure depends whether you have a fail safe pass phrase to unlock or not:

>> If you have a pass phrase set for the LUKS devices, perform the steps below

The operations have to be performed on all LUKS devices (in the example below there is only one LUKS device: /dev/vda3).

  1. Unlock manually the devices using the pass phrase

  2. Collect the slot that holds the Clevis token

    # clevis luks list -d /dev/vda3
    0: tpm2 '{"hash":"sha256","key":"ecc","pcr_bank":"sha1","pcr_ids":"7"}'
    

    From above, we can see that the slot is use is slot 0.

  3. Regenerate the binding

    # clevis luks regen -d /dev/vda3 -s 0
    Regenerating binding (device /dev/vda3, slot 0):
    Pin: tpm2, Config: '{"hash":"sha256","key":"ecc","pcr_bank":"sha1","pcr_ids":"7"}'
    Do you want to proceed? [ynYN] Y
    Enter existing LUKS password: <YOUR PASS PHRASE>
    Warning: Value 512 is outside of the allowed entropy range, adjusting it.
    Binding regenerated successfully
    

>> If you don't have a pass phrase set for the LUKS devices, perform the recovery steps below

  1. Boot the system onto the DVD into Troubleshooting mode

    Troubleshooting -->
        Rescue a Red Hat Enterprise Linux system
    
  2. Drop to the shell without trying to mount the root device, by selecting 3) Skip to shell

    1) Continue
    2) Read-only mount
    3) Skip to shell
    4) Quit (Reboot)
    
    Please make a selection from the above: 3
    
  3. Mount the UEFI file system as /boot/efi and collect the date of the boot loader file

    In the example below, the UEFI partition is hosted on /dev/vda1.

    sh-4.4# mkdir /boot/efi && mount /dev/vda1 /boot/efi
    sh-4.4# ls -ld /boot/efi/EFI/redhat /boot/efi/EFI/redhat/shimx64.efi
    drwxr-xr-x. 3 root root   4096 Jul  9 07:40 /boot/efi/EFI/redhat
    -rwxr-xr-x. 1 root root 924888 Apr  7 15:17 /boot/efi/EFI/redhat/shimx64.efi
    

    In the example above, we can see that shimx64.efi is dated Apr 7 15:17 and the top directory /boot/efi/EFI/redhat was modified on Jul 9 07:40 which hints that some update of the shim-x64 packages was likely performed on Jul 9.

  4. On a Linux system, browse This content is not included.Red Hat's Package Browser to download the two latest releases of the shim-x64 package the extract the packages to check the dates of the files being shipped

    We assume here that shim-x64-15.4-2.el8_1 is the latest release (as tagged on the Package Browser page) and shim-x64-15-16.el8 is the previous one:

    4.1. Extract the packages

     mydesktop $ cd Downloads
     mydesktop $ LATEST=shim-x64-15.4-2.el8_1
     mydesktop $ PREVIOUS=shim-x64-15-16.el8
     mydesktop $ for pkg in $LATEST $PREVIOUS; do (mkdir $pkg && cd $pkg && rpm2cpio ../$pkg.x86_64.rpm | cpio -icdmu); done
    

    4.2. Check the dates

     mydesktop $ for pkg in $LATEST $PREVIOUS; do ls -l $pkg/boot/efi/EFI/redhat/shimx64.efi; done
     -rwx------. 1 rmetrich rmetrich 924888 Apr  7 17:17 shim-x64-15.4-2.el8_1/boot/efi/EFI/redhat/shimx64.efi
     -rwx------. 1 rmetrich rmetrich 1244488 Sep 22  2020 shim-x64-15-16.el8/boot/efi/EFI/redhat/shimx64.efi
    

    In the example above, we can see that shim-x64-15.4-2.el8_1 matches the date found on the system being rescued (note that the timezone may differ).
    We can hence expect that shim-x64 was updated to shim-x64-15.4-2.el8_1 and the previous package we need to recover the system is the previous one: shim-x64-15-16.el8.

  5. Set up the network and collect the shim-x64 stored on the Linux system

    Note: in the rescue environment, there is no sshd daemon, so you need to collect the package using ftp or scp command.
    The network has to be set up using ip link / ip addr commands in case of static networking or dhclient <dev> in case of DHCP.

    sh-4.4# dhclient enp1s0
    sh-4.4# scp rmetrich@mydesktop:Downloads/shim-x64-15-16.el8.x86_64.rpm /tmp/
    
  6. Extract the package to overwrite the files in /boot/efi and verify the date of the boot loader file

    sh-4.4# cd / && rpm2cpio /tmp/shim-x64-15-16.el8.x86_64.rpm | cpio -icdmu
    sh-4.4# ls -ld /boot/efi/EFI/redhat/shimx64.efi
    -rwxr-xr-x. 1 root root 1244488 Sep 22  2020 /boot/efi/EFI/redhat/shimx64.efi
    

    In the example above, we overwrote the file with the previous version, dated Sep 22 2020.

  7. Exit from the rescue and reboot on the hard disk

    sh-4.4# exit
    

    If the system doesn't boot with LUKS automatic unlocking, redo the steps with a more ancient shim-x64 package.

    Otherwise perform the final steps below to set up a fail safe pass phrase on the LUKS devices.

    The operations have to be performed on all LUKS devices (in the example below /dev/vda3 is the LUKS device).

  8. Collect the slot that holds the Clevis token

    # clevis luks list -d /dev/vda3
    0: tpm2 '{"hash":"sha256","key":"ecc","pcr_bank":"sha1","pcr_ids":"7"}'
    

    From above, we can see that the slot is use is slot 0.

  9. Collect the Clevis pass phrase stored in the above slot

    # echo $(clevis luks pass -d /dev/vda3 -s 0)
    <CLEVISPASSPHRASE>
    
  10. Add a fail safe pass phrase on the LUKS device

    # cryptsetup luksAddKey /dev/vda3
    Enter any existing passphrase: <CLEVISPASSPHRASE>
    Enter new passphrase for key slot: <SOME FAIL SAFE PASS PHRASE>
    Verify passphrase: <SOME FAIL SAFE PASS PHRASE>
    
  11. Reinstall the shim-x64 package to make sure it's now the newer available package

    # yum -y reinstall shim-x64
    
  12. Reboot the system and follow the instructions again

    This time you will meet the first case: If you have a pass phrase set for the LUKS devices, unlock manually the devices then regenerate the bindings.

Root Cause

It may happen that the PCR registers used to bind the LUKS devices against changed value.
Most PCR registers are not stable and should not be used, that's why Red Hat recommends only using PCR 7.

PCR 7 is quite stable but may change in the near future due to SBAT vulnerability fix which will like change the SbatLevelRT UEFI variable value.
PCR 7 also changes value when updating from RHEL 8.x.0 "DVD level" to more recent update of 8.x, due to updating the shim-x64 package, which was performed to introduce the SbatLevelRT UEFI variable.

Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.