How to set up Network-Bound Disk Encryption with multiple LUKS devices (Clevis+Tang unlocking)

Updated

This article provides additional details to the official Red Hat Documentation Configuring Automated Unlocking Of Encrypted Volumes Using Policy-Based Decryption.

After reading the official documentation, you may then read this article to gain more knowledge on the implementation details and restrictions applying to this solution.

This article applies to RHEL 7.5 and later.

Table of contents

1. Prerequisites

We assume that the Tang server has already been installed and configured.

We also assume that the official Red Hat Documentation Configuring Automated Unlocking Of Encrypted Volumes Using Policy-Based Decryption has been read carefully.

We finally assume that the necessary packages have been installed on the system:

# yum -y install clevis-dracut

In the rest of the document, the Tang server will be referenced as TANGSERVER, its signing key as TANGSIGNINGKEY and the LUKS password as LUKSKEY.

The signing key can be retrieved using the following command:

# clevis encrypt tang '{"url":"http://TANGSERVER"}' <<< FOO
The advertisement contains the following signing keys:

TANGSIGNINGKEY

Do you wish to trust these keys? [ynYN] n

2. Binding the LUKS devices to Tang server

Usually, all LUKS devices will be bound to the same Tang server, but it's not mandatory of course.

To easily bind all LUKS devices to the Tang server, the following script can be used:

# for dev in $(lsblk -p -n -s -r | awk '$6 == "crypt" { getline; print $1 }' | sort -u); do
	clevis luks bind -f -k- -d $dev tang '{"url":"http://TANGSERVER","thp":"TANGSIGNINGKEY"}' <<< "LUKSKEY" 
done

Note: on RHEL7, you are advised to wipe out the key slots in inactive state prior to binding, see KCS 5636821 for details.

In the previous example, the LUKS key is "LUKSKEY". The script lists the devices hosting the LUKS device then calls the binding command automatically, without any confirmation from the user.

Depending on how the system was installed, there may be one or more LUKS devices per Volume Group or one LUKS device per Logical Volume.

2.1. LUKS on LVM

In this scenario, the system has been installed with LUKS on LVM. This happens when, in the installer, the administrator didn't choose any automatic partitioning and directly clicked on the + (Plus) button to create his own partitions.

When executing the script listed above, the dev variable will usually contain device mapper names related to a logical volume, as shown in the example below:

# lsblk -p -n -s -r | awk '$6 == "crypt" { getline; print $1 }' | sort -u
/dev/mapper/rhel-06
/dev/mapper/rhel-00
/dev/mapper/rhel-01
/dev/mapper/rhel-05
/dev/mapper/rhel-04
/dev/mapper/rhel-03
/dev/mapper/rhel-07
/dev/mapper/rhel-02
/dev/mapper/rhel-08

# lsblk
NAME                                            MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sr0                                              11:0    1  4.2G  0 rom   
vda                                             252:0    0   20G  0 disk  
├─vda1                                          252:1    0    1G  0 part  /boot
└─vda2                                          252:2    0   19G  0 part  
  ├─rhel-00                                     253:0    0    2G  0 lvm   
  │ └─luks-50e5d2d4-a66d-49d8-a082-cfbfc64de1f3 253:10   0    2G  0 crypt /
  ├─rhel-01                                     253:1    0    1G  0 lvm   
  │ └─luks-363925b7-390f-4ee8-b12f-385b607da96e 253:11   0    1G  0 crypt [SWAP]
  ├─rhel-02                                     253:2    0    1G  0 lvm   
  │ └─luks-e5121b4e-8bd7-4976-9b8f-2838d302ccb2 253:16   0    1G  0 crypt /tmp
  ├─rhel-03                                     253:3    0    2G  0 lvm   
  │ └─luks-e4cb239a-7328-4be0-b789-bc6376ccd36f 253:14   0    2G  0 crypt /var
  ├─rhel-04                                     253:4    0    1G  0 lvm   
  │ └─luks-c692c476-1a7a-4584-8170-8017bbc96680 253:13   0    1G  0 crypt /var/log
  ├─rhel-05                                     253:5    0    1G  0 lvm   
  │ └─luks-4d53f707-045f-4cb3-b810-638827f589f9 253:12   0    1G  0 crypt /var/log/audit
  ├─rhel-06                                     253:6    0    2G  0 lvm   
  │ └─luks-e3009a89-3d8d-40ce-b731-a7b949aed248 253:9    0    2G  0 crypt /usr
  ├─rhel-07                                     253:7    0    1G  0 lvm   
  │ └─luks-7671f775-9ea8-4e14-be6a-0c65eef92e48 253:15   0    1G  0 crypt /home
  └─rhel-08                                     253:8    0    8G  0 lvm   
    └─luks-623573fb-6f92-472a-b5fb-95783bfa4dcf 253:17   0    8G  0 crypt /opt

When having such setup, it will be necessary to decrypt all LUKS devices during the initramfs phase.
This will be explained in 3. Understanding which file systems are to be considered as "root volumes".

2.2. LVM on LUKS

In this scenario, the system has been installed with LVM on LUKS. This happens when, in the installer, the administrator chooses automatic partitioning then modified the partitions if necessary.

When executing the script listed above, the dev variable will usually contain raw device names, as shown in the example below:

# lsblk -p -n -s -r | awk '$6 == "crypt" { getline; print $1 }' | sort -u
/dev/vda2

# lsblk
NAME                                    MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sr0                                      11:0    1 1024M  0 rom   
vda                                     252:0    0   20G  0 disk  
├─vda1                                  252:1    0    1G  0 part  /boot
└─vda2                                  252:2    0   19G  0 part  
  └─luks-9443aec0-73e9-484c-8668-ef77e647c0f5
                                        253:0    0   19G  0 crypt 
    ├─rhel-root                         253:1    0    2G  0 lvm   /
    ├─rhel-swap                         253:2    0    2G  0 lvm   [SWAP]
    ├─rhel-usr                          253:3    0    2G  0 lvm   /usr
    ├─rhel-var                          253:4    0    2G  0 lvm   /var
    ├─rhel-var_log                      253:5    0    1G  0 lvm   /var/log
    ├─rhel-var_log_audit                253:6    0    1G  0 lvm   /var/log/audit
    ├─rhel-tmp                          253:7    0    1G  0 lvm   /tmp
    ├─rhel-home                         253:8    0    1G  0 lvm   /home
    └─rhel-opt                          253:9    0    7G  0 lvm   /opt

When having such setup, it will only be necessary to decrypt the LUKS devices making the LVM Volume Group.

3. Understanding which file systems are to be considered as "root volumes"

Clevis requires the network to be up to decrypt a LUKS device. Hence, Clevis can run only during 2 phases of the boot:

  • While in the initramfs (with network enabled)
  • After switching root once network has been brought up

Due to this, all LUKS devices holding a file system that needs to start prior the network is up after switching root must be considered as "root volumes".
For example, mount points such as /tmp, /var and /usr/local need to be mounted early after switching root.

Additionally, all mount points that are used by services running prior to the network (services defined with Before=local-fs.target dependency for example, which is the default) need to be mounted early after switching root.

In a nutshell, everything that is not tagged with _netdev in /etc/fstab is likely a "root volume".

In the example above, /var/log and /var/log/audit need to be considered as "root volumes" because /var/log/audit is used by auditd which is defined with Before=local-fs.target dependency. /var/log must also be considered as it holds /var/log/audit.

Also, in the example above, /opt needs to be considered as a "root volume" since it may be referenced by services units in /usr/lib/systemd/system or /etc/systemd/system delivered by the Red Hat Software Collection.

4. Understanding which file systems are to be considered as "non-root volumes"

"Non-root volumes" are the LUKS devices holding other file systems explictly mounted after the network is used. This is usually done by adding _netdev to /etc/fstab.
This usually applies to iSCSI devices or devices managed by non-system tools, such as Gluster or VxVM.

When configuring Clevis to unlock these devices, make sure to also add _netdev to corresponding entry in /etc/crypttab, or else ordering cycles will appear and the system will not boot properly.

Finally, enable the Clevis service through letting systemd watch the communication path:

# systemctl enable clevis-luks-askpass.path

5. Deciding how the LUKS information should be provided on the kernel command line

In order for Clevis to decrypt the LUKS devices early during boot, it is necessary to pass rd.luks.uuid and rd.lvm.lv or rd.lvm.vg parameters to the kernel command line.
These parameters will then help the initramfs's dracut-initqueue component setup the system to decrypt the devices.

In particular, the rd.lvm.lv parameter is used to let dracut-initqueue activate the specified LVM volume. The rd.luks.uuid parameter is used to decrypt the LUKS device on top of the LVM logical volume or the LUKS device below the LVM Volume Group.

There are multiple ways of specifying these parameters.

5.1. Specifying LUKS and LVM information on the kernel command line

Usually, upon installation, the Anaconda installer updated /etc/default/grub with the necessary parameters to decrypt manually the mandatory devices used to boot, which are usually root, swap and sometimes /usr if it is on a dedicated volume.

As an example, after installation, the following may be found in /etc/default/grub:

GRUB_CMDLINE_LINUX="... rd.luks.uuid=luks-9443aec0-73e9-484c-8668-ef77e647c0f5 rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rd.lvm.lv=rhel/usr ..."

When configuring Clevis, all "non-root" volumes must be specified as well, otherwise the system won't boot. This is explained below.

5.1.1. Rebuilding the initramfs

First, you need to rebuild the initramfs for Clevis use, as shown below:

# dracut -f --regenerate-all

You should then verify that the Clevis hooks are installed:

# lsinitrd /boot/initramfs-$(uname -r).img etc/cmdline.d/99clevis.conf
rd.neednet=1

This assumes that DHCP is used to setup the network while in the initramfs.

If this is not the case, you need to configure the network using a Dracut drop-in or directly on the kernel command line. This is explained in 9.7. Configuring manual enrollment of LUKS-encrypted root volumes.

5.1.2. Collecting the LUKS and LVM information

In order to automatically collect the list of rd.luks.uuid parameters suitable for Clevis to work, you may use the script shown below:

# for luksdev in /dev/mapper/luks-*; do echo -n "rd.luks.uuid=$(basename $luksdev) "; done; echo

rd.luks.uuid=luks-363925b7-390f-4ee8-b12f-385b607da96e rd.luks.uuid=luks-4d53f707-045f-4cb3-b810-638827f589f9 rd.luks.uuid=luks-50e5d2d4-a66d-49d8-a082-cfbfc64de1f3 rd.luks.uuid=luks-623573fb-6f92-472a-b5fb-95783bfa4dcf rd.luks.uuid=luks-7671f775-9ea8-4e14-be6a-0c65eef92e48 rd.luks.uuid=luks-c692c476-1a7a-4584-8170-8017bbc96680 rd.luks.uuid=luks-e3009a89-3d8d-40ce-b731-a7b949aed248 rd.luks.uuid=luks-e4cb239a-7328-4be0-b789-bc6376ccd36f rd.luks.uuid=luks-e5121b4e-8bd7-4976-9b8f-2838d302ccb2

You additionally need to collect the list of rd.lvm.lv parameters, unless you specify the rd.lvm.vg parameter instead (in such case, only specify Volume Groups).

This can be done using the script shown below:

# for dev in $(lsblk -p -n -s -r | awk '$6 == "crypt" { getline; print $1 }' | sort -u); do
	read lv vg <<< "$(lvs --noheadings -o lv_name,vg_name $dev 2>/dev/null)"
	[ -n "$vg$lv" ] && echo -n "rd.lvm.lv=$vg/$lv "
done; echo

rd.lvm.lv=rhel/00 rd.lvm.lv=rhel/01 rd.lvm.lv=rhel/02 rd.lvm.lv=rhel/03 rd.lvm.lv=rhel/04 rd.lvm.lv=rhel/05 rd.lvm.lv=rhel/06 rd.lvm.lv=rhel/07 rd.lvm.lv=rhel/08

If you prefer to activate early all the necessary LVM Volume Groups instead, you may use the script shown below:

# (for dev in $(lsblk -p -n -s -r | awk '$6 == "crypt" { getline; print $1 }' | sort -u); do
	read vg <<< "$(lvs --noheadings -o vg_name $dev 2>/dev/null)"
	[ -n "$vg" ] && echo -n "rd.lvm.vg=$vg "
done; echo) | sort -u

rd.lvm.vg=rhel

5.1.3. Editing the Grub configuration file

With the rd.luks.uuid and rd.lvm.lv values collected above, we need to edit /etc/default/grub's GRUB_CMDLINE_LINUX line to append the values, as shown in the example below:

  • Original line excerpt

    GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/00 rd.luks.uuid=luks-50e5d2d4-a66d-49d8-a082-cfbfc64de1f3"

  • New line excerpt

    GRUB_CMDLINE_LINUX="crashkernel=auto rd.luks.uuid=luks-363925b7-390f-4ee8-b12f-385b607da96e rd.luks.uuid=luks-4d53f707-045f-4cb3-b810-638827f589f9 rd.luks.uuid=luks-50e5d2d4-a66d-49d8-a082-cfbfc64de1f3 rd.luks.uuid=luks-623573fb-6f92-472a-b5fb-95783bfa4dcf rd.luks.uuid=luks-7671f775-9ea8-4e14-be6a-0c65eef92e48 rd.luks.uuid=luks-c692c476-1a7a-4584-8170-8017bbc96680 rd.luks.uuid=luks-e3009a89-3d8d-40ce-b731-a7b949aed248 rd.luks.uuid=luks-e4cb239a-7328-4be0-b789-bc6376ccd36f rd.luks.uuid=luks-e5121b4e-8bd7-4976-9b8f-2838d302ccb2 rd.lvm.lv=rhel/00 rd.lvm.lv=rhel/01 rd.lvm.lv=rhel/02 rd.lvm.lv=rhel/03 rd.lvm.lv=rhel/04 rd.lvm.lv=rhel/05 rd.lvm.lv=rhel/06 rd.lvm.lv=rhel/07 rd.lvm.lv=rhel/08"

Finally, rebuild the Grub menu:

# grub2-mkconfig -o /etc/grub2.cfg

Important Considerations for Edits to GRUB

  • If the system is in UEFI mode, command grub2-mkconfig -o /etc/grub2-efi.cfg should be used instead.
  • With RHEL 9, add the --update-bls-cmdline flag to ensure the changes are propagated to the default configurations as per our documentation; e.g. grub2-mkconfig -o /etc/grub2-efi.cfg --update-bls-cmdline
  • With RHEL 8, depending on how long the new GRUB_CMDLINE_LINUX line is, you may face an issue where Grub complains about not having enough space in /boot/grub2/grubenv to store the parameters. If this happens, you need to set GRUB_ENABLE_BLSCFG to false* in /etc/default/grub.

5.2. Specifying LUKS and LVM information into the initramfs

As we could see in 5.1. Specifying LUKS and LVM information on the kernel command line, specifying all the rd.luks.uuid and rd.lvm.lv parameters is not convenient, in particular when having a LUKS on LVM setup.

Using the proposed method below, it is possible to embed these parameters automatically into the initramfs.

Note however that the initramfs produced MUST be a "host only" initramfs. This is the default, unless dracut -N is used or dracut-config-generic package is installed on the system.

5.2.1. Editing /etc/fstab

dracut builds automatically the /etc/crypttab file embedded into the initramfs by looking into common directories (e.g. / and /lib64) and examining /etc/fstab content.

By default, dracut only specify the root (/) device, the swap and potentially /usr device. This is sufficient for password unlocking, but not when using Clevis.

Indeed, when using Clevis, as explained in 3. Understanding which file systems are to be considered as "root volumes", almost all volumes must be decrypted early during boot.

To make this happen, we need to tell dracut to mount the file systems in the initramfs, by specifying the x-initrd.mount parameter to all the "root volumes" in the options field.

This is suboptimal because in theory only unlocking needs to be done, but there is no other way to provide this information for now, hence, a This content is not included.BZ has been filed recently to improve this.

In our example with LUKS on LVM setup, the new /etc/fstab file will look like the one shown below (excerpts):

/dev/mapper/luks-7671f775-9ea8-4e14-be6a-0c65eef92e48 /home                   xfs     defaults,x-systemd.device-timeout=0,x-initrd.mount 0 0
/dev/mapper/luks-623573fb-6f92-472a-b5fb-95783bfa4dcf /opt                    xfs     defaults,x-systemd.device-timeout=0,x-initrd.mount 0 0
/dev/mapper/luks-e4cb239a-7328-4be0-b789-bc6376ccd36f /var                    xfs     defaults,x-systemd.device-timeout=0,x-initrd.mount 0 0
/dev/mapper/luks-c692c476-1a7a-4584-8170-8017bbc96680 /var/log                xfs     defaults,x-systemd.device-timeout=0,x-initrd.mount 0 0
/dev/mapper/luks-4d53f707-045f-4cb3-b810-638827f589f9 /var/log/audit          xfs     defaults,x-systemd.device-timeout=0,x-initrd.mount 0 0
/dev/mapper/luks-e5121b4e-8bd7-4976-9b8f-2838d302ccb2 /tmp                    xfs     defaults,x-systemd.device-timeout=0,x-initrd.mount 0 0

Note: as shown above, it is not necessary to add x-initrd.mount parameter for the root device and /usr devices because they are always mounted early during boot.

On LVM on LUKS setups, it is usually not necessary to add any x-initrd.mount parameter because usually the "root volumes" are all part of the same Volume Group, which is already activated during boot.

5.2.2. Creating the dracut configuration drop-in

We now need to tell dracut to embed the rd.luks.uuid and rd.lvm.lv parameters into the initramfs, as dedicated files in /etc/cmdline.d. To do so, we need to configure dracut to use hostonly_cmdline mode, which is done by creating the following configuration drop-in:

# mkdir -p /etc/dracut.conf.d
# cat > /etc/dracut.conf.d/99-kcs4500491.conf << EOF
hostonly_cmdline="yes"
EOF

5.2.3. Rebuilding the initramfs

We can now rebuild the initramfs:

# dracut -f --regenerate-all

Note: If using RHEL 7, remove the --regenerate-all flag.

We should then verify that the Clevis hooks are installed:

# lsinitrd /boot/initramfs-$(uname -r).img etc/cmdline.d/99clevis.conf
rd.neednet=1

Additionnally, we should verify that the /etc/cmdline.d drop-ins are present in the initramfs:

# lsinitrd /boot/initramfs-$(uname -r).img | grep "etc/cmdline.d/90"
-rw-r--r--   1 root     root          496 Oct 15 09:57 etc/cmdline.d/90crypt.conf
-rw-r--r--   1 root     root          181 Oct 15 09:57 etc/cmdline.d/90lvm.conf

# lsinitrd /boot/initramfs-$(uname -r).img etc/cmdline.d/90crypt.conf
 rd.luks.uuid=luks-363925b7-390f-4ee8-b12f-385b607da96e rd.luks.uuid=luks-50e5d2d4-a66d-49d8-a082-cfbfc64de1f3 rd.luks.uuid=luks-e4cb239a-7328-4be0-b789-bc6376ccd36f rd.luks.uuid=luks-e5121b4e-8bd7-4976-9b8f-2838d302ccb2 rd.luks.uuid=luks-4d53f707-045f-4cb3-b810-638827f589f9 rd.luks.uuid=luks-c692c476-1a7a-4584-8170-8017bbc96680 rd.luks.uuid=luks-7671f775-9ea8-4e14-be6a-0c65eef92e48 rd.luks.uuid=luks-e3009a89-3d8d-40ce-b731-a7b949aed248 rd.luks.uuid=luks-623573fb-6f92-472a-b5fb-95783bfa4dcf

# lsinitrd /boot/initramfs-$(uname -r).img etc/cmdline.d/90lvm.conf
 rd.lvm.lv=rhel/01 
 rd.lvm.lv=rhel/00 
 rd.lvm.lv=rhel/03 
 rd.lvm.lv=rhel/02 
 rd.lvm.lv=rhel/05 
 rd.lvm.lv=rhel/04 
 rd.lvm.lv=rhel/07 
 rd.lvm.lv=rhel/06 
 rd.lvm.lv=rhel/08

Note: if this is not the case, verify that you don't have the dracut-config-generic package installed on the system. If the issue persists, read 6.2. /etc/cmdline.d/90* files are not included in the initramfs.

5.2.4. Editing the Grub configuration file

The final step consists in rebuilding the Grub menu, after making sure that no rd.luks.uuid remains on the kernel command line.

Indeed, due to This content is not included.BZ systemd-cryptsetup-generator ignores LUKS devices, preventing the system to boot the system won't boot otherwise.

Edit /etc/default/grub's GRUB_CMDLINE_LINUX line to remove occurrences of rd.luks.uuid and rd.lvm.* as shown below:

  • Original line excerpt

    GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/00 rd.luks.uuid=luks-50e5d2d4-a66d-49d8-a082-cfbfc64de1f3"

  • New line excerpt

    GRUB_CMDLINE_LINUX="crashkernel=auto"

Finally, rebuild the Grub menu:

# grub2-mkconfig -o /etc/grub2.cfg

Important Considerations for Edits to GRUB

  • If the system is in UEFI mode, command grub2-mkconfig -o /etc/grub2-efi.cfg should be used instead.
  • With RHEL 9, add the --update-bls-cmdline flag to ensure the changes are propagated to the default configurations as per our documentation; e.g. grub2-mkconfig -o /etc/grub2-efi.cfg --update-bls-cmdline

5.2.5. Kickstart sample

For convenience, the kickstart sample below recapitulates all the operations done in the steps above.

%post
sed -i "s/x-systemd.device-timeout=0/x-systemd.device-timeout=0,x-initrd.mount/g" /etc/fstab

mkdir -p /etc/dracut.conf.d
cat > /etc/dracut.conf.d/99-kcs4500491.conf << EOF
hostonly_cmdline="yes"
EOF
dracut -f --regenerate-all

sed -i -e "s/rd.luks.uuid=[^ ]*//g" -e "s/rd.lvm.lv=[^ ]*//g" -e "s/  */ /g" /etc/default/grub
if [ -e "/sys/firmware/efi" ]; then
  grub2-mkconfig -o /etc/grub2-efi.cfg
else
  grub2-mkconfig -o /etc/grub2.cfg
fi
%end

6. Troubleshooting

6.1. "Error communicating with the server!" message

Usually, early during the boot we can see some Error communicating with the server! messages on the console. These messages are expected because the network is brought up in parallel to Clevis.
Hence, if the system boots anyway, these messages can be ignored.

However, if the system doesn't boot, this indicates that the Tang server is not reachable, usually due to network configuration (e.g. missing routes).

To debug, this, monitor the HTTP traffic on the Tang server to see if the requests are made.

  • If the request is not seen on the Tang server side, verify your network settings, in particular if static networking is used on the Clevis client

  • If the request is seen, this may indicate an issue within the Clevis code running in the initramfs.

    1. Boot the system with rd.debug debug rd.break on the kernel command line
    2. Wait for some time for the messages to be printed repeatedly
    3. Enter the password manually or wait for the Dracut emergency prompt to appear
    4. Collect the journal
    5. Open a Support Case if necessary

6.2. /etc/cmdline.d/90* files are not included in the initramfs

When using the 5.2. Specifying LUKS and LVM information into the initramfs method, it may happen that the necessary files are not embedded in the initramfs.

First, make sure that the dracut-config-generic package is not installed on the system:

# rpm -q dracut-config-generic

Assuming the package is not present, you need to execute dracut in debug mode to find out what happens:

# dracut -f --debug /tmp/initramfs.img $(uname -r) 2>/tmp/dracut.debug

Search for hostonly_cmdline variable assignment and open a Support Case if necessary.

7. Bugs and limitations

At the time of writing this article, the following bugs and limitations were found. Unless stated otherwise, they apply to both RHEL 7.5+ and RHEL 8.

SBR
Product(s)
Category
Components
Article Type