Modifying the Red Hat OpenStack Platform overcloud image with virt-customize

Updated

Red Hat OpenStack Platform director provides an image for the overcloud. This image contains a base set of software components that integrate to form various overcloud roles, such as Compute, Controller, and Storage nodes. You might need to modify the overcloud image to suit your needs, such as installing additional components on Controller nodes.

This document describes a series of actions to use the virt-customize tool to modify an existing overcloud image to augment an existing Controller node. For example, you can use these procedures to install additional ML2 plugins, Block Storage (cinder) back ends, or monitoring agents not shipped with the initial image.

Installing virt-customize

RHEL 9

The guestfs-tools package contains the virt-customize tool. Install the guestfs-tools package from the rhel-9-for-x86_64-appstream-rpms repository.

$ sudo subscription-manager repos --enable=rhel-9-for-x86_64-appstream-rpms
$ sudo dnf install guestfs-tools

RHEL 8

The libguestfs-tools package contains the virt-customize tool. Install the libguestfs-tools package from the rhel-8-for-x86_64-appstream-rpms repository:

$ sudo subscription-manager repos --enable=rhel-8-for-x86_64-appstream-rpms
$ sudo dnf install libguestfs-tools

Downloading the overcloud image

You need copies of the overcloud images to modify. Obtain these images by installing the RPM package.

RHOSP 17.x

If required, enable the openstack-17.1-for-rhel-9-x86_64-rpms repository first, then get the overcloud-hardened-uefi-full.qcow2 image.

$ sudo subscription-manager repos --enable=openstack-17.1-for-rhel-9-x86_64-rpms
$ sudo dnf install rhosp-director-images-uefi-x86_64

Extract the images from the archive installed by the RPM:

$ mkdir ~/overcloud-images
$ tar -xvf /usr/share/rhosp-director-images/overcloud-hardened-uefi-full-latest.tar -C ~/overcloud-images

RHOSP 16.x

If required, enable the openstack-16.2-for-rhel-8-x86_64-rpms repository first, then
get the overcloud-full.qcow2 image.

$ sudo subscription-manager repos --enable=openstack-16.2-for-rhel-8-x86_64-rpms
$ sudo dnf install rhosp-director-images

Extract the images from the archive installed by the RPM:

$ mkdir ~/overcloud-images
$ tar -xvf /usr/share/rhosp-director-images/overcloud-full-latest-16.2.tar -C ~/overcloud-images

Inspecting the overcloud image

You can explore the contents of the overcloud image using the guestfish tool. For RHOSP 17.x the image name is overcloud-hardened-uefi-full.qcow2; on RHOS 16.x the image name is overcloud-full.qcow2.

Example guestfish - RHOSP 17.x overcloud-hardened-uefi-full.qcow2 image:

$ guestfish -r -a overcloud-hardened-uefi-full.qcow2
Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: ‘help’ for help on commands
      ‘man’ to read the manual
      ‘quit’ to quit the shell

><fs> run
><fs> list-filesystems 
/dev/mapper/vg-lv_thinpool: unknown
/dev/mapper/vg-lv_thinpool-tpool: unknown
/dev/mapper/vg-lv_thinpool_tdata: unknown
/dev/mapper/vg-lv_thinpool_tmeta: unknown
/dev/sda1: vfat
/dev/sda2: unknown
/dev/sda3: ext4
/dev/vg/lv_audit: xfs
/dev/vg/lv_home: xfs
/dev/vg/lv_log: xfs
/dev/vg/lv_root: xfs
/dev/vg/lv_srv: xfs
/dev/vg/lv_tmp: xfs
/dev/vg/lv_var: xfs
><fs> mount /dev/vg/lv_root /
><fs> mount /dev/sda3 /boot
><fs> mount /dev/sda1 /boot/efi
><fs> mount /dev/vg/lv_var /var/
><fs> ls /etc/cloud/cloud.cfg.d/
/etc/cloud/cloud.cfg.d/05_logging.cfg    /etc/cloud/cloud.cfg.d/10_etc_hosts.cfg  /etc/cloud/cloud.cfg.d/README            
><fs> cat /etc/cloud/cloud.cfg.d/10_etc_hosts.cfg 
manage_etc_hosts: false
><fs> exit

Example guestfish - RHOSP 16.x overcloudfull.qcow2 image:

$ LIBGUESTFS_BACKEND=direct guestfish -a overcloud-full.qcow2

Welcome to guestfish, the guest filesystem shell for
editing virtual machine filesystems and disk images.

Type: ‘help’ for help on commands
      ‘man’ to read the manual
      ‘quit’ to quit the shell

><fs> run
><fs> list-filesystems
/dev/sda: xfs
><fs> mount /dev/sda /
><fs> ls /etc/cloud/cloud.cfg.d/
05_logging.cfg
10_etc_hosts.cfg
README
><fs> cat /etc/cloud/cloud.cfg.d/10_etc_hosts.cfg
manage_etc_hosts: false

><fs> exit

Customizing the overcloud image

NOTE: RHOSP 16.x (RHEL 8)

  • The examples below operate on the overcloud-hardened-uefi-full.qcow2 image. When modifying the RHOS 16.x image, replace the image with overcloud-full.qcow2.
  • The SELinux relabeling is not happening by default, make sure to either include the --selinux-relabel option when running virt-customize, or do the relabeling as a separate step after performing the other modifications.
  • Depending on how libvirt is set up, you may need to run virt-customize using the direct back end mode. To do this set the LIBGUESTFS_BACKEND environment variable to direct by running export LIBGUESTFS_BACKEND=direct before running virt-customize commands.

The following procedures help you customize the overcloud image.

Changing kernel console arguments

Since RHOSP 17.1.4, all kernel console logging arguments are removed because console logging can cause unacceptable latency issues in Compute workloads. Therefore, the kernel console arguments are no longer logged during kernel boot to the graphics console tty0, or to the first serial console ttyS0. In releases earlier than 17.1.4, both tty0 and ttyS0 may be enabled in the overcloud image. Use the following command to remove them from the image to prevent latency from console logging:

$ virt-customize -a overcloud-hardened-uefi-full.qcow2 --run-command "sed -i 's/\<console=[[:alnum:],]*[[:space:]]*//g' /etc/default/grub" --run-command "grub2-mkconfig"

If kernel boot fails to complete it may be necessary to temporarily enable console logging to help identify the problem. Use the following command to enable graphics console logging on the image:

$ virt-customize -a overcloud-hardened-uefi-full.qcow2 --run-command "sed -i -E 's/(GRUB_CMDLINE_LINUX_DEFAULT=")(.+)(")/\1\2 console=tty0"/' /etc/default/grub" --run-command "grub2-mkconfig"

You can verify the resut of Changing kernel console arguments by virt-cat commands

$ virt-cat -a overcloud-hardened-uefi-full.qcow2 /etc/default/grub

Disable cloud-init network configuration

If you want to disable the network configuration part of cloud-init, upload a file to the /etc/cloud/cloud.cfg.d/ directory in the image.

$ cat 15_disable_network_config.conf

network:
  config: disabled

Copy the configuration file on to the image:

$ virt-customize -a overcloud-hardened-uefi-full.qcow2 --upload 15_disable_network_config.conf:/etc/cloud/cloud.cfg.d/
[   0.0] Examining the guest ...
[   5.7] Setting a random seed
[   5.7] Uploading: 15_disable_network_config.conf to /etc/cloud/cloud.cfg.d/
[   5.7] SELinux relabelling
[  22.6] Finishing off

The --upload option copies the local file 15_disable_network_config.conf to the /etc/cloud/cloud.cfg.d/ directory on the overcloud image.

Setting the root password

Set the password for the root user on image:

$ virt-customize -a overcloud-hardened-uefi-full.qcow2 --root-password password:test
[   0.0] Examining the guest ...
[   5.5] Setting a random seed
[   5.6] Setting passwords
[   6.5] SELinux relabelling
[  23.5] Finishing off

This provides administration-level access for your nodes through the console.

Registering the image

Register your image temporarily to enable Red Hat repositories relevant to your customizations:

$ virt-customize -a overcloud-hardened-uefi-full.qcow2 --run-command 'subscription-manager register --username=[username] --password=[password]'
[   0.0] Examining the guest ...
[  10.0] Setting a random seed
[  10.0] Running: subscription-manager register --username=[username] --password=[password]
[  23.2] SELinux relabelling
[  51.5] Finishing off

Make sure to replace the [username] and [password] with your Red Hat customer account details. This runs the following command on the image:

subscription-manager register --username=[username] --password=[password]

This registers your overcloud image to the Red Hat Content Delivery Network:

Attaching a subscription and enabling Red Hat Repositories

Find a list of pool ID from your account's subscriptions:

$ sudo subscription-manager list

Choose a subscription pool ID and attach it to the image:

$ virt-customize -a overcloud-hardened-uefi-full.qcow2 --run-command 'subscription-manager attach --pool [subscription-pool]'
[   0.0] Examining the guest ...
[  12.0] Setting a random seed
[  12.0] Running: subscription-manager attach --pool [subscription-pool]
[  23.2] SELinux relabelling
[  51.5] Finishing off
[  52.0] Finishing off

Make sure to replace the [subscription-pool] with your chosen subscription pool ID. This runs the following command on the image:

subscription-manager attach --pool [subscription-pool]

This add the pool to the image, which allows you to enable Red Hat repositories with the following command:

$ subscription-manager repos --enable=[repo-id]

Copying a custom repository file

Adding third-party software to the image requires additional repositories. For example, the following is an example repo file that contains configuration to use example repository content:

$ cat example.repo

[example]
name=Example Repository
baseurl=https://repo.example.org/content/repositories/example-el-9-x86_64/
gpgcheck=0

Copy the repository file on to the image:

$ virt-customize -a overcloud-hardened-uefi-full.qcow2 --upload example.repo:/etc/yum.repos.d/
[   0.0] Examining the guest ...
[   4.8] Setting a random seed
[   4.8] Uploading: example.repo to /etc/yum.repos.d/
[   4.8] SELinux relabelling
[  20.2] Finishing off

The --upload option copies the repository file to /etc/yum.repos.d/ on the overcloud image.

Important: Red Hat does not offer support for software from non-certified vendors. Check with your Red Hat support representative that the software you aim to install is supported.

Installing RPMs

Use the virt-customize command to install packages to the image:

$ virt-customize -a overcloud-hardened-uefi-full.qcow2 --install example-package
[   0.0] Examining the guest ...
[  11.0] Setting a random seed
[  11.0] Installing packages: example-package
[  15.8] SELinux relabelling
[  91.0] Finishing off

The --install option allows you to specify a package to install.

Cleaning the Subscription Pool

After installing the necessary packages to customize the image, remove your subscriptions and unregister the image:

$ virt-customize -a overcloud-hardened-uefi-full.qcow2 --run-command 'subscription-manager remove --all'
[   0.0] Examining the guest ...
[   5.5] Setting a random seed
[   5.5] Running: subscription-manager remove --all
[  12.0] SELinux relabelling
[  28.4] Finishing off

This removes all subscription pools from the image.

Unregister the image

Finally, unregister the image. This is so the overcloud deployment process can deploy the image to your nodes and register each of them individually.

$ virt-customize -a overcloud-hardened-uefi-full.qcow2 --run-command 'subscription-manager unregister'
[   0.0] Examining the guest ...
[   5.7] Setting a random seed
[   5.7] Running: subscription-manager unregister
[   7.9] SELinux relabelling
[  25.6] Finishing off

Relabeling the SElinux contexts

This can re run as an individual step, or add --selinux-relabel to the command after which the SElinux contexts shall be relabeled:

virt-customize -a overcloud-hardened-uefi-full.qcow2 --selinux-relabel -v

NOTE: This is only required on the virt-customize in RHEL 8. More recent versions always finish with relabeling, so --selinux-relabel has no effect.

Uploading the images to director

After modifying the image, upload it to director. Make sure to source the stackrc file so that you can access director from the command line:

RHOSP 17.x

$ source stackrc
$ openstack overcloud image upload --update-existing  --whole-disk --image-path <path to image directory>

RHOSP 16.x

$ source stackrc
$ openstack overcloud image upload --update-existing --image-path <path to image directory>
Product(s)
Category
Components
Tags
Article Type