XFS ftype=0 prevents upgrading to a version of OpenStack Director with containers

Solution Verified - Updated

Environment

  • Customers who originally deployed using Red Hat OpenStack Director 7 or Red Hat OpenStack Director 8 and followed the upgrade path to Red Hat OpenStack Director 10 or Red Hat Openstack Director 11.

Issue

  • Users who have previously deployed Red Hat OpenStack using Director 8 or lower, the deployed systems may have an older version of the XFS file system which prevents them from being upgraded to Red Hat OpenStack Director 12 or Red Hat OpenStack Director 13.

  • Customers attempting to upgrade from 11 to 12, must check the xfs configuration on the overcloud nodes prior to performing the upgrade.

  • Customer must also check that their nodes are not affected if attempting to upgrade from 10 to 13 using the Fast-forward Upgrade process.

Resolution

NOTE: Prior to performing these actions, the systems affected will have to be completely shut down. Any services running on these machines may be impacted during this outage. Running VMs on compute nodes should be migrated using the Migrating VMs from an Overcloud Compute Node

  • After ensuring that the systems have been upgraded to the latest version of director, the XFS file system that contains /var/lib must be reformatted with ftype=1. This means a full backup and restore must be performed on the systems. For the sake of documentation, the provided procedure assumes that there is only a single root XFS partition. The documentation assumes that a bootable RHEL 7.5 DVD or USB can be used to run the rescue procedure on the affected systems.

XFS Backup and Restore using a local disk

This procedure assumes a local disk (usb or internal hard drive) is available and mountable under the rescue procedure and contains enough space to hold a full backup of the affected disk.

  1. Boot from RHEL 7.5 DVD or USB

  2. Select "Troubleshooting"

  3. Select "Rescue a Red Hat Enterprise Linux system"

  4. Enter "3" to select "3) Skip to shell"

  5. Run the following procedure in the shell.

    a. Mount local backup disk

     mkdir /mnt/sysbackup
 mount /dev/BACKUP_DISK /mnt/sysbackup

An example `/dev/BACKUP_DISK` would be `/dev/sdb1`. This will vary depending on your system. The backup disk does not need to be XFS formated as long as it can be mounted and written to under the rescue shell.

    b. Collect disk UUID for the affected root disk

     export $(blkid -o export /dev/ROOTFS_PARTITION | grep UUID)
 echo $UUID

An example `/dev/ROOTFS_PARTITION` could be `/dev/sda1`. This will vary depending on your system.

    c. Mount root filesystem partition and verify ftype=0

     mount /dev/ROOTFS_PARTITION /mnt/sysimage
 xfs_info /mnt/sysimage

    d. Run xfsdump on the mount. Read additional xfs backup and restore documentation

     xfsdump -l 0 -F -f /mnt/sysbackup/rootfs.xfsdump /mnt/sysimage

Before proceededing, make sure that "xfsdump: Dump Status: SUCCESS" is returned by the xfsdump process.

    e. Unmount disk and reformat root filesystem partition.

    **WARNING: This will destroy the data on the disk.**

 umount /mnt/sysimage
 mkfs.xfs -f -m uuid=$UUID /dev/ROOTFS_PARTITION

Before proceeding make sure ftype=1 is in the output of the filesystem creation.

*NOTE:* You must provide the uuid at format time. Failure to do so may lead to an unbootable system. See [BZ#1399487](https://bugzilla.redhat.com/show_bug.cgi?id=1399487).

    f. Verify the UUID for the root filesystem partition matches the UUID from Step b.

     blkid /dev/ROOTFS_PARTITION

    e. Mount the the root filesystem partition.

     mount /dev/ROOTFS_PARTITION /mnt/sysimage

    f. Restore the backup using xfsrestore.

     xfsrestore -f /mnt/sysbackup/rootfs.xfsdump /mnt/sysimage

Before proceeding, make sure that "xfsrestore: Restore Status: SUCCESS" is returned by the xfsrestore process.

If the filesystem label did not restore, do so manually:

 umount /dev/ROOTFS_PARTITION
 xfs_admin -L <PREVIOUS_LABEL> /dev/ROOTFS_PARTITION
 mount /dev/ROOTFS_PARTITION /mnt/sysimage

    g. Reinstall the bootloader

     cd /mnt/sysimage
 mount -t proc proc proc/
 mount -t sysfs sys sys/
 mount -o bind /dev dev/
 chroot /mnt/sysimage
 grub2-install /dev/PRIMARY_DISK
 exit

An example `/dev/PRIMARY_DISK` could be `/dev/sda`. This will vary depending on your system.

    h. Cleanup and power down the system

     cd /
 umount /mnt/sysimage/{dev,proc,sys,} /mnt/sysbackup
 poweroff

  6. Remove the RHEL 7.5 DVD or USB device

  7. Boot machine and validate the file system is now configured with ftype=1

     xfs_info /

XFS Backup and Restore using a remote network system via SSH

This procedure assumes a remote linux system is available via SSH and an output location contains enough space to hold a full backup of the affected disk.

  1. Boot from RHEL 7.5 DVD or USB

  2. Select "Troubleshooting"

  3. Select "Rescue a Red Hat Enterprise Linux system"

  4. Enter "3" to select "3) Skip to shell"

  5. Run the following procedure in the shell.

    a. Enable basic network configuration

     ip addr add IFADDR dev IFNAME
 ip route add default via ROUTE

An example `IFADDR` could be `192.168.24.12/24`, `IFNAME` could be `enp0s3`, and `ROUTE` could be `192.168.24.1`. These values will vary depending on your system.

    b. Generate ssh keys and add them to the remote host.

     ssh-keygen
 scp /root/.ssh/id_rsa.pub <user>@<backuphost>:backup.pub
 ssh <user>@<backuphost> 'cat backup.pub >> ~/.ssh/authorized_keys'

The ssh key created must configured without a pass phrase. This key is only available for the life of this rescue process and is not persisted once the system is rebooted. The public key can be safely removed from the backup host once this process has been completed.

    c. Validate ssh connecitivty without a password.

     ssh <user>@<backuphost> 'whoami'

This should return the `<user>`.

    e. Collect disk UUID for the affected root disk

     export $(blkid -o export /dev/ROOTFS_PARTITION | grep UUID)
 echo $UUID

An example `/dev/ROOTFS_PARTITION` could be `/dev/sda1`. This will vary depending on your system.

    f. Mount root filesystem partition and verify ftype=0

     mount /dev/ROOTFS_PARTITION /mnt/sysimage
 xfs_info /mnt/sysimage

    d. Run xfsdump on the mount and write it out to the remote host over ssh. Read additional xfs backup and restore documentation

     xfsdump -l 0 -J - /mnt/sysimage | ssh user@backuphost 'cat >BACKUPLOCATION/rootfs.xfsdump'

The `BACKUPLOCATION` should be a directory writable by the `user` on `backuphost`. For example this could be `~/backup/` which would be the `backup` directory in the `user`'s home folder.
Before proceededing, make sure that "xfsdump: Dump Status: SUCCESS" is returned by the xfs dump process.

    e. Unmount disk and reformat root filesystem partition.

    **WARNING: This will destroy the data on the disk.**

 umount /mnt/sysimage
 mkfs.xfs -f -m uuid=$UUID /dev/ROOTFS_PARTITION

Before proceeding make sure ftype=1 is in the output of the filesystem creation.

*NOTE:* You must provide the uuid at format time. Failure to do so may lead to an unbootable system. See [BZ#1399487](https://bugzilla.redhat.com/show_bug.cgi?id=1399487).

    f. Verify the UUID for the root filesystem partition matches the UUID from Step e.

     blkid /dev/ROOTFS_PARTITION

    g. Mount the the root filesystem partition.

     mount /dev/ROOTFS_PARTITION /mnt/sysimage

    h. Restore the backup using xfsrestore.

     ssh user@backuphost 'cat BACKUPLOCATION/rootfs.xfsdump' | xfsrestore -J - /mnt/sysimage

Before proceeding, make sure that "xfsrestore: Restore Status: SUCCESS" is returned by the xfsrestore process.

If the filesystem label did not restore, do so manually:

 umount /dev/ROOTFS_PARTITION
 xfs_admin -L <PREVIOUS_LABEL> /dev/ROOTFS_PARTITION
 mount /dev/ROOTFS_PARTITION /mnt/sysimage

    i. Reinstall the bootloader

     cd /mnt/sysimage
 mount -t proc proc proc/
 mount -t sysfs sys sys/
 mount -o bind /dev dev/
 chroot /mnt/sysimage
 grub2-install /dev/PRIMARY_DISK
 exit

An example `/dev/PRIMARY_DISK` could be `/dev/sda`. This will vary depending on your system.

    j. Cleanup and power down the system

     cd /
 umount /mnt/sysimage/{dev,proc,sys,}
 poweroff

  6. Remove the RHEL 7.5 DVD or USB device

  7. Boot machine and validate the file system is now configured with ftype=1

     xfs_info /

Root Cause

OpenStack Director deploys containers with the overlayFS storage configuration. When using XFS, overlayFS requires that the ftype be set to 1 when formatting the storage This content is not included.BZ#1206277. Prior to RHEL7.3, the default value for ftype was set to 0 which means any systems provisioned using RHEL7.2 or older will have ftype=0. This setting is only able to be configured when the filesystem is created which means it cannot be used with containers. RHEL7.3 included xfsprogs version 4.5.0 which changed the default of ftype from 0 to 1. Systems provisioned with RHEL7.3 or newer are not affected.

Diagnostic Steps

In order to check to see if a given node would be affected, run xfs_info /var/lib on the mounted filesystem.

Affected node:

[heat-admin@controller-0 ~]$ xfs_info /var/lib | grep ftype
naming   =version 2              bsize=4096   ascii-ci=0 ftype=0

Unaffected node (running xfs):

[heat-admin@controller-0 ~]$ xfs_info /var/lib | grep ftype
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1

Unaffected node (running ext4):

[heat-admin@controller-0 ~]$ xfs_info / | grep ftype
xfs_info: / is not a mounted XFS filesystem

A basic ansible role Content from gitlab.com is not included.ansible-role-xfs-ftype is available that performs these checks based on a provided inventory.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.