Leapp fails to upgrade the system in reboot phase and prints "Message: Failed to append extra arguments to kernel command line."

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 7
    • leapp

Issue

  • Upgrading a RHEL 7 system to RHEL 8 fails in the reboot phase, drops to emergency mode and prints the following error message:

    ============================================================
                           ERRORS
============================================================
2021-06-05 15:20:04.141285 [ERROR] Actor: kernelcmdlineconfig
Message: Failed to append extra arguments to kernel command line.
Summary:
    Details: Command ['grubby', '--update-kernel=/boot/vmlinuz-4.18.0-305.3.1.el8_4.x86_64', '--args=net.ifnames=0'] failed with exit code 1.
============================================================
                       END OF ERRORS
============================================================

Resolution

This issue has multiple causes. To find out which scenario applies to your system, follow the steps in the Diagnostic Steps section.

1) The issue happens when the /etc/default/grub file doesn't end with an end-of-line character (Carriage Return).

  1. Boot the system on RHEL 7 (this assumes you can rollback or restore your system from backup)

  2. Add the required character to the file, as shown below:

    # echo >> /etc/default/grub

  3. Perform the upgrade again (no need to execute the preupgrade phase)

2) The issue happens when the /etc/default/grub file has the immutable attribute.

  1. Boot the system on RHEL 7 (this assumes you can rollback or restore your system from backup)

  2. Remove the immutable attribute:

    # chattr -i /etc/default/grub

  3. Perform the upgrade again (no need to execute the preupgrade phase)

3) The issue happens when the package grub2-tools is installed but is not present in the RPM database.

  1. Boot the system on RHEL 7 (this assumes you can rollback or restore your system from backup)

  2. Reinstall the grub2-tools package:

    # yum install grub2-tools

  3. Perform the upgrade again (no need to execute the preupgrade phase)

4) The issue happens when multiple grub2-tools package is present in the RPM database.

  1. Remove the old grub2-tools-* present by using below command to mitigate the issue.

     # yum erase grub2-tools-*    
 EX: # yum erase grub2-tools-2.02-0.87.el7_9.9.x86_64

5) Perform the below task based os UEFI or BIOS system.

  1. Review /etc/default/grub for Syntax Errors
    Start by examining the /etc/default/grub file to ensure there are no syntax mistakes in the GRUB_CMDLINE_LINUX. Any inconsistency in this file can cause the aforementioned error.

  2. Rebuild the grub.cfg File
    Once any potential errors in the /etc/default/grub file have been addressed, proceed with rebuilding the grub configuration file. Depending on the type of machine, follow the steps below:

    • For BIOS-based machines:
      Issue the following command as root:
    # grub2-mkconfig -o /boot/grub2/grub.cfg
    • For UEFI-based machines:
      Issue the following command as root:
    # grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

Root Cause

1) The issue happens when the /etc/default/grub file doesn't end with an end-of-line character (Carriage Return).

Due to not having an end-of-line character, /etc/default/grub doesn't get updated correctly which leads to hitting This content is not included.BZ#1955099.

2) The issue happens when the /etc/default/grub file has the immutable attribute.

During the upgrade phase a script is run to append GRUB_ENABLE_BLSCFG=true to /etc/default/grub, but as the file has the immutable attribute the changes are not saved, which leads to hitting This content is not included.BZ#1955099.

3) The issue happens when the package grub2-tools is installed but is not present in the RPM database.

In this situation, during the upgrade operation the package grub2-tools is marked to be installed instead of to be upgraded. Due to this, the script that appends GRUB_ENABLE_BLSCFG=true to /etc/default/grub is not executed, which leads to hitting This content is not included.BZ#1955099.

This content is not included.BZ#1955099 - Updating the kernel on leapp'ed systems doesn't create the initramfs depending on Grub BLS state.

4) The issue happens when multiple grub2-tools package is present in the RPM database.

In this scenario, during leapp upgrade, there are multiple grub2-tools packages present in the RPM database which leads to this error.

Diagnostic Steps

1) The issue happens when the /etc/default/grub file doesn't end with an end-of-line character (Carriage Return).

  • Verify that /etc/default/grub has an end-of-line character (Carriage Return)

    # tail -n 1 /etc/default/grub | hexdump -C
00000000  47 52 55 42 5f 53 45 52  49 41 4c 5f 43 4f 4d 4d  |GRUB_SERIAL_COMM|
00000010  41 4e 44 3d 22 73 65 72  69 61 6c 20 2d 2d 73 70  |AND="serial --sp|
00000020  65 65 64 3d 31 31 35 32  30 30 20 2d 2d 75 6e 69  |eed=115200 --uni|
00000030  74 3d 30 20 2d 2d 77 6f  72 64 3d 38 20 2d 2d 70  |t=0 --word=8 --p|
00000040  61 72 69 74 79 3d 6e 6f  20 2d 2d 73 74 6f 70 3d  |arity=no --stop=|
00000050  31 22 0a                                          |1".|
00000053

    In the example above, the last character is 0a in the hexadecimal column, which is end-of-line, hence the system won't hit the issue upon upgrade.

2) The issue happens when the /etc/default/grub file has the immutable attribute.

  • Check the attributes of the file.

    # lsattr /etc/default/grub
----i--------------- /etc/default/grub

    In the example above, i indicates that the file has the immutable attribute, so the system will hit the issue upon upgrade.

3) The issue happens when the package grub2-tools is installed but is not present in the RPM database.

  • Verify if the package grub2-tools is present in the RPM database.

    # rpm -q grub2-tools
grub2-tools-2.02-0.87.el7_9.11.x86_64

    In the example above, grub2-tools is present. If the command returned no output, the system would hit the issue upon upgrade.

4) The issue happens when multiple grub2-tools package is present in the RPM database.

  • Verify if the package grub2-tools is present in the RPM database.

    # rpm -q grub2-tools
grub2-tools-2.02-0.87.el7_9.9.x86_64
grub2-tools-2.02-0.87.el7_9.11.x86_64
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.