RHEL7: "leapp upgrade" fails in reboot phase with no message and enters emergency prompt

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 7
    • leapp

Issue

  • Upon rebooting to perform the real upgrade, the upgrade fails without further message, emergency prompt is entered

    Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.

:/#

Resolution

Follow the procedure in the Diagnostic Steps section first to make sure that this is a match.

Case where you are reading this solution before rebooting to perform the real upgrade

  1. Edit /var/log/leapp/answerfile

    You need to remove or comment out the unexpected line authselect select [...] from authselect_check section.
    Additionally, you need to explicitly uncomment the line confirm = True (or set confirm = False, depending on your configuration settings).

    [authselect_check]
# Title:              None
# Reason:             Confirmation
# ========================== authselect_check.confirm =========================
# Label:              Configure PAM and nsswitch.conf with the following authselect call?
# Description:        If yes, suggested authselect profile will be applied on your system to [...]
# Type:               bool
# Default:            True
# Available choices: True/False
# Unanswered question. Uncomment the following line with your answer
confirm = True

  2. Run or re-run leapp upgrade command

    # leapp upgrade [...]

  3. Confirm that the /var/log/leapp/answerfile.userchoices file contains the expected value for authselect_check section

    [authselect_check]
confirm = True

  4. Reboot to perform the upgrade for real

    # reboot

Case where you are already at the emergency prompt in reboot phase

There is no possibility to mitigate the issue in-place, you need to restore the system to RHEL 7 state and redo the upgrade after having fixed the /var/log/leapp/answerfile file content.

This issue is fixed in leapp-0.13.0-1.el7_9.noarch (<https://access.redhat.com/errata/RHBA-2021:4603>).
Please also check "8.3. Known issues (BZ#2014015)" section.

Root Cause

The issue is tracked by This content is not included.BZ 2014015 - authselectapply actor crashes, causing upgrade to fail in reboot phase.

Diagnostic Steps

The procedure depends whether you are already encountering the issue or not.

Case where you are reading this solution before rebooting to perform the real upgrade

  1. Execute leapp preupgrade command if not already executed

    # leapp preupgrade

  2. Check that the /var/log/leapp/answerfile file contains an invalid *authselect select... line for [authselect_check] block

    [authselect_check]
# Title:              None
# Reason:             Confirmation
# ========================== authselect_check.confirm =========================
# Label:              Configure PAM and nsswitch.conf with the following authselect call?
authselect select sssd with-fingerprint --force

# Description:        If yes, suggested authselect profile will be applied on your system to [...]
# Type:               bool
# Default:            True
# Available choices: True/False
# Unanswered question. Uncomment the following line with your answer
# confirm = True

    Here above the invalid line authselect select [...] is present in authselect_check section, which will lead to a failure near the end of the reboot phase.
    Proceed to Resolution to mitigate this issue.

Case where you are already at the emergency prompt in reboot phase

  1. Check that the upgrade.service unit failed

    :/# systemctl --failed
  UNIT            LOAD   ACTIVE SUB    DESCRIPTION                             
● upgrade.service loaded failed failed System Upgrade                          

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

1 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.

    If the unit didn't fail, you are not hitting the issue described in this document. Please open a case on the Customer Portal referencing this document.

  2. Search in the console output or journal file for a Traceback line

    :/# journalctl -u upgrade.service | grep -A 12 Traceback
[...] upgrade[609]: Traceback (most recent call last):
[...] upgrade[609]:   File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
[...] upgrade[609]:     self.run()
[...] upgrade[609]:   File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
[...] upgrade[609]:     self._target(*self._args, **self._kwargs)
[...] upgrade[609]:   File "/root/tmp_leapp_py3/leapp/repository/actor_definition.py", line 72, in _do_run
[...] upgrade[609]:     actor_instance.run(*args, **kwargs)
[...] upgrade[609]:   File "/root/tmp_leapp_py3/leapp/actors/__init__.py", line 335, in run
[...] upgrade[609]:     self.process(*args)
[...] upgrade[609]:   File "/etc/leapp/repos.d/system_upgrade/el7toel8/actors/authselectapply/actor.py", line 31, in process
[...] upgrade[609]:     decision = next(self.consume(AuthselectDecision))
[...] upgrade[609]: StopIteration
[...] upgrade[609]: =================================================================================================

    Here above the journal shows a traceback for actor authselectapply, which is the issue described in this document.
    Proceed to Resolution.

    If there is no such traceback, open a case on the Customer Portal referencing this document.

SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.