Why does leapp upgrade fail with error "Command ['chroot', '/var/lib/leapp/el8userspace', '/bin/bash', '-c', 'su - -c update-ca-trust'] failed with exit code 127" ?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 7

    • leapp-repository
  • Red Hat Enterprise Linux 8

    • leapp-repository
    • fapolicyd

Issue

  • Leapp Upgrade with local repository fails with below traceback error:

    Risk Factor: high (error)
    Title: Actor target_userspace_creator unexpectedly terminated with exit code: 1
    Summary: Traceback (most recent call last):
      File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 74, in _do_run
        actor_instance.run(*args, **kwargs)
      File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 289, in run
        self.process(*args)
      File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/actor.py", line 58, in process
        userspacegen.perform()
      File "/usr/lib/python2.7/site-packages/leapp/utils/deprecation.py", line 42, in process_wrapper
        return target_item(*args, **kwargs)
      File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 1246, in perform
        _create_target_userspace(context, indata, indata.packages, indata.files, target_repoids)
      File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 1108, in _create_target_userspace
        _prep_repository_access(context, target_path)
      File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 629, in _prep_repository_access
        run(["chroot", target_userspace, "/bin/bash", "-c", "su - -c update-ca-trust"])
      File "/usr/lib/python2.7/site-packages/leapp/libraries/stdlib/__init__.py", line 192, in run
    result=result
    CalledProcessError: Command ['chroot', '/var/lib/leapp/el8userspace', '/bin/bash', '-c', 'su - -c update-ca-trust'] failed with exit code 127. 
    

Resolution

Follow the procedure described in the Diagnostic Steps section.

If this is a match, proceed to appropriate solution, otherwise or if in doubt, open a case on the Customer Portal referencing this solution and follow the procedure described in Other case.


Case of install_weak_deps=False

Remove install_weak_deps=False from /etc/dnf/dnf.conf or /etc/leapp/files/dnf.conf files.


Case of invalid repository

The "BaseOS" repository must contain metadata created for the target OS release (hence RHEL8 when upgrading from RHEL7 to RHEL8).
Make sure to regenerate the metadata from a host running the target OS release, e.g. use createrepo on a RHEL8 system when upgrading to RHEL7.

Usually we do not recommend using custom repositories at all. If the system to be upgraded is not connected and the upgrade is to be done using the RHEL DVD, the user should use the leapp upgrade --iso <Path to ISO.iso> scheme instead of extracting the DVD and creating his own custom repository from the DVD content.


Case of fapolicyd being active

Stop the service:

# systemctl stop fapolicyd

Other case

Execute leapp preupgrade under strace, open a case on the Customer Portal referencing thie solution and provide the strace file (/tmp/leapp.strace) for review:

# yum -y install strace
# strace -fttTvyy -s 256 -o /tmp/leapp.strace -- leapp preupgrade ...

Root Cause

In order to update certificates, the su utility, shipped by util-linux package, is used.
Usually the package is automatically pulled when installing the minimal container, because dnf satisfies "Recommends" dependencies, unless install_weak_deps=False is specified in the DNF configuration file.

Alternatively, the issue may also occur if the repository "BaseOS" is a custom repository created using createrepo on RHEL7, which doesn't handle "Recommends" option found in RHEL8 packages at all. Because of this, util-linux will not be pulled either.

The issue has been resolved via errata:RHBA-2024:9503

Diagnostic Steps

  1. Confirm that su binary was not installed in the minimal container

    # grep '/bin/bash: su: command not found' var/log/leapp/leapp-*.log
    var/log/leapp/leapp-upgrade.log:2024-06-14 15:05:23.272 DEBUG    PID: 17018 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: /bin/bash: su: command not found
    
  2. Check if install_weak_deps keyword is specified in the DNF configuration files

    # grep install_weak_deps /etc/dnf/dnf.conf /etc/leapp/files/dnf.conf 2>/dev/null
    /etc/dnf/dnf.conf:install_weak_deps=False
    

    If the keyword is found in one of the files, proceed to Resolution - Case of install_weak_deps=False.

  3. Check is the "BaseOS" repository metadata contains "Recommends" information

    # zgrep "<rpm:recommends>" /var/lib/leapp/el8userspace/var/cache/dnf/BaseOS-*/repodata/*-primary.xml.*
    

    If no output is displayed, this means that the "BaseOS" repository was not created appropriately, proceed to Resolution - Case of invalid repository.

  4. For RHEL8 to RHEL9 upgrades, check if fapolicyd is started

    # systemctl is-active fapolicyd
    active
    

    If it's active, proceed to Resolution - Case of fapolicyd being active.

SBR
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.