Why does leapp upgrade fail with error "Command ['chroot', '/var/lib/leapp/el8userspace', '/bin/bash', '-c', 'su - -c update-ca-trust'] failed with exit code 1" ?

Environment

• Red Hat Enterprise Linux 7
  • leapp-upgrade-el7toel8-0.20.0-9.el7_9.noarch
• Red Hat Enterprise Linux 8 and later
  • leapp-upgrade-el8toel9-0.20.0-2.el8.noarch

Issue

• On RHEL7, leapp preupgrade command fails with the following error, as reported in /var/log/leapp/leapp-report.txt

  Risk Factor: high (error)
  Title: Actor target_userspace_creator unexpectedly terminated with exit code: 1
  Summary: Traceback (most recent call last):
    File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 74, in _do_run
      actor_instance.run(*args, **kwargs)
    File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 289, in run
      self.process(*args)
    File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/actor.py", line 58, in process
      userspacegen.perform()
    File "/usr/lib/python2.7/site-packages/leapp/utils/deprecation.py", line 42, in process_wrapper
      return target_item(*args, **kwargs)
    File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 1268, in perform
      _create_target_userspace(context, indata, indata.packages, indata.files, target_repoids)
    File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 1117, in _create_target_userspace
      _prep_repository_access(context, target_path)
    File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 629, in _prep_repository_access
      run(["chroot", target_userspace, "/bin/bash", "-c", "su - -c update-ca-trust"])
    File "/usr/lib/python2.7/site-packages/leapp/libraries/stdlib/__init__.py", line 192, in run
      result=result
  CalledProcessError: Command ['chroot', '/var/lib/leapp/el8userspace', '/bin/bash', '-c', 'su - -c update-ca-trust'] failed with exit code 1.
  

• On RHEL8, leapp preupgrade command fails with the following error, as reported in /var/log/leapp/leapp-report.txt

  Risk Factor: high (error)
  Title: Cannot set the container mode for the subscription-manager.
  Summary:
  Key: 06583cd4c6f9e1629a9c1937643c3897618f33ef
  

Resolution

On RHEL7, this solution deals with the error message

• "CalledProcessError: Command ['chroot', '/var/lib/leapp/el8userspace', '/bin/bash', '-c', 'su - -c update-ca-trust'] failed with exit code 1."

• not with "CalledProcessError: Command ['chroot', '/var/lib/leapp/el8userspace', '/bin/bash', '-c', 'su - -c update-ca-trust'] failed with exit code 127." which is handled by Solution Why does leapp upgrade fail with error "Command ['chroot', '/var/lib/leapp/el8userspace', '/bin/bash', '-c', 'su - -c update-ca-trust'] failed with exit code 127" ?

Follow the procedure described in the Diagnostic Steps section.
If this is a match, proceed further, otherwise or if in doubt, open a case on the Customer Portal referencing this solution and provide the /var/log/leapp/leapp-preupgrade.log file for review.

Case where /etc/machine-id is missing

Generate a machine id for the system and retry:

# systemd-machine-id-setup
# leapp preupgrade ...

Other cases

Open a case on the Customer Portal to improve this solution regarding new unknown cases.

Root Cause

leapp uses systemd-nspawn internally to execute commands in containers.
In some cases, systemd-nspawn itself fails, instead of the command it's supposed to execute, typically because /etc/machine-id or other critical files are missing on the host.
Unfortunately leapp cannot distinguish between systemd-nspawn and command failure, causing the issue to show up later "randomly".
Hardening of leapp execution is tracked by Jira This content is not included.RHEL-50161.

Diagnostic Steps

1. Execute leapp preupgrade in debug mode

   # leapp preupgrade --debug
   

2. Extract the logs related to "leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator" actor

   # grep -a "leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator" /var/log/leapp/leapp-preupgrade.log > /var/log/leapp/leapp-preupgrade.log.target_userspace_creator
   

3. Search in the extract for "Failed to retrieve machine ID"

   # grep -a "Failed to retrieve machine ID" /var/log/leapp/leapp-preupgrade.log.target_userspace_creator
   2024-07-23 10:46:37.206 DEBUG    PID: 18387 leapp.workflow.TargetTransactionFactsCollection.target_userspace_creator: Failed to retrieve machine ID: No such file or directory
   

• If you have some output, proceed to Resolution - Case where /etc/machine-id is missing

• If you don't have any output, proceed to Resolution - Other cases

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.