Why does leapp upgrade fails in emergency with error "UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1: ordinal not in range(128)" ?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 7
    • leapp-0.17.0-1.el7_9.noarch
    • leapp-upgrade-el7toel8-0.20.0-2.el7_9.noarch

Issue

System upgrade failed in emergency mode with below error

[  937.868119] localhost upgrade[1705]:   yum-langpacks-0.4.2-7.el7.noarch
[  941.078514] localhost upgrade[1705]:   yProcess Process-255:
[  941.078514] localhost upgrade[1705]: Traceback (most recent call last):
[  941.124397] localhost upgrade[610]:   File "/usr/lib/python2.7/site-packages/leapp/repository/actor_definition.py", line 74, in _do_run
[  941.124397] localhost upgrade[610]:     actor_instance.run(*args, **kwargs)
[  941.124397] localhost upgrade[610]:   File "/usr/lib/python2.7/site-packages/leapp/actors/__init__.py", line 289, in run
[  941.124397] localhost upgrade[610]:     self.process(*args)
[  941.124397] localhost upgrade[610]:   File "/usr/share/leapp-repository/repositories/system_upgrade/common/actors/dnfupgradetransaction/actor.py", line 63, in process
[  941.124397] localhost upgrade[610]:     shutil.rmtree(userspace.path)
[  941.124397] localhost upgrade[610]:   File "/usr/lib64/python2.7/shutil.py", line 247, in rmtree
[  941.124397] localhost upgrade[610]:     elif onerror is None:
[  941.124397] localhost upgrade[610]:   File "/usr/lib64/python2.7/shutil.py", line 247, in rmtree
[  941.124397] localhost upgrade[610]:     elif onerror is None:
[  941.124397] localhost upgrade[610]:   File "/usr/lib64/python2.7/shutil.py", line 241, in rmtree
[  941.124397] localhost upgrade[610]:     is false and onerror is None, an exception is raised.
[  941.124397] localhost upgrade[610]:   File "/usr/lib64/python2.7/posixpath.py", line 80, in join
[  941.124397] localhost upgrade[610]:     # Trailing '/'es are stripped from head unless it is the root.
[  941.124397] localhost upgrade[610]: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 1: ordinal not in range(128)
[  941.137924] localhost upgrade[610]: ========================================================================================================
[  941.137924] localhost upgrade[610]: Actor dnf_upgrade_transaction unexpectedly terminated with exit code: 1 - Please check the above details
[  941.137924] localhost upgrade[610]: ========================================================================================================

Resolution

If you are reading this document prior to executing leapp upgrade

Follow the procedure described in the Diagnostic Steps section to confirm you have no problematic file on your file systems that would lead to hitting the issue.

If you already hit the issue in reboot phase

  1. Revert the system to the old backup or snapshot prior to the leapp inplace upgrade

  2. Clear any leapp cache in /var/lib/leapp as the affected file may be copied to the leapp temporary upgrade environment at this location

    # rm -rf /var/lib/leapp/el8userspace

  3. Follow the procedure described in the Diagnostic Steps section to remove the problematic files

Root Cause

Based on the python trace, the actor which is running is dnfupgradetransaction and its performing shutil.rmtree(userspace.path) meaning likely removing /var/lib/leapp/el8userspace.
When its trying to remove this directory and all the subdirectories and files, for some file path it has encountered a non-ascii character.

Diagnostic Steps

  1. Check the local file systems for any problematic files

    # for i in `df --local -P | rev | cut -f1 -d' ' | rev | tail -n +2`;do echo 'Checking FS:' $i; find $i -xdev -print; done | grep -a $'\xe2'

    NOTE: Here the option $'\xe2' is used to search the hex character e2. Its the bash representation of the character. More details in bash man page under section QUOTING.

    If the above command lists files, these have to be removed prior to executing leapp upgrade again.

    e.g. command returned:

    /etc/yum.repos.d/���p

  2. For each file listed above, remove the file using find command and inode number

    • List the file to find the inode number

      # ls -li /etc/yum.repos.d/
total 944
 795312 -rw-r--r--. 1 root root      0 Jul 10 22:00 �??p
 795329 -rw-r-----. 1 root root 887964 Jul  9 21:39 redhat.repo

    • Print the file using find command to verify

      # find /etc/yum.repos.d/ -xdev -inum 795312 -print
/etc/yum.repos.d/�??p

    • Delete the file

      # find /etc/yum.repos.d/ -xdev -inum 795312  -delete
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.