Why does leapp preupgrade fail with `sqlite3.OperationalError: unable to open database file` traceback error ?

Environment

• Red Hat Enterprise Linux Server 8
• Red Hat Enterprise Linux Server 7
  • Leapp

Issue

• leapp preupgrade command fails with below traceback error when upgrading from RHEL 7 to RHEL 8 :

  # leapp preupgrade
  Traceback (most recent call last):
    File "/usr/bin/leapp", line 9, in <module>
      load_entry_point('leapp==0.12.1', 'console_scripts', 'leapp')()
    File "/usr/lib/python2.7/site-packages/leapp/cli/__init__.py", line 18, in main
      cli.command.execute('leapp version {}'.format(VERSION))
    File "/usr/lib/python2.7/site-packages/leapp/utils/clicmd.py", line 105, in execute
      args.func(args)
    File "/usr/lib/python2.7/site-packages/leapp/utils/clicmd.py", line 127, in called
      self.target(args)
    File "/usr/lib/python2.7/site-packages/leapp/cli/upgrade/__init__.py", line 352, in preupgrade
      e.store()
    File "/usr/lib/python2.7/site-packages/leapp/utils/audit/__init__.py", line 79, in store
      self.do_store(connection)
    File "/usr/lib/python2.7/site-packages/leapp/utils/audit/__init__.py", line 126, in do_store
      (self.context, self.configuration, self.stamp, self.kind))
  sqlite3.OperationalError: unable to open database file
  

• leapp preupgrade command fails with below traceback error when upgrading from RHEL 8 to RHEL 9 :

    Traceback (most recent call last):
      File "/usr/lib/python3.6/site-packages/leapp/repository/actor_definition.py", line 74, in _do_run
      File "/usr/lib/python3.6/site-packages/leapp/actors/__init__.py", line 289, in run
      File "/etc/leapp/repos.d/system_upgrade/common/actors/targetuserspacecreator/actor.py", line 58, in process
      File "/usr/lib/python3.6/site-packages/leapp/utils/deprecation.py", line 42, in process_wrapper
      File "/etc/leapp/repos.d/system_upgrade/common/actors/targetuserspacecreator/libraries/userspacegen.py", line 256, in perform
      File "/usr/lib64/python3.6/contextlib.py", line 99, in __exit__
      File "/etc/leapp/repos.d/system_upgrade/common/libraries/overlaygen.py", line 593, in create_source_overlay
      File "/etc/leapp/repos.d/system_upgrade/common/libraries/overlaygen.py", line 326, in cleanup_scratch
      File "/usr/lib64/python3.6/logging/__init__.py", line 1296, in debug
      File "/usr/lib64/python3.6/logging/__init__.py", line 1444, in _log
      File "/usr/lib64/python3.6/logging/__init__.py", line 1454, in handle
      File "/usr/lib64/python3.6/logging/__init__.py", line 1516, in callHandlers
      File "/usr/lib64/python3.6/logging/__init__.py", line 865, in handle
      File "/usr/lib/python3.6/site-packages/leapp/logger/__init__.py", line 40, in emit
      File "/usr/lib/python3.6/site-packages/leapp/logger/__init__.py", line 45, in _do_emit
      File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 87, in store
      File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 73, in get_connection
      File "/usr/lib/python3.6/site-packages/leapp/cli/commands/upgrade/util.py", line 26, in wrapper
      File "/usr/lib/python3.6/site-packages/leapp/utils/audit/__init__.py", line 60, in create_connection
  sqlite3.OperationalError: unable to open database file
  

Resolution

1) The issue happens due to problems with /var:

1. Verify that /var/lib/leapp directory exists

   If it doesn't, reinstall leapp package:

   # yum -y reinstall leapp
   

2. If /var filesystem is mounted in read only mode then remount it in read write mode

   Refer to How can I mount a read-only filesystem as read-write? for details.

3. Check the /var filesystem disk space utilization. If disk space is full then remove some unwanted files to free disk space.

4. If there is nothing to delete then extend the /var logical volume or create a dedicated /var/lib/leapp logical volume.

   Refer to How to extend a logical volume and its filesystem online in Red Hat Enterprise Linux? for details.

2) The issue happens when Leapp hits the per-process file descriptor limit:

1. Raise the limit with ulimit. This change is not permanent:

   # ulimit -n 16384
   

2. Verify that the new limit has been applied successfully:

   # ulimit -n
   

   • For more information about file limits, refer to How to increase number of open files limit in Red Hat Enterprise Linux ?

Root Cause

• The /var filesystem is mounted as read only.
• The /var filesystem is 100% full.
• Leapp hits the per-process file descriptor limit.
• Please note this traceback can happen in other circumstances, in particular: Leapp can fail when there are too many LV partitions

Diagnostic Steps

1) The issue happens due to problems with /var:

• Check the disk space utilization and mounted filesystem status:

      # df -h | grep var
      /dev/mapper/vgname-lvname   4.7G  4.7G   20K 100% /var
  
      # cat /proc/mounts | grep var
  

• Check for existence of /var/lib/leapp directory

  # ls -ld /var/lib/leapp
  

2) The issue happens when Leapp hits the per-process file descriptor limit:

1. Use Strace to log the syscalls called and the signals received by the process:

   # strace -fttTvyyo /tmp/leapp.strace -s 128 <leapp-command>
   

2. Check the output file /tmp/leapp.strace for EMFILE errors:

   # grep "1 EMFILE" /tmp/leapp.strace
   
   70588 16:36:26.909214 epoll_create(1023) = -1 EMFILE (Too many open files) <0.000006>
   70588 16:36:26.909394 open("/var/lib/leapp/leapp.db", O_RDWR|O_CREAT|O_CLOEXEC, 0644) = -1 EMFILE (Too many open files) <0.000023>
   70588 16:36:26.909445 open("/var/lib/leapp/leapp.db", O_RDONLY|O_CLOEXEC) = -1 EMFILE (Too many open files) <0.000017>
   

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.