How to rebuild RPM database on a Red Hat Enterprise Linux system?

Solution Verified - Updated

Environment

Issue

  • rpm or yum commands fail

  • Common Errors:

    • cannot open Packages database in /var/lib/rpm
    • unexpected file type or format
    • damaged header instance
    • rpmdb: Lock table is out of available locker entries
    • rpm -qa shows nothing or hangs

  • Attempted to do an rpm -q on a couple of packages and got errors like this:

    rpmdb: /var/lib/rpm/Packages: unexpected file type or format  
error: cannot open Packages index using db3 - Invalid argument (22)

  • On doing rpm query or install, it is failing with following error:

    error: rpmdbNextIterator: skipping h#    1601 Header V4 RSA/SHA1 signature: BAD, key ID 2874ffbe

Resolution

Note: On RHEL 6, before attempting to rebuild the rpm database, first refer to the following known issue: Update to nss-softokn 3.14.3-19 breaks RPM.

For RHEL 9 and later

For RHEL 8 and earlier

1) Stale lock file cleanup - files needed for updating are being held open.

Note: There are two methods (1A & 1B) to correct this.
Note: As always it is best to start by taking a backup before attempting to repair.

# tar -zcvf /tmp/rpmdb-$(date +%Y-%m-%d_%H-%M-%S).tar.gz /var/lib/rpm

Note: This tar backup can be used if the attempt to recover the RPM database has issues.

1A) Reboot system (preferred)

  1. The best course of action is to simply reboot the machine, as the system cleans up stale locks during the boot. The cleanup occurs at this stage as the system is assured there is no process that would have a lock on the RPM database.

1B) Reboot not feasible (more risky than reboot)

  1. Ensure there are no processes with the RPM database files open. One way to do this is using lsof & ps as the root user:
# ps aux | grep -E "rpm|yum|dnf|up2date"
# lsof | grep /var/lib/rpm
  1. If, and only if, there is no RPM database activity going on it is safe to delete the lock files.
    WARNING: If the system is in multi-user mode it is possible that a process may open the RPM database at any time. Therefore, the recommended method for clearing stale locks is the reboot which cleans up the locks prior to entering multi-user mode.
# rm -f /var/lib/rpm/__db*

2) DB corruption recovery process - The RPM database has become corrupt and recovery is desired.

Note: If you are coming here from 1B), you can go to step 2-2.

  1. As always it is best to start by taking a backup before attempting to repair. 

    # tar -zcvf /tmp/rpmdb-$(date +%Y-%m-%d_%H-%M-%S).tar.gz /var/lib/rpm

    Note: This tar backup can be used if the attempt to recover the RPM database has issues.

  2. Verify integrity of the Packages file:

    2-1. Delete lock files.

    # cd /var/lib/rpm
# rm -f __db*      # to avoid stale locks
# /usr/lib/rpm/rpmdb_verify Packages

    2-2. Verify Packages

    # /usr/lib/rpm/rpmdb_verify Packages

    If the command shown in method 2-2 returns without any error, then we can be sure about the integrity of the rpm database. You can also issue the following command to be sure of the success of the rpmdb_verify command.

    $ /usr/lib/rpm/rpmdb_verify Packages
$ echo $?
0

    A return value of 0 from echo $? means that the last command has been executed successfully.

  3. If the above verify reports success, go to step 6.

  4. If the above verify reports any errors, a dump and load of the database is required.

    # mv Packages Packages.orig
# /usr/lib/rpm/rpmdb_dump Packages.orig | /usr/lib/rpm/rpmdb_load Packages

  5. Verify the newly loaded Packages file, as well as rpm output.

    # /usr/lib/rpm/rpmdb_verify Packages
# rpm -qa >/dev/null    # Validating there are no errors
# rpm -qa               # Validating list of installed RPMs

  6. If no errors and everything looks good, rebuild the RPM indexes

    # rpm -vv --rebuilddb

  7. Once again verify RPM database:

    # cd /var/lib/rpm
# /usr/lib/rpm/rpmdb_verify Packages

Notes:

  • If the above recovery steps fail or throw errors, restoring /var/lib/rpm directory from a recent system backup may be your best option. Remember to not overwrite the tar backup as it may still be valuable.
  • If you have a rpm database in other directory except for /var/lib/rpm and it's corrupted, you will need to do the same things to the rpm database, not one in /var/lib/rpm. The rpm command has --root option to specify the root directory for the rpm database. Especially, please take care of the rpm -vv --rebuilddb command to rebuild the rpm database. For example, if you are using /opt/var/lib/rpm for a directory including rpm database, run this command rpm -vv --rebuilddb --root /opt.
  • To reconstruct the RPM database, a Packages file (typically located at /var/lib/rpm/Packages) is required. Without that file, it might be possible to copy the RPM database from another healthy system which was installed similarly, with the same RHEL minor version. After making a copy of the RPM database, you should execute rpm -Va to find out potential discrepancies, and then reinstall those packages.

3) Other resources:

Root Cause

There are multiple reasons which can lead to rpm database corruption. Refer to: Causes and Prevention of RPM Database Corruption in RHEL.

Diagnostic Steps

  • yum update fails with the following:
[root@localhost ~]# yum update
rpmdb: PANIC: fatal region error detected; run recovery
error: db3 error(-30974) from dbenv->open: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed
[root@localhost ~]# yum clean all
rpmdb: PANIC: fatal region error detected; run recovery
error: db3 error(-30974) from dbenv->open: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm
CRITICAL:yum.main:

Error: rpmdb open failed
  • rpm shows damaged header instances:
$ rpm -Va
error: rpmdb: damaged header instance #<header number> retrived, skipping
error: rpmdb: damaged header instance #<header number> retrived, skipping
(above message repeated ad infinitum)
  • rpm commands terminate with the error below:
rpmdb: unable to join the environment  
error: db3 error(11) from dbenv->open: Resource temporarily unavailble  
error: cannot open Packages index using db3 - Resource temporarily unavailable (11)  
error: cannot open Packages database in /var/lib/rpm
  • rpm and yum commands return the following:
$ rpm -qa  
...  
rpmdb: Lock table is out of available locker entries  
error: db4 error(22) from db->close: Invalid argument
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.