Update to nss-softokn 3.14.3-19 breaks RPM

Solution Verified - Updated 5 Aug 2024

Environment

Red Hat Enterprise Linux 6.6 and older
nss-softokn-3.14.3-19.el6_6 rpm

Issue

bad signature errors when running rpm commands:

# rpm -qa 
error: rpmdbNextIterator: skipping h#     496 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD
error: rpmdbNextIterator: skipping h#     241 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD
error: rpmdbNextIterator: skipping h#     498 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD
error: rpmdbNextIterator: skipping h#     243 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD
error: rpmdbNextIterator: skipping h#     500 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD
...

"Problem with the SSL CA cert" errors when download rpm files in yum:

# yum update -y
[...]
  system-config-firewall-base-1.2.27-7.2.el6_6.noarch: failed to retrieve getPackage/system-config-firewall-base-1.2.27-7.2.el6_6.noarch.rpm from rhel-x86_64-server-6
error was [Errno 14] PYCURL ERROR 77 - "Problem with the SSL CA cert (path? access rights?)"

rpm query commands don't show any packages installed, or only show gpg-pubkey packages as follows:

# rpm -q kernel
error: rpmdbNextIterator: skipping h#     206 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD
package kernel is not installed
# rpm -qa 2>/dev/null
gpg-pubkey-2fa658e0-45700c69
gpg-pubkey-fd431d51-4ae0493b
#

Resolution

Create a /usr/lib*/libfreeblpriv3.so symlink
- [64-bit systems] Run the following command:
```
# ln -sv libfreebl3.so /usr/lib64/libfreeblpriv3.so
```
- [32-bit systems] Run the following command:
```
# ln -sv libfreebl3.so /usr/lib/libfreeblpriv3.so
```
Check the output of rpm -qa or any other rpm -q command
- If the output looks good, normal operations can resume
  However, it is recommended to update the nss-softokn* packages (RHBA-2015:0110), e.g.:
```
# yum update nss-softokn\*
```
- If output is not as expected:
  - Check the size of the /var/lib/rpm/Packages file with ls -lh -- if its size can be measured in K instead of M, it has been wiped by someone running an rpm --rebuilddb command (probably as advised by rpm or yum)
    1. The only way to recover from this is by replacing the Packages file from a backup or an identical system
    2. After replacing /var/lib/rpm/Packages, run rpm --rebuilddb and then re-check rpm -qa output
    3. Finally, check rpm -Va output to look for inconsistencies (which could be caused by restoring a Packages file from a system that was not actually identical

Root Cause

The nss-softokn-3.14.3-19.el6_6 package was built with an incorrect dependency on nss-softokn-freebl version 3.14.3, instead of the correct 3.14.3-19 version
If a system installs nss-softokn-3.14.3-19.el6_6, it will not get the appropriately matched version of nss-softokn-freebl as a dependency and rpm operations will cease to work until the symlink work-around is implemented
- This happened to many machines before Red Hat released an updated nss-softokn package; however, it could only happen for machines which did selective updates (e.g., yum update --security)
- Now that there are newer versions of nss-softokn, this can only happen if a machine explicitly asks for the nss-softokn-3.14.3-19.el6_6 package

A machine facing this nss-softokn problem is usually discovered to have an issue during yum or rpm operations and in some of those situations, rpm helpfully suggests the following:
```
Could not determine what version of Red Hat Linux you are running.
If you get this error, try running 

        rpm --rebuilddb
```
The above error is printed because rpm can't access the rpm database and decides it must be corrupt -- if that were actually the case, running that command would be a good idea; however in this situation the problem is simply that rpm doesn't have a library it needs for db operations, so running the above command will actually irreparably destroy the /var/lib/rpm/Packages file. As described in the Resolution, this can only be fixed be implementing the symlink, restoring the Packages file from a backup, and THEN doing an rpm --rebuilddb.

Diagnostic Steps

The following would likely be seen on a system facing this issue.

Check for rpmdbNextIterator signature errors in rpm -q output

Package listing reports errors as follows

# rpm -qa
error: rpmdbNextIterator: skipping h#     257 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD
error: rpmdbNextIterator: skipping h#       2 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD
error: rpmdbNextIterator: skipping h#     259 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD
error: rpmdbNextIterator: skipping h#       4 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD
...

NOTE: these errors will be hidden if someone has already rebuilt the database with rpm --rebuilddb

Check for normal rpm output

Re-check installed pacakges, hiding errors, and only a handful of packages are shown

# rpm -q kernel
error: rpmdbNextIterator: skipping h#     103 Header V3 RSA/SHA256 Signature, key ID fd431d51: BAD
package kernel is not installed
# rpm -qa 2>/dev/null
gpg-pubkey-2fa658e0-45700c69
gpg-pubkey-fd431d51-4ae0493b

Check yum history for installs of nss-softokn
- Last installed version is nss-softokn-3.14.3-19.el6_6 according to yum history
```
yum history packages-list nss-softokn
```
- NOTE: yum history will show no history and only error out with RPM error. ... try running rpm --rebuilddb if someone has already rebuilt the database
- Alternatively, the same might be visible in the yum log
```
grep nss-softokn /var/log/yum.log
```
Check for a /usr/lib*/libfreeblpriv3.so symlink
- The following command shows no libfreeblpriv3.so file:
```
ls -l /usr/lib{,64}/*freeb*so
```

SBR

Anaconda

Product(s)

Red Hat Enterprise Linux

Components

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.