[Satellite 6.x] 'WARNING: Snapshot rootvg-pgsql--snap changed state to: Invalid and should be removed.' warning during Satellite backup
Environment
Red Hat Satellite 6.x
Issue
While performing Satellite backup via the following command:
# satellite-maintain backup snapshot --skip-pulp-content /var/tmp/satellite-backup/
the backup completes successfully:
The backup is completing successfully
Done with backup: 2021-06-06 20:55:52 +0100
**** BACKUP Complete, contents can be found in: /var/tmp/satellite-backup ****
However, the following warnings appear in /var/log/messages on the Satellite server:
Jun 6 20:46:15 satellite-serv lvm[3254]: WARNING: Snapshot rootvg-pgsql--snap is now 91.39% full.
Jun 6 20:46:15 satellite-serv lvm[3254]: WARNING: Snapshot rootvg-pgsql--snap is now 91.39% full.
Jun 6 20:46:15 satellite-serv kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Jun 6 20:46:15 satellite-serv kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Jun 6 20:46:15 satellite-serv lvm[3254]: WARNING: Snapshot rootvg-pgsql--snap changed state to: Invalid and should be removed.
Jun 6 20:46:15 satellite-serv lvm[3254]: WARNING: Snapshot rootvg-pgsql--snap changed state to: Invalid and should be removed.
Jun 6 20:46:15 satellite-serv lvm[3254]: Unmounting invalid snapshot rootvg-pgsql--snap from /var/snap/pgsql.
Jun 6 20:46:15 satellite-serv lvm[3254]: Unmounting invalid snapshot rootvg-pgsql--snap from /var/snap/pgsql.
Resolution
Save Satellite backups on a different VG than the one from which the data is backed up.
For more KB articles/solutions related to Red Hat Satellite 6.x backup issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x backup-related Issues
Root Cause
When you use snapshots to perform Satellite backup, this kind of backup utilizes LVM snapshots to back up the /var/lib/pulp/, /var/lib/mongodb/, and /var/lib/pgsql/ directories. This is done to ensure consistency of the backup. After the snapshot is created, its contents are copied to the backup directory. Based on whether the '--incremental' option is used or not, the backup is either full or incremental. After that, the backup operation will delete the LVM snapshots.
A LVM snapshot should be big enough in order to store any changes made in the original LV during its lifetime. So, if we're saving new data to the volume, that data is consuming snapshot space. Any changes that happen on the filesystem due to normal satellite operations are also consuming snapshot space (and that's normal and usually not a lot). By default, the snapshot size is 2GB (unless foreman-maintain was called with options to make it bigger).
This kind of warning happens when the snapshot space is running low and then LVM is supposed to try extending the snapshot size:
Jun 6 20:46:15 rook lvm[3254]: WARNING: Snapshot rootvg-pgsql--snap is now 91.39% full.
Jun 6 20:46:15 rook lvm[3254]: WARNING: Snapshot rootvg-pgsql--snap is now 91.39% full.
However, if for some reason the snapshot size cannot be increased, then it will be invalidated:
Jun 6 20:46:15 rook kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Jun 6 20:46:15 rook kernel: device-mapper: snapshots: Invalidating snapshot: Unable to allocate exception.
Jun 6 20:46:15 rook lvm[3254]: WARNING: Snapshot rootvg-pgsql--snap changed state to: Invalid and should be removed.
Jun 6 20:46:15 rook lvm[3254]: WARNING: Snapshot rootvg-pgsql--snap changed state to: Invalid and should be removed.
Jun 6 20:46:15 rook lvm[3254]: Unmounting invalid snapshot rootvg-pgsql--snap from /var/snap/pgsql.
Jun 6 20:46:15 rook lvm[3254]: Unmounting invalid snapshot rootvg-pgsql--snap from /var/snap/pgsql.
Diagnostic Steps
Using the lvdisplay command, verify whether the Satellite backups are stored on the same VG as the source data or on a separate VG.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.