[Satellite6] Satellite services are not restarted after the backup

Environment

• Red Hat Satellite 6.5

Issue

• Satellite offline backup using lvm snapshop failed, Exit5:

D, [2020-09-26 04:42:21+1200 #10720] DEBUG -- : Invoking tar from /misc/backup/satellite6/online-daily
D, [2020-09-26 04:42:21+1200 #10720] DEBUG -- : Running command tar --selinux --create --file=/misc/backup/satellite6/online-daily/satellite-backup-2020-09-26-04-40-03/config_files.tar.gz --gzip --exclude=/var/lib/foreman/public/assets --listed-incremental=/misc/backup/satellite6/online-daily/satellite-backup-2020-09-26-04-40-03/.config.snar --ignore-failed-read 
:
I, [2020-09-26 05:10:42+1200 #10720]  INFO -- : --- Execution step 'Backup config files' finished ---
I, [2020-09-26 05:10:42+1200 #10720]  INFO -- : --- Execution step 'Create and mount snapshot of Mongo DB' [backup-snapshot-mount-mongo] started ---
D, [2020-09-26 05:10:42+1200 #10720] DEBUG -- : Running command hostname -f with stdin nil
D, [2020-09-26 05:10:42+1200 #10720] DEBUG -- : output of the command:
 prlsat03.nz.thenational.com
D, [2020-09-26 05:10:42+1200 #10720] DEBUG -- : Running command findmnt -n --target /var/lib/mongodb -o SOURCE,FSTYPE with stdin nil
D, [2020-09-26 05:10:42+1200 #10720] DEBUG -- : output of the command:
 /dev/mapper/vg_data_01-var_lib_mongodb xfs
D, [2020-09-26 05:10:42+1200 #10720] DEBUG -- : Running command lvcreate -nmongodb-snap -L2G -s /dev/mapper/vg_data_01-var_lib_mongodb with stdin nil
D, [2020-09-26 05:10:43+1200 #10720] DEBUG -- : output of the command:
 Logical Volume "mongodb-snap" already exists in volume group "vg_data_01"
E, [2020-09-26 05:10:43+1200 #10720] ERROR -- : Failed executing lvcreate -nmongodb-snap -L2G -s /dev/mapper/vg_data_01-var_lib_mongodb, exit status 5:
 Logical Volume "mongodb-snap" already exists in volume group "vg_data_01" (ForemanMaintain::Error::ExecutionError)
/usr/share/gems/gems/foreman_maintain-0.4.10/lib/foreman_maintain/utils/command_runner.rb:54:in `execution_error'

• Logical volume snapshot backup is marked as invalid:

[Mon Sep 28 06:41:12 2020] device-mapper: snapshots: Snapshot is marked invalid.
[Mon Sep 28 06:41:12 2020] Buffer I/O error on dev dm-15, logical block 7602160, async page read

[Thu Oct  8 09:41:15 2020] Buffer I/O error on dev dm-11, logical block 275251184, async page read
[Thu Oct  8 09:41:15 2020] Buffer I/O error on dev dm-15, logical block 7602160, async page read
[Thu Oct  8 09:41:15 2020] Buffer I/O error on dev dm-19, logical block 26214384, async page read

lrwxrwxrwx. 1 root root       8 Sep 28 19:41 vg_data_01-pulp--snap -> ../dm-11
lrwxrwxrwx. 1 root root       8 Sep 28 19:41 vg_data_01-pgsql--snap -> ../dm-15
lrwxrwxrwx. 1 root root       8 Sep 28 19:41 vg_data_01-var_lib_pgsql -> ../dm-13

Resolution

• The logical volume is invalid or could be corrupted, refer to your Storage team/provider to investigate further.
  Alternatively, use RHV or VMware snapshot for backup.

For more KB articles/solutions related to Red Hat Satellite 6.x backup issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x backup-related Issues

Root Cause

• The logical volume is invalid, could be corrupted.

Diagnostic Steps

• To do a dry run filesystem check:

  # xfs_repair -n /dev/mapper/vg_data_01-var_lib_pgsql

   -n   No modify mode. Specifies that xfs_repair should not modify the filesystem but should only scan the filesystem and indicate what repairs would have been made.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.