Satellite installer replaced some configuration files

Solution Verified - Updated 14 Jun 2024

Environment

Satellite 6 Server or Capsule
Satellite installer
Puppet

Issue

Satellite installer (satellite-installer, katello-installer, foreman-installer, capsule-installer) replaced some configuration files, how to restore the originals?

Resolution

Luckily, the installer maintains copies of all configuration files. First, identify which files were overwritten.

First, take a look at the journal / syslog to figure out what md5sum the overwritten file had, by looking for lines like:

/Stage[main]/Dhcp/File[/etc/dhcp/dhcpd.conf]: Filebucketed /etc/dhcp/dhcpd.conf to puppet with sum 622d9820b8e764ab124367c68f5fa3a1

The above example is for an overwritten /etc/dhcp/dhcpd.conf with md5sum 622d9820b8e764ab124367c68f5fa3a1. We can restore it by running the following command:

puppet filebucket restore --local --bucket /var/lib/puppet/clientbucket /etc/dhcp/dhcpd.conf 622d9820b8e764ab124367c68f5fa3a1

On 6.4/6.5

puppet filebucket restore --local --bucket /opt/puppetlabs/puppet/cache/clientbucket /var/www/html/pub/katello-rhsm-consumer 4b26eaa5f202fe3628ef27050e89624b

Or as described in docs Installing Satellite from a Connected network

# puppet filebucket -l \
restore /etc/dhcp/dhcpd.conf 622d9820b8e764ab124367c68f5fa3a1

Before restoring a file, be aware the installer during an upgrade might added some configuration option mandatory for the upgraded Satellite, that is missing in your config (this usually does not happen to DNS or DHCP config). It is recommended to compare the configuration file before and after restoring if this did not happen, and if so, manually add the new option supposed to be added by upgrade.

There is an even easier option to restore a config from filebucket. If you grab Content from github.com is not included.this script from Github, you can interactively restore filebucketed files by running (for our example):

./clientbucket.rb /etc/dhcp/dhcpd.conf

The script will show you the available backups, complete with a timestamp and the option to diff the current version on the filesystem with the backed up version.

On older versions of Satellite (pre-6.2), the above method might not work. Instead, run this command to find filenames and their MD5 sums, use /var/log/capsule-installer/capsule-installer.log when on capsule.

$ grep Filebucketed /var/log/katello-installer/katello-installer.log | sed 's/^.*Filebucketed //'
/etc/dhcp/dhcpd.conf to puppet with sum cf06f277ef451311e185642a77d11bb4
/etc/dhcp/dhcpd.hosts to puppet with sum 3513cb4e5389ab9e0d5757bae7fbeb51
/etc/default/pulp_workers to puppet with sum b9dab6df913154b4528c69cce09c2dd4
/etc/httpd/conf.d/userdir.conf to puppet with sum d4a2620683cc3ff2315c685f9f354265
/etc/httpd/conf.d/welcome.conf to puppet with sum a01d5e7293cc81f258f80a3560f8fca0
/etc/httpd/conf.d/README to puppet with sum 20b886e8496027dcbc31ed28d404ebb1
/etc/httpd/conf.d/25-puppet.conf to puppet with sum 6830430d125fa039618a52d0194f165c
/etc/puppet/puppet.conf to puppet with sum d897cc893a6b5bf6ffb317cc7e5d6e16
/etc/foreman-proxy/settings.d/tftp.yml to puppet with sum 2a03225c1c66bafc31d965605b7b122b
/etc/qpid-dispatch/qdrouterd.conf to puppet with sum e730270269e88726148a2b51d87a7a55

Each installer run rotates logs by one number, so .log is the latest run, .1.log is previous run etc.

[root@scirhsat6 ~]# find /var/log/katello-installer/
/var/log/katello-installer/
/var/log/katello-installer/katello-installer.log
/var/log/katello-installer/katello-installer.3.log
/var/log/katello-installer/katello-installer.2.log
/var/log/katello-installer/katello-installer.1.log

Now proceed with restoration. Puppet provides a backup directory of all changed files, it maitains them in directory structure based on MD5 sums of content:

[root@scirhsat6 ~]# find /var/lib/puppet/clientbucket 
/var/lib/puppet/clientbucket
/var/lib/puppet/clientbucket/d
/var/lib/puppet/clientbucket/d/b
/var/lib/puppet/clientbucket/d/b/5
/var/lib/puppet/clientbucket/d/b/5/7
/var/lib/puppet/clientbucket/d/b/5/7/e
/var/lib/puppet/clientbucket/d/b/5/7/e/0
/var/lib/puppet/clientbucket/d/b/5/7/e/0/e
/var/lib/puppet/clientbucket/d/b/5/7/e/0/e/a
/var/lib/puppet/clientbucket/d/b/5/7/e/0/e/a/db57e0ea6abf9ffdfb75fa6bd1d6119a
/var/lib/puppet/clientbucket/d/b/5/7/e/0/e/a/db57e0ea6abf9ffdfb75fa6bd1d6119a/paths
/var/lib/puppet/clientbucket/d/b/5/7/e/0/e/a/db57e0ea6abf9ffdfb75fa6bd1d6119a/contents

The "paths" file contains file path and "contents" contain the actual content. Instead of manually finding the files and coping them over, there is a command that does the job. To see content of a file from backup database, issue:

puppet filebucket -l --bucket /var/lib/puppet/clientbucket get cf06f277ef451311e185642a77d11bb4 | head

This should present contents of the file backup. WARNING: There are several puppet versions with bugs that does not find the correct content and returns nothing. If this is the case, scroll down to Alternative restore. To restore via puppet the file do this:

puppet filebucket -l --bucket /var/lib/puppet/clientbucket restore cf06f277ef451311e185642a77d11bb4 /etc/dhcp/dhcpd.conf

Alternative restore

Download helper interactive ruby utility that guides you with restoration of any historical version of a file and run it for each individual file found via the grep command above:

wget https://raw.githubusercontent.com/andytinycat/puppet-clientbucket-restore/master/clientbucket.rb
ruby clientbucket.rb /etc/dhcp/dhcpd.conf
[0]: cf06f277ef451311e185642a77d11bb4 2017-02-14 09:12:40 -0500
------------------------
Pick a file, or x to exit: 0
Done

Repeat for all individual files.

Root Cause

Satellite installer is puppet based, it maintains desired state in input parameters which are also known as "answers" (answer file) and each time it is executed it makes all the necessary changes to get to that state. When running the installer again (e.g. after upgrade), it is highly recommended to provide -n -v options (dry run, verbose) to see changes that will be made.

Typically, dhcpd configuration will be overwritten and all subnets added after installation will be lost, in that case disable dhcp module and run dry run again, until all configuration file changes are resolved. Then execute it.

Diagnostic Steps

Some configuration files were overwritten, symptoms are various. Typically, new DHCP reservation cannot be made because subnet is missing.

Product(s)

Red Hat Satellite

Components

foreman

Tags

satellite_6

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.