How to properly clean redundant and unneeded host facts data stored in Satellite 6?

Solution Verified - Updated

Environment

Red Hat Satellite 6

Issue

  • Host Facts are stored during host registration and ongoing periodic uploads during normal operation. If there are too many facts stored in Satellite, it might affect the performance of the Satellite server. Most users of Satellite don't use all the facts reported and stored in the database. There are often cases when there are are tens of thousands of facts per host which can lead to very large tablesizes in the Satellite 6 database.

  • Recently, with the increase in adoption of OpenShift and containers, the number of facts reported by hosts are going up rapidly in customer environments. For example, this can happen if many network interfaces are created and deleted. We already filter out some facts (by default) which caused issues in the past, but there are some new fact offenders that were found recently while working with some of our large customers.

  • Satellite performance is poor

  • you find a large number of long running queries in PostgreSQL associated with fact_names or fact_values.

    • Note: Enable log_min_duration_statement in PostgreSQL configuration to log long running queries.

  • our systems were reporting to Satellite6 redundantly detailed facts that slowed down facts processing

  • setting "Ignore interfaces with matching identifier" and/or "Exclude pattern for facts stored in satellite" was updated accordingly to prevent storing new redundant facts

  • how to prune out stored facts?

  • Satellite upgrade fails with the below error.

    Check number of fact names in database (WARNING)
Host (ID 1043) has 10658 fact values which is more than 10000
This can cause slow fact processing.

Resolution

  • Get the list of fact_names and find the top facts contributors.

    # su - postgres -c "psql foreman -c \"select name from fact_names;\"" > pg_fact_names.log
# awk -F "::" '{ print $1 }' pg_fact_names.log | sort | uniq -c | sort -nr | head -n5
 407683  mountpoints
 345778  partitions
  46945  disks
  11158  net
   6581  networking

  • Find the fact names which you don't use and add it to the filter exclude list. In this example, customer is not using partitions* and mountpoints* .

SATELLITE 6.10 AND BELOW

  • Go to Satellite UI -> Administer -> Settings -> Provisioning -> Exclude pattern for facts stored in satellite -> Add the following to the already existing list: partitions*, mountpoints*

SATELLITE 6.11+

  • Go to Satellite UI -> Administer -> Settings -> facts -> Exclude pattern for facts stored in satellite -> Add the following to the already existing list: partitions*, mountpoints*

After this, we can clean the facts from the database. Two rake scripts shall be used. To prevent concurrent uploading of facts (what would make the scripts slower and error-prone), Satellite must be set into a maintenance mode, such that no clients can access it (a firewall rule will prevent that):

# foreman-maintain maintenance-mode start    # some older Sat versions use  foreman-maintain advanced procedure run maintenance-mode-enable
# foreman-rake interfaces:clean
# foreman-rake facts:clean
# foreman-maintain maintenance-mode stop   # some older Sat versions use  foreman-maintain advanced procedure run maintenance-mode-disable

Note:
The interfaces:clean and facts:clean rake tasks might need to run for a significant long time to completion. If the Satellite server cannot be kept in the maintenance mode long enough for these tasks to complete, it is safe to abort them at any time and the only consequence would be that they did not finish deleting all that they could. These tasks can be restarted at a later time to continue with the cleanup.

After completion, verify the removal of facts (note the mountpoints and partitions are gone):

# su - postgres -c "psql foreman -c \"select name from fact_names;\"" > pg_fact_names.log
# awk -F "::" '{ print $1 }' pg_fact_names.log | sort | uniq -c | sort -nr | head -n5
  46945  disks
  11158  net
   6581  networking

Be aware that

  • Satellite is put into maintenance mode during the procedure, causing client activity (WebUI, Hosts requests,..) is blocked
  • the rake script can take significant amount of time - depending on number of facts or interfaces to clean, it can run up to several hours

NOTE: If errors are observed during facts cleanup there is a hotfix available that is landing in Satellite 6.6.2 to resolve cleanup errors with large amounts of facts:

This content is not included.Foreman-rake facts:cleanup batch/rolling support

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.