Red Hat Satellite 6: key log files and how to enable debug logging (Foreman, Katello, Puppet, Pulp, Candlepin, Hammer, etc.)

Solution Verified - Updated

Environment

  • Red Hat Satellite 6

Issue

  • How to increase debug logging on Red Hat Satellite 6.
  • More debugging information is needed from Foreman production.log, Katello, Puppet, Pulp, Candelpin, Hammer, or Capsule
  • Logging on Red Hat Satellite 6 is way too verbose and generating too much data. How do I restrict it?

Resolution

Notice: This solution describes ways how to change logging verbosity of Satellite6 components. To change selective debugs within foreman/katello only, please follow this solution.

Note:
Try to make sure that you did not run out of space , as the Red Hat Satellite is an embedded application, see Sat 6.7: Admin Guide:Chaper 11.4 and This content is not included.Recover from a "/var" that ran out of space on Satellite 6

df -h

Satellite 6.2 or later

Starting from Red Hat Satellite 6.2, a utility called "hammer admin logging" is available to set debug or production logging levels per individual Satellite components or all of them. This tool only works on Hammer CLI executed on Satellite Server or Capsule directly as it changes configuration files locally. Usage is as follows:

# hammer admin logging --help
Usage:
    hammer admin logging [OPTIONS]

Options:
 --no-backup                   Skip configuration backups creation.
 --prefix PATH                 Operate on prefixed environment (e.g. chroot).
 -a, --all                     Apply to all components.
 -c, --components COMPONENTS   Components to apply, use --list to get them.
                               Comma separated list of values. Values containing comma should be double quoted
 -d, --level-debug             Increase verbosity level to debug.
 -h, --help                    print help
 -l, --list                    List available components.
 -n, --dry-run                 Do not apply specified changes.
 -p, --level-production        Decrease verbosity level to standard.

To enable debugging logging level on all components:

# hammer admin logging --all --level-debug
# satellite-maintain service restart

To flip configuration back to production mode:

# hammer admin logging --all --level-production
# satellite-maintain service restart

Note:
1.hammer admin logging changes the configuration files only. It does not apply these change, which require restarting relevant services. Hence, a manual restart of the services is required to really apply the change.
2.hammer admin logging --all --level-debug can increase the amount of usage of logs in candlepin and foreman.
3. Setting the log level does not require any reboot of satellite/capsule server simply restart the satellite services will makes the changes effective.

Every time the tool changes a configuration file, a backup copy is made. To list all backup copies in chronological order:

# ls /etc/foreman/settings.yaml* -1
/etc/foreman/settings.yaml
/etc/foreman/settings.yaml.ovut7d~
/etc/foreman/settings.yaml.ovut88~
/etc/foreman/settings.yaml.ovut8d~
/etc/foreman/settings.yaml.ovutb5~

To evaluate changes on a file use GNU diff tool:

# diff /etc/foreman/settings.yaml.ovut7d~ /etc/foreman/settings.yaml
33c33
<   :level: info
---
>   :level: debug

To see list of all backups per one execution:

# find /etc /var/lib/pgsql -name \*.ovut7d~
/etc/qpid/qpidd.conf.ovut7d~
/etc/pulp/server.conf.ovut7d~
/etc/sysconfig/virt-who.ovut7d~
/etc/xinetd.d/tftp.ovut7d~
/etc/dhcp/dhcpd.conf.ovut7d~
/etc/named.conf.ovut7d~
/etc/puppet/puppet.conf.ovut7d~
/etc/foreman-proxy/settings.yml.ovut7d~
/etc/foreman/settings.yaml.ovut7d~
/etc/candlepin/candlepin.conf.ovut7d~
/var/lib/pgsql/data/postgresql.conf.ovut7d~

To see list of all components and changed configuration files:


# hammer admin logging --list
-

COMPONENT  | AUTO-DETECTED BY EXISTENCE OF            | DESTINATIONS                                     
-|-|-
postgresql | /var/lib/pgsql/data/postgresql.conf      | syslog /var/lib/pgsql/data/pg_log/               
rails      | /etc/foreman/settings.yaml               | /var/log/foreman/production.log                  
proxy      | /etc/foreman-proxy/settings.yml          | /var/log/foreman-proxy/proxy.log                 
puppet     | /etc/puppet/puppet.conf                  | /var/log/puppet/masterhttp.log                   
puppet5    | /etc/puppetlabs/puppetserver/logback.xml | /var/log/puppetlabs/puppetserver/puppetserver.log
dhcpd      | /etc/dhcp/dhcpd.conf                     | syslog /var/log/dhcpd-debug.log                  
named      | /etc/named.conf                          | syslog                                           
tftp       | /etc/xinetd.d/tftp                       | syslog                                           
redis      | /etc/opt/rh/rh-redis5/redis.conf         | /var/log/redis/redis.log                         
qpidd      | /etc/qpid/qpidd.conf                     | syslog                                           
tomcat     | /etc/candlepin/candlepin.conf            | /var/log/candlepin/ /var/log/tomcat/             
pulpcore   | /etc/pulp/settings.py                    | syslog                                           

virt-who   | /etc/sysconfig/virt-who                  | syslog                                           
-|-|-

To set debug level only on the Tomcat component:

# hammer admin logging --components tomcat --level-debug

Be aware that -c/--components option does work only in Satellite 6.4 or later.

Satellite 6.0 or 6.1

The "hammer admin logging" tool is only available in Satellite version 6.2+ therefore individual components must be configured separately.

Foreman and Katello

  • Modify the /usr/share/foreman/config/environments/production.rb file and ensure the following line exists:
config.log_level = :debug
  • Restart two services on Satellite:
# service httpd restart
# service foreman-tasks restart
  • Verify that more verbose messages are shown in /var/log/foreman/production.log

Enabling selective foreman/katello logging

Since Satellite 6.2, selective logging of katello and foreman is available. Follow this solution for more. Note that default debugs might not enable all areas to debug.

Puppet

Pulp

# log_level: INFO

to:

log_level: DEBUG
  • Apply the change by restarting appropriate services:
# for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i restart; done
  • If setting pulp to Debug, with rsyslog you may encounter a situation where many logs are discarded and missed. If that is the case, you can disable rsyslog Rate-Limiting or create a new logfile called pulp.log and then inside this file, add the following:
# vi /etc/rsyslog.d/pulp.conf

:programname, startswith, "pulp" -/var/log/pulp.log
 & ~

Save this file and restart rsyslog and then Pulp.

# service rsyslog restart
# for i in pulp_resource_manager pulp_workers pulp_celerybeat; do service $i restart; done

Check content of log file /var/log/pulp.log.

Candlepin

  • Add to /etc/candlepin/candlepin.conf line:

    log4j.logger.org.candlepin=DEBUG

  • Restart the tomcat service (tomcat6 if running on RHEL6 and tomcat if RHEL7) :

    # service tomcat restart

  • Verify debug output in log file /var/log/candlepin/candlepin.log

  • If the candlepin log files are found to be too verbose with the default debug settings, the following can be configured in /etc/candlepin/candlepin.conf on agreement with a Red Hat support representative:

    log4j.logger.org.candlepin.resource.ConsumerResource=WARN
log4j.logger.org.candlepin.resource.HypervisorResource=WARN

Capsule/foreman-proxy

  • Uncomment the DEBUG line in /etc/foreman-proxy/settings.yml:
     # WARN, DEBUG, Error, Fatal, INFO, UNKNOWN
     #:log_level: DEBUG
  • Restart the foreman-proxy service :
# service foreman-proxy restart
  • Observe /var/log/foreman-proxy/proxy.log

Hammer

  • Change the following line in /etc/hammer/cli_config.yml:
   :log_level: 'error'

to:

   :log_level: 'debug'
  • Watch for debug output in log: ~/.foreman/log/hammer.log (the log directory is adjustible in the cli_config.yml file, since Red Hat Satellite 6.2 the default is ~/.hammer/log/hammer.log)

Elasticsearch

(Note that Elasticsearch is removed since Satellite 6.2)

To enable debugging of katello<->elasticsearch communication, enable tire debugs in /etc/foreman/plugins/katello.yaml, under/inside common block:

  logging:
    loggers:
      root:
        level: debug
      tire_rest:
        enabled: true

Ensure the space indentation is preserved. To apply the change, restart the Satellite services:

satellite-maintain service restart

Logs will appear in /var/log/foreman/production.log.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.