How do I configure logging for the various components of a RHEL 6 High Availability or Resilient Storage cluster?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 6 with the High Availability or Resilient Storage Add On

Issue

  • Are there any debug options for RHEL High availability clusters?
  • How can I enable additional diagnostic messages in my cluster?
  • Can I get more information from DLM when a problem occurs?
  • How do I turn on debugging for luci?
  • I want to increase the verbosity of cluster daemons

Resolution

General Configuration


Most settings are controlled via `/etc/cluster/cluster.conf`. The [configuration must be propagated throughout the cluster after making any changes to this file](/site/solutions/5949). Certain changes may require [a restart of the cluster in order to apply](https://access.redhat.com/site/solutions/39903).

Debug Logging for All Daemons

To enable debugging for all cluster daemons, set logging debug="on" in /etc/cluster/cluster.conf. Logging is directed automatically to a specific file for each daemon and the log files are located in the following directory /var/log/cluster/

       <cluster config_version="42" name="rh6cluster">    
          <logging debug="on"/>
           ...  
        </cluster>

Debug Logging for Individual Daemons


To adjust the logging configuration for a specific cluster processes, add a `` element to the `` section of `/etc/cluster/cluster.conf` specific to that daemon. 
    <cluster config_version="42" name="rh6cluster">
        <logging>
             <logging_daemon name="qdiskd" debug="on"/>
             <logging_daemon name="fenced" debug="on"/>
             <logging_daemon name="dlm_controld" debug="on"/>
             <logging_daemon name="gfs_controld" debug="on"/>
             <logging_daemon name="rgmanager" debug="on"/>
             <logging_daemon name="corosync" debug="on"/>
        </logging>
    [...]
    </cluster>

Pacemaker Clusters


[`pacemaker` offers additional settings to control logging for the daemons it provides](https://access.redhat.com/site/solutions/970083).

DLM Debug Logging


There are two options that can be enabled for `dlm` debugging that can be added to the `/etc/cluster/cluster.conf` file. They are `log_debug` for logging kernel debug messages and `plock_debug` for logging posix lock messages. Here is an example snippet of a `/etc/cluster/cluster.conf file with the options added. 
<cluster config_version="42" name="rh6cluster">
          <dlm log_debug="1" plock_debug="1"/>
          ...
</cluster>

Logging of kernel debug messages must be enabled in order to capture dlm messages produced by the above settings. The following is an example of a rule for rsyslog or syslog that captures all kernel messages, including the debug log level.

kern.*                                  /var/log/kernel.log

cman Init Service Issues


To debug an issue when starting/stopping/statusing/etc the `cman` init service (`/etc/init.d/cman`), then breakpoints can be used, so that at the particular break-point then the rest of the script `/etc/init.d/cman` will not be executed. The break-point arguments are listed below: 
- setup: The configfs setup, load modules, set network parameters (incl virtd/xend workarounds)
- join: Joins the cluster (cman_tool join)
- quorum: Runs a node's unfence block, starts qdiskd, and waits for a quorum to form.
- daemons: starts requisite daemons - groupd, fenced, dlm_controld, ocfs2_controld, cman_notifyd, gfs_controld, etc.
- (no argument) - starts all of the above plus joins the fence domain and starts fence_xvmd 

An example of using breakpoints to troubleshoot an issue with quorum(which includes setup, join, and quorum steps):

```
# service cman start quorum
Do debugging, and once finished resume the cman start where the breakpoint left off
# service cman start daemons
```

Conga


Logging can be enabled for the `luci` component of the Conga administration interface with [`luci-0.26.0-13.el6`](http://rhn.redhat.com/errata/RHBA-2012-0766.html) and later, using the *Admin* menu in the upper right corner, then the *Server Logging* tab.

Fencing


Logging for the `fenced` daemon is controlled through the settings in the *Logging Configuration for Individual Daemons* section above. Individiual fence agents may also offer options to print verbose messages, or output to a debug file. See the man page for the specific agent in question to see what options it supports.

Root Cause

If you are opening a ticket with Red Hat support for a Red Hat Enterprise Linux Server 6 (with the High Availability Add on) issue then please see this article on what information should be gather: What data do I need to collect for Red Hat support to troubleshoot a High Availability or Resilient Storage issue?

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.