How can I enable debug mode for qdiskd in a RHEL 5 cluster?

Environment

• Red Hat Enterprise Linux (RHEL) 5 with the High Availability Add On
• A configuration utilizing a quorum device (<quorumd> in /etc/cluster/cluster.conf)

Issue

• How can I enable debug mode for qdiskd in cluster?
• How can I obtain more data from qdiskd to assist in diagnosing unexpected behavior?

Resolution

syslog approach (recommended)

To cause the quorum disk daemon qdiskd to log more information to syslog:

1. Add the log_level and log_facility attributes to the quorumd element in /etc/cluster/cluster.conf (7 is the highest log_level, and the log_facility should be a syslog facility):

    <quorumd interval="1" tko="5" votes="2" log_level="7" log_facility="local5" label="rh5qdisk">
   

2. Create a separate log file for qdiskd, such as /var/log/qdiskd.log:

    # touch /var/log/qdiskd.log
   

3. Implement a debug logging level for qdiskd in /etc/syslog.conf, matching the above log_facility and file. For example:

    local5.*                 /var/log/qdiskd.log
   

4. Propagate the configuration change to the other nodes.

5. Restart qdiskd and syslog services.
   WARNING: Restarting qdiskd will result in a temporary loss of the quorum device votes. This should only be performed while enough node-votes are available to maintain quorum without the quorum device.

    # service syslog restart
    # service qdiskd restart
   

Logging output will be captured to /var/log/qdiskd.log.

status_file alternative

In a small number of cases, a status_file may be useful to gather real-time information from qdiskd. Do not use this approach unless recommended by Red Hat Support.

• NOTE: The status_file attribute should only be used for diagnostic purposes, and is generally not recommended for production systems. This option introduces the possibility of I/O deadlocks in qdiskd, which could lead to evictions and/or membership changes.
• Because status_file is a real-time log of the state of qdiskd, it is often not very useful for capturing the source of problems for later review. It must be monitored for a specific value or change in order to be beneficial, and as such is typically not used as a general diagnostic tool for qdiskd.

If it would be useful, a status_file can be configured on the quorumd element:

        <quorumd interval="1" tko="5" votes="2" log_level="7" label="r5qdisk" status_file="/tmp/qdiskd.status">

Again, the configuration must be propagated throughout the cluster, and qdiskd service restart.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.