What data do I gather if processes accessing GFS1 are hung on RHEL4?
The following steps can be used when working with Red Hat support to assist in the diagnosis and resolution of GFS locking problems.
This article applies only to:
- GFS
- Red Hat Cluster Suite 4 and Red Hat GFS 6.1
Perform these steps to gather information needed to analyze a cluster having hang issues with a GFS file system. Usually only one mounted file system causes a problem that needs to be analyzed. If this is the case, we only need information for that particular GFS mount point and device. However, if more than one GFS file system is having problems, please gather information for each of the affected GFS file systems. Once a hang occurs, do not modify the GFS file system so that we can have a consistent view of the device until the analysis is complete.
On every node in the cluster, please dump the GFS locks for all(or just a particular GFS filesystem) the mounted GFS file systems, then collect the sysrq-t information. If we do not have all this information then we will have an incomplete diagnosis of the current view of the locks.
1) Run these commands in a script session to gather various information about the cluster. Hit control-D to exit the script session. The "script" command will then write the file.
Gather this information before doing the next section if possible, because some of the steps below could prevent getting all the information. If the issue is random, try to collect it immediately after the hang occurs. If that's not possible, collect it after all the information below has been collected.
$ script $(hostname)-cluster_data-$(date +%Y%m%d.%H%M.%s).txt
$ set -x
$ hostname
$ date
$ cman_tool version
$ cman_tool status
$ cman_tool services
$ cman_tool -af nodes
$ ccs_tool lsnode
$ ccs_tool lsfence
$ clustat -fl
$ ps axwwo user,pid,%cpu,%mem,vsz,rss,wchan=WIDE-WCHAN-COLUMN,stat,start,cputime,comm
Run the next command for each GFS mountpoint that we looking at. mountpoint is name of mount.
$ gfs_tool gettune /mount/point > $(hostname)-gettune.$(date +%Y%m%d.%H%M.%s).txt
$ exit
Perform the following steps when the hang has occurred, on all nodes in the cluster.
2) If you have recreation steps then document them as shown below. If only one node is needed to do the recreation then you will only need to supply the one file.
*This step can be skipped if reproducer steps are not known. *
Use a command called "script" to capture the commands and output of the reproduction of the problem if possible.
$ script $(hostname)-script_recreation.txt
$ hostname
$ date
Do recreation steps below
This command will create a file called "$(hostname)-script_recreation.txt" after you are done doing the commands to reproduce the problem.
Hit control-D to exit. The "script" command will then write the file.
Run the script and/or recreate the GFS file system hang
3) Dump the locks the capture the information
The cman command will give you a list of running services, including the name of the GFS file system. You need this name to dump the locks. This is done by setting the /proc file to read those locks from the GFS file system into a file. Then we capture the locks in a file.
$ mkdir /tmp/$(hostname)-cluster-debug/
$ cman_tool services
$ gfs_tool lockdump <GFS filesystem mount point> >/tmp/$(hostname)-cluster-debug/lockdump-<mountpoint>.txt
$ echo "<GFS filesystem name>" > /proc/cluster/dlm_locks
$ cat /proc/cluster/dlm_locks > /tmp/$(hostname)-cluster-debug/$(hostname)_dlmlocks-<GFS filesystem name>.txt
$ tar jcvf /tmp/$(hostname)-cluster-debug.tar.bz2 /tmp/$(hostname)-cluster-debug
4) Perform sysrq -t once the hang has occurred.
The "-t" option will print the program calling sequence (backtrace) to /var/log/messages for every process running on the system.
sysrq-t will cause the system for a moment to pause and dump its threads, or process list to the console. If your system has an excess amount call sysrq-t can cause the system to miss a heartbeat or two. This could lead to the node *being fenced. *
If you are having hang issues, you may want to temporarily increase the post_fail_delay in /etc/cluster/cluster.conf until you get good data to analyze.
How to delay fencing for sysrq data capture?
Run this command twice, 30 seconds apart each time, so we can see the state of each processes and what they're doing.
What is the SysRq facility and how do I use it?
5) Gather a fresh sosreport from all nodes after all the other steps have been performed.
This step may be done after all the other information has been gathered. The sosreports can be gathered after the system has been rebooted (if it has been manually rebooted or fenced).
What is an sosreport and how do I create it in Red Hat Enterprise Linux?
6) Send in all the requested information and follow the steps below so that there is unified structure for reviewing the data(There could be multiple files for some of these):
**
**
-
$(hostname)-cluster_data-$(date +%Y%m%d.%H%M.%s).txt
-
$(hostname)-script_recreation.txt
-
sosreport that contains the sysrq-t info
-
$(hostname)-gettune.$(date +%Y%m%d.%H%M.%s).txt
-
$(hostname)-cluster-debug.tar.bz2
Here is an example of how to setup the directory and compress it to send to Red Hat Support. All the files that were generated with this document should be in those directories. This means that files located on other nodes to be copied over to this directory:
$ mkdir /root/clusterdumps<br />$ mkdir /root/clusterdumps/sosreports /root/clusterdumps/cluster-data /root/clusterdumps/gettune<br />$ scp all the files to one machine in the respective directories.
The layout should look something like this where "@dirname" indicates a directory.
$ ls /root/clusterdumps/cluster-data
$ ls /root/clusterdumps
@sosreports @gettune @cluster-data node1-script_recreation.txt node1-cluster-debug.tar.bz2 node2-cluster-debug.tar.bz2
node1-cluster_data-20100215.1315.1266257733.txt node2--cluster_data-20100215.1316.1266257801.txt
$ ls /root/clusterdumps/sosreport
sosreport-node1-592066-cd351f.tar.bz2 sosreport-node2-612066-cd351c.tar.bz2
$ ls /root/clusterdumps/gettune
node1-gettune-20100215.1315.1266257733.txt node2-gettune-20100215.1316.1266257801.txt
Once the directory structure is setup correctly like the example above, tarball the directory for upload and analysis.
$ cd /root
$ tar -jcvf clusterdumps.tar.bz2 clusterdump