How to collect volume profile info on client node in RHGS ?
Environment
- Red Hat Enterprise Linux 7.4
- Red Hat Gluster Storage 3.x
Issue
- How to determine the number of read and write calls made from gluster volume mount path ?
- How to determine the amount of data being read and written on gluster volume mount path ?
- How can we check the FOP latency on client node ?
Resolution
The volume profile information can be collected for volume mounted over FUSE protocol. To collect client side volume profile, set xattr trusted.io-stats-dump on volume mount path.
# setfattr -n trusted.io-stats-dump -v <client-profile-log-file-path> /mnt/<gluster-vol-mount-path>
Use attached clientprofcollect.sh script to collect client side volume profile every 5 minutes (please change casenum, timercnt and sleep count accordingly in script for longer runs).
# sh clientprofcollect.sh /mnt/ecvol/
Created tmp dir to collect logs : /tmp/glusterc3-node2_clientprofile_data
Run#1
Collecting client side volume profile for /mnt/ecvol/ in /tmp/glusterc3-node2_clientprofile_data/012121212_glusterc3-node2_client_profile_ecvol.2018_06_26-09_02_23.log
Run#2
Collecting client side volume profile for /mnt/ecvol/ in /tmp/glusterc3-node2_clientprofile_data/012121212_glusterc3-node2_client_profile_ecvol.2018_06_26-09_07_24.log
Run#3
Collecting client side volume profile for /mnt/ecvol/ in /tmp/glusterc3-node2_clientprofile_data/012121212_glusterc3-node2_client_profile_ecvol.2018_06_26-09_12_25.log
Run#4
Collecting client side volume profile for /mnt/ecvol/ in /tmp/glusterc3-node2_clientprofile_data/012121212_glusterc3-node2_client_profile_ecvol.2018_06_26-09_17_23.log
Run#5
Collecting client side volume profile for /mnt/ecvol/ in /tmp/glusterc3-node2_clientprofile_data/012121212_glusterc3-node2_client_profile_ecvol.2018_06_26-09_23_24.log
Run#6
Collecting client side volume profile for /mnt/ecvol/ in /tmp/glusterc3-node2_clientprofile_data/012121212_glusterc3-node2_client_profile_ecvol.2018_06_26-09_58_24.log
-------------------------------------------------------
Please compress dir /tmp/glusterc3-node2_clientprofile_data and attach to case 012121212
-------------------------------------------------------
Root Cause
The client side volume profile data also helps to find details on I/O workload from client mount path and also latency seen by various FOPs which can help to determine exact cause of the performance issue if any.
Diagnostic Steps
The client-side volume profile has below details captured :
=== Cumulative stats ===
Duration : 5141 secs
BytesRead : 8401084
BytesWritten : 8559359
Block Size : 2B+ 256B+ 512B+
Read Count : 0 0 8
Write Count : 1 9 10
Block Size : 1024B+ 2048B+ 4096B+
Read Count : 0 4 4
Write Count : 1 8 2081
Block Size : 16384B+ 32768B+ 65536B+
Read Count : 3 3 1
Write Count : 0 0 0
Block Size : 131072B+
Read Count : 62
Write Count : 0
Fop Call Count Avg-Latency Min-Latency Max-Latency
--- ---------- ----------- ----------- -----------
STAT 126 87.61 us 2.00 us 1449.00 us
MKDIR 26 12395.42 us 9979.00 us 24632.00 us
TRUNCATE 28 3752.93 us 3323.00 us 5740.00 us
OPEN 43 63.49 us 8.00 us 1098.00 us
READ 85 1139.98 us 15.00 us 3204.00 us
WRITE 2110 27.15 us 10.00 us 1331.00 us
STATFS 7 1012.86 us 937.00 us 1123.00 us
FLUSH 56 2924.98 us 1028.00 us 7701.00 us
GETXATTR 2 1172.50 us 1166.00 us 1179.00 us
REMOVEXATTR 28 3638.46 us 3227.00 us 6121.00 us
CREATE 28 2808.14 us 2230.00 us 4868.00 us
FSTAT 90 407.61 us 2.00 us 1129.00 us
LOOKUP 670926 2303.73 us 1.00 us 53182.00 us
RELEASE 56 0 us 0 us 0 us
------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
Current open fd's: 4 Max open fd's: 18 time 2018-05-23 12:45:07.688516
==========Open File Stats========
COUNT: FILE NAME
2 /app/system/regular/temp/1112222.zip
2 /app/system/regular/temp/33334444.zip
2 /app/system/regular/temp/44445555.zip
2 /app/system/regular/temp/55556666.zip
==========Read File Stats========
COUNT: FILE NAME
69 /app/system/regular/temp/1112222.zip
==========Write File Stats========
COUNT: FILE NAME
2067 /app/system/regular/temp/1112222.zip
13 /app/system/regular/temp/33334444.zip
2 /app/system/regular/temp/44445555.zip
2 /app/system/regular/temp/55556666.zip
==========Directory open stats========
COUNT: DIRECTORY NAME
========Directory readdirp Stats=======
COUNT: DIRECTORY NAME
========Read Throughput File Stats=====
TIMESTAMP THROUGHPUT(KBPS) FILE NAME
2018-05-23 12:33:59.000000 2912.00 /app/system/regular/temp/1112222.zip
2018-05-23 12:31:04.000000 2184.00 /app/system/regular/temp/33334444.zip
2018-05-23 12:33:16.000000 14.00 /app/system/regular/temp/44445555.zip
2018-05-23 12:33:47.000000 11.00 /app/system/regular/temp/55556666.zip
======Write Throughput File Stats======
TIMESTAMP THROUGHPUT(KBPS) FILE NAME
2018-05-23 12:33:57.000000 409.00 /app/system/regular/temp/1112222.zip
2018-05-23 12:33:16.000000 288.00 /app/system/regular/temp/33334444.zip
2018-05-23 12:31:04.000000 273.00 /app/system/regular/temp/44445555.zip
2018-05-23 12:33:36.000000 109.00 /app/system/regular/temp/55556666.zip
=== Interval 14 stats ===
Duration : 300 secs
BytesRead : 0
BytesWritten : 2900
Block Size : 256B+
Read Count : 0
Write Count : 8
Fop Call Count Avg-Latency Min-Latency Max-Latency
--- ---------- ----------- ----------- -----------
STAT 37 5.08 us 3.00 us 14.00 us
MKDIR 1 10792.00 us 10792.00 us 10792.00 us
TRUNCATE 8 3586.50 us 3346.00 us 3859.00 us
OPEN 8 15.75 us 10.00 us 26.00 us
WRITE 8 1194.12 us 1133.00 us 1331.00 us
FLUSH 16 2881.62 us 1060.00 us 4850.00 us
REMOVEXATTR 8 3504.25 us 3227.00 us 3675.00 us
CREATE 8 2640.75 us 2358.00 us 3323.00 us
LOOKUP 104301 2272.73 us 1.00 us 5478.00 us
RELEASE 16 0 us 0 us 0 us
------ ----- ----- ----- ----- ----- ----- ----- ----- ----- ----- -----
SBR
Product(s)
Components
Category
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.