How to check which RBD images are used by which clients?

Environment

• Red Hat Ceph Storage 1.3.2

Issue

• How to check which RBD images are used by which clients ?
• What is the rbd status command ?
• If given a client.id how can I determine the IP address that the client.id represents?

Resolution

Determine what clients are using a specific RBD image

• Use the rbd status command to check the current clients which are using the given RBD image.
• For example, the below given RBD image is an Openstack instance root disk:

# rbd -p vms ls | grep 9c714d3f-9471-4cc3-aee2-f8d7745d6bba_disk
9c714d3f-9471-4cc3-aee2-f8d7745d6bba_disk

• Use the rbd status command to check to which compute node this instance belong:

# rbd status vms/9c714d3f-9471-4cc3-aee2-f8d7745d6bba_disk
Watchers:
	watcher=192.168.123.30:0/1037939 client.13539154 cookie=140387036137344

• This Openstack instance belongs to a compute node with the 192.168.123.30 IP address.
• The same command also checks the KRBD mapped RBD images.

Determine what client the client.id number represents

• Often when looking at the ceph log files it's common to see a client.xxxxx number referenced during an RBD op, for example:

2016-04-27 07:05:53.891381 7ffaddce0700  0 log [WRN] : slow request 30.710408 seconds old, received at 2016-04-27 07:05:23.180896: osd_op(client.248112341.0:3192955 rbd_data.bc77eeb4fb420c9.00000000000003a1 [stat,set-alloc-hint object_size 2097152 write_size 2097152,write 1048576~4096] 3.773b74e2 ack+ondisk+write+??? e840353) v5 currently waiting for subops from 18,439

The client.id in the above output is: client.248112341.

• To determine what IP address this references, iterate through the RBD header objects and list the watchers for each RBD. The following command can accomplish this, where -p is the pool name where the RBD images live.

# for each in `rados -p rbd ls | grep .rbd`; do echo $each: && rados -p rbd listwatchers $each && echo -e '\n'; done 
 rbdtest.rbd:
 watcher=10.13.212.167:0/2064500421 client.4667 cookie=1
 
 rbdtest2.rbd:
 watcher=10.13.212.163:0/2064500421 client.4668 cookie=2

• Then use a tool such as grep to search through the results for the client.id referenced in the log files.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.