Tracing RADOS Block Device (RBD) Workloads with the RBD Replay Feature

Tracing with the RBD Replay Feature

Tracing is a method for analyzing the system behavior and interactions between components on the system. Data captured by tracing is typically used for performance tuning and debugging purposes.

The RBD Replay toolkit provides utilities for capturing and replaying RADOS Block Device (RBD) workloads. It uses the Linux Trace Toolkit: next generation (LLTng) tracing framework.

RBD Replay does not record any data read from or written to the RBD disks, it only captures the metadata associated with the disk access patterns. One of the important benefits of RBD Replay is that it allows users to submit this metadata back to Red Hat engineers or upstream Ceph community members, who then use the metadata for performance improvements, regression testing, or benchmarking.

For more information about LLTng and tracing in general, see the Content from lttng.org is not included.LLTng official documentation.

Installation

On a client, install the lttng-tools package. Execute the following command:

$ sudo yum install lttng-tools

The ceph-common package provides the RBD Replay utilities. Ensure that ceph-common is installed on the client:

$ rpm -q ceph-common

Note that the librdb package on the client must be Ceph version 1.3.1 or later.

Using RBD Replay

To use RBD Replay, first capture a trace using the lltng utility. Then process the captured trace with the rbd-replay-prep utility in order to prepare the trace to be replayed by the rbd-replay utility. Finally, replay the trace with rbd-replay.

Applications providing mandatory access control mechanism, for example SELinux or AppArmor, must be either disabled or set to permissive mode for the QEMU process in order for rbd-replay to work.

The steps below show the detailed procedure how to capture and replay an RBD workload:

1. Create a directory for capturing traces, for example ~/traces/:

   $ mkdir -p ~/traces

2. Create a tracing session. The -o command-line option specifies an output path for traces, in this case the ~/traces/ directory.

   $ lttng create -o traces librbd

3. Enable a tracing event. Use the -u command-line option to apply the event to the user-space tracer.

   $ lttng enable-event -u 'librbd:*'

4. Make sure to capture the pthread_id context. Use the -u command-line option again to apply the event to the user-space tracer. Also, specify the type context, in this case pthread_id by using the -t command-line option:

   $ lttng add-context -u -t pthread_id

5. Start the tracing session:

   $ lttng start

6. Perform the steps to run an RBD workload you want to trace.

7. Stop the tracing session:

   $ lttng stop

8. Process the trace with rbd-replay-prep:

   $ rbd-replay-prep ~/traces/ust/uid/*/* replay.bin

9. Replay the trace with rbd-replay. IMPORTANT: rbd-replay destroys data by default. To keep the captured data, use the --read-only option:

   $ rbd-replay --read-only replay.bin 

The replayed workload does not have to be against the same RBD image or even the same cluster as the captured workload. To account for differences, use the --pool and --map-image options of rdb-replay.

For more information about rbd-replay-prep and rbd-replay see the rbd-replay-prep(8) and rbd-replay(8) manual pages.