What is SystemTap and how to use it?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (all versions)

Issue

  • What is SystemTap and how to use it?

Resolution

SystemTap is an innovative tool which allows for simplified information gathering on the running Linux kernel. The purpose of using SystemTap is to obtain information on either performance issues or functional problems (bugs). By using SystemTap, developers and system administrators can debug problems and gather profiling and performance data without having to create and install instrumented kernels or other packages.

In essence, SystemTap provides the infrastructure (a command line interface and scripting language) needed to gather information. The actual job that SystemTap performs relies on user-developed scripts tailored to a specific purpose. Currently, there are a number of example SystemTap scripts pre-made for general use.

The operation of SystemTap is quite simple. The stap command takes as an argument the name of a SystemTap file (called a probe). There may be additional command line arguments passed depending on the probe. SystemTap translates the probe into C and compiles the C source as a kernel module. It then inserts the resulting kernel module into the running kernel to perform the probe functions defined in the script. The output is printed to the console or can be redirected to file.

Requirements

Because SystemTap compiles scripts from C code and launches probes for kernel instrumentation, it requires several packages in order to function. See the Installing SystemTap chapter of the SystemTap Beginners Guide for detailed installation instructions.

The easiest way to satisfy the requirements is to simply subscribe the system to the relevant debuginfo channels in RHSM, then run the following commands which should set up the environment for SystemTap:

# yum install systemtap
# stap-prep

Note: The message Debuginfo automatic downloading is not configured via $DEBUGINFOD_URLS is not an error! This is telling you that debugging symbols won't be downloaded from an external URL, but that doesn't matter because you have the debuginfo package correctly installed which provides the debugging symbols.

To set the environment up manually, in addition to the systemtap package, the following packages must also be installed:

  • kernel-devel for the running kernel
  • kernel-debuginfo for the running kernel
  • kernel-debuginfo-common for the running kernel
  • gcc
  • systemtap

For example, for a 2.6.32-71.18.2.el6.x86_64 kernel you'll need the following kernel packages along with the latest gcc and systemtap packages.

kernel-debuginfo-2.6.32-71.18.2.el6.x86_64
kernel-debuginfo-common-x86_64-2.6.32-71.18.2.el6.x86_64
kernel-devel-2.6.32-71.18.2.el6.x86_64

Installing debuginfo can be done by following the steps in the below article.

The gcc and kernel-devel packages are available on Red Hat Network and can be installed using yum (RHEL 5 onwards).

Once packages are installed, confirm everything is in place once by running the following command :

# rpm -qa | egrep -e kernel-'(devel|debug)' -e systemtap
Legacy

On RHEL 4, use up2date. For Red Hat Enterprise Linux (RHEL) 4, the kernel-debuginfo package is not available via up2date and must be installed from the ISO image available on Red Hat Network or from This content is not included.This content is not included.https://ftp.redhat.com/pub/redhat/linux/updates/enterprise/4AS/en/os/Debuginfo/ (substitute 4AS in the URL with 4WS, 4ES, or 4Desktop, depending on the variant of RHEL you have installed. Note: Red Hat Enterprise Linux 4 does not have kernel-debuginfo-common package.

For RHEL 5, you can download the kernel-debuginfo and kernel-debuginfo-common packages from This content is not included.This content is not included.https://ftp.redhat.com/pub/redhat/linux/enterprise/5Server/en/os/.

Using Systemtap

You can verify the SystemTap environment with the stap-report command.

An example script is shown below:

#! /usr/bin/env stap

# Using statistics and maps to examine kernel memory allocations

global kmalloc

probe kernel.function("__kmalloc") {
        kmalloc[execname()] <<< $size
}

# Exit after 10 seconds
probe timer.ms(10000) { exit () }

probe end {
        foreach ([name] in kmalloc) {
                printf("Allocations for %s\\n", name)
                printf("Count:   %d allocations\\n", @count(kmalloc[name]))
                printf("Sum:     %d Kbytes\\n", @sum(kmalloc[name])/1000)
                printf("Average: %d bytes\\n", @avg(kmalloc[name]))
                printf("Min:     %d bytes\\n", @min(kmalloc[name]))
                printf("Max:     %d bytes\\n", @max(kmalloc[name]))
                print("\\nAllocations by size in bytes\\n")
                print(@hist_log(kmalloc[name]))
                printf("-------------------------------------------------------\\n\\n");
        }
}

This script, drawn from the SystemTap project wiki, can be used to print information kernel memory allocations of the system. The script can be invoked as follows:

stap kmalloc2.stp

For issues during the compile or loading of the module within the stap command, append the parameter -vv to show more verbose output.

SystemTap will then translate the probe into C, compile the C program, and insert the probe into the running kernel. Truncated output is below:

-------------------------------------------------------

Allocations for httpd
Count:   10 allocations
Sum:     0 Kbytes
Average: 0 bytes
Min:     0 bytes
Max:     0 bytes

Allocations by size in bytes
value |-------------------------------------------------- count
    0 |@@@@@@@@@@                                         10
    1 |                                                    0
    2 |                                                    0

-------------------------------------------------------

Allocations for sendmail
Count:   2 allocations
Sum:     0 Kbytes
Average: 24 bytes
Min:     24 bytes
Max:     24 bytes

Allocations by size in bytes
value |-------------------------------------------------- count
    4 |                                                   0
    8 |                                                   0
   16 |@@                                                 2
   32 |                                                   0
   64 |                                                   0

-------------------------------------------------------

This is a simple example that just touches the surface of the capabilities offered by SystemTap. System Administrators could use this information to better understand kernel memory allocation on the running system and adjust kernel tuning parameters accordingly. Application developers can use this information as an overview of which applications are receiving more kernel memory allocations, which can be used as a starting point for deeper application profiling.

Additional Information about SystemTap

Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.