Kernel tracing using eBPF

Overview

Red Hat Enterprise Linux 7.6 introduces the Extended Berkeley Packet Filter tool (eBPF) as a Technology Preview. eBPF is an in-kernel virtual machine that allows a code execution in the kernel space, in the restricted sandbox environment with access to a limited set of functions. The virtual machine executes a special eBPF assembly-like code. The code is then loaded to the kernel (it needs to pass the safety verification) and translated to the native machine code with just-in-time compilation.

eBPF can be used to access the kernel private data, therefore it is available only to a user with the CAP_SYS_ADMIN capability. eBPF can be used through a new system call bpf(2). See the bpf(2) man page for more information.

All eBPF programs with a specified type have to be loaded using the BPF_PROG_LOAD command argument. The following program types are enabled in Red Hat Enterprise Linux 7:

BPF_PROG_TYPE_KPROBE
BPF_PROG_TYPE_TRACEPOINT
BPF_PROG_TYPE_PERF_EVENT

Writing eBPF programs

Although the eBPF programs can be written in the eBPF assembly-like language, it is recommended to use the Clang and LLVM Toolset, which allows compilation of code written in the C programming language to the eBPF assembly code. Clang and LLVM Toolset is available as a Technology Preview. For more information please refer to This content is not included.Installing Clang and LLVM Toolset Chapter in Red Hat Developer Tools Product Documentation.

The eBPF programs can access internal kernel data, which are not stable interface. For this reason, all eBPF programs need to be written and compiled against the kernel on which they will run. If a kernel is updated, all eBPF programs have to be recompiled and adjusted for the new kernel as well.

BPF Compiler Collection

Red Hat Enterprise Linux 7 provides BPF Compiler Collection (bcc) which simplifies creation of new tools using eBPF. bcc also provides Python binding to ease creation of eBPF front-end programs. Red Hat Enterprise Linux 7 only supports BCC and pcp-pmda-bcc for the 64-bit AMD and Intel CPU architectures.

bcc-tools package provides many pre-created programs built using bcc which can also serve as examples for creation additional programs. These tools can be installed from the base Red Hat Enterprise Linux 7 channel using yum(1):

yum install bcc-tools

After the package is installed the tools are delivered under /usr/share/bcc/tools.

Other eBPF related tools

Red Hat Enterprise Linux 7 also contains the bpftool package that is used to list and show the eBPF programs and maps loaded in the running kernel.

SystemTap 3.3 includes support for the SystemTap’s new BPF backend (stapbpf). To use the new backend instead of the standard one, based on kernel modules, run stap with --runtime=bpf option:

# stap ---runtime=bpf sample.stp

sosreport(1) now contains a plug-in which uses bpftool to list and disassemble all eBPF programs loaded on the machine where the sos packaged is installed.