How to gather a performance profile from the kubelet on OpenShift 4

Environment

• Red Hat OpenShif Container Platform (RHOCP) 4

Issue

• how to get a golang pprof from the kubelet
• how to gather a performance profile from a node

Resolution

• The kubelet exposes profiling data in the format expected by the golang pprof tool via an HTTP endpoint. This endpoint is enabled by default, and is available over its default localhost address in OpenShift. The endpoint can be used to collect CPU and memory usage profiles, which can be later on analyzed manually.

# find out the node whose kubelet you would like to profile
oc get nodes

# get the profile via must gather, the profile collection lasts by default 30s and will also gather a cri-o performance profile
oc adm must-gather -- gather_profiling_node <node_name>

• It is also important to gather a second profile from the same machine at a later time. Typically 5 minutes later is adequate for the second profile. This second profile aids in debugging memory leaks and other performance related issues over time

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.