How to get profile data from openshift process (until v3.4)?

Solution Unverified - Updated

Environment

  • OpenShift Enterprise
    • 3.4 or earlier

NOTE: For v3.5 or later, you can refer to How to get profile data from openshift process?

Issue

  • How to get profile data from openshift process like memory, CPU load and blocking?
  • Setting OPENSHIFT_PROFILE has any concern, like performance or security?

Resolution

Table of Contents

Overview

  • You can set OPENSHIFT_PROFILE=<PROFILE> in your /etc/sysconfig/atomic-openshift-{node, master} and generate profile data.
  • OpenShift supports cpu, mem and web forOPENSHIFT_PROFILE . For example, you can set them as in /etc/sysconfig/atomic-openshift-node:
OPENSHIFT_PROFILE=web
  • We recommend that OPENSHIFT_PROFILE=web instead of cpu and memory, since web can generate mem, cpu and block all.

Usage

How to setup?

-Note: some tools may require golang (which is offered out of rhel-7-server-optional-rpms repositories)

  • Add OPENSHIFT_PROFILE to your /etc/sysconfig/atomic-openshift-{node,master} or /etc/sysconfig/atomic-openshift-master-{api,controllers} for multi-master clusters.

e.g. web

OPENSHIFT_PROFILE=web

# systemctl restart atomic-openshift-{node,master}

How to get profile?

web profile

  • You can access http://127.0.0.1:6060/debug/pprof and see the profile:
profiles:
0   block
1358    goroutine
1178    heap
13  threadcreate
  • You can run curl or go tool pprof to get above profiles.
block (stack trace of blocking event)
curl http://127.0.0.1:6060/debug/pprof/block?debug=1

NOTE: OpenShift v3.1.16 has Content from github.com is not included.this bug and you can't use block profile via web console.

goroutine
  • similar to java thread dump.
  • We recommend full goroutine stack dump rather than goroutine.
curl http://127.0.0.1:6060/debug/pprof/goroutine?debug=1
heap
  • heap dump & memory usage.
  • We recommend you to use go tool pprof 
curl http://127.0.0.1:6060/debug/pprof/heap?debug=1

or

go tool pprof --text http://localhost:6060/debug/pprof/heap
threadcreate
curl http://127.0.0.1:6060/debug/pprof/threadcreate?debug=1
full goroutine stack dump
  • similar to Java's threaddump
curl http://127.0.0.1:6060/debug/pprof/goroutine?debug=2
Trace 30s CPU
go tool pprof $(which openshift) http://127.0.0.1:6060/debug/pprof/profile
Heap dump & total allocations
go tool pprof --alloc_space $(which openshift) http://127.0.0.1:6060/debug/pprof/heap

NOTE: To install golang, please refer How to install golang on RHEL 7?

cpu, memory and block profile

  • You will get *.pprof under /var/lib/origin/
/var/lib/origin/{cpu, memory, block}.pprof

How to analyze?

FAQ

  • Q. Are there any performance degradation with running openshift process with setting OPENSHIFT_PROFILE=?

  • A. Negligible from what we've seen, the data is continuously published and the pprof tooling will scrape the local web-endpoint. (Of course, it will effect when you get stack dump by go tool pprof.)

  • Q. Do we have any security concern?

  • A. Other than the possibility of someone getting local network access, we wouldn't expect an impact to production.

  • Q. Is it possible to enable it for integrated docker registry?

  • A. Yes, it is. You can refer to How to enable and collect pprof for OpenShift integrated registry?

Reference

SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.