How to enable and collect pprof for OpenShift integrated registry?

Environment

• OpenShift Container Platform
  • 3.7

Issue

• How to enable and pprof for OpenShift integrated registry?
• docker registry keeps increasing memory usage. I am suspecting memory leak. How to debug it?

Resolution

Please follow below steps

1. Set OPENSHIFT_PROFILE=web on deploymentConfig of docker-registry

(Please be careful. When you set it, docker registry pod will be re-deployed.)

  # oc set env dc/docker-registry OPENSHIFT_PROFILE=web

2. ssh into the Node host

  // Check which node docker-registry pod deployed
  # oc get pod -o wide
  # ssh 

3. Install golang package

  # yum install golang --enablerepo=rhel-7-server-optional-rpms

4. Collect pprof

  # nsenter -t `pgrep dockerregistry` -n -- go tool pprof -inuse_space --text http://localhost:6060/debug/pprof/heap
  # nsenter -t `pgrep dockerregistry` -n -- go tool pprof -inuse_objects --text http://localhost:6060/debug/pprof/heap  
  # nsenter -t `pgrep dockerregistry` -n -- go tool pprof -alloc_space --text http://localhost:6060/debug/pprof/heap

• You will see heap usage output like [1]. So, could you please provide us with the outputs?

# nsenter -t `pgrep dockerregistry` -n -- go tool pprof -inuse_space --text http://localhost:6060/debug/pprof/heap
Fetching profile from http://localhost:6060/debug/pprof/heap
Saved profile in /root/pprof/pprof.localhost:6060.inuse_objects.inuse_space.005.pb.gz
3982.71kB of 3982.71kB total (  100%)
Dropped 233 nodes (cum <= 19.91kB)
      flat  flat%   sum%        cum   cum%
  809.97kB 20.34% 20.34%   809.97kB 20.34%  github.com/openshift/origin/vendor/golang.org/x/net/http2.(*ClientConn).frameScratchBuffer
  591.75kB 14.86% 35.20%   591.75kB 14.86%  crypto/elliptic.initTable
  528.17kB 13.26% 48.46%   528.17kB 13.26%  regexp.(*bitState).reset
  515.26kB 12.94% 61.39%   515.26kB 12.94%  encoding/json.typeFields
     513kB 12.88% 74.27%      513kB 12.88%  github.com/openshift/origin/vendor/golang.org/x/net/http2/hpack.addDecoderNode
  512.28kB 12.86% 87.14%   512.28kB 12.86%  github.com/openshift/origin/vendor/github.com/go-openapi/spec.(*SchemaOrBool).UnmarshalJSON
  512.28kB 12.86%   100%   512.28kB 12.86%  reflect.mapassign
 ... (snip) ...

Root Cause

• You can enable the profile by:

# oc set env dc/docker-registry OPENSHIFT_PROFILE=web

OPENSHIFT_PROFILE_PORT and OPENSHIFT_PROFILE_HOST are also available. Please refer to the detail Content from github.com is not included.in the code

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.