Troubleshooting OpenShift Container Platform 3.x: Integrated Container Registry

Environment

• OpenShift Container Platform 3

Issue

• Having issue with deploying the OpenShift v3 Internal Container Image Registry
• What information should I look for when troubleshooting my OpenShift Container Registry

Diagnostic Steps

• Health Checking
• Testing
• Information for Support
• Other Knowledge Content

Health Checking

• A basic health check to verify that the internal registry is running and responding to its service address is to "ping" its /healthz path. Under normal circumstances this should return a HTTP 200 response. This is to be run on a master/node in the cluster.

# RegistryAddr=$(oc get svc docker-registry -n default -o 'jsonpath={.spec.clusterIP}:{.spec.ports[0].port}')

# curl -vk $RegistryAddr/healthz
OR
# curl -vk https://$RegistryAddr/healthz

• The registry pod IP should be checked as well if the service fails.

# oc get pods -n default -l docker-registry=default -o wide 

# curl -vk https://$POD_IP:5000/healthz

Testing the registry

• Test out your registry by pushing an image to it. Refer to our documentation for steps. How to login in and push to the Integrated Container Registry

• If attaching a volume to the registry make sure that the registry can write to the volume.

# oc project default 
# oc rsh `oc get pods -o name -l docker-registry`

$ touch /registry/test
$ ls -la /registry/ 
$ rm /registry/test
$ exit

• If the registry is insecure make sure you have edited the /etc/sysconfig/docker file and added --insecure-registry 172.30.0.0/16 to the OPTIONS parameter on the node. This is part of the host preparation for an OpenShift node.

• If a http proxy is configured and used for the cluster, ensure that the no_proxy variable is configured for all docker and Openshift services as mentioned below. Docker does not accept CIDR notation for no_proxy, so the full service IP address is needed when configuring no_proxy. See our documentation for more information on setting this up.
  Files:
  /etc/hosts
  /etc/sysconfig/atomic-openshift-master-controllers
  /etc/sysconfig/atomic-openshift-master-api
  /etc/ansible/facts.d/openshift.fact
  /etc/ansible/hosts
  /etc/origin/master/master-config.yaml

Information to gather when opening up a support ticket

If opening up at This content is not included.support ticket please provide or be prepared to provide the following:

• Login in as system:admin on the master host

  # oc login -u system:admin  --config=/etc/origin/master/admin.kubeconfig 

• Collect the output of the following

# oc status -n default
# oc get nodes,hostsubnet,all,events -n default
# oc logs <docker-registry-pod> -n default
# oc describe all -l docker-registry -n default
# oc get dc <registry> -o yaml -n default 
# oadm diagnostics 

• Gathering a sosreport from the node where the registry pod lives may also be needed. The Node can be found by running oc describe on the docker-registry pod.

# sosreport -k docker.all=on

Links to other useful Registry Knowledge Solutions

Registry

• How to retain or manually set the service IP address for docker-registry in Openshift 3
• Server error pushing to OpenShift internal registry
• How to configure a persistent volume for the Integrated Docker Registry in OpenShift 3
• OpenShift builds fail trying to push image using a wrong IP address for the registry

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.