OpenShift Cluster Metrics needs the newest version available that matches the cluster version

Environment

• OpenShift Enterprise 3.1-3.2
• OpenShift Container Platform 3.3+

Issue

• I am having issues with metrics and am running older metrics images

Resolution

• Scale the metrics component replicationcontrollers (RC) down to zero

oc scale rc heapster --replicas=0
oc scale rc hawkular-metrics --replicas=0
for cassrc in $(oc get rc -n openshift-infra -o jsonpath='{range .items[*].metadata}{.name}{"\n"}{end}' | grep cassandra); do echo -e "\n\nscaling down $cassrc"; oc scale rc $cassrc --replicas=0; done

• Make sure the image version specified in the RC loosely matches the version of OpenShift you are running (3.2.1.1-3.2.1.17 all should identify as 3.2.1 in the RCs), check using the below command and then modify them if they do not match:

oc get rc -o jsonpath='{range .items[*]}{"RC Name: "}{.metadata.name}{"\n  Image: "}{.spec.template.spec.containers[*].image}{"\n"}{end}'

• Then scale the components back up to the previous numbers (most commonly one each), one at a time to let the various components finish starting up before moving on to the next one.

for cassrc in $(oc get rc -n openshift-infra -o jsonpath='{range .items[*].metadata}{.name}{"\n"}{end}' | grep cassandra); do echo -e "\n\nscaling down $cassrc"; oc scale rc $cassrc --replicas=1; done
# Wait
oc scale rc hawkular-metrics --replicas=1
# Wait
oc scale rc heapster --replicas=1

Diagnostic Steps

• To Determine the exact version of the This page is not included, but the link has been rewritten to point to the nearest parent document.Cluster Metrics that is running in your cluster, more specifically than the vague image version listed in the RC, you will need to look at the output of docker images on the nodes running the metrics components, and compare the "IMAGE ID" there to the "Docker Image ID" listed under the "Tags" tab for that image on the This content is not included.Container Catalog.
• A quick oneline command that will look at the pods running in the project you are currently in, determine the nodes they are all running on, ssh to them, and collect the output from docker images:

for node in $(oc get pods -o jsonpath='{range .items[*].spec}{.nodeName}{" "}{end}'); do echo $node; ssh $node "docker images";done

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.