Check OpenShift Cluster Metrics Cassandra Health

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform 3.3+
  • Red Hat OpenShift Enterprise 3.1-3.2

Issue

  • I am having issues with my cluster metrics
  • It looks like Cassandra might be causing issues for my metrics

Resolution

  • The This content is not included.metrics-cassandra image provides a built-in method to check its current health. To check this health, you will need to remote into the pod and run nodetool status.

    • The main section of this to look at for a quick health check is the first section of information on last line in the output
    • This is going to be a two character string that is either U or D and then N, L, J, or M
    • These represent Up or Down and Normal, Leaving, Joining, or Moving

  • Assuming you have at least one hawkular-cassandra pod running in your openshift-infra project, the following command should collect the output of nodetool status from each hawkular-cassandra pod:

for casspod in $(oc get pods -n openshift-infra -o jsonpath='{range .items[*].metadata}{.name}{"\n"}{end}' | grep cassandra); do echo -e "\n\n$casspod"; oc -n openshift-infra exec $casspod -- nodetool status; done
  • Example output:


hawkular-cassandra-1-<HASH>
Picked up JAVA_TOOL_OPTIONS: -Duser.home=/home/jboss -Duser.name=jboss
Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address     Load       Tokens       Owns (effective)  Host ID                               Rack
UN  <IP>        43.44 MB   256          100.0%            <HASH>                                rack1

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.