Hawkular-metrics shows empty charts after a few minutes

Environment

• OpenShift Enterprise 3.1-3.2
• OpenShift Container Platform 3.3+

Issue

• After deploying hawkular-metrics, the charts works fine, showing information in "metrics" tab correctly. But after three or four minutes, no more information is displayed.

Resolution

To resolve this, you will need to increase the rate that Heapster collects information to a value higher than 5 seconds. In versions leading up to, and including, 3.2.0 this value is the stats_resolution, but in 3.2.1 and on, the value was renamed to metrics_resolution:

 # oc edit rc heapster
...
        - --tls_key=/secrets/heapster.key
        - --tls_client_ca=/secrets/heapster.client-ca
        - --allowed_users=%allowed_users%
        - --stats_resolution=30s
        ####- --metrics_resolution=30s
...

Root Cause

• The problem is that heapster collects metrics from kubelet/cadvisor every 5 seconds by default. After each collection, heapster scrapes the data. This process can take more than 5 seconds when the system is under load, resulting in empty stats.
• As of 3.5+, the kubelet/cadvisor collects metrics every 10 seconds by default, you should specify the metrics_resolution more than 10s.
  • This content is not included.Metrics Resolution of Heapster Image Should be 30s to Match cAdvisor
  • Content from kubernetes.io is not included.kubelet Options

--housekeeping-interval duration
  Interval between container housekeepings (default 10s)

Diagnostic Steps

• Verify timestamps of "starting to scrape data from sources start" and "completed scraping data from sources" entries in heapster log

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.