Metric graph does not show data for more then last seven days

Solution Verified - Updated

Environment

  • Red Hat JBoss Operations Network (JON) 3.3

Issue

  • Metric graph does not show data for more then last seven days;
  • Metric graph does not show 3 months data;

Resolution

Restart the JBoss ON Server(s) and make sure that DataCalcJob is properly scheduled:

10:30:52,527 INFO  [org.rhq.enterprise.server.scheduler.EnhancedSchedulerImpl] (pool-6-thread-1) Scheduled cron job [org.rhq.enterprise.server.scheduler.jobs.DataCalcJob:org.rhq.enterprise.server.scheduler.jobs.DataCalcJob] to fire next at [Mon Nov 23 11:00:00 CET 2015] with the cron string [0 0 * * * ?].
10:30:52,575 INFO  [org.rhq.enterprise.server.scheduler.EnhancedSchedulerImpl] (pool-6-thread-1) Scheduled cron job [380816c0--d225257a:syncContentSource] to fire next at [Tue Dec 01 00:00:00 CET 2015] with the cron string [0 0 0 1 * ?].

After JBoss ON Server restart, confirm that data compression is correctly executed hourly - for instance:

13:00:00,016 INFO  [org.rhq.enterprise.server.scheduler.jobs.DataCalcJob] (RHQScheduler_Worker-5) Data Calc Job STARTING
13:00:00,019 INFO  [org.rhq.enterprise.server.scheduler.jobs.DataCalcJob] (RHQScheduler_Worker-5) Measurement data compression starting at Mon Nov 23 13:00:00 CET 2015
13:00:00,020 INFO  [org.rhq.server.metrics.aggregation.AggregationManager] (RHQScheduler_Worker-5) Starting metrics data aggregation
...
14:00:00,010 INFO  [org.rhq.enterprise.server.scheduler.jobs.DataCalcJob] (RHQScheduler_Worker-5) Data Calc Job STARTING
14:00:00,011 INFO  [org.rhq.enterprise.server.scheduler.jobs.DataCalcJob] (RHQScheduler_Worker-5) Measurement data compression starting at Mon Nov 23 14:00:00 CET 2015
14:00:00,011 INFO  [org.rhq.server.metrics.aggregation.AggregationManager] (RHQScheduler_Worker-5) Starting metrics data aggregation
...

Leave JBoss ON Server running for at least 10 days and then confirm that metric graph properly show data for last 8 or 9 days.

Note: The following alert could be set up to alert if DataCalcJob stops running completely:

  1. From JBoss ON UI -> navigate to JBossAS7 Standalone Servers -> EAP (x.x.x.x:6990) RHQ Server -> RHQ Server Subsystems -> Measurement Subsystem -> Monitoring -> Schedules;
  2. Scroll down the list, select the Measurement Compression Time per Minute metric and set it to be collected every 10 minutes -> click the Enable button.
  3. On that same resource Measurement Subsystem -> Alerts tab -> Definitions -> select New button and then define alert condition:

Condition Type: Measurement Absolute Value Threshold;
Metric: Measurement Compression Time per Minute;
Comparator: = (Equal of)
Metric Value: 0 ms

  1. On the Dampening tab select:

Dampening: Last N Evaluations
Occurrences: 12
Evaluations: 12

  1. Save the changes;

This Alert will be fired if metric compaction hasn't run in 2 hours.

Root Cause

In every correctly working JBoss ON, version 3.2 and later, the server.log file will contain messages like:

00:00:00,066 INFO  [org.rhq.enterprise.server.scheduler.jobs.DataCalcJob] (RHQScheduler_Worker-1) Measurement data compression starting at Tue Nov 17 00:00:00 GMT 2015
...
00:00:00,066 INFO  [org.rhq.server.metrics.aggregation.AggregationManager] (RHQScheduler_Worker-1) Starting metrics data aggregation

Above messages should be logged once every hour and this is when JON Server does data compression and data aggregation and generates one_hour_metric, six_hour_metrics and twenty_four_hour_metrics that are used to draw graphs.

When creating graphs, the algorithm is:

1. If the time period is for less then the past 7 days, raw data is used...
2. If the time period extends past 7 days ago but less then 14 days ago, then hourly aggregates are used (one_hour_metric)...
3. If that time period extends past 14 days ago but less then 30 days ago, 6 hour aggregates are used (six_hour_metrics)...
4. And finally, if more then 30 days ago but less then 365 days ago, daily aggregates are used (twenty_four_hour_metrics).

Data compression job is done by DataCalcJob cron job which is scheduled during JBoss ON Server start up and something like the following should be in the server.log file:

10:30:52,527 INFO  [org.rhq.enterprise.server.scheduler.EnhancedSchedulerImpl] (pool-6-thread-1) Scheduled cron job [org.rhq.enterprise.server.scheduler.jobs.DataCalcJob:org.rhq.enterprise.server.scheduler.jobs.DataCalcJob] to fire next at [Mon Nov 23 11:00:00 CET 2015] with the cron string [0 0 * * * ?].
10:30:52,575 INFO  [org.rhq.enterprise.server.scheduler.EnhancedSchedulerImpl] (pool-6-thread-1) Scheduled cron job [380816c0--d225257a:syncContentSource] to fire next at [Tue Dec 01 00:00:00 CET 2015] with the cron string [0 0 0 1 * ?].

If, for any reason, DataCalcJob is not scheduled properly, data compression will not be done and one_hour_metric, six_hour_metrics and twenty_four_hour_metrics will not contain data needed for metric graphs.

Diagnostic Steps

Check the server.log file generated during latest JBoss ON Server restart and make sure that DataCalcJob is properly scheduled. If there is any error/exception when scheduling this job, data compression will not be done.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.