After upgrade from JBoss ON 3.2.3 to 3.3.0, some rhq column families are unavailable and compaction operations fail

Environment

• Red Hat JBoss Operations Network (JON) - upgrade done from 3.2.3 to 3.3.0 version;
• In 3.2.3 version - scheduled Force Major Column Family Compaction operation for folders: metrics_index, one_hour_metrics, raw_metrics, schema_version, six_hour_metrics and twenty_four_hour_metrics.
• Above operation is scheduled to be executed now and repeated indefinitely.

Issue

• After upgrade from JBoss ON 3.2.3 to JBoss ON 3.3.0, some column families are reported as unavailable: one_hour_metrics, six_hour_metrics, twenty_four_hour_metrics and metrics_index.
• On disk the corresponding directories are not accessed any more and previously scheduled manual compaction task fails.
• Manual compaction operation fails after JBoss ON upgrade from 3.2.3 to 3.3.0
• Manual compaction operation fails with the exception:

org.mc4j.ems.connection.EmsInvocationException: Exception on invocation of [forceTableCompaction]javax.management.RuntimeMBeanException: java.lang.IllegalArgumentException: Unknown table/cf pair (rhq.metrics_index)
	at org.mc4j.ems.impl.jmx.connection.bean.operation.DOperation.invoke(DOperation.java:133)
	at org.rhq.plugins.cassandra.util.KeyspaceService.compact(KeyspaceService.java:101)
	at org.rhq.plugins.cassandra.KeyspaceComponent.compactKeyspace(KeyspaceComponent.java:168)
	at org.rhq.plugins.cassandra.ColumnFamilyComponent.invokeOperation(ColumnFamilyComponent.java:70)
...
Caused by: javax.management.RuntimeMBeanException: java.lang.IllegalArgumentException: Unknown table/cf pair (rhq.metrics_index)
...
Caused by: java.lang.IllegalArgumentException: Unknown table/cf pair (rhq.metrics_index)
	at org.apache.cassandra.db.Table.getColumnFamilyStore(Table.java:168)
	at org.apache.cassandra.service.StorageService.getValidColumnFamilies(StorageService.java:2407)
	at org.apache.cassandra.service.StorageService.forceTableCompaction(StorageService.java:2246)
...

Resolution

This issue has been logged in the This content is not included.Red Hat Bugzilla 1213812 and it will be fixed in a future releases of JBoss ON.

In the meantime, in order to work around this, remove from the inventory one_hour_metrics, six_hour_metrics, twenty_four_hour metrics and metrics_index from ColumnFamilies and create a new compaction operation that will take in account aggregate_metrics and metrics_idx folders as well.

Root Cause

The structure of the rhq-data folder has been changed in JBoss ON 3.3.0 and instead to have three different aggregate tables: one_hour_metrics, six_hour_metrics and twenty_four_hour metrics there is a new aggregate_metrics table/folder under rhq-data/data/rhq. This change has been logged in the This content is not included.Red Hat Bugzilla 1135602.

In the same way, metrics_index folder has been replaced with metrics_idx and the change has been logged in the This content is not included.Red Hat Bugzilla 1135630.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.