Deletion of partition events in JBoss ON results in OutOfMemoryError when there are more then a million of partition events in the database

Solution Unverified - Updated

Environment

  • Red Hat JBoss Operations Network (ON) 3.3.4
  • JBoss ON has stored over a million partition events;

Issue

  • Operation Timed out and Crash when deleting all partition events
  • When trying to delete all partition events from JON (over 1 million events), the operation times out and JON subsequently crashes after spiking the CPU.
  • Deletion of partition events in JBoss ON results in OutOfMemoryError when there are more then a million of partition events in the database
  • Attempt to delete huge number of partition events (more then 1 million events in the database) causes:
10:29:07,685 ERROR [org.jboss.as.controller.management-operation] (http-/0.0.0.0:7080-72) JBAS014612: Operation ("read-attribute") failed - address: ([("deployment" => "rhq.ear")]): java.lang.OutOfMemoryError: GC overhead limit exceeded
	at java.util.LinkedHashMap.newNode(LinkedHashMap.java:256) [rt.jar:1.8.0_66]
	at java.util.HashMap.putVal(HashMap.java:630) [rt.jar:1.8.0_66]
	at java.util.HashMap.put(HashMap.java:611) [rt.jar:1.8.0_66]
	at java.util.HashSet.add(HashSet.java:219) [rt.jar:1.8.0_66]
	at org.jboss.as.controller.registry.AbstractModelResource.getChildren(AbstractModelResource.java:115) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.jboss.as.controller.OperationContextImpl.readResourceFromRoot(OperationContextImpl.java:680) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.jboss.as.controller.OperationContextImpl.readResource(OperationContextImpl.java:628) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.jboss.as.controller.operations.global.ReadAttributeHandler.doExecuteInternal(ReadAttributeHandler.java:153) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.jboss.as.controller.operations.global.ReadAttributeHandler.doExecute(ReadAttributeHandler.java:96) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.jboss.as.controller.operations.global.GlobalOperationHandlers$AbstractMultiTargetHandler.execute(GlobalOperationHandlers.java:249) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:702) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:537) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:188) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.jboss.as.controller.ModelControllerImpl$3.execute(ModelControllerImpl.java:563) [jboss-as-controller-7.5.3.Final-redhat-2.jar:7.5.3.Final-redhat-2]
	at org.rhq.common.jbossas.client.controller.JBossASClient.execute(JBossASClient.java:283) [rhq-jboss-as-dmr-client-4.12.0.JON330GA.jar:4.12.0.JON330GA]
	at org.rhq.common.jbossas.client.controller.DeploymentJBossASClient.getDeploymentPath(DeploymentJBossASClient.java:99) [rhq-jboss-as-dmr-client-4.12.0.JON330GA.jar:4.12.0.JON330GA]
	at org.rhq.enterprise.server.core.CoreServer.getEarDeploymentDir(CoreServer.java:187) [rhq-server.jar:4.12.0.JON330GA-redhat-4]
	at sun.reflect.GeneratedMethodAccessor156.invoke(Unknown Source) [:1.8.0_66]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_66]
	at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_66]
	at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) [rt.jar:1.8.0_66]
	at sun.reflect.GeneratedMethodAccessor120.invoke(Unknown Source) [:1.8.0_66]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.8.0_66]
	at java.lang.reflect.Method.invoke(Method.java:497) [rt.jar:1.8.0_66]
	at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) [rt.jar:1.8.0_66]
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) [rt.jar:1.8.0_66]
	at com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) [rt.jar:1.8.0_66]
	at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) [rt.jar:1.8.0_66]

Resolution

To workaround this issue, purge partition events in batches. This can be done by selecting groups of rows -- clicking on the first row and shift clicking a few pages down to select a few thousand at a time -- and then clicking Remove Selected.

It may also be necessary to increase the JBoss ON server JVM's maximum heap memory setting.

Root Cause

When partition events deletion is requested, the keys for the entries that will be deleted are retained in memory. If there is a million or more of partition events, the JBoss ON server JVM may temporarily require several gigabytes of heap to perform the purge.

This issue has been captured in This content is not included.Red Hat Bugzilla 1297702 and will be reviewed in a future release.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.