Multiple restarts of a JON agent's plug-in container can cause OutOfMemoryError: PermGen space when JBoss EAP 6 management plug-in is installed

Solution Unverified - Updated

Environment

  • Red Hat JBoss Operations Network (ON) 3.1.2, 3.2
  • Red Hat JBoss Enterprise Application Platform (EAP) 6 management plug-in for JBoss ON provided with:
    • JBoss ON 3.2
    • EAP Plugin Pack for JBoss ON 3.1.2 Hotfix-03, Hotfix-04, Hotfix-05
    • JDG Plugin Pack for JBoss ON 3.1.2 Hotfix-03, Hotfix-04, Hotfix-05
    • JPP Plugin Pack for JBoss ON 3.1.2 Hotfix-03, Hotfix-04, Hotfix-05
  • Inventory contains one or more JBoss EAP 6 server resources
  • Agent managing JBoss EAP server resource has had any of the following operations invoked multiple times:
    • Restart Agent;
    • Restart Plugin Container;
    • Update All Plugins;

Issue

  • Agents become unresponsive and run out of PermGen space when recycled

  • Restart operation for agent exhausts its PermGen space and becomes unresponsive

  • Agent log contains the following error:

      org.rhq.core.clientapi.agent.PluginContainerException: Could not build component for Resource [Resource[id=10009, uuid=6c6dd970-753b-45bd-9fc0-fdda829248db, type={JBossAS}JBossAS Server, key=/opt/jboss/jboss-on/jon-server/jbossas/server/default, name=loleary:2099 RHQ Server, parent=myagent.example.com, version=AS 4.2.3.GA]]
      at org.rhq.core.pc.inventory.InventoryManager.prepareResourceForActivation(InventoryManager.java:1644)
      at org.rhq.core.pc.inventory.InventoryManager.activateAndUpgradeResourceRecursively(InventoryManager.java:3075)
      ...
  Caused by: java.lang.OutOfMemoryError: PermGen space
      at java.lang.ClassLoader.defineClass1(Native Method)
      at java.lang.ClassLoader.defineClass(ClassLoader.java:634)
      at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
      at java.net.URLClassLoader.defineClass(URLClassLoader.java:277)
      at java.net.URLClassLoader.access$000(URLClassLoader.java:73)
      at java.net.URLClassLoader$1.run(URLClassLoader.java:212)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
      at org.rhq.core.pc.plugin.PluginClassLoader.loadClass(PluginClassLoader.java:63)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:264)
      at org.rhq.core.pc.plugin.PluginComponentFactory.instantiateClass(PluginComponentFactory.java:259)
      at org.rhq.core.pc.plugin.PluginComponentFactory.buildResourceComponent(PluginComponentFactory.java:164)
      at org.rhq.core.pc.inventory.InventoryManager.prepareResourceForActivation(InventoryManager.java:1642)
      ... 10 more

  WARN  [RHQ Agent Restart Thread] (rhq.core.pc.inventory.InventoryManager)- Cannot give activated resource its discovery component. Cause: org.rhq.core.clientapi.agent.PluginContainerException: Cannot get discovery component proxy for [org.rhq.modules.plugins.jbossas7.SubsystemDiscovery@b5f3fb0]
  ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.AutoDiscoveryExecutor)- Error in auto discovery
  java.lang.OutOfMemoryError: PermGen space
      at java.lang.ClassLoader.defineClass1(Native Method)
      at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)
      at java.lang.ClassLoader.defineClass(ClassLoader.java:615)
  ERROR [InventoryManager.discovery-1] (rhq.core.pc.inventory.RuntimeDiscoveryExecutor)- Error in runtime discovery
  java.lang.OutOfMemoryError: PermGen space

Resolution

To prevent the out of memory condition, do not use the agent's Restart Agent, Restart Plugin Container, or Update All Plugins operation. Instead, restart the agent process by first shutting it down and then start it up normally.

For JBoss ON 3.1.2, this issue has been resolved in the latest plug-in pack hotfix.

Root Cause

ASConnection has a static thread pool which runs connection cleaners to prevent connection leaks. However, the static thread pool is never shutdown resulting in each subsequent restart of the plug-in container leaving behind a running thread pool with its own context classloader.

For more details see This content is not included.bug 1077943.

Diagnostic Steps

  • Obtain a thread dump from the agent. If the thread dump contains more then one thread with the name ASConnection Cleaner-1 this issue applies.

  • The following object query language (OQL) query can be used against a heap dump that was generated by an agent having this issue:

      select { name: t.name? t.name.toString() : "null", thread: t } 
  from instanceof java.lang.Thread t
  where /ASConnection Cleaner-1/(t.name.toString())

    If more then one row is returned, this issue applies.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.