JON: Server becomes unavailable due to OutOfMemoryError: Java heap space when multiple users are accessing the web UI

Solution Verified - Updated

Environment

  • JBoss Operations Network (JON) 2.3.1
  • Multiple clients accessing JON UI

Issue

  • Our JBoss Operations Network (JON) server is experiencing out of memory issues and finally down
  • Server becomes unavailable and the following error appears every second or so in the server log:
    ERROR [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/jboss-remoting-servlet-invoker].[ServerInvokerServlet]] Servlet.service() for servlet ServerInvokerServlet threw exception
javax.management.RuntimeErrorException
     at org.jboss.mx.interceptor.ReflectedDispatcher.handleInvocationExceptions(ReflectedDispatcher.java:178)
     at org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:163)
     at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
     at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
     at org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
     at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
     at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:201)
     at $Proxy381.processRequest(Unknown Source)
     at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.processRequest(ServerInvokerServlet.java:128)
     at org.jboss.remoting.transport.servlet.web.ServerInvokerServlet.doPost(ServerInvokerServlet.java:157)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
     at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
     at org.rhq.helpers.rtfilter.filter.RtFilter.doFilter(RtFilter.java:124)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
     at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:96)
     at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
     at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
     at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:230)
     at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:175)
     at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:182)
     at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:84)
     at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
     at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
     at org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:157)
     at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
     at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:262)
     at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:844)
     at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
     at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:446)
     at java.lang.Thread.run(Thread.java:595)
Caused by: java.lang.OutOfMemoryError: Java heap space

Resolution

This issue has been identified as This content is not included.Bug 577294. It has been resolved in JON 2.4 and later and in JON 2.3.1 Patch-3. Either upgrade to JON 2.4 or apply the latest JON server patch.

Root Cause

The JON UI navigation tree is retaining navigation elements using org.ajax4jsf.util.LRUMaps. The implementation in use results in high memory usage per client session. With one or two client sessions, this overhead is not noticeable but with 10 or more clients accessing the JON UI or a single client with a prolonged session, the result can lead to huge heap requirements. For details, see This content is not included.This content is not included.https://bugzilla.redhat.com/show_bug.cgi?id=577294.

Diagnostic Steps

  • Obtain a Java heap dump when the condition occurs - See How do I create a Java heap dump?
    • Analyse the heap dump - See How do I analyze a Java heap dump?
      • A Leak Suspect Report should reveal:
        One instance of "org.apache.catalina.session.StandardManager" loaded by 
"org.jboss.mx.loading.UnifiedClassLoader3 @ 0x2b4a473b2f40" occupies 304,068,272 
(14.54%) bytes. The memory is accumulated in one instance of "org.ajax4jsf.util.LRUMap" 
loaded by "org.jboss.web.tomcat.service.WebAppClassLoader @ 0x2b4a48cf0240".

        Keywords
        
org.jboss.web.tomcat.service.WebAppClassLoader @ 0x2b4a48cf0240
        
org.jboss.mx.loading.UnifiedClassLoader3 @ 0x2b4a473b2f40
        
org.apache.catalina.session.StandardManager
        
org.ajax4jsf.util.LRUMap

Product(s)
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.