Memory leak in JBoss EAP 6 when highly-nested JSP tags are used / Heap retention in org.jboss.as.web.deployment.WebInjectionContainer

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6.0.0

Issue

Resolution

  • This is fixed in JBoss EAP 6.0.1 (specifically JBossWeb 7.0.17). Please upgrade to EAP 6.0.1.

  • Until EAP 6.0.1 is released, one can use the following workarounds:

  • Add the following jvm option in your JAVA_OPTS:

      -Dorg.apache.jasper.Constants.USE_INSTANCE_MANAGER_FOR_TAGS=false

or

  • Disable tag-pooling in the web subsystem's jsp-configuration set as per https://access.redhat.com/solutions/889563:

      <subsystem xmlns="urn:jboss:domain:web:1.0" default-virtual-server="default-host">
              <configuration>
                  <jsp-configuration tag-pooling="false"/>
              </configuration>
  ...

    • That can be set via CLI like so:

        /profile=PROFILE_NAME/subsystem=web/configuration=jsp-configuration:add(tag-pooling=false)

    • Note that this can increase JSP compiled size. For large JSPs, they may need to be refactored to disable tag pooling without hitting the 65535 byte limit.

    • And because of the following This content is not included.bug, tmp (which contains compiled jsp objects) may not get cleaned up. So when making this change on a version prior to EAP 6.3.0, you should also delete the JBoss tmp directory between the restart as well. This will ensure old JSP class files previously compiled with the old true tag-pooling setting are discarded so that they will be recompiled with tag-pooling disabled.

    • Also, note that this option from the web subsystem is ignored by any app that configures the org.apache.jasper.servlet.JspServlet within its WEB-INF/web.xml.

  • If using an executor and seeing this behavior, consider removing your keepalive-time settings from this executor to disable the thread recycling. Or remove the executor from your connectors and configure the connector's max-connections parameter instead. Or if you need the executor with the keepalive-time thread reclamation, set the following system properties:

      -Dorg.apache.jasper.Constants.USE_INSTANCE_MANAGER_FOR_TAGS=false -Dorg.apache.jasper.Constants.INJECT_TAGS=false

  • Tag pool overhead grows with the number of threads and the number of JSPs since every JSP will create a pool for every thread that services a request for it. Thus, you'd expect the number of indivdual tag pools to be equal to #ofJSPs*#ofThreads. Those individual tag pools are sized by the org.apache.jasper.Constants.MAX_POOL_SIZE with a default of 4 so the total overall pooled tag count would be 4*#ofJSPs*#ofThreads. Even with the above memory leak bug fix, this overhead may be too great so tag-pooling likely will need to be disabled with large JSP and/or thread counts.

Root Cause

Diagnostic Steps

  • The heap dump shows heap retention in org.jboss.as.web.deployment.WebInjectionContainer.
Class Name                                                                           | Shallow Heap | Retained Heap | Percentage
---------------------------------------------------------------------------------------------------------------------------------
org.jboss.as.web.deployment.WebInjectionContainer @ 0x781ef6440                      |           32 | 1,137,297,960 |     78.20%
|- org.jboss.as.web.deployment.ConcurrentReferenceHashMap @ 0x781f3d660              |           48 | 1,137,288,344 |     78.20%
|  '- org.jboss.as.web.deployment.ConcurrentReferenceHashMap$Segment[2] @ 0x781f3d690|           24 | 1,137,288,296 |     78.20%
|     |- org.jboss.as.web.deployment.ConcurrentReferenceHashMap$Segment @ 0x7822f4d68|           56 |   569,281,688 |     39.14%
|     |- org.jboss.as.web.deployment.ConcurrentReferenceHashMap$Segment @ 0x781f3d6a8|           56 |   568,004,360 |     39.06%
|     |- org.opencms.jsp.CmsJspTagContentLoad @ 0x7850544e8                          |          112 |         1,504 |      0.00%
|     |- org.apache.taglibs.standard.tag.el.fmt.BundleTag @ 0x785054558              |           56 |            80 |      0.00%
|     |- org.apache.taglibs.standard.tag.el.fmt.BundleTag @ 0x7877191f8              |           56 |            80 |      0.00%
|     |- org.apache.taglibs.standard.tag.el.fmt.BundleTag @ 0x78898b870              |           56 |            80 |      0.00%
---------------------------------------------------------------------------------------------------------------------------------
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.