Solving Java heap retention in BodyContentImpl in JBoss EAP

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 7.x
  • 6.x
  • 5.x
  • 4.x

Issue

• org.apache.jasper.runtime.BodyContentImpl object is holding 47.85% of memory from heap dump.
• Continuous full garbage collections causing the application to be unresponsive.
• Having to recycle the site every 5 hours to avoid OutOfMemoryError.

Resolution

The steps to solve this issue in JBoss EAP 7, JBoss EAP 6 (and 5), and JBoss EAP 4.

JBoss EAP 7.x

The org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER environment variable is available again in JBoss EAP 7. Set this property to true to release larger buffers, as in:

 org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true

This can be set on the standalone.conf (for standalone mode), or domain.conf (for domain mode). Example adding at the bottom of the .conf file:

JAVA_OPTS="$JAVA_OPTS -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true"

JBoss EAP 6.x and 5.x

The org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER environment variable is no longer available in EAP 5 and 6 since the buffers were revamped and grow less aggressively.

The BodyContentImpl instances are stored in an array called "outs" in PageContextImpl. Set the environment variable org.apache.jasper.runtime.JspFactoryImpl.USE_POOL to false to disable the pooling of PageContextImpl objects:

-Dorg.apache.jasper.runtime.JspFactoryImpl.USE_POOL=false

The trade-off in limiting the buffer is the possibility of increased garbage collection activity due to the character array being re-allocated.

JBoss EAP 4.x

Set the org.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER environment variable to true (it is false by default) to allow a new BodyContentImpl character array to be allocated for a JSP page if the buffer needed to render the last page was larger than the default buffer size (512K).

-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true

The BodyContentImpl instances are stored in an array called "outs" in PageContextImpl. An additional step to avoid retention is to set the environment variable org.apache.jasper.runtime.JspFactoryImpl.USE_POOL to false to disable the pooling of PageContextImpl objects:

-Dorg.apache.jasper.runtime.JspFactoryImpl.USE_POOL=false

Root Cause

The issue can be caused by any of the following:

• BodyContentImpl character buffer is allowed to grow as large as needed to render a JSP page
• BodyContentImpl objects are pooled between pages.
• Before the next JSP page is rendered, the BodyContentImpl clear() method is called:

nextChar = 0;
if (LIMIT_BUFFER && (cb.length > Constants.DEFAULT_TAG_BUFFER_SIZE)) {
    bufferSize = Constants.DEFAULT_TAG_BUFFER_SIZE;
    cb = new char[bufferSize];
}

The position of the buffer is reset to 0, then if the LIMIT_BUFFER flag is set and the character buffer size leftover from rendering the last JSP page is larger than the 512K limit, a new character buffer is allocated, leaving the old one free to be garbage collected.

Diagnostic Steps

1. Get a heap dump
2. Analyze it
3. Verify the retention is in org.apache.jasper.runtime.BodyContentImpl

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.