JBoss - Java heap retention in InternalOutputBuffer

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)
    • 5.x
    • 6.x
  • JBoss Web Server
    • Tomcat

Issue

  • JBoss ran out of memory and most of the heap was used by org.apache.coyote.http11.InternalOutputBuffer and their buf byte[]s
  • Tomcat has high heap usage in org.apache.coyote.http11.Http11InputBuffer and org.apache.coyote.http11.Http11OutputBuffer objects.

Resolution

  • Lower the maxHttpHeaderSize
  • Increase heap to meet needs of your configured maxHttpHeaderSize and number of connections
  • Once an http processor is created with these buffers to serve a request, tomcat caches it for potential future reuse to decrease garbage allocation and collection. It caches these up to the connector's processorCache setting (default is a limit of 200). You could lower this processorCache to keep fewer processors cached and lower they're buffer overhead in heap. But this would mean more GC overhead with any load that exceeds that processorCache size and you would still have to plan your max heap to support (2 * maxHttpHeaderSize * maxThreads) to account for peak overhead possible under max load.

Root Cause

  • maxHttpHeaderSize configures the size that http buffers are initiated to. So the overall buffer overhead would increase with load (maxHttpHeaderSize * connection count)

Diagnostic Steps

  • Check the maxHttpHeaderSize in the heap dump and ensure it matches the buffer sizes seen:

      SELECT s.maxHttpHeaderSize FROM org.apache.coyote.http11.Http11Protocol s
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.