OutOfMemory exceptions on EAP 7.2.0 with memory retention in io.undertow.server.protocol.http.HttpOpenListener

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)
    • 7.2.0

Issue

  • The EAP instance is slowly running out of memory over time, needs to be restarted daily.

  • Messages on JMS queues stops to be consumed.

  • Looking at a heap dump, we can see memory in HttpServerConnections accumulating in the HttpOpenListener's connections map:

Class Name                                                                | Shallow Heap | Retained Heap | Percentage | Retained Heap
--------------------------------------------------------------------------------------------------------------------------------------
io.undertow.server.protocol.http.HttpOpenListener @ 0x6c39be4e8           |           48 | 2,221,973,536 |     72.73% | 2,221,973,536
|- java.util.Collections$SetFromMap @ 0x6c39be518                         |           24 | 2,221,973,336 |     72.73% |              
|  |- java.util.concurrent.ConcurrentHashMap @ 0x6c39be530                |           64 | 2,221,973,288 |     72.73% |              
|  |  |- java.util.concurrent.ConcurrentHashMap$Node[131072] @ 0x74a4f3e70|      524,304 | 2,221,973,152 |     72.73% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x6f9cb9018     |           32 |       172,088 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x71b5700c8     |           32 |       165,920 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x71aae0f38     |           32 |       165,576 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x701960c00     |           32 |       165,320 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x72007d900     |           32 |       165,208 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x71a2ae2f0     |           32 |       165,040 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x6f2d7c548     |           32 |       164,208 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x74f0ada50     |           32 |       163,536 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x74ea19d98     |           32 |       161,744 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x6f66b9c88     |           32 |       159,328 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x711386590     |           32 |       159,256 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x71a6121a0     |           32 |       159,184 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x74ebc3fe0     |           32 |       159,184 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x719f3d190     |           32 |       159,152 |      0.01% |              
|  |  |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0x710879300     |           32 |       159,048 |      0.01% |              
--------------------------------------------------------------------------------------------------------------------------------------

Resolution

Apply the latest available JBoss EAP 7.2.x upgrade patch.1

Root Cause

You are facing this issue because of the bug in undertow API's. This issue is captured in JIRA1, this issue is fixed in undertow "2.0.18.Final" version. JBoss EAP 7.2.0 uses "2.0.15.Final" so you have to upgrade EAP version to 7.2.1 (or later) release in order to upgrade the undertow version.

SBR
Product(s)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.