HttpServerExchange accumulation with async requests on EAP 7

Solution Unverified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP) 7.x

Issue

  • We see a growing number of io.undertow.server.HttpServerExchanges retained like so with async requests leading to an OutOfMemoryError:
Class Name                                                                                                      | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
org.xnio.nio.WorkerThread @ 0x5000732b8  default I/O-11 Thread                                                  |          269 |          152 |            47,344 |         8,464
|- <Java Local>, selector sun.nio.ch.EPollSelectorImpl @ 0x5022a3208                                            |          262 |           72 |            46,112 |        19,464
|  '- fdToKey java.util.HashMap @ 0x5022eb8c0                                                                   |          262 |           48 |            46,112 |        19,200
|     '- table java.util.HashMap$Node[512] @ 0x6359c80f0                                                        |          262 |        2,064 |            46,112 |        19,152
|        |- [30] java.util.HashMap$Node @ 0x7744cb260                                                           |            2 |           32 |               352 |            96
|        |  |- next java.util.HashMap$Node @ 0x265839df8                                                        |            1 |           32 |               176 |            48
|        |  |- value sun.nio.ch.SelectionKeyImpl @ 0x6e560f040                                                  |            1 |           40 |               176 |            40
|        |  |  '- attachment org.xnio.nio.NioSocketConduit @ 0x7139e2638                                        |            1 |           64 |               176 |            64
|        |  |     '- writeReadyHandler io.undertow.protocols.ssl.SslConduit$SslWriteReadyHandler @ 0x584ed0a20  |            1 |           24 |               176 |            40
|        |  |        '- delegateHandler org.xnio.conduits.WriteReadyHandler$ChannelListenerHandler @ 0x572e48ec8|            1 |           16 |               176 |            16
|        |  |           '- channel org.xnio.conduits.ConduitStreamSinkChannel @ 0x771e5c8b0                     |            1 |           32 |               176 |           256
|        |  |              '- conduit io.undertow.conduits.ChunkedStreamSinkConduit @ 0x761d02ed8               |            1 |           56 |               176 |           224
|        |  |                 '- attachable io.undertow.server.HttpServerExchange @ 0x6abeb1408                 |            1 |          176 |               176 |     3,091,280

Resolution

  • Apply a timeout to associated AsyncContexts created by the application for the problem requests

Root Cause

  • AsyncContext is created without timeout and never has complete or dispatch called on it.

Diagnostic Steps

  • Use the following OQL to check overall requests and any associate AsyncContext
SELECT exchange.requestURI.toString(), asyncStarted, asyncContext.timeout, asyncContext.complete, asyncContext.dispatched FROM io.undertow.servlet.spec.HttpServletRequestImpl
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.