EAP 7 hits Direct buffer OOME with large response headers on HTTP2

Environment

• JBoss Enterprise Application Platform (EAP) 7.x

Issue

• At times, we see java.lang.OutOfMemoryError: Direct buffer memory errors when large response headers are sent via HTTP2 on EAP 7 and direct memory usage rapidly spikes

Resolution

• Consider disabling HTTP2 as a workaround
• Remove any unexpectedly large response headers

Root Cause

• HpackEncoder.encode may be failing to progress through header data properly so it is repeatedly cycling over data repeatedly to flood and exhaust the available DirectByteBuffer space.

Diagnostic Steps

• From a heap dump, check that the byte buffers are retained under a local io.undertow.connector.PooledByteBuffer:

Class Name                                                                  | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
----------------------------------------------------------------------------------------------------------------------------------------------
org.xnio.nio.WorkerThread @ 0x6c3c96c28  default I/O-8 Thread               |            1 |          152 |                24 |    41,100,320
'- <Java Local> io.undertow.connector.PooledByteBuffer[209205] @ 0x734200000|            1 |      836,840 |                24 |    41,002,616
   '- [207931] io.undertow.server.XnioByteBufferPool$1 @ 0x7bfaa6328        |            1 |           24 |                24 |           192
----------------------------------------------------------------------------------------------------------------------------------------------

• Check the thread stack of that associated thread and see it is processing in io.undertow.protocols.http2.HpackEncoder.encode:

default I/O-8
  at io.undertow.protocols.http2.HPackHuffman.encode(Ljava/nio/ByteBuffer;Ljava/lang/String;Z)Z (HPackHuffman.java:456)
  at io.undertow.protocols.http2.HpackEncoder.writeHuffmanEncodableValue(Ljava/nio/ByteBuffer;Lio/undertow/util/HttpString;Ljava/lang/String;)V (HpackEncoder.java:277)
  at io.undertow.protocols.http2.HpackEncoder.encode(Lio/undertow/util/HeaderMap;Ljava/nio/ByteBuffer;)Lio/undertow/protocols/http2/HpackEncoder$State; (HpackEncoder.java:219)
  at io.undertow.protocols.http2.Http2DataStreamSinkChannel.encodeContinuationFrame(Lio/undertow/util/HeaderMap;Lio/undertow/connector/PooledByteBuffer;)Lio/undertow/protocols/http2/HpackEncoder$State; (Http2DataStreamSinkChannel.java:280)
  at io.undertow.protocols.http2.Http2DataStreamSinkChannel.createFrameHeaderImpl()Lio/undertow/server/protocol/framed/SendFrameHeader; (Http2DataStreamSinkChannel.java:146)
  at io.undertow.protocols.http2.Http2StreamSinkChannel.generateSendFrameHeader()Lio/undertow/server/protocol/framed/SendFrameHeader; (Http2StreamSinkChannel.java:56)
  at io.undertow.protocols.http2.Http2FramePriority.insertFrame(Lio/undertow/protocols/http2/AbstractHttp2StreamSinkChannel;Ljava/util/List;)Z (Http2FramePriority.java:71)
  at io.undertow.protocols.http2.Http2FramePriority.insertFrame(Lio/undertow/server/protocol/framed/AbstractFramedStreamSinkChannel;Ljava/util/List;)Z (Http2FramePriority.java:34)
  at io.undertow.server.protocol.framed.AbstractFramedChannel.flushSenders()V (AbstractFramedChannel.java:603)
  at io.undertow.server.protocol.framed.AbstractFramedChannel$5.run()V (AbstractFramedChannel.java:747)
  at io.undertow.server.protocol.framed.AbstractFramedChannel$1.run()V (AbstractFramedChannel.java:144)
  at org.xnio.nio.WorkerThread.safeRun(Ljava/lang/Runnable;)V (WorkerThread.java:612)
  at org.xnio.nio.WorkerThread.run()V (WorkerThread.java:479)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.