JVM crash in DirectByteBuffer.put from Undertow SslConduit.doWrap/ServletOutputStreamImpl write
Environment
- JBoss Enterprise Application Platform (EAP) 7.x
Issue
- Our JVM crashes when a custom application thread tries to write on a ServletOutputStream like below:
Stack: [0x00007f7f634d2000,0x00007f7f635d3000], sp=0x00007f7f635d0eb0, free space=1019k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
v ~StubRoutines::jbyte_disjoint_arraycopy
J 26178 C2 java.nio.DirectByteBuffer.put(Ljava/nio/ByteBuffer;)Ljava/nio/ByteBuffer; (269 bytes) @ 0x00007f7fc0c19526 [0x00007f7fc0c191c0+0x366]
J 27604 C2 sun.security.ssl.EngineWriter.writeRecord(Lsun/security/ssl/EngineOutputRecord;Lsun/security/ssl/EngineArgs;Lsun/security/ssl/Authenticator;Lsun/security/ssl/CipherBox;)Ljavax/net/ssl/SSLEngineResult$HandshakeStatus; (87 bytes) @ 0x00007f7fc0f84ef4 [0x00007f7fc0f84ce0+0x214]
J 30358 C2 sun.security.ssl.SSLEngineImpl.writeRecord(Lsun/security/ssl/EngineOutputRecord;Lsun/security/ssl/EngineArgs;)Ljavax/net/ssl/SSLEngineResult$HandshakeStatus; (88 bytes) @ 0x00007f7fbedf83e8 [0x00007f7fbedf8380+0x68]
J 31285 C2 io.undertow.protocols.ssl.SslConduit.wrapAndFlip([Ljava/nio/ByteBuffer;II)Ljavax/net/ssl/SSLEngineResult; (98 bytes) @ 0x00007f7fc066d650 [0x00007f7fc066cfa0+0x6b0]
J 31444 C2 io.undertow.protocols.ssl.SslConduit.doWrap([Ljava/nio/ByteBuffer;II)J (654 bytes) @ 0x00007f7fc1586d10 [0x00007f7fc1586ba0+0x170]
J 31425 C2 io.undertow.conduits.ChunkedStreamSinkConduit.doWrite(Ljava/nio/ByteBuffer;)I (514 bytes) @ 0x00007f7fbf95b818 [0x00007f7fbf95b200+0x618]
J 30825 C2 io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(Z)V (110 bytes) @ 0x00007f7fc06213b0 [0x00007f7fc06212c0+0xf0]
J 30437 C2 io.netty.channel.socket.http.HttpTunnelingServlet$OutboundConnectionHandler.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (104 bytes) @ 0x00007f7fc0e1160c [0x00007f7fc0e10b40+0xacc]
J 29241 C2 io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V (9 bytes) @ 0x00007f7fc0c24354 [0x00007f7fc0c241a0+0x1b4]
J 27462 C2 io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(Lio/netty/channel/AbstractChannelHandlerContext;Ljava/lang/Object;)V (53 bytes) @ 0x00007f7fc0f32868 [0x00007f7fc0f32700+0x168]
J 31540 C1 io.netty.channel.local.LocalChannel.finishPeerRead0(Lio/netty/channel/local/LocalChannel;)V (90 bytes) @ 0x00007f7fc0915c34 [0x00007f7fc0915880+0x3b4]
J 28954 C2 io.netty.channel.local.LocalChannel$5.run()V (12 bytes) @ 0x00007f7fc00b512c [0x00007f7fc00b50e0+0x4c]
J 30216% C2 io.netty.channel.DefaultEventLoop.run()V (33 bytes) @ 0x00007f7fc0522b68 [0x00007f7fc0522b00+0x68]
- We otherwise see many UT010029 excpetions repeating like below:
WARN [io.netty.channel.socket.http.HttpTunnelingServlet] (Server-Worker-2-206) UT010029: Stream is closed: java.io.IOException: UT010029: Stream is closed
at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:137)
...
Resolution
- Note that the servlet spec expects request/response objects are generally processed within the original container request thread. So forked threads from a servlet, and if done, it is up to the application to ensure safety as the request/response objects are not expected to be thread safe objects per the servlet spec.
- Update to EAP 7.3.6+ for additional thread safety in undertow
Root Cause
- This content is not included.JBEAP-20636
- One thread closes out the connection and streams and releases underlying DirectByteBuffers while another forked thread is concurrently trying to write on the ServletOutputStream and its DirectByteBuffer.
- This can often times fail more safely with one of these exceptions:
UT010029: Stream is closed: java.io.IOException: UT010029: Stream is closed
at io.undertow.servlet.spec.ServletOutputStreamImpl.write(ServletOutputStreamImpl.java:137)
...
UT000091: Buffer has already been freed: java.lang.IllegalStateException: UT000091: Buffer has already been freed
at io.undertow.server.DefaultByteBufferPool$DefaultPooledBuffer.getBuffer(DefaultByteBufferPool.java:260)
at io.undertow.protocols.ssl.SslConduit.wrapAndFlip(SslConduit.java:982)
at io.undertow.protocols.ssl.SslConduit.doWrap(SslConduit.java:913)
at io.undertow.protocols.ssl.SslConduit.write(SslConduit.java:393)
at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:655)
at io.undertow.conduits.ChunkedStreamSinkConduit.doWrite(ChunkedStreamSinkConduit.java:166)
at io.undertow.conduits.ChunkedStreamSinkConduit.write(ChunkedStreamSinkConduit.java:128)
...
java.nio.channels.ClosedChannelException
at io.undertow.protocols.ssl.SslConduit.write(SslConduit.java:391)
at io.undertow.server.protocol.http.HttpResponseConduit.write(HttpResponseConduit.java:655)
at io.undertow.conduits.ChunkedStreamSinkConduit.doWrite(ChunkedStreamSinkConduit.java:166)
at io.undertow.conduits.ChunkedStreamSinkConduit.write(ChunkedStreamSinkConduit.java:128)
at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:150)
at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:240)
at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:2107)
at io.undertow.servlet.spec.ServletOutputStreamImpl.writeBufferBlocking(ServletOutputStreamImpl.java:574)
at io.undertow.servlet.spec.ServletOutputStreamImpl.flushInternal(ServletOutputStreamImpl.java:489)
at io.undertow.servlet.spec.ServletOutputStreamImpl.flush(ServletOutputStreamImpl.java:476)
- But depending on the timing of the multi-theaded actions, the write attempting thread may already be past these exception points when the streams and buffers are closed by the other thread.
Components
Category
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.