NullPointerException/IllegalArgumentException can happen at channel.write(buffer) because of a race condition inside AsyncSenderImpl in JBoss EAP 7

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 7.x

Issue

The following NullPointerException or IllegalArgumentException happened. This happened very raraly. What caused this issue?

ERROR [org.xnio.listener] (default I/O-3) XNIO001007: A channel event listener threw an exception: java.lang.NullPointerException
    at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:154) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1]
    at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:187)
    at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:2011)
    at io.undertow.io.AsyncSenderImpl.invokeOnComplete(AsyncSenderImpl.java:398)
    at io.undertow.io.AsyncSenderImpl.access$400(AsyncSenderImpl.java:41)
    at io.undertow.io.AsyncSenderImpl$3.handleEvent(AsyncSenderImpl.java:454)
    at io.undertow.io.AsyncSenderImpl$3.handleEvent(AsyncSenderImpl.java:440)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1]
    at io.undertow.channels.DetachableStreamSinkChannel$SetterDelegatingListener.handleEvent(DetachableStreamSinkChannel.java:285)
    at io.undertow.channels.DetachableStreamSinkChannel$SetterDelegatingListener.handleEvent(DetachableStreamSinkChannel.java:272)
    at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1]
    at org.xnio.conduits.WriteReadyHandler$ChannelListenerHandler.writeReady(WriteReadyHandler.java:65) [xnio-api-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1]
    at io.undertow.server.protocol.ajp.AjpServerResponseConduit$AjpServerWriteReadyHandler.writeReady(AjpServerResponseConduit.java:496)
    at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:94) [xnio-nio-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1]
    at org.xnio.nio.WorkerThread.run(WorkerThread.java:571) [xnio-nio-3.4.7.Final-redhat-1.jar:3.4.7.Final-redhat-1]

or 

ERROR [stderr] (default task-4) Exception in thread "default task-44" java.lang.NullPointerException
ERROR [stderr] (default task-4)    at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:154)
ERROR [stderr] (default task-4)    at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:187)
ERROR [stderr] (default task-4)    at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:2011)
ERROR [stderr] (default task-4)    at io.undertow.io.AsyncSenderImpl.invokeOnComplete(AsyncSenderImpl.java:398)
ERROR [stderr] (default task-4)    at io.undertow.io.AsyncSenderImpl.send(AsyncSenderImpl.java:162)
ERROR [stderr] (default task-4)    at io.undertow.server.handlers.resource.PathResource$1ServerTask.run(PathResource.java:178)
ERROR [stderr] (default task-4)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
ERROR [stderr] (default task-4)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
ERROR [stderr] (default task-4)    at java.lang.Thread.run(Thread.java:748)

or

ERROR [stderr] (default task-13) Exception in thread "default task-13" java.lang.IllegalArgumentException
ERROR [stderr] (default task-13)    at java.nio.Buffer.position(Buffer.java:244)
ERROR [stderr] (default task-13)    at sun.nio.ch.IOUtil.write(IOUtil.java:158)
ERROR [stderr] (default task-13)    at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
ERROR [stderr] (default task-13)    at org.xnio.nio.NioSocketConduit.write(NioSocketConduit.java:162)
ERROR [stderr] (default task-13)    at io.undertow.conduits.BytesSentStreamSinkConduit.write(BytesSentStreamSinkConduit.java:70)
ERROR [stderr] (default task-13)    at io.undertow.conduits.AbstractFramedStreamSinkConduit.doWrite(AbstractFramedStreamSinkConduit.java:140)
ERROR [stderr] (default task-13)    at io.undertow.conduits.AbstractFramedStreamSinkConduit.write(AbstractFramedStreamSinkConduit.java:108)
ERROR [stderr] (default task-13)    at io.undertow.server.protocol.ajp.AjpServerResponseConduit.write(AjpServerResponseConduit.java:298)
ERROR [stderr] (default task-13)    at io.undertow.server.protocol.ajp.AjpServerResponseConduit.write(AjpServerResponseConduit.java:357)
ERROR [stderr] (default task-13)    at org.xnio.conduits.ConduitStreamSinkChannel.write(ConduitStreamSinkChannel.java:154)
ERROR [stderr] (default task-13)    at io.undertow.channels.DetachableStreamSinkChannel.write(DetachableStreamSinkChannel.java:187)
ERROR [stderr] (default task-13)    at io.undertow.server.HttpServerExchange$WriteDispatchChannel.write(HttpServerExchange.java:2011)
ERROR [stderr] (default task-13)    at io.undertow.io.AsyncSenderImpl.invokeOnComplete(AsyncSenderImpl.java:398)
ERROR [stderr] (default task-13)    at io.undertow.io.AsyncSenderImpl.send(AsyncSenderImpl.java:162)
ERROR [stderr] (default task-13)    at io.undertow.server.handlers.resource.PathResource$1ServerTask.run(PathResource.java:178)
ERROR [stderr] (default task-13)    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
ERROR [stderr] (default task-13)    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
ERROR [stderr] (default task-13)    at java.lang.Thread.run(Thread.java:748)

Resolution

There is a possible race condition issue that multiple threads touch the same buffer in Undertow's Sender implementation (AsyncSenderImpl). So, that can cause NullPointerException or IllegalArgumentException on the buffer during writing to the channel for serving static contents through the <file> handler (or DefaultServlet). Probably, response data corruption could happen in this case.

The following JIRA has been raised. The fix will be incorporated in EAP 7.2.9 and EAP 7.3.2 onwards:

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.