I/O threads experience increased CPU usage for https-listener when task worker thread pool is exhausted in JBoss EAP 7

Solution Verified - Updated 12 Mar 2025

Environment

JBoss Enterprise Application Platform (EAP)
- 7.x
- 8.x

Issue

We ran into a situation of exhausted task threads from long-running application requests that were blocked and waiting for a long time. But at that time, we saw increased CPU from the default I/O threads in JBoss EAP 7. The high CPU threads can be seen cycling over SslConduit.read calls from the AlpnOpenListener or HttpReadListener:

12:56:10,512 INFO  [stdout] (default I/O-17) io.undertow.protocols.ssl.SslConduit.doUnwrap(SslConduit.java:720)
12:56:10,512 INFO  [stdout] (default I/O-17) io.undertow.protocols.ssl.SslConduit.read(SslConduit.java:588)
12:56:10,512 INFO  [stdout] (default I/O-17) org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
12:56:10,512 INFO  [stdout] (default I/O-17) io.undertow.server.protocol.http.AlpnOpenListener$AlpnConnectionListener.handleEvent(AlpnOpenListener.java:356)
12:56:10,512 INFO  [stdout] (default I/O-17) io.undertow.server.protocol.http.AlpnOpenListener$AlpnConnectionListener.handleEvent(AlpnOpenListener.java:341)
12:56:10,512 INFO  [stdout] (default I/O-17) org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
12:56:10,512 INFO  [stdout] (default I/O-17) org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
12:56:10,512 INFO  [stdout] (default I/O-17) io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1291)
12:56:10,512 INFO  [stdout] (default I/O-17) org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
12:56:10,512 INFO  [stdout] (default I/O-17) org.xnio.nio.WorkerThread.run(WorkerThread.java:591)

10:14:29,798 INFO  [stdout] (default I/O-23) io.undertow.protocols.ssl.SslConduit.doUnwrap(SslConduit.java:850)
10:14:29,798 INFO  [stdout] (default I/O-23) io.undertow.protocols.ssl.SslConduit.read(SslConduit.java:587)
10:14:29,798 INFO  [stdout] (default I/O-23) org.xnio.conduits.AbstractStreamSourceConduit.read(AbstractStreamSourceConduit.java:51)
10:14:29,798 INFO  [stdout] (default I/O-23) io.undertow.conduits.BytesReceivedStreamSourceConduit.read(BytesReceivedStreamSourceConduit.java:67)
10:14:29,798 INFO  [stdout] (default I/O-23) org.xnio.conduits.ConduitStreamSourceChannel.read(ConduitStreamSourceChannel.java:127)
10:14:29,798 INFO  [stdout] (default I/O-23) io.undertow.server.protocol.http.HttpReadListener.handleEventWithNoRunningRequest(HttpReadListener.java:158)
10:14:29,798 INFO  [stdout] (default I/O-23) io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:136)
10:14:29,798 INFO  [stdout] (default I/O-23) io.undertow.server.protocol.http.HttpReadListener.handleEvent(HttpReadListener.java:59)
10:14:29,798 INFO  [stdout] (default I/O-23) org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
10:14:29,798 INFO  [stdout] (default I/O-23) org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
10:14:29,798 INFO  [stdout] (default I/O-23) io.undertow.protocols.ssl.SslConduit$SslReadReadyHandler.readReady(SslConduit.java:1274)
10:14:29,798 INFO  [stdout] (default I/O-23) io.undertow.protocols.ssl.SslConduit$1.run(SslConduit.java:183)
10:14:29,798 INFO  [stdout] (default I/O-23) io.undertow.protocols.ssl.SslConduit$2.run(SslConduit.java:198)
10:14:29,798 INFO  [stdout] (default I/O-23) org.xnio.nio.WorkerThread.safeRun(WorkerThread.java:612)

Resolution

Either of the following can be used as workaround:

Update to EAP 7.4.16+ or 8.0.1.1+
Use a request limit filter to keep idle task worker threads available for the handshake tasks even if application requests would otherwise exhaust the pool. So doubling the default task-max-threads for instance (512 for example) and then setting max-concurrent-requests limit to the original task-max-threads (256 in this case):
```
/subsystem=io/worker=default:write-attribute(name=task-max-threads,value=512)
/subsystem=undertow/configuration=filter/request-limit=mylimit:add(max-concurrent-requests=256, queue-size=10000)
/subsystem=undertow/server=default-server/host=default-host/filter-ref=mylimit:add()
```
See also the knowledge article "How to configure max concurrent requests limit for specific application in EAP 7 for the details.
Separate the task worker thread pool for the web application that executes long-running processing and causes thread pool exhaustion. See the knowledge article "How to specify a custom task worker thread pool for the specific application in JBoss EAP 7 for the details.

Root Cause

This content is not included.JBEAP-25579
This content is not included.JBEAP-25705
This content is not included.UNDERTOW-2304
SSL handshake tasks are delegated to the same task threads as application requests. Due to task thread exhaustion from application requests, handshake tasks are backlogged and not promptly completed. This means Content from github.com is not included.SslConduit.doUnwrap returns 0 here to calling read operations while awaiting those outstanding handshake tasks and other methods repeat that read attempt due to the returned 0.
With HTTP2, the looping reading operations can persist indefinitely but they can clear after a connection close is detected without HTTP2.

SBR

Webservers

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.