maximum connections reached on Windows running several EAP 7 instances together
Environment
- JBoss Enterprise Application Platform (EAP) 7.x
- Windows
Issue
- We run many EAP 7 instances together on Windows with default thread pool sizings in the IO subsystem. We begin to see high localhost socket counts equally across all the instances leading to availability issues as requests start to fail with the following error:
ERROR [io.undertow.request] (default task-7980) UT005023: Exception handling request to /app: java.lang.RuntimeException: java.io.IOException: Unable to establish loopback connection
at io.undertow.servlet.spec.HttpServletRequestImpl.parseFormData(HttpServletRequestImpl.java:779)
at io.undertow.servlet.spec.HttpServletRequestImpl.getParameter(HttpServletRequestImpl.java:653)
at javax.servlet.ServletRequestWrapper.getParameter(ServletRequestWrapper.java:194)
...
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:326)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:812)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.IOException: Unable to establish loopback connection
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:94)
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:61)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.PipeImpl.<init>(PipeImpl.java:171)
at sun.nio.ch.SelectorProviderImpl.openPipe(SelectorProviderImpl.java:50)
at java.nio.channels.Pipe.open(Pipe.java:155)
at sun.nio.ch.WindowsSelectorImpl.<init>(WindowsSelectorImpl.java:127)
at sun.nio.ch.WindowsSelectorProvider.openSelector(WindowsSelectorProvider.java:44)
at org.xnio.nio.NioXnio$DefaultSelectorCreator.open(NioXnio.java:272)
at org.xnio.nio.NioXnio.getSelector(NioXnio.java:258)
at org.xnio.nio.SelectorUtils.await(SelectorUtils.java:44)
at org.xnio.nio.NioSocketConduit.awaitReadable(NioSocketConduit.java:358)
at org.xnio.conduits.AbstractSourceConduit.awaitReadable(AbstractSourceConduit.java:66)
at io.undertow.conduits.ReadDataStreamSourceConduit.awaitReadable(ReadDataStreamSourceConduit.java:101)
at io.undertow.conduits.FixedLengthStreamSourceConduit.awaitReadable(FixedLengthStreamSourceConduit.java:285)
at org.xnio.conduits.ConduitStreamSourceChannel.awaitReadable(ConduitStreamSourceChannel.java:151)
at io.undertow.channels.DetachableStreamSourceChannel.awaitReadable(DetachableStreamSourceChannel.java:77)
at io.undertow.server.HttpServerExchange$ReadDispatchChannel.awaitReadable(HttpServerExchange.java:2161)
at io.undertow.server.handlers.form.FormEncodedDataDefinition$FormEncodedDataParser.parseBlocking(FormEncodedDataDefinition.java:254)
at io.undertow.servlet.spec.HttpServletRequestImpl.parseFormData(HttpServletRequestImpl.java:777)
... 52 more
Caused by: java.net.SocketException: No buffer space available (maximum connections reached?): bind
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at sun.nio.ch.PipeImpl$Initializer$LoopbackConnector.run(PipeImpl.java:121)
at sun.nio.ch.PipeImpl$Initializer.run(PipeImpl.java:76)
... 71 more
Resolution
- Size IO subsystem threads lower instead of allowing all JVMs to scale with CPU. You may size these then instead like:
<subsystem xmlns="urn:jboss:domain:io:3.0">
<worker name="default" io-threads="2*#ofCPUcores/JVMcount" task-max-threads="16*#ofCPUcores/JVMcount"/>
<buffer-pool name="default"/>
</subsystem>
- Use EAP 7.2+ so idle threads can be released to reduce selector ports instead of thread pools automatically filling
- Note that EAP 7.3.2 and earlier may face selector growth between GC cycles due to EAP 7 has growing amounts of pipe and eventpoll file handles between full GC cycles. So the latest EAP 7.3.3+ would be best or on 7.3.2 and earlier, set your task-core-threads equal to task-max-threads. You can set both to 16*#ofCPUcores so there is no reduction in max request threads:
<subsystem xmlns="urn:jboss:domain:io:3.0">
<worker name="default" task-core-threads="16*#ofCPUcores" task-max-threads="16*#ofCPUcores" />
- Consider if the windows
TcpNumConnectionsorMaxUserPortregistry values need to be increased to support higher number of resulting ports
Root Cause
- Each I/O and request task thread will cache its own nio selector. Each of those selectors has its own wake up pipe of two locally connected ports. So this results in localhost ports in count equal to thread count * 2.
- The IO subsystem by default sizes task threads to 16 * #ofCPUcores and I/O threads to 2 * #ofCPUcores. So selector sockets from those threads also scale with CPU counts as a result. If running multiple JVMs together with the default, they all assume they should scale to CPU counts for selector ports to scale to 18 * #ofCPUcores * #ofJVMs.
- With EAP 7.1 or earlier, this can be worse as the server will arbitrarily fill thread pools to their max
Diagnostic Steps
- Capture
netstat -aonoutput and a heap dump from a point of high socket counts for inspection - Run the following OQL on the heap dump to check localhost sockets (2130706433 is the byte representation for 127.0.0.1):
SELECT remoteAddress.holder.addr.holder.address, remoteAddress.holder.port, localAddress.holder.addr.holder.address, localAddress.holder.port FROM sun.nio.ch.SocketChannelImpl WHERE (remoteAddress.holder.addr.holder.address = 2130706433)
remoteAddress.holder.addr.holder.address | remoteAddress.holder.port | localAddress.holder.addr.holder.address | localAddress.holder.port
-------------------------------------------------------------------------------------------------------------------------------------------
2,130,706,433 | 65,298 | 2,130,706,433 | 65,299
2,130,706,433 | 65,299 | 2,130,706,433 | 65,298
2,130,706,433 | 65,296 | 2,130,706,433 | 65,297
2,130,706,433 | 65,297 | 2,130,706,433 | 65,296
2,130,706,433 | 65,294 | 2,130,706,433 | 65,295
2,130,706,433 | 65,295 | 2,130,706,433 | 65,294
2,130,706,433 | 64,869 | 2,130,706,433 | 64,870
2,130,706,433 | 64,870 | 2,130,706,433 | 64,869
2,130,706,433 | 64,862 | 2,130,706,433 | 64,863
2,130,706,433 | 64,863 | 2,130,706,433 | 64,862
2,130,706,433 | 64,832 | 2,130,706,433 | 64,833
2,130,706,433 | 64,833 | 2,130,706,433 | 64,832
2,130,706,433 | 64,820 | 2,130,706,433 | 64,821
2,130,706,433 | 64,821 | 2,130,706,433 | 64,820
2,130,706,433 | 64,810 | 2,130,706,433 | 64,811
2,130,706,433 | 64,811 | 2,130,706,433 | 64,810
2,130,706,433 | 64,804 | 2,130,706,433 | 64,805
2,130,706,433 | 64,805 | 2,130,706,433 | 64,804
...
- You should find these match the ports found for that pid in netstat:
TCP 127.0.0.1:64804 127.0.0.1:64805 ESTABLISHED 204336
TCP 127.0.0.1:64805 127.0.0.1:64804 ESTABLISHED 204336
TCP 127.0.0.1:64810 127.0.0.1:64811 ESTABLISHED 204336
TCP 127.0.0.1:64811 127.0.0.1:64810 ESTABLISHED 204336
TCP 127.0.0.1:64820 127.0.0.1:64821 ESTABLISHED 204336
TCP 127.0.0.1:64821 127.0.0.1:64820 ESTABLISHED 204336
TCP 127.0.0.1:64832 127.0.0.1:64833 ESTABLISHED 204336
TCP 127.0.0.1:64833 127.0.0.1:64832 ESTABLISHED 204336
TCP 127.0.0.1:64862 127.0.0.1:64863 ESTABLISHED 204336
TCP 127.0.0.1:64863 127.0.0.1:64862 ESTABLISHED 204336
TCP 127.0.0.1:64869 127.0.0.1:64870 ESTABLISHED 204336
TCP 127.0.0.1:64870 127.0.0.1:64869 ESTABLISHED 204336
TCP 127.0.0.1:65294 127.0.0.1:65295 ESTABLISHED 204336
TCP 127.0.0.1:65295 127.0.0.1:65294 ESTABLISHED 204336
TCP 127.0.0.1:65296 127.0.0.1:65297 ESTABLISHED 204336
TCP 127.0.0.1:65297 127.0.0.1:65296 ESTABLISHED 204336
TCP 127.0.0.1:65298 127.0.0.1:65299 ESTABLISHED 204336
TCP 127.0.0.1:65299 127.0.0.1:65298 ESTABLISHED 204336
- Trace those socket references to their GC roots and you may find they are reference by I/O or task thread local selectors:
Class Name | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
------------------------------------------------------------------------------------------------------------------------------------------------
java.lang.Thread @ 0x4f894eee0 default task-16 Thread | 2 | 120 | 224 | 43,008
'- threadLocals java.lang.ThreadLocal$ThreadLocalMap @ 0x4fa1e3740 | 2 | 24 | 224 | 42,632
'- table java.lang.ThreadLocal$ThreadLocalMap$Entry[128] @ 0x4d3e984a0 | 2 | 528 | 224 | 42,608
'- [127] java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x4e2fe1018 | 2 | 32 | 224 | 32
'- value org.xnio.nio.NioXnio$FinalizableSelectorHolder @ 0x4e2fe1038| 2 | 16 | 224 | 14,592
'- selector sun.nio.ch.WindowsSelectorImpl @ 0x4e2fe1048 | 2 | 128 | 224 | 14,576
'- wakeupPipe sun.nio.ch.PipeImpl @ 0x4e2fe1350 | 2 | 24 | 224 | 1,200
|- sink sun.nio.ch.SinkChannelImpl @ 0x4e2fe1598 | 1 | 56 | 112 | 616
|- source sun.nio.ch.SourceChannelImpl @ 0x4e2fe1368 | 1 | 56 | 112 | 560
------------------------------------------------------------------------------------------------------------------------------------------------
Components
Category
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.