Undertow server port stops listening when socket accept error happens on EAP 7.0.5+

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP) 7.0.5+

Issue

  • JBoss is running but not accessible through the undertow server listener port (http, ajp, or https). Checking a netstat listing, we see the server socket is missing and no longer in a listening state.
  • After the exceptions below, connections are refused and http 8080 port stop listening: 
<http_8080 Accept><org.xnio.nio.tcp.server> Exception accepting request, closing server channel TCP server (NIO) <6fa28018>: java.net.SocketException: Bad file descriptor
        at sun.nio.ch.Net.localPort(Native Method)
        at sun.nio.ch.Net.localAddress(Net.java:479)
        at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:133)
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:266)
        at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:477)
        at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
: java.net.SocketException: Bad file descriptor
        at sun.nio.ch.Net.localPort(Native Method) [rt.jar:1.8.0_212]
        at sun.nio.ch.Net.localAddress(Net.java:479) [rt.jar:1.8.0_212]
        at sun.nio.ch.SocketChannelImpl.<init>(SocketChannelImpl.java:133) [rt.jar:1.8.0_212]
        at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:266) [rt.jar:1.8.0_212]
        at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:477)
        at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
        at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)

<http_8080 Accept><org.xnio.nio.tcp.server> Exception accepting request, closing server channel TCP server (NIO) <7fad216f>: java.io.IOException: Too many open files
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
	at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:477)
	at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)
: java.io.IOException: Too many open files
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method) [rt.jar:1.8.0_212]
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422) [rt.jar:1.8.0_212]
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250) [rt.jar:1.8.0_212]
	at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:477)
	at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)

Resolution

  • Increase your file descriptor limit (open files) to avoid exhaustion. Investigate and address any file/socket leak concerns.
  • Update to EAP 7.2 CP05 when available and enable the alternative queue acceptor adding the the JVM argument -Dxnio.nio.alt-queued-server=true. This property This content is not included.now defaults to true on 7.3 Update 3+.

Root Cause

Diagnostic Steps

  • Enable the following DEBUG logging in your logging subsystem to see info on any socket accept exceptions:

            <logger category="org.xnio.nio.tcp.server">
            <level name="DEBUG"/>
        </logger>

    then, you can see the following DEBUG log message when IOException happens at socket accept:

    DEBUG [org.xnio.nio.tcp.server] (default Accept) Exception accepting request, closing server channel TCP server (NIO) <7a2b6f37>: java.io.IOException: Too many open files
	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
	at org.xnio.nio.QueuedNioTcpServer.handleReady(QueuedNioTcpServer.java:477)
	at org.xnio.nio.QueuedNioTcpServerHandle.handleReady(QueuedNioTcpServerHandle.java:38)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)

    this DEBUG log message will log at ERROR level since EAP 7.2.2 or later by Content from issues.jboss.org is not included.JBEAP-16566.

  • To check for any file or socket leak concerns contributing to the issue, capture the following:

    • output of the following commands from the issue time:

      • netstat -vatnp > netstat.out
      • lsof -p $JBOSS_PID > lsof.out
      • cat /proc/$JBOSS_PID/limits > limits.out

    • A full heap dump from the issue time, which you can capture with your JDK's jmap command per <https://access.redhat.com/solutions/21109#jmapjavaprocess>:

        jmap -dump:format=b,file=heap.hprof JAVA_PID

    • Inspect states of org.wildfly.extension.undertow.HttpListenerService or org.wildfly.extension.undertow.HttpsListenerService. Check its underlying org.xnio.nio.QueuedNioTcpServer's sun.nio.ch.ServerSocketChannelImpl. Is this has a state of 1 or open value of false, it indicates it has been closed due to some prior issue. The following OQLs can check this concern:

SELECT sslServer.tcpServer.realServer.channel.open FROM org.wildfly.extension.undertow.HttpsListenerService 

SELECT sslServer.tcpServer.channel.open FROM org.wildfly.extension.undertow.HttpsListenerService 

SELECT server.channel.open FROM org.wildfly.extension.undertow.HttpListenerService 

SELECT server.channel.open FROM org.wildfly.extension.undertow.AjpListenerService
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.