JBoss EAP 7 http connector leaks many "can't identify protocol" sockets

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 7.0

Issue

Too many open files errors on JBoss EAP 7. The lsofoutput shows many can't identify protocol sockets.

Resolution

The cause for this issue is known Undertow bugs, therefore, Upgrade to EAP 7.0 CP5 or later that contain the fixes.

Root Cause

Those are known issues on Undertow, which was introduced on JBoss EAP 7:

• Content from issues.jboss.org is not included.UNDERTOW-926 - Undertow connection may not be closed on IO exception in some circumstances
• Content from issues.jboss.org is not included.JBEAP-7745 (JBoss EAP 7.1.0) - Undertow connection may not be closed on IO exception in some circumstances
• Content from issues.jboss.org is not included.JBEAP-7942 (JBoss EAP 7.0.z) - UNDERTOW-926 - Undertow connection may not be closed on IO exception in some circumstances

Diagnostic Steps

1. Capture lsof output from the issue and a heap dump.
2. Run the following OQL to check open http port states
3. Compare to the leaked connection counts:

SELECT s.localAddress.holder.port, s.isInputOpen, s.isOutputOpen, s.readyToConnect, s.socket.closed, s.socket.bound, s.socket.created, fdVal FROM sun.nio.ch.SocketChannelImpl s WHERE (s.localAddress.holder.port = $HTTP_PORT)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.