One CLOSE_WAIT connection remains every mod_cluster MCMP STATUS request in JBoss EAP 6.x/7.x

Solution Unverified - Updated 17 Apr 2018

Environment

Red Hat JBoss Enterprise Application Platform (EAP) 6.x, 7.x
mod_cluster

Issue

mod_cluster DefaultMCMPHandler leaves one CLOSE_WAIT connection per each httpd/mod_cluster instance every MCMP STATUS request in JBoss EAP 6.x/7.x.
Anytime I checked nestat -tunapl command on JBoss machine, JBoss EAP CLOSE_WAIT holds one connection per each httpd/mod_cluster instance. It looks a CLOSE_WAIT connection remains 10 seconds and another new CLOSE_WAIT connection remains after the previous CLOSE_WAIT connection disappears:

$ while true; do date; sudo netstat -tunapl|grep <JBOSS_JAVA_PID>; echo "-----------"; echo ""; sleep 5; done
Tue Apr 17 14:39:53 JST 2018
tcp        0      0 127.0.0.1:9999          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:4447          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:9990          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:8009          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47152         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47166         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47148         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47136         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47142         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47158         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47172         ESTABLISHED 27379/java
tcp        1      0 127.0.0.1:38974         127.0.0.1:6666          CLOSE_WAIT  27379/java          // <-- !!
tcp        0      0 127.0.0.1:8009          127.0.0.1:47138         ESTABLISHED 27379/java
-----------

Tue Apr 17 14:39:58 JST 2018
tcp        0      0 127.0.0.1:9999          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:4447          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:9990          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:8009          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47152         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47166         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47148         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47136         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47142         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47158         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47172         ESTABLISHED 27379/java
tcp        1      0 127.0.0.1:38974         127.0.0.1:6666          CLOSE_WAIT  27379/java          // <-- !!
tcp        0      0 127.0.0.1:8009          127.0.0.1:47138         ESTABLISHED 27379/java
-----------

Tue Apr 17 14:40:03 JST 2018
tcp        0      0 127.0.0.1:9999          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:4447          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:9990          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:8009          0.0.0.0:*               LISTEN      27379/java
tcp        1      0 127.0.0.1:38990         127.0.0.1:6666          CLOSE_WAIT  27379/java          // <-- !!
tcp        0      0 127.0.0.1:8009          127.0.0.1:47152         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47166         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47148         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47136         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47142         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47158         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47172         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47138         ESTABLISHED 27379/java
-----------

Tue Apr 17 14:40:09 JST 2018
tcp        0      0 127.0.0.1:9999          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:4447          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:9990          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:8009          0.0.0.0:*               LISTEN      27379/java
tcp        1      0 127.0.0.1:38990         127.0.0.1:6666          CLOSE_WAIT  27379/java          // <-- !!
tcp        0      0 127.0.0.1:8009          127.0.0.1:47152         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47166         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47148         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47136         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47142         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47158         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47172         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47138         ESTABLISHED 27379/java
-----------

Tue Apr 17 14:40:14 JST 2018
tcp        0      0 127.0.0.1:9999          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:8080          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:4447          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:9990          0.0.0.0:*               LISTEN      27379/java
tcp        0      0 127.0.0.1:8009          0.0.0.0:*               LISTEN      27379/java
tcp        1      0 127.0.0.1:38996         127.0.0.1:6666          CLOSE_WAIT  27379/java          // <-- !!
tcp        0      0 127.0.0.1:8009          127.0.0.1:47152         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47166         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47148         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47136         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47142         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47158         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47172         ESTABLISHED 27379/java
tcp        0      0 127.0.0.1:8009          127.0.0.1:47138         ESTABLISHED 27379/java
-----------

Resolution

This CLOSE_WAIT connection will be cleaned when DefaultMCMPHandler sends the next MCMP STATUS request, so there's no functional impact. This is not a critical and harmless as neither connection leak nor file descriptor leak happens. As this is harmless, please just ignore this CLOSE_WAIT connection.
The following JIRA was opened to improve mod_cluster DefaultMCMPHandler#sendRequest() to handle "Connection: close" response header and close a connection correctly. CLOSE_WAIT connection will not remain in the future release of JBoss EAP 7.:
- Content from issues.jboss.org is not included.MODCLUSTER-653
- Content from issues.jboss.org is not included.JBEAP-14530

Root Cause

mod_cluster DefaultMCMPHandler#sendRequest() does not close a connection correctly when Apache httpd closes a connection and responds to MCMP STATUS request with "Connection: close" response header. (As KeepAlive Off is set by default in httpd.conf, Apache httpd closes a connection against every MCMP STATUS with "Connection: close" response header.) Therefore, CLOSE_WAIT connection remains after every MCMP STATUS command.

SBR

Webservers

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Learn more

Tags

jboss

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.