JBoss EAP 6 or 7 starts slowly when one of the mod_cluster proxies is down

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6.x
    • 7.x
  • Apache
    • mod_cluster 1.x

Issue

  • In EAP 6/7 when using more than one Apache in proxy-list, JBoss starts slowly if at least one Apache is down. From thread dumps, we see start up operations frequently persisting in calls like below:
"ServerService Thread Pool -- 72" #134 prio=5 os_prio=0 tid=0x00007f84581fa9f0 nid=0x6718 runnable [0x00007f83c1561000]
   java.lang.Thread.State: RUNNABLE
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
        - locked <0x00000000eb0a4700> (a java.io.InputStreamReader)
        at java.io.InputStreamReader.read(InputStreamReader.java:184)
        at java.io.BufferedReader.fill(BufferedReader.java:161)
        at java.io.BufferedReader.readLine(BufferedReader.java:324)
        - locked <0x00000000eb0a4700> (a java.io.InputStreamReader)
        at java.io.BufferedReader.readLine(BufferedReader.java:389)
        at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:529)
        at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:605)
        - locked <0x00000000fcaac598> (a org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler$Proxy)
        at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:429)
        at org.jboss.modcluster.ModClusterService.enable(ModClusterService.java:375)
        at org.jboss.modcluster.ModClusterService.start(ModClusterService.java:361)
        at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapter.onDeploymentStart(UndertowEventHandlerAdapter.java:136)
        - locked <0x00000000fcaac7a8> (a org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapter)
        at org.wildfly.extension.undertow.Host$1.invoke(Host.java:192)
        at org.wildfly.extension.undertow.UndertowService.fireEvent(UndertowService.java:249)
        - locked <0x00000000fcaa4598> (a java.util.Collections$SynchronizedList)
        at org.wildfly.extension.undertow.Host.registerDeployment(Host.java:189)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:103)
        at org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:82)
  • JBoss EAP starts slowly when one Apache is down

Resolution

  • In modcluster subsystem, the socket timeout is set to 20 seconds by default (see further details in This content is not included.Mod_cluster Subsystem). If you need you can set a lower value in modcluster subsystem by executing the following command via JBoss EAP CLI. Example using full profile:
/profile=full/subsystem=modcluster/mod-cluster-config=configuration/:write-attribute(name=socket-timeout,value=3)
  • Resulting xml in domain.xml:
            <subsystem xmlns="urn:jboss:domain:modcluster:1.2">
               <mod-cluster-config advertise-socket="modcluster" proxy-list="apache1:6666,apache2:6666" socket-timeout="3" connector="ajp">
                    <dynamic-load-provider>
                        <load-metric type="busyness"/>
                    </dynamic-load-provider>
               </mod-cluster-config>
            </subsystem>
  • Upgrade to EAP 7.1.2+ for improvement on mod_cluster's handling of a down proxy

Root Cause

  • One Apache set in the proxy-list is down for maintenance or does not respond.
  • This can be exasperated by a timing issue between periodic STATUS attempts from mod_cluster to its proxies and the enable-app attempts from app start ups. MCMPs should only be sent to a proxy in a good state. The one exception is the periodic retry INFO attempt that will see if the proxy is back up. When that happens, it places a bad proxy back in a OK state until that attempt finishes. So while such retry attempts take place during app start ups, the app start up is being delayed as it tries to enable itself on a bad proxy.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.