EAP 7 start up with many applications using mod_cluster delayed in org.wildfly.extension.undertow.UndertowService.fireEvent

Solution Unverified - Updated 14 Jun 2024

Environment

JBoss Enterprise Application Platform (EAP) 7.x

Issue

We use mod_cluster and deploy a large amount of applications on EAP 7. Start up is very slow. Thread dumps show deployments waiting here:

"ServerService Thread Pool -- 368" #4342 prio=5 os_prio=0 tid=0x000055fc03b25800 nid=0xdc1c waiting for monitor entry [0x00007f17c63cf000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.wildfly.extension.undertow.UndertowService.fireEvent(UndertowService.java:285)
	- waiting to lock <0x00000002cef91808> (a java.util.Collections$SynchronizedList)
	at org.wildfly.extension.undertow.Host.registerDeployment(Host.java:268)

The lock owner is blocked here:

"ServerService Thread Pool -- 168" #1860 prio=5 os_prio=0 tid=0x000055fc03708800 nid=0xd0d7 waiting for monitor entry [0x00007f17f0263000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:539)
	- waiting to lock <0x00000002c4b0ae38> (a org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler$Proxy)
	at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:373)
	at org.jboss.modcluster.ModClusterService.enable(ModClusterService.java:326)
	at org.jboss.modcluster.ModClusterService.start(ModClusterService.java:312)
	at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapterService.onStart(UndertowEventHandlerAdapterService.java:129)
	- locked <0x00000002c2ce8108> (a org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapterService)
	at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapterService.onDeploymentStart(UndertowEventHandlerAdapterService.java:145)
	at org.wildfly.extension.undertow.Host.lambda$registerDeployment$0(Host.java:268)
	at org.wildfly.extension.undertow.Host$$Lambda$1111/1883027328.invoke(Unknown Source)
	at org.wildfly.extension.undertow.UndertowService.fireEvent(UndertowService.java:286)
	- locked <0x00000002cef91808> (a java.util.Collections$SynchronizedList)
	at org.wildfly.extension.undertow.Host.registerDeployment(Host.java:268)
	at org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:99)

And that is blocked by this thread awaiting a mod_cluster STATUS update:

"UndertowEventHandlerAdapterService - 1" #106 prio=5 os_prio=0 tid=0x000055fbf83e1800 nid=0xc666 runnable [0x00007f1813e46000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
	at sun.security.ssl.InputRecord.read(InputRecord.java:503)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990)
	- locked <0x00000006ed984718> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948)
	at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
	- locked <0x00000006ed984948> (a sun.security.ssl.AppInputStream)
	at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
	at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
	at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
	- locked <0x00000006eda0bd30> (a java.io.InputStreamReader)
	at java.io.InputStreamReader.read(InputStreamReader.java:184)
	at java.io.BufferedReader.fill(BufferedReader.java:161)
	at java.io.BufferedReader.readLine(BufferedReader.java:324)
	- locked <0x00000006eda0bd30> (a java.io.InputStreamReader)
	at java.io.BufferedReader.readLine(BufferedReader.java:389)
	at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:468)
	at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:552)
	- locked <0x00000002c4b0ae38> (a org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler$Proxy)
	at org.jboss.modcluster.mcmp.impl.DefaultMCMPHandler.sendRequest(DefaultMCMPHandler.java:373)
	at org.jboss.modcluster.ModClusterService.status(ModClusterService.java:398)
	at org.wildfly.mod_cluster.undertow.UndertowEventHandlerAdapterService.run(UndertowEventHandlerAdapterService.java:178)

Resolution

To correct such delays, you can disable request queuing during start up on EAP 7.2+:

<host name="default-host" alias="localhost" queue-requests-on-start="false" default-response-code="503" >

Upgrading to 7.2.8+ or 7.3.1+ may help by avoiding unneeded context registration while the server is suspended per The application contexts are enabled to mod_cluster for servers that are started in suspended mode in JBoss EAP 7.2

Root Cause

If proxying via http, mod_cluster by default will attempt an OPTIONS ping request to JBoss with such STATUS updates. But EAP isn't fully started so the listener will queue the request. That delays the STATUS responses from httpd to JBoss so the large number of app deployments are delayed when they need to send ENABLE requests to httpd because the STATUS requests are often pending for long periods and blocking them.

SBR

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.