EAP 6 start up stalls on EJB creation

Solution Verified - Updated 6 Aug 2024

Environment

Red Hat JBoss Enterprise Application Platform (EAP)
- 6.x

Issue

JBoss stalls during start up, with MSC threads stuck creating an EJB:

"MSC service thread 1-1" prio=10 tid=0x00007fd19464d800 nid=0xa9a in Object.wait() [0x00007fd18d631000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x000000009df05b00> (a org.jboss.as.ejb3.component.stateless.StatelessSessionComponent)
	at java.lang.Object.wait(Object.java:503)
	at org.jboss.as.ee.component.BasicComponent.waitForComponentStart(BasicComponent.java:109)
	- locked <0x000000009df05b00> (a org.jboss.as.ejb3.component.stateless.StatelessSessionComponent)
	at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:127)
	at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:85)
	at org.jboss.as.ejb3.component.stateless.StatelessSessionComponent$1.create(StatelessSessionComponent.java:67)
	at org.jboss.as.ejb3.component.stateless.StatelessSessionComponent$1.create(StatelessSessionComponent.java:64)
	at org.jboss.as.ejb3.pool.AbstractPool.create(AbstractPool.java:60)
	at org.jboss.as.ejb3.pool.strictmax.StrictMaxPool.get(StrictMaxPool.java:123)

Application are not started correctly if there are multiple SingletonServices, the server hung with the following thread

"MSC service thread 1-1" prio=10 tid=0x00007ff3b04a5000 nid=0x40a0 in Object.wait() [0x00007ff399ad6000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000c92d4100> (a org.jboss.as.ejb3.component.singleton.SingletonComponent)
	at java.lang.Object.wait(Object.java:503)
	at org.jboss.as.ee.component.BasicComponent.waitForComponentStart(BasicComponent.java:117)
	- locked <0x00000000c92d4100> (a org.jboss.as.ejb3.component.singleton.SingletonComponent)
	at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:147)
	at org.jboss.as.ee.component.BasicComponent.constructComponentInstance(BasicComponent.java:135)
	at org.jboss.as.ee.component.BasicComponent.createInstance(BasicComponent.java:90)
	at org.jboss.as.ejb3.component.singleton.SingletonComponent.getComponentInstance(SingletonComponent.java:122)
	- locked <0x00000000c92d43a8> (a java.lang.Object)
	at org.jboss.as.ejb3.component.singleton.SingletonComponentInstanceAssociationInterceptor.processInvocation(SingletonComponentInstanceAssociationInterceptor.java:47)

Resolution

Correct any bad cyclical depedencies
Ensure resources your EJBs depend on successfully bind in JNDI and ensure your JNDI name for your EJB resource references are correct
If there are no idle MSC threads visible in the dump, increase your MSC thread pool (default size is 2 * #ofCPUcores:

-Dorg.jboss.server.bootstrap.maxThreads=10

Root Cause

EJB start is stalled awaiting dependencies. This could be due to:
- Bad unresolvable/cyclical dependencies
  - Due to Content from issues.jboss.org is not included.WFLY-917, JBoss start can stall on bad EJB dependencies rather than erroring out
- Incorrect JNDI names used in resource dependencies or expected resource is never bound in JDNI
- No available MSC threads to process further start up events that would resolve those depedencies
  - a cause might be the deployment order in the application, the 'core' is last in the order but other modules depend on this are started earlier in the sequence

Diagnostic Steps

Review dependencies established in ejb-jar.xmls and jboss-ejb3.xmls
Take a thread dump and check whether there are MSC service threads waiting on dependencies
Capture a CLI service dump, for example to check for missing dependencies:

/host=master/server=server-one/core-service=service-container:dump-services

SBR

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Troubleshoot

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.