JBoss EAP start up deadlocks on Elytron ExpressionResolver

Solution Verified - Updated 20 Jan 2025
Environment

JBoss Enterprise Application Platform (EAP)
- 7.x
- 8.x
Issue

We've configured JBoss to use an elytron expression resolver. After this, we randomly see startup hang until it times out and thread dumps show it has reached a deadlock between a RuntimeExpressionResolver HashSet lock and an ElytronDoohickey lock:
Found one Java-level deadlock:
=============================
"ServerService Thread Pool -- 63":
  waiting to lock monitor 0x00007fbf08013bf8 (object 0x00000006b32b4ca0, a java.util.HashSet),
  which is held by "ServerService Thread Pool -- 38"
"ServerService Thread Pool -- 38":
  waiting for ownable synchronizer 0x00000006b2f61d58, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
  which is held by "ServerService Thread Pool -- 42"
"ServerService Thread Pool -- 42":
  waiting to lock monitor 0x00007fbf08013bf8 (object 0x00000006b32b4ca0, a java.util.HashSet),
  which is held by "ServerService Thread Pool -- 38"

Java stack information for the threads listed above:
===================================================
"ServerService Thread Pool -- 63":
        at org.jboss.as.server.RuntimeExpressionResolver.resolvePluggableExpression(RuntimeExpressionResolver.java:115)
        - waiting to lock <0x00000006b32b4ca0> (a java.util.HashSet)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionString(ExpressionResolverImpl.java:348)
        at org.jboss.as.controller.ExpressionResolverImpl.parseAndResolve(ExpressionResolverImpl.java:253)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionStringRecursively(ExpressionResolverImpl.java:150)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionsRecursively(ExpressionResolverImpl.java:90)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressions(ExpressionResolverImpl.java:71)
        at org.jboss.as.controller.ModelControllerImpl.resolveExpressions(ModelControllerImpl.java:932)
        at org.jboss.as.controller.OperationContextImpl.resolveExpressions(OperationContextImpl.java:1297)
        at org.jboss.as.controller.ParallelBootOperationContext.resolveExpressions(ParallelBootOperationContext.java:446)
        at org.jboss.as.controller.AttributeDefinition$1.resolveExpressions(AttributeDefinition.java:603)
        at org.jboss.as.controller.AttributeDefinition.resolveValue(AttributeDefinition.java:667)
        at org.jboss.as.controller.AttributeDefinition.resolveModelAttribute(AttributeDefinition.java:626)
        at org.jboss.as.controller.AttributeDefinition.resolveModelAttribute(AttributeDefinition.java:600)
        at org.wildfly.extension.undertow.UndertowSubsystemAdd.performBoottime(UndertowSubsystemAdd.java:106)
        at org.jboss.as.controller.AbstractBoottimeAddStepHandler.performRuntime(AbstractBoottimeAddStepHandler.java:119)
        at org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:164)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1063)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:784)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:470)
        at org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:384)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348)
        at java.lang.Thread.run(Thread.java:750)
        at org.jboss.threads.JBossThread.run(JBossThread.java:513)
"ServerService Thread Pool -- 38":
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000006b2f61d58> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
        at org.wildfly.extension.elytron.ElytronDoohickey.apply(ElytronDoohickey.java:80)
        at org.wildfly.extension.elytron.ElytronDoohickey.apply(ElytronDoohickey.java:49)
        at org.wildfly.extension.elytron.expression.ElytronExpressionResolver.resolveCredentialStore(ElytronExpressionResolver.java:285)
        at org.wildfly.extension.elytron.expression.ElytronExpressionResolver.resolveExpressionInternal(ElytronExpressionResolver.java:121)
        at org.wildfly.extension.elytron.expression.ElytronExpressionResolver.resolveExpression(ElytronExpressionResolver.java:76)
        at org.jboss.as.controller.extension.ExtensionRegistry$FutureExpressionResolverExtension.resolveExpression(ExtensionRegistry.java:1288)
        at org.jboss.as.server.RuntimeExpressionResolver.resolveExpression(RuntimeExpressionResolver.java:152)
        at org.jboss.as.server.RuntimeExpressionResolver.resolvePluggableExpression(RuntimeExpressionResolver.java:118)
        - locked <0x00000006b32b4ca0> (a java.util.HashSet)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionString(ExpressionResolverImpl.java:348)
        at org.jboss.as.controller.ExpressionResolverImpl.parseAndResolve(ExpressionResolverImpl.java:253)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionStringRecursively(ExpressionResolverImpl.java:150)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionsRecursively(ExpressionResolverImpl.java:90)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressions(ExpressionResolverImpl.java:71)
        at org.jboss.as.controller.ModelControllerImpl.resolveExpressions(ModelControllerImpl.java:932)
        at org.jboss.as.controller.OperationContextImpl.resolveExpressions(OperationContextImpl.java:1297)
        at org.jboss.as.controller.ParallelBootOperationContext.resolveExpressions(ParallelBootOperationContext.java:446)
        at org.jboss.as.controller.AttributeDefinition$1.resolveExpressions(AttributeDefinition.java:603)
        at org.jboss.as.controller.AttributeDefinition.resolveValue(AttributeDefinition.java:667)
        at org.jboss.as.controller.AttributeDefinition.resolveModelAttribute(AttributeDefinition.java:626)
        at org.jboss.as.controller.AttributeDefinition.resolveModelAttribute(AttributeDefinition.java:600)
        at org.jboss.as.connector.subsystems.datasources.ConnectionPropertyAdd.performRuntime(ConnectionPropertyAdd.java:55)
        at org.jboss.as.controller.AbstractAddStepHandler.performRuntime(AbstractAddStepHandler.java:343)
        at org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:164)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1063)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:784)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:470)
        at org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:384)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348)
        at java.lang.Thread.run(Thread.java:750)
        at org.jboss.threads.JBossThread.run(JBossThread.java:513)
"ServerService Thread Pool -- 42":
        at org.jboss.as.server.RuntimeExpressionResolver.resolvePluggableExpression(RuntimeExpressionResolver.java:115)
        - waiting to lock <0x00000006b32b4ca0> (a java.util.HashSet)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionString(ExpressionResolverImpl.java:348)
        at org.jboss.as.controller.ExpressionResolverImpl.parseAndResolve(ExpressionResolverImpl.java:253)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionStringRecursively(ExpressionResolverImpl.java:150)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressionsRecursively(ExpressionResolverImpl.java:90)
        at org.jboss.as.controller.ExpressionResolverImpl.resolveExpressions(ExpressionResolverImpl.java:71)
        at org.jboss.as.controller.ModelControllerImpl.resolveExpressions(ModelControllerImpl.java:932)
        at org.jboss.as.controller.OperationContextImpl.resolveExpressions(OperationContextImpl.java:1297)
        at org.jboss.as.controller.ParallelBootOperationContext.resolveExpressions(ParallelBootOperationContext.java:446)
        at org.jboss.as.controller.AttributeDefinition$1.resolveExpressions(AttributeDefinition.java:603)
        at org.jboss.as.controller.AttributeDefinition.resolveValue(AttributeDefinition.java:667)
        at org.jboss.as.controller.AttributeDefinition.resolveModelAttribute(AttributeDefinition.java:626)
        at org.jboss.as.controller.AttributeDefinition.resolveModelAttribute(AttributeDefinition.java:600)
        at org.wildfly.extension.elytron.SecretKeyCredentialStoreDefinition$SecretKeyDoohickey.resolveRuntime(SecretKeyCredentialStoreDefinition.java:230)
        at org.wildfly.extension.elytron.ElytronDoohickey.resolveRuntime(ElytronDoohickey.java:150)
        at org.wildfly.extension.elytron.ElytronDoohickey.resolveRuntime(ElytronDoohickey.java:135)
        at org.wildfly.extension.elytron.DoohickeyAddHandler.performRuntime(DoohickeyAddHandler.java:74)
        at org.jboss.as.controller.AbstractAddStepHandler$1.execute(AbstractAddStepHandler.java:164)
        at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:1063)
        at org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:784)
        at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:470)
        at org.jboss.as.controller.ParallelBootOperationStepHandler$ParallelBootTask.run(ParallelBootOperationStepHandler.java:384)
        at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
        at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1348)
        at java.lang.Thread.run(Thread.java:750)
        at org.jboss.threads.JBossThread.run(JBossThread.java:513)

Found 1 deadlock.
Resolution

As a workaround, avoid the use of expressions in the expression resolover or its credential store.
Root Cause

An expression is used for an attribute of the expression resolver or its credential store like the path.
This content is not included.WFCORE-7139
This content is not included.JBEAP-28895
This content is not included.JBEAP-28894
This content is not included.JBEAP-28893
SBR
JBoss Base AS
Product(s)
Red Hat JBoss Enterprise Application Platform
Components
jbossas
Category
Troubleshoot
Tags
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.