Deployment operation causes the domain controller to become non-responsive in EAP 6

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 6.4.x

Issue

• After a deployment with --force flag, domain is non responsive, no management operations can be run, need to restart the whole domain.

Resolution

• Upgrade to CP14 and set property 'org.jboss.msc.directionalExecutor'

    /system-property=org.jboss.msc.directionalExecutor:add(value=true)
  

Root Cause

• Force deployment on deployments with many optional dependencies and 3rd level dependencies

Diagnostic Steps

• Deploy a war with the --force option, and check:

  • Management console non responsive

  • Unable to run any CLI commands

  • Thread dumps show the Stability monitor running beyond the blocking.timeout setting

    "ServerService Thread Pool -- 160" prio=10 tid=0x00007fd2f4134000 nid=0x122f in Object.wait() [0x00007fd2caf3a000]
    java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    waiting on <0x000000055468a2d8> (a java.lang.Object)
    at java.lang.Object.wait(Object.java:461)
    at org.jboss.msc.service.StabilityMonitor.awaitStability(StabilityMonitor.java:354)
    locked <0x000000055468a2d8> (a java.lang.Object)
    at org.jboss.msc.service.StabilityMonitor.awaitStability(StabilityMonitor.java:297)
    at org.jboss.as.controller.ContainerStateMonitor.awaitStability(ContainerStateMonitor.java:125)
    at org.jboss.as.controller.ModelControllerImpl.awaitContainerStability(ModelControllerImpl.java:740)
    at org.jboss.as.controller.OperationContextImpl.ensureWriteLockForRuntime(OperationContextImpl.java:523)
    at org.jboss.as.controller.OperationContextImpl.removeService(OperationContextImpl.java:373)
    at org.jboss.as.server.deployment.DeploymentHandlerUtil$4.execute(DeploymentHandlerUtil.java:267)
    at org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:708)
    at org.jboss.as.controller.AbstractOperationContext.doCompleteStep(AbstractOperationContext.java:543)
    at org.jboss.as.controller.AbstractOperationContext.completeStepInternal(AbstractOperationContext.java:338)
    at org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:314)
    at org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1144)
    at org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:331)
    at org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:201)

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.