Session blocked when using distributable applications and cross-context forward/include in EAP 7.1

Solution Verified - Updated 14 Jun 2024

Environment

JBoss Enterprise Application Platform (EAP)
- 7

Issue

When using distributable applications (session replication using <distributable/> in the web.xml) and doing cross-context access between them (one application includes or forwards to a resource from another application) the application gets blocked. The resulting exception is similar to:

  ERROR [io.undertow.request] (default task-11) UT005023: Exception handling request to /webapp1/servlet1: java.lang.RuntimeException: java.lang.RuntimeException: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key SessionCreationMetaDataKey(x9ah3VdYkbEentlDFomBMhRR_eilSE7UzlgPtVDI) and requestor GlobalTransaction:<null>:22:local. Lock is held by GlobalTransaction:<null>:14:local
          at io.undertow.servlet.spec.ServletContextImpl.invokeAction(ServletContextImpl.java:934)
          at io.undertow.servlet.spec.RequestDispatcherImpl.setupIncludeImpl(RequestDispatcherImpl.java:299)
          at io.undertow.servlet.spec.RequestDispatcherImpl.include(RequestDispatcherImpl.java:277)
          at com.example.Servlet1.service(Servlet1.java:48)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          ...
  Caused by: java.lang.RuntimeException: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key SessionCreationMetaDataKey(x9ah3VdYkbEentlDFomBMhRR_eilSE7UzlgPtVDI) and requestor GlobalTransaction:<null>:22:local. Lock is held by GlobalTransaction:<null>:14:local
          at io.undertow.servlet.spec.RequestDispatcherImpl.includeImpl(RequestDispatcherImpl.java:382)
          at io.undertow.servlet.spec.RequestDispatcherImpl.access$300(RequestDispatcherImpl.java:54)
          at io.undertow.servlet.spec.RequestDispatcherImpl$4.call(RequestDispatcherImpl.java:302)
          at io.undertow.servlet.spec.RequestDispatcherImpl$4.call(RequestDispatcherImpl.java:299)
          ...
  Caused by: org.infinispan.util.concurrent.TimeoutException: ISPN000299: Unable to acquire lock after 15 seconds for key SessionCreationMetaDataKey(x9ah3VdYkbEentlDFomBMhRR_eilSE7UzlgPtVDI) and requestor GlobalTransaction:<null>:22:local. Lock is held by GlobalTransaction:<null>:14:local
          at org.infinispan.util.concurrent.locks.impl.DefaultLockManager$KeyAwareExtendedLockPromise.lock(DefaultLockManager.java:239)
          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.lockAndRecord(AbstractLockingInterceptor.java:193)
          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.checkPendingAndLockKey(AbstractTxLockingInterceptor.java:193)
          at org.infinispan.interceptors.locking.AbstractTxLockingInterceptor.lockOrRegisterBackupLock(AbstractTxLockingInterceptor.java:116)
          at org.infinispan.interceptors.locking.PessimisticLockingInterceptor.visitDataReadCommand(PessimisticLockingInterceptor.java:71)
          at org.infinispan.interceptors.locking.AbstractLockingInterceptor.visitGetKeyValueCommand(AbstractLockingInterceptor.java:80)
          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
          at org.infinispan.interceptors.TxInterceptor.enlistReadAndInvokeNext(TxInterceptor.java:346)
          at org.infinispan.interceptors.TxInterceptor.visitGetKeyValueCommand(TxInterceptor.java:331)
          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
          at org.infinispan.interceptors.base.CommandInterceptor.invokeNextInterceptor(CommandInterceptor.java:99)
          at org.infinispan.interceptors.InvocationContextInterceptor.handleAll(InvocationContextInterceptor.java:110)
          at org.infinispan.interceptors.InvocationContextInterceptor.handleDefault(InvocationContextInterceptor.java:79)
          at org.infinispan.commands.AbstractVisitor.visitGetKeyValueCommand(AbstractVisitor.java:85)
          at org.infinispan.commands.read.GetKeyValueCommand.acceptVisitor(GetKeyValueCommand.java:43)
          at org.infinispan.interceptors.InterceptorChain.invoke(InterceptorChain.java:335)
          at org.infinispan.cache.impl.CacheImpl.get(CacheImpl.java:414)
          at org.infinispan.cache.impl.DecoratedCache.get(DecoratedCache.java:443)
          at org.infinispan.cache.impl.AbstractDelegatingCache.get(AbstractDelegatingCache.java:287)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.getValue(InfinispanSessionMetaDataFactory.java:74)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:64)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionMetaDataFactory.findValue(InfinispanSessionMetaDataFactory.java:36)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:60)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionFactory.findValue(InfinispanSessionFactory.java:38)
          at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:249)
          at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:209)
          at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:773)
          at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:368)
          at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:373)
          at com.example.Servlet2.service(Servlet2.java:32)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
          at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
          ...

Applications do not undeploy/stop, they are blocked in a similar stack to the one commented in solution: EAP 7.1 shutdown blocked by DistributableSessionManager.stop.

  "ServerService Thread Pool -- 107" #2334 prio=5 os_prio=0 tid=0x000000001aee0000 nid=0x53e8 waiting on condition [0x00007fad3b6f0000]
     java.lang.Thread.State: WAITING (parking)
      at sun.misc.Unsafe.park(Native Method)
      - parking to wait for  <0x00000006dc2b7ae8> (a java.util.concurrent.locks.StampedLock)
      at java.util.concurrent.locks.StampedLock.acquireWrite(StampedLock.java:1119)
      at java.util.concurrent.locks.StampedLock.writeLockInterruptibly(StampedLock.java:413)
      at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.stop(DistributableSessionManager.java:90)
      - locked <0x00000006dc2b7ac0> (a org.wildfly.clustering.web.undertow.session.DistributableSessionManager)
      ...

Resolution

The problem has been fixed in EAP 7.1.CP6. The combination of distributable sessions and cross-context forwards or includes triggers the problem, the only workaround right now is avoiding the use of both features at the same time.

Root Cause

The root cause is jira Content from issues.jboss.org is not included.JBEAP-15508. Tagged to be fixed in 7.1.CP6.

SBR

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

cluster

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.