"AMQ229014: Did not receive data from invm:0 within the -1ms connection TTL" occurs due to a race condition in JBoss EAP 7.x / 8.0.x

Solution Verified - Updated 17 May 2024

Environment

Red Hat JBoss Enterprise Application Platform (EAP) 7.x / 8.0.x
ActiveMQ Artemis

Issue

The following WARN `AMQ212037: Connection failure to invm:0 has been detected: AMQ229014: Did not receive data from invm:0 within the -1ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]" can occur due to a race condition.

WARN  [org.apache.activemq.artemis.core.client] (Thread-7 (ActiveMQ-server-org.apache.activemq.artemis.core.server.impl.ActiveMQServerImpl$6@2c1c0981)) AMQ212037: Connection failure to invm:0 has been detected: AMQ229014: Did not receive data from invm:0 within the -1ms connection TTL. The connection will now be closed. [code=CONNECTION_TIMEDOUT]

Also, the following ERROR message due to "ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219010: Connection is destroyed]" can happen at the same time:

ERROR [io.undertow.request] (default task-1) UT005023: Exception handling request to /helloworld-jms/producer: javax.jms.JMSRuntimeException: AMQ219010: Connection is destroyed
  at org.apache.activemq.artemis.jms.client.JmsExceptionUtils.convertToRuntimeException(JmsExceptionUtils.java:88)
  at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:326)
  at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:309)
  at org.wildfly.extension.messaging.activemq.deployment.injection.AbstractJMSContext.createContext(AbstractJMSContext.java:60)
  at org.wildfly.extension.messaging.activemq.deployment.injection.AbstractJMSContext.getContext(AbstractJMSContext.java:48)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)
  at org.jboss.weld.bean.proxy.AbstractBeanInstance.invoke(AbstractBeanInstance.java:38)
  at org.jboss.weld.bean.proxy.ProxyMethodHandler.invoke(ProxyMethodHandler.java:106)
  at org.wildfly.extension.messaging.activemq.deployment.injection.RequestedJMSContext$Proxy$_$$_WeldClientProxy.getContext(Unknown Source)
  at org.wildfly.extension.messaging.activemq.deployment.injection.InjectedJMSContext.getDelegate(InjectedJMSContext.java:92)
  at org.wildfly.extension.messaging.activemq.deployment.injection.JMSContextWrapper.createProducer(JMSContextWrapper.java:66)
  at org.jboss.as.quickstarts.servlet.JMSProducerServlet.doGet(JMSProducerServlet.java:102)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:503)
  at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
  at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
  at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
  at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
  at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
  at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
  at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
  at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
  at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
  at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
  at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
  at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
  at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
  at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
  at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:275)
  at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79)
  at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134)
  at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131)
  at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
  at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
  at org.wildfly.extension.undertow.security.SecurityContextThreadSetupAction.lambda$create$0(SecurityContextThreadSetupAction.java:105)
  at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1555)
  at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1555)
  at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1555)
  at org.wildfly.extension.undertow.deployment.UndertowDeploymentInfoService$UndertowThreadSetupAction.lambda$create$0(UndertowDeploymentInfoService.java:1555)
  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:255)
  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79)
  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100)
  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852)
  at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
  at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:1990)
  at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1486)
  at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1377)
  at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
  at java.lang.Thread.run(Thread.java:750)
Caused by: javax.jms.JMSException: AMQ219010: Connection is destroyed
  at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:465)
  at org.apache.activemq.artemis.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:439)
  at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:300)
  at org.apache.activemq.artemis.core.protocol.core.impl.ActiveMQClientProtocolManager.createSessionContext(ActiveMQClientProtocolManager.java:249)
  at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionChannel(ClientSessionFactoryImpl.java:1415)
  at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSessionInternal(ClientSessionFactoryImpl.java:712)
  at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl.createSession(ClientSessionFactoryImpl.java:318)
  at org.apache.activemq.artemis.jms.client.ActiveMQConnection.authorize(ActiveMQConnection.java:684)
  at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createConnectionInternal(ActiveMQConnectionFactory.java:920)
  at org.apache.activemq.artemis.jms.client.ActiveMQConnectionFactory.createContext(ActiveMQConnectionFactory.java:321)
  ... 59 more
Caused by: ActiveMQNotConnectedException[errorType=NOT_CONNECTED message=AMQ219010: Connection is destroyed]
  ... 69 more

Resolution

This is a race condition issue which has been reported as This content is not included.JBEAP-26395 for JBoss EAP 7.x, This content is not included.JBEAP-26490 for JBoss EAP 8.0.x, and Content from issues.apache.org is not included.ARTEMIS-4571 for upstream. This will be fixed in future releases (tentatively JBoss EAP 7.4.16+ and JBoss EAP 8.0.2+).
A workaround is to set connection-ttl to a long enough time that never expires in <connection-factory> for in-vm in the JBoss EAP configuration file (standalone-full-*.xml or domain.xml). Just for example, connection-ttl="31536000000" (1 year):
```
<connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm" connection-ttl="31536000000"/>
```
Note: connection-ttl can accept "long" value. However, since the timeout checker makes comparisons with long >= long + long as in if (now >= entry.lastCheck + entry.ttl) where now and lastCheck come from System.currentTimeMillis(), the right side will overflow when connection-ttl is huge (e.g. close to Long.MAX_VALUE), and the connection timeout always happens. Therefore, connection-ttl should be a reasonable time (large enough that never expires for long-running, but small enough to be safe from overflow) for the workaround.

Root Cause

This content is not included.JBEAP-26395 for JBoss EAP 7.4.x
This content is not included.JBEAP-26490 for JBoss EAP 8.0.x
Content from issues.apache.org is not included.ARTEMIS-4571

SBR

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

Category

Supportability

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.