JMS thread hangs on socketWrite during client message delivery

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)
    • JBoss Messaging
    • JBoss Remoting

Issue

  • We can see a JMS thread hang for quite some time on a socketWrite during a client message delivery:
    "WorkerThread#14[127.0.0.1:50175]" prio=10 tid=0x000000005c708000 nid=0x590 runnable [0x0000000047dd9000]
   java.lang.Thread.State: RUNNABLE
     at java.net.SocketOutputStream.socketWrite0(Native Method)
     at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
     at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
     at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
     at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
     - locked <0x00000000c02a38a0> (a java.io.BufferedOutputStream)
     at java.io.DataOutputStream.flush(DataOutputStream.java:106)
     at org.jboss.jms.wireformat.ClientDelivery.write(ClientDelivery.java:93)
     at org.jboss.jms.wireformat.JMSWireFormat.write(JMSWireFormat.java:237)
     at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.versionedWrite(MicroSocketClientInvoker.java:940)
     at org.jboss.remoting.transport.socket.MicroSocketClientInvoker.transport(MicroSocketClientInvoker.java:575)
     at org.jboss.remoting.transport.bisocket.BisocketClientInvoker.transport(BisocketClientInvoker.java:422)
     at org.jboss.remoting.MicroRemoteClientInvoker.invoke(MicroRemoteClientInvoker.java:141)
     at org.jboss.remoting.Client.invoke(Client.java:1858)
     at org.jboss.remoting.Client.invoke(Client.java:718)
     at org.jboss.remoting.Client.invokeOneway(Client.java:769)
     at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallback(ServerInvokerCallbackHandler.java:829)
     at org.jboss.remoting.callback.ServerInvokerCallbackHandler.handleCallbackOneway(ServerInvokerCallbackHandler.java:700)
     at org.jboss.jms.server.endpoint.ServerSessionEndpoint.performDelivery(ServerSessionEndpoint.java:1467)
     at org.jboss.jms.server.endpoint.ServerSessionEndpoint.handleDelivery(ServerSessionEndpoint.java:1379)
     - locked <0x00000000c02fcd98> (a org.jboss.jms.server.endpoint.ServerSessionEndpoint)
     at org.jboss.jms.server.endpoint.ServerConsumerEndpoint.handle(ServerConsumerEndpoint.java:327)
     - locked <0x00000000c034c078> (a java.lang.Object)
     at org.jboss.messaging.core.impl.RoundRobinDistributor.handle(RoundRobinDistributor.java:119)
     at org.jboss.messaging.core.impl.MessagingQueue$DistributorWrapper.handle(MessagingQueue.java:583)
     at org.jboss.messaging.core.impl.ClusterRoundRobinDistributor.handle(ClusterRoundRobinDistributor.java:79)
     at org.jboss.messaging.core.impl.ChannelSupport.deliverInternal(ChannelSupport.java:670)
     at org.jboss.messaging.core.impl.MessagingQueue.deliverInternal(MessagingQueue.java:506)
     at org.jboss.messaging.core.impl.ChannelSupport.handle(ChannelSupport.java:251)
     - locked <0x00000000c034c370> (a java.lang.Object)
     at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.routeInternal(MessagingPostOffice.java:2228)
     at org.jboss.messaging.core.impl.postoffice.MessagingPostOffice.route(MessagingPostOffice.java:515)
     at org.jboss.jms.server.endpoint.ServerConnectionEndpoint.sendMessage(ServerConnectionEndpoint.java:777)

Resolution

  • JBoss Remoting 2.2.3.SP1 and later now supports the configuration of socket write timeouts through remoting-biscoket-service.xml.  Set this accordingly to timeout slow/hanging socket writes in a timely fashion.
  • EAP 4.3.0_CP06 uses JBoss Remoting 2.2.3 so EAP 4.3.0_CP06 and earlier does not support this write timeout.
  • If on a version that does not support this write timeout, upgrade to a later version that does or configure the OS side TCP/IP socket timeouts to a lower value to time these threads out in a timely fashion.

Root Cause

  • The socket by default has no write timeout and so this can hang indefinitely or until the OS TCP/IP socket timeouts cause it to error out.

Diagnostic Steps

SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.