Intermittent problems after updating to EAP 6.4 CP17

Solution Verified - Updated 14 Jun 2024

Environment

Red Hat JBoss Enterprise Application Platform (EAP)
- 6.4 CP17

Issue

jboss-cli hangs after update to JBoss EAP 6.4 CP17
Loss of jms messages when posting into a local queue
jboss-cli randomly fails to connect to the controller, JBREM000201: Received invalid message on Remoting connection

Server instances will not boot (and sometimes not shutdown) properly. The server will get a timeout when trying to connect to the host controller locally (port 9999) while booting the server instance, and fail the boot process.

  [Server:server-two] 15:17:17,337 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 2) MSC000001: Failed to start service jboss.server-boot-operations: org.jboss.msc.service.StartException in service jboss.server-boot-operations: java.net.ConnectException: JBAS012174: Could not connect to remote://127.0.0.1:9999. The connection failed
  [Server:server-two] 	at org.jboss.as.server.mgmt.domain.ServerBootOperationsService$1.run(ServerBootOperationsService.java:73) [jboss-as-server-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]
  [Server:server-two] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_92]
  [Server:server-two] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_92]
  [Server:server-two] 	at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_92]
  [Server:server-two] 	at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1]
  [Server:server-two] Caused by: java.net.ConnectException: JBAS012174: Could not connect to remote://127.0.0.1:9999. The connection failed
  [Server:server-two] 	at org.jboss.as.protocol.ProtocolConnectionUtils.connectSync(ProtocolConnectionUtils.java:133) [jboss-as-protocol-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]
  [Server:server-two] 	at org.jboss.as.protocol.ProtocolConnectionManager$EstablishingConnection.connect(ProtocolConnectionManager.java:256) [jboss-as-protocol-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]
  [Server:server-two] 	at org.jboss.as.protocol.ProtocolConnectionManager.connect(ProtocolConnectionManager.java:70) [jboss-as-protocol-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]
  [Server:server-two] 	at org.jboss.as.server.mgmt.domain.HostControllerConnection.openConnection(HostControllerConnection.java:130) [jboss-as-server-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]
  [Server:server-two] 	at org.jboss.as.server.mgmt.domain.HostControllerClient.resolveBootUpdates(HostControllerClient.java:107) [jboss-as-server-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]
  [Server:server-two] 	at org.jboss.as.server.mgmt.domain.ServerBootOperationsService$1.run(ServerBootOperationsService.java:69) [jboss-as-server-7.5.17.Final-redhat-4.jar:7.5.17.Final-redhat-4]
  [Server:server-two] 	... 4 more
  [Server:server-two] Caused by: java.io.IOException: JBREM000201: Received invalid message on Remoting connection 78baee97 to localhost/127.0.0.1:9999
  [Server:server-two] 	at org.jboss.remoting3.remote.ClientConnectionOpenListener$Capabilities.handleEvent(ClientConnectionOpenListener.java:530) [jboss-remoting-3.3.10.Final-redhat-1.jar:3.3.10.Final-redhat-1]

Continuous GC after upgrading to EAP 6.4.17

  "Remoting "svc-4-comecimpolicy" read-1" prio=10 tid=0x00007f60f81c2800 nid=0xf7f runnable [0x00007f60e4c13000] java.lang.Thread.State: RUNNABLE
  at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
  at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
  at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
  at sun.nio.ch.IOUtil.read(IOUtil.java:197)
  at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:384)
  - locked <0x000000078e662e48> (a java.lang.Object)
  at org.xnio.nio.AbstractNioStreamChannel.read(AbstractNioStreamChannel.java:249)
  at org.jboss.remoting3.remote.RemotingMessageChannel.receive(RemotingMessageChannel.java:126)
  - locked <0x000000078e663070> (a java.lang.Object)
  at org.jboss.remoting3.remote.ServerConnectionOpenListener$Initial.handleEvent(ServerConnectionOpenListener.java:243)
  - locked <0x000000078e6630a0> (a java.util.ArrayDeque)
  at org.jboss.remoting3.remote.ServerConnectionOpenListener$Initial.handleEvent(ServerConnectionOpenListener.java:140)
  at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
  at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
  at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
  at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
  at org.xnio.nio.NioHandle.run(NioHandle.java:90)
  at org.xnio.nio.WorkerThread.run(WorkerThread.java:198)

Very high CPU after CP17

  "Remoting "DEVPC028118_2100:MANAGEMENT" read-1" Id=98 RUNNABLE
  at java.lang.Object.<init>(Object.java:37)
  at org.xnio.ByteBufferSlicePool$PooledByteBuffer.<init>(ByteBufferSlicePool.java:111)
  at org.xnio.ByteBufferSlicePool.allocate(ByteBufferSlicePool.java:98)
  at org.jboss.remoting3.remote.RemoteConnection.allocate(RemoteConnection.java:76)
  at org.jboss.remoting3.remote.ServerConnectionOpenListener$Initial.handleEvent(ServerConnectionOpenListener.java:223)
  at org.jboss.remoting3.remote.ServerConnectionOpenListener$Initial.handleEvent(ServerConnectionOpenListener.java:140)
  at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
  at org.xnio.channels.TranslatingSuspendableChannel.handleReadable(TranslatingSuspendableChannel.java:189)
  at org.xnio.channels.TranslatingSuspendableChannel$1.handleEvent(TranslatingSuspendableChannel.java:103)
  at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:72)
  at org.xnio.nio.NioHandle.run(NioHandle.java:90)
  at org.xnio.nio.WorkerThread.run(WorkerThread.java:198)

Resolution

Apply CP 18 (6.4.18) or later. CP 18 can be downloaded from here: <This content is not included.https://access.redhat.com/jbossnetwork/restricted/softwareDetail.html?softwareId=54561&product=appplatform&version=6.4&downloadType=patches>

Root Cause

Bug fix in CP17 caused the regression:This content is not included.BZ-1491965 which is fixed in CP18.

SBR

JBoss Base AS

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Learn more

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.