JBoss thread/internal module by accident uses the external custom module
Environment
Red Hat JBoss Enterprise Application Platform 6.4.0_CP06
Issue
Occasionally, the error below is seen during the jboss start and EAR deployments. And it causes the EARs failing to deploy.
2016-06-23 19:46:20,872 ERROR [org.jboss.msc.service.fail] (ServerService Thread Pool -- 68) MSC000001: Failed to start service jboss.jgroups.channel.shmcoreserv: org.jboss.msc.service.StartException in service jboss.jgroups.channel.shmcoreserv: java.lang.ClassCastException: org.jgroups.protocols.UDP cannot be cast to org.jgroups.stack.Protocol
at org.jboss.as.clustering.jgroups.subsystem.ChannelService.start(ChannelService.java:75)
at org.jboss.as.clustering.msc.AsynchronousService$1.run(AsynchronousService.java:86)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_95]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_95]
at java.lang.Thread.run(Thread.java:745) [rt.jar:1.7.0_95]
at org.jboss.threads.JBossThread.run(JBossThread.java:122) [jboss-threads-2.1.2.Final-redhat-1.jar:2.1.2.Final-redhat-1]
Caused by: java.lang.ClassCastException: org.jgroups.protocols.UDP cannot be cast to org.jgroups.stack.Protocol
at org.jgroups.stack.Configurator.createLayer(Configurator.java:434) [jgroups-3.6.1.Final.jar:3.6.1.Final]
at org.jgroups.stack.Configurator.createProtocols(Configurator.java:388) [jgroups-3.6.1.Final.jar:3.6.1.Final]
at org.jgroups.stack.Configurator.setupProtocolStack(Configurator.java:55) [jgroups-3.6.1.Final.jar:3.6.1.Final]
at org.jgroups.stack..Configurator.setupProtocolStack(Configurator.java:88) [jgroups-3.6.1.Final.jar:3.6.1.Final]
at org.jgroups.stackProtocolStack.setup(ProtocolStack.java:465) [jgroups-3.6.1.Final.jar:3.6.1.Final]
at org.jgroups.JChannel.init(JChannel.java:786) [jgroups-3.6.1.Final.jar:3.6.1.Final]
at org.jgroups.JChannel.
at org.jboss.as.clustering.jgroups.MuxChannel.
at org.jboss.as.clustering.jgroups.JChannelFactory.createChannel(JChannelFactory.java:77)
at org.jboss.as.clustering.jgroups.subsystem.ChannelService.start(ChannelService.java:70)
... 5 more
It's a jboss thread, but from the 4th last line we see it's calling the jgroups-3.6.1.Final.jar.
jgroups-3.6.1.Final.jar is packaged in our own custom module - com/xxxxxxxx/yyy/zzzz/sdk/infinispan/7.1.1.Final. And the jgroups shipped with JBoss is org/jgroups/main/jgroups-3.2.13.Final-redhat-1.jar.
Resolution
Use EAP 6.4.0_CP15 as that has the fix ( https://bugzilla.redhat.com/show_bug.cgi?id=1430921 ) in it.
Root Cause
JGroups in EAP 6 uses the TCCL first when loading protocols, which may cause it to load the wrong class if the TCCL is not set correctly. A correction has been made to ensure the TCCL is set correctly and is safer
This does not affect upstream or EAP 7, as JGroups has changed the order it searches classloaders. The EAP 7 approach will not be backported because it results in a change in behaviour, thus breaks backward compatibility.
Diagnostic Steps
When this occurs interogate with a byteman script[1] to report on the classpath and possibly a tattletale report[2].
[1] https://access.redhat.com/solutions/31283
[2] https://access.redhat.com/solutions/800073
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.