Monitoring clustered level 2 cache invalidation for Hibernate / Infinispan in EAP 6

Environment

• Red Hat JBoss Enterprise Application Platform (EAP) 6
• Hibernate 4
• Infinispan

Issue

• Trace inbound / outbound communication between clustered servers for level 2 cache (L2C) invalidation
• Need to verify that L2C entity invalidation messages are broadcast and received by other nodes in a cluster

Resolution

Enable TRACE for org.infinispan (captures the most information) or narrow TRACE to the following categories:

• org.infinispan.remoting.rpc.RpcManagerImpl
• org.infinispan.remoting.transport.jgroups.JGroupsTransport
• org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher
• org.infinispan.remoting.InboundInvocationHandlerImpl
• org.infinispan.commands.remote.BaseRpcInvokingCommand
• org.infinispan.interceptors.CallInterceptor
• org.infinispan.commands.write.InvalidateCommand

Diagnostic Steps

Outbound Log Entries

... TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (<thread id>) <local node>/hibernate broadcasting call InvalidateCommand{keys=[org.myorg.MyEvent#1]} to recipient list null
... TRACE [org.infinispan.remoting.transport.jgroups.JGroupsTransport] (<thread id>) dests=null, command=SingleRpcCommand{cacheName='...org.myorg.MyEvent', command=InvalidateCommand{keys=[org.myorg.MyEvent#1]}}, mode=SYNCHRONOUS, timeout=17500
... TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (<thread id>) Replication task sending SingleRpcCommand{cacheName='...org.myorg.MyEvent', command=InvalidateCommand{keys=[org.myorg.MyEvent#1]}} to single recipient <sending node>/hibernate with response mode GET_ALL
...
... TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (<thread id>) Response: null
... TRACE [org.infinispan.remoting.rpc.RpcManagerImpl] (<thread id>) Response(s) to SingleRpcCommand{cacheName='...org.myorg.MyEvent', command=InvalidateCommand{keys=[org.myorg.MyEvent#1]}} is {<receiving node>/hibernate=null}

Inbound Log Entries

... TRACE [org.infinispan.remoting.transport.jgroups.CommandAwareRpcDispatcher] (<thread id>) Attempting to execute command: SingleRpcCommand{cacheName='...org.myorg.MyEvent', command=InvalidateCommand{keys=[org.myorg.MyEvent#1]}} [sender=<sending node>/hibernate]
... TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (<thread id>) Calling perform() on SingleRpcCommand{cacheName='...org.myorg.MyEvent', command=InvalidateCommand{keys=[org.myorg.MyEvent#1]}}
... TRACE [org.infinispan.commands.remote.BaseRpcInvokingCommand] (<thread id>) Invoking command InvalidateCommand{keys=[org.myorg.MyEvent#1]}, with originLocal flag set to false
...
... TRACE [org.infinispan.interceptors.CallInterceptor] (<thread id>) Executing command: InvalidateCommand{keys=[org.myorg.MyEvent#1]}.
... TRACE [org.infinispan.commands.write.InvalidateCommand] (<thread id>) Invalidating keys [org.myorg.MyEvent#1]
...
... TRACE [org.infinispan.remoting.InboundInvocationHandlerImpl] (<thread id>) About to send back response null for command SingleRpcCommand{cacheName='...org.myorg.MyEvent', command=InvalidateCommand{keys=[org.myorg.MyEvent#1]}}

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.