High CPU Usage By Infinispan In EAP 7

Solution Unverified - Updated

Environment

Red Hat JBoss Enterprise Application Platform (EAP)

  • 7.x

Issue

  • We are seeing high CPU utilization by Infinspan in EAP 7.
  • Infinispan uses high CPU in EAP 7

Resolution

This issue is under review. See Content from issues.jboss.org is not included.Content from issues.jboss.org is not included.https://issues.jboss.org/browse/JBEAP-5687 for more information.

Diagnostic Steps

  • During high CPU usage, obtain high CPU data coupled with thread dumps as described in https://access.redhat.com/solutions/24830.
  • Use the JVMPeg Lab to provided analysis of CPU data and thread dumps obtained in previous step.
  • Notice high CPU utilization threads:
"ForkJoinPool.commonPool-worker-18" #1405 daemon prio=5 os_prio=0 tid=0x00007f36fc3a6800 nid=0xe9c9 runnable [0x00007f3481946000]
java.lang.Thread.State: RUNNABLE
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.infinispan.commons.util.ReflectionUtil.invokeAccessibly(ReflectionUtil.java:168)
    at org.infinispan.distribution.group.GroupManagerImpl$GroupMetadataImpl.getGroup(GroupManagerImpl.java:53)
    at org.infinispan.distribution.group.GroupManagerImpl.getGroup(GroupManagerImpl.java:98)
    at org.infinispan.distribution.group.GroupFilter.accept(GroupFilter.java:23)
    at org.infinispan.filter.CompositeKeyFilter.accept(CompositeKeyFilter.java:31)
    at org.infinispan.container.DefaultDataContainer.lambda$executeTask$228(DefaultDataContainer.java:531)
    at org.infinispan.container.DefaultDataContainer$$Lambda$201/1045562576.accept(Unknown Source)
    at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ForEachMappingTask.compute(EquivalentConcurrentHashMapV8.java:4868)
    at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers:
- None

"ForkJoinPool.commonPool-worker-5" #1589 daemon prio=5 os_prio=0 tid=0x00007f363c35d000 nid=0xeadd runnable [0x00007f345979a000]
java.lang.Thread.State: RUNNABLE
    at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$BulkTask.advance(EquivalentConcurrentHashMapV8.java:4755)
    at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ForEachMappingTask.compute(EquivalentConcurrentHashMapV8.java:4867)
    at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers:
- None

"ForkJoinPool.commonPool-worker-16" #1782 daemon prio=5 os_prio=0 tid=0x00007f363c572000 nid=0xebdd runnable [0x00007f344d3d9000]
java.lang.Thread.State: RUNNABLE
    at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ForEachMappingTask.compute(EquivalentConcurrentHashMapV8.java:4868)
    at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers:
- None

"ForkJoinPool.commonPool-worker-25" #6395 daemon prio=5 os_prio=0 tid=0x00007f3950189000 nid=0xde30 runnable [0x00007f36ed51e000]
java.lang.Thread.State: RUNNABLE
    at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:946)
    at org.infinispan.distribution.group.GroupManagerImpl.getMetadata(GroupManagerImpl.java:128)
    at org.infinispan.distribution.group.GroupManagerImpl.getGroup(GroupManagerImpl.java:96)
    at org.infinispan.distribution.group.GroupFilter.accept(GroupFilter.java:23)
    at org.infinispan.filter.CompositeKeyFilter.accept(CompositeKeyFilter.java:31)
    at org.infinispan.container.DefaultDataContainer.lambda$executeTask$228(DefaultDataContainer.java:531)
    at org.infinispan.container.DefaultDataContainer$$Lambda$237/466301799.accept(Unknown Source)
    at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ForEachMappingTask.compute(EquivalentConcurrentHashMapV8.java:4868)
    at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers:
- None

"ForkJoinPool.commonPool-worker-30" #6532 daemon prio=5 os_prio=0 tid=0x00007f399426f000 nid=0xded6 runnable [0x00007f3605cdb000]
java.lang.Thread.State: RUNNABLE
    at org.infinispan.filter.CompositeKeyFilter.accept(CompositeKeyFilter.java:31)
    at org.infinispan.container.DefaultDataContainer.lambda$executeTask$228(DefaultDataContainer.java:531)
    at org.infinispan.container.DefaultDataContainer$$Lambda$237/466301799.accept(Unknown Source)
    at org.infinispan.commons.util.concurrent.jdk8backported.EquivalentConcurrentHashMapV8$ForEachMappingTask.compute(EquivalentConcurrentHashMapV8.java:4868)
    at java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:731)
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
    at java.util.concurrent.ForkJoinPool$WorkQueue.execLocalTasks(ForkJoinPool.java:1040)
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1058)
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157)
Locked ownable synchronizers:
- None
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.