High CPU in sun.nio.ch.EPollArrayWrapper.epollWait in JBoss Data Grid 6.1

Environment

• Red Hat JBoss Data Grid (JDG)
  • 6.1
• Oracle Java
  • 7u40
• OpenJDK 1.8
  • Update 51

Issue

• Threads consuming high cpu have the following stack trace:

"A-index-8b" #62 prio=5 os_prio=0 tid=0x00007f8532a2d000 nid=0xdd83 runnable [0x00007f84f49b7000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x0000000737fc7a88> (a sun.nio.ch.Util$2)
        - locked <0x0000000737fc7a70> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00000007378e2cb8> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at com.abc.MyClass.select(MyClass.java:400)
        at com.abc.MyClass.run(MyClass.java:602)
        at java.lang.Thread.run(Thread.java:745)

• High CPU usage in JDG 6.1. There is the following thread in Thread Dump:

"HotRodServerWorker-161" prio=10 tid=0x00007f1995b89000 nid=0x13ab runnable [0x00007f17140bf000]
  java.lang.Thread.State: RUNNABLE
  at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
  at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
  at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
  at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
  - locked <0x00007f1ae34fc1e8> (a sun.nio.ch.Util$2)
  - locked <0x00007f1ae34fc1d8> (a java.util.Collections$UnmodifiableSet)
  - locked <0x00007f1ae34fc1f8> (a sun.nio.ch.EPollSelectorImpl)
  at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
  at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
  at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
  at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
  at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
  at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
  at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
  at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
  at java.lang.Thread.run(Thread.java:722)

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

One possible reason that there is a JDK issue in relation to Netty. If seeing such calls from netty, it may help setting the following JVM argument:

-Dorg.jboss.netty.epollBugWorkaround=true

This known issue related to Netty can be seen in Content from github.com is not included.this link.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.