High CPU in ScheduledThreadPoolExecutors after they are shutdown

Environment

• JBoss Enterprise Application Platform (EAP)

Issue

• We're seeing high CPU in java and it looks like it is caused by ScheduledThreadPoolExecutor that has been shutdown.
• JBoss is running EJB timers that show high CPU persisting in the following calls:

"EJB-Timer-1 [target=jboss.j2ee:ear=myapp.ear,jar=myapp.jar,name=MyEJB,service=EJB3]" prio=10 tid=0x00002aaac0141800 nid=0x3d7 runnable [0x00000000541cc000]
   java.lang.Thread.State: RUNNABLE
	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.poll(ScheduledThreadPoolExecutor.java:602)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:943)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
	at java.lang.Thread.run(Thread.java:662)

Resolution

• This fix should available in the latest java 7 update.
• If needed, upgrade to EAP 5.2.0+ and EAP 6.0.0+, which are java 7 compatible.

Root Cause

• The problem is here in the executor source:

  Runnable getTask()
  {
    while (true)
      try
      {
        int i = this.runState;
        if (i > 1)
          return null;
        Runnable localRunnable;
        if (i == 1)
          localRunnable = (Runnable)this.workQueue.poll();    // line 943
        else if ((this.poolSize > this.corePoolSize) || (this.allowCoreThreadTimeOut))
          localRunnable = (Runnable)this.workQueue.poll(this.keepAliveTime, TimeUnit.NANOSECONDS);
        else
          localRunnable = (Runnable)this.workQueue.take();
        if (localRunnable != null)
          return localRunnable;
        if (workerCanExit()) {
          if (this.runState >= 1)
            interruptIdleWorkers();
          return null;
        }

        continue;
      } catch (InterruptedException localInterruptedException) {
      }
  }

• When runstate is 1 (meaning the executor has been shutdown), the worker threads poll the workQueue continuously with no wait until the queue is empty. With a scheduled executor, workQueue.poll() won't return anything until an entry has passed its scheduled time and is ready for execution. Thus it can take quite some time for old scheduled tasks to be cleared out and the scheduled executor's worker threads chew up the CPU polling the queue until that time.
• This is caused by a Content from bugs.sun.com is not included.JDK bug

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.