Timer stuck sporadically if a database persistence is used

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 6.x
    • 7.0.x

Issue

  • For a persistent timer there are some messages like below that the timer is not executed due to overlap with an already running one, this works normally, but sometimes the timer is not longer active after this. We need to restart the server to get this fixed.
11:50:53,001 WARN  [org.jboss.as.ejb3.timer] (EJB default - 7) WFLYEJB0043: A previous execution of timer [id=0203e061-b8da-4349-bd93-7f5316d59d6d timedObjectId=ejb31-timer.ejb31-timer.SimpleScheduleSingletonTimerBean auto-timer?:true persistent?:true timerService=org.jboss.as.ejb3.timerservice.TimerServiceImpl@58a23128 initialExpiration=null intervalDuration(in milli sec)=0 nextExpiration=Wed Jun 14 11:50:53 CEST 2017 timerState=IN_TIMEOUT info=null] is still in progress, skipping this overlapping scheduled execution at: Wed Jun 14 11:50:53 CEST 2017.
  • Seeing occasional issue with a persistent scheduler job updating next_date column in jboss_ejb_timer database table with past date. This causes scheduler job to not run again.

Resolution

The issue will be fixed with a CP release for

  • EAP 7.1
  • EAP 7.0.7 or later
  • EAP 6.4.16 or later

Root Cause

There is a race condition if a timer is executed on a node and take a longer time to run the timeout method.
If the timer finish at the same time the next schedule starts the state check and persistence update can cause an inconsistent state.

This is often caused by having to less threads to handle timers (and EJB remote access) at the same time or long running GC.
As workaround the timer can use a separate Thread-pool or tune the JVM memory settings.

This is tracked by
Content from issues.jboss.org is not included.EAP7.0 JBEAP-11519
This content is not included.EAP6 bug 1461416

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.