Stale Hibernate entities are found in some clustered nodes in EAP

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 5
  • 6
• Hibernate
  • Hibernate 3 / JBossCache
  • Hibernate 4 / Infinispan

Issue

• Clustering has been configured for multiple server JVMs
• Level 2 caching is enabled for persistence units deployed in each server
• When entities are concurrently accessed in multiple server JVMs, stale entities are randomly found in one or more JVMs
• Once stale, entities remain in the level 2 cache until they are explicitly or implicitly (e.g. due to timeout) evicted

Resolution

One of the following approaches may be taken to address the issue

• In EAP 6.4.0 CP07 and later 6.4 releases, setting the system property infinispan.invalidation.on_commit to true will shift the timing of invalidate to the point that the database commit is complete and updated data may be safely read. Note that this is a weaker guarantee than pessimistic locking but it will prevent "indefinitely stale" states.
• Stale entities may be explicitly refreshed by calling Content from docs.oracle.com is not included.EntityManager.refresh(Object) outside a transaction
• In JPA 2.0 applications, stale entities may be explicitly evicted by calling Content from docs.oracle.com is not included.Cache.evict(Class type, Object key)
  • The (level 2) cache instance may be obtained via the Content from docs.oracle.com is not included.EntityManagerFactory
• For some databases, it may be possible to prevent stale reads by using a transaction isolation level of REPEATABLE_READ or SERIALIZABLE for EAP 5 datasources and EAP 6 datasources
  • This will cause a database read between the SQL row update and commit to block until the database commit completes which should prevent (re-)load of stale state
  • The behavior of reads in REPEATABLE_READ and SERIALIZABLE isolation may vary with database
  • For some databases, it may be necessary to perform reads with explicit locks to avoid stale reads
• EAP 7.0/Hibernate 5.0/Infinispan 8.1 and later are Content from hibernate.atlassian.net is not included.not susceptible to this vulnerability.
• See also StaleObjectStateException for discussion of a related issue and solution

Root Cause

When a transaction commits changes for an entity it performs the following, in order:

1. Execute SQL update against the database
2. Publish invalidation message to all clustered servers
3. Perform database commit

If logic running in a different server (from the transaction above) loads the entity from the database between steps 2 and 3 above (i.e. after evict has occurred locally in response to the invalidation message but before the commit completes) it is possible to load (or re-load) the entity in the exact state in which it exists prior to the database commit above.

Since no subsequent action takes place, the newly loaded (or re-loaded) entity becomes stale the instant the database commit initiated by the updating server completes. This entity will remain stale until evicted or explicitly refreshed.

Diagnostic Steps

• Enable Hibernate logging, including both the SQL and values.
  • Check whether multiple threads in the JVM are updating the same entity concurrently
  • Check whether multiple nodes/JVMs may be updating the same entity concurrently
• Enable transaction logging for EAP 5 and EAP 6
• See Monitoring clustered level 2 cache invalidation for Hibernate / Infinispan in EAP 6
• See Monitoring clustered level 2 cache invalidation for Hibernate / JBossCache in EAP 5

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.