Limitation of expiration with maxIdle for RDG cache entries in clustered environments

Environment

• Red Hat Data Grid (JDG, RDG)
  • 7.0, 7.1, 7.2
  • 6

Issue

• Are there drawbacks if clustered caches use expiration with maxIdle?
• In case of node shutdown or crash sometimes entries are lost if maxIdle is used

Resolution

Because of the current limitations it is not recommended to use maxIdle based expiration for clustered environments until DG 7.3.
From DG 7.3.0 onwards there is no longer a limitation for max-idle in a cluster.

But based on the use case it might be considered whether a maxIdle expiration can be used in a clustered environment with the limitations.

A possible workaround can be to use lifespan based expiration and refresh the entry by using a put operation with the same value. This will correctly update any owner and cache store consistently.
But the use of near or L1 caching will not have any performance benefit then.

To optimize the behavior and minimize the affect to cache performance there is a feature request to configure how exact maxIdle expiration should behave. The request might take longer and if there is an interest please open a support case with the requirements and reference this solution, or comment on this solution, to raise the priority.

Root Cause

The meta data to handle maxIdle is the timestamp for the last use of an entry.
Due to performance reasons this attribute is not fully replicated accross the cluster which can cause issues.

lifespan expiration

The creation time will be updated for each put operation, this include the creation timestamp and the offset for lifespan.
It will be updated on all nodes and cachestores bases on the configuration.
On expiration the owner will lock this entry and remove all occurences in one action.

maxIdle expiration

The lastUse timestamp will be updated on the local node by any read, all other nodes are only updated if the entry is updated!
This is because of performance considerations as a simple get should not write updates to every owner.
The entry will expire based on the lastUsed timestamp and the maxIdle offset individually on each node!

• Any owner might remove the entry based on the local lastUsed timestamp
  • If the cache is used remote via smart HotRod routing the clients will access the primary owner only
    • as result only the primary owner will update the entry on get's and keep it. Backup will be deleted after the maxIdle time as there is no update!
    • as result there is no backup and if the primary owner is shutting down or crashed the entry is lost!
  • if the cache is used embedded or without smart routing each node might refresh the maxIdle but it is not ensured that all primary and backups are keeping the entry
    • this might remove the entry from some or any owner
    • as result the entry could be lost if nodes are shutting down
• HotRod near caching for remote clients
  • clients which have near caching configured will use the entry at client side without notifying the server of any access
• Embedded mode and Level1 (L1) caching
  • If the node is an owner it will update the lastUsed timestamp only local
  • If the node is not an owner it checks the L1 cache and might request the entry from one owner
    This might not update any access time or only for one owner
• If a cache store is used a get will not write an update because of the performance
  • If the cluster is restarted the lastUse timestamp will be wrong
    Beside that the maxIdle time still calculate the downtime so the entry will expire if the cluster is down for the idle time

related documentation

Content from infinispan.org is not included.Infinispan User Guide.
JDG Administration Guide - expiration

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.