High volume of memory cached by Infinispan for PutFromLoadValidator$PendingPut and TransactionImple in EAP 6

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP) 6
  • Hibernate 4
  • Infinispan 5

Issue

  • A high volume of memory is retained by an Infinispan cache/container (87% of the total memory) leading to an eventual OutOfMemoryException
  • For the memory retaining java.util.HashMap$Node instances, the value is PutFromLoadValidator$PendingPut and the key is TransactionImple
Class Name                                                                                                              | ... | Retained Heap | Percentage
----------------------------------------------------------------------------------------------------------------------------------------------------------
org.infinispan.container.DefaultDataContainer @ 0xfffffffee263e180                                                      | ... |   642,118,928 |     87.30%
'- java.util.concurrent.ConcurrentHashMap @ 0xfffffffee263e1a8                                                          | ... |   642,118,888 |     87.30%
   |- java.util.concurrent.ConcurrentHashMap$Node[256] @ 0xfffffffee4d74680                                             | ... |   642,118,808 |     87.30%
   |  |- java.util.concurrent.ConcurrentHashMap$Node @ 0xfffffffee55a4e10                                               | ... |   206,041,616 |     28.01%
   |  |  |- org.infinispan.container.entries.TransientCacheEntry @ 0xfffffffee55a4e50                                   | ... |   206,041,456 |     28.01%
   |  |  |  '- org.infinispan.container.entries.TransientCacheValue @ 0xfffffffee55a4e68                                | ... |   206,041,432 |     28.01%
   |  |  |     '- org.hibernate.cache.infinispan.access.PutFromLoadValidator$PendingPutMap @ 0xfffffffee55a4e88         | ... |   206,041,400 |     28.01%
   |  |  |        |- java.util.HashMap @ 0xfffffffee55a4ea0                                                             | ... |   206,041,328 |     28.01%
   |  |  |        |  '- java.util.HashMap$Node[32768] @ 0xfffffffefbb55c48                                              | ... |   206,041,280 |     28.01%
   |  |  |        |     |- java.util.HashMap$Node @ 0xfffffffee6bfa758                                                  | ... |        69,104 |      0.01%
   |  |  |        |     |  |- java.util.HashMap$Node @ 0xfffffffeec4ec578                                               | ... |        59,232 |      0.01%
   |  |  |        |     |  |- com.arjuna.ats.internal.jta.transaction.arjunacore.TransactionImple @ 0xfffffffee6bfa778  | ... |         9,816 |      0.00%
   |  |  |        |     |  |- org.hibernate.cache.infinispan.access.PutFromLoadValidator$PendingPut @ 0xfffffffee6c06d18| ... |            24 |      0.00%
   ...
----------------------------------------------------------------------------------------------------------------------------------------------------------
  • The data container's shortest path to the GC root is
Class Name                                                                                     | ... | Retained Heap
--------------------------------------------------------------------------------------------------------------------
java.lang.Thread @ 0xfffffffee2640198  TxCleanupService,pending-puts,local Thread              | ... |           320
'- <Java Local> java.util.concurrent.ScheduledThreadPoolExecutor @ 0xfffffffee2640028          | ... |           360
   '- threadFactory org.infinispan.transaction.TransactionTable$1 @ 0xfffffffee2640518         | ... |            16
      '- this$0 org.infinispan.transaction.TransactionTable @ 0xfffffffee263fe90               | ... |           312
         '- commandsFactory org.infinispan.commands.CommandsFactoryImpl @ 0xfffffffee263e118   | ... |           104
            '- dataContainer org.infinispan.container.DefaultDataContainer @ 0xfffffffee263e180| ... |   642,118,928
               '- entries java.util.concurrent.ConcurrentHashMap @ 0xfffffffee263e1a8          | ... |   642,118,888
--------------------------------------------------------------------------------------------------------------------

Resolution

Root Cause

  • Attempts to load a cacheable entity (e.g. using find by ID queries) that did not exist in the database
  • Transactional load attempts were executed at a rate of 1 per minute (or greater)
  • Query related structures/data were left in memory and due to the sustained frequency of access, were unable to timeout
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.