JBoss - Java heap retention in FileCleaningTracker$Tracker

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform
  • Apache commons IO

Issue

  • We start to receive OutOfMemoryErrors under load.  The heap dump shows much of the heap at the time is consumed by many instances of org.apache.commons.io.FileCleaningTracker$Tracker.  These tracker objects are stored in FileCleaningTrackers Vector called "trackers".

Resolution

  • Ensure the Reaper thread is present, healthy, and clearing out tracker entries as expected.
  • Reduce the load and number of Tracker entries generated to allow the Reaper thread to keep up.
  • Content from issues.apache.org is not included.https://issues.apache.org/jira/browse/IO-220

    • Under heavy load, the Vector.remove called to clear out these tracker entries becomes very inefficient as it causes the Vector to shift and reindex the large amount of remaining data.  Threads adding entries are forced to wait on the slow performing removes as well.  As the Vector.remove starts to slow and fails to keep up, the Vector size grows, forcing Vector.remove to perform even slower and slower.

    • Upgrade to apache commons io 2.1 or later where the Trackers are stored in a HashSet instead of a Vector.

Root Cause

  • The Tracker entries should be cleared out continuously from this Vector over time by the FileCleaningTracker Reaper thread.
  • The FileCleaningTracker Reaper thread is missing, terminated, or stalled and thus not clearing out the trackers Vector.
  • Heavy load is generating Tracker entries and adding them to the Vector faster than the Reaper thread is clearing them out.

Diagnostic Steps

Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.