A multi-threaded application not seem to use Transparent HugePages in RHEL-6

Solution Unverified - Updated

Environment

  • Red Hat Enterprise Linux 6
  • glibc
  • Multi-threaded applications

Issue

  • A custom application multi-threaded application seems to allocate a lot of memory in arenas, but all of that memory seems to be backed by regular pages and not Transparent HugePages (THP), as seen from /proc/meminfo.
  • Waiting for a while results in consolidation of the pages into THP, but the allocation should have been backed by THP from the beginning.

Resolution

This is expected behaviour and the expectation that thread arenas allocated by glibc be backed by THP from the beginning is actually undesirable for a general case.

Root Cause

Allocations being backed by THP come at a cost, which is larger sections of memory being paged in/out due to a fault. So THP is beneficial in cases where allocations are large and are being used at the same time. That however is not the case for the glibc thread arena. When an arena is allocated, it is not meant to be used as a single chunk and all of it is not meant to be addressable on the first request itself. The arenas are used as heaps to service small requests unrelated to each other and hence it only makes sense that individual requests are only given access at a time from the larger PROT_NONE arena.

These individual allocation requests could come from different unrelated malloc() requests, possibly even from different threads in the application and there is no reason for glibc to assume that all of those smaller allocations are related and would (almost) always be needed together. There is also the case where the arena is never
really used fully and in the worst case, one ends up using up all of 128M from the system memory for an allocation of a few bytes, where it could have gotten away with a smaller (4k) wastage instead.

The THP defragmentation kernel thread does the consolidation if necessary and that currently looks like the optimal general approach.

SBR
Product(s)
Components

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.