Calls to mmap() returning allocation failure (ENOMEM) on RHEL 8

Environment

• Red Hat Enterprise Linux 8+

Issue

• Calls to mmap() that worked in previous versions are now returning a memory allocation failure / out of memory (ENOMEM)
• No change to ulimit -d (data segment) between versions, compile flags, etc.

Resolution

A new data segment ulimit will need to be determined for each application. It should be the previous ulimit -d value plus the largest size of a mmap() that the application will utilize.

Root Cause

• The kernel has changed how it treats memory allocation by mmap(). From the malloc man page:

Prior to Linux [kernel] 4.7 allocations performed using mmap(2) were unaffected by the RLIMIT_DATA resource limit; since Linux 4.7, this limit is also enforced for allocations performed using mmap(2).

• As Red Hat Enterprise Linux 7 uses Linux kernel 3.10, it did not have this behavior.

• Red Hat Enterprise Linux 8 uses Linux kernel 4.10, and does have this behavior.

• Additionally, from the setrlimit man page:

   RLIMIT_AS

          This is the maximum size of the process's virtual memory
          (address space).  The limit is specified in bytes, and is
          rounded down to the system page size.  This limit affects
          calls to brk(2), mmap(2), and mremap(2), which fail with
          the error ENOMEM upon exceeding this limit.  In addition,
          automatic stack expansion fails (and generates a SIGSEGV
          that kills the process if no alternate stack has been made
          available via sigaltstack(2)).  Since the value is a long,
          on machines with a 32-bit long either this limit is at
          most 2 GiB, or this resource is unlimited.

   RLIMIT_DATA
          This is the maximum size of the process's data segment
          (initialized data, uninitialized data, and heap).  The
          limit is specified in bytes, and is rounded down to the
          system page size.  This limit affects calls to brk(2),
          sbrk(2), and (since Linux 4.7) mmap(2), which fail with
          the error ENOMEM upon encountering the soft limit of this
          resource.

Diagnostic Steps

The man pages note "malloc" and other memory allocation calls are also susceptible to the Rlimit value, and will report "ENOMEM" if the limit is exceeded:

Content from man7.org is not included.malloc(3) - Linux manual page

   calloc(), malloc(), realloc(), and reallocarray() can fail with
   the following error:

   ENOMEM Out of memory.  Possibly, the application hit the
          RLIMIT_AS or RLIMIT_DATA limit described in getrlimit(2).

Failures such as this do not result in a system out of memory event, but in a failure affecting only the process or application that reached the limit. The process typically dies with a coredump.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.