How to disable the Out of memory or oom-killer?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux (RHEL) 4
  • Red Hat Enterprise Linux (RHEL) 5
  • Red Hat Enterprise Linux (RHEL) 6
  • Red Hat Enterprise Linux (RHEL) 7
  • Red Hat Enterprise Linux (RHEL) 8
  • Red Hat Enterprise Linux (RHEL) 9
  • Red Hat Enterprise Linux (RHEL) 10

Issue

  • How to disable the Out of memory or oom-killer?
  • How do I determine and configure the likelihood that a process will be killed in a out-of-memory situation?

Resolution

Disabling OOM-KILLER

Red Hat Enterprise Linux 4

  • Red Hat Enteprise Linux 4.2 and newer releases have the /proc/sys/vm/oom-kill tunable. Set this to 0 to disable the oom-killer.

    • /proc/sys/vm/oom-kill

      # cat /proc/sys/vm/oom-kill
1
# echo 0 > /proc/sys/vm/oom-kill
# cat /proc/sys/vm/oom-kill
0

    • /etc/sysctl.conf

      vm.oom-kill = 0
Red Hat Enterprise Linux 5/6/7/8/9/10

Red Hat Enteprise Linux 5, 6, 7,8, 9 an 10 do not have the ability to completely disable OOM-KILLER. Please see the following section for tuning OOM-KILLER operation within RHEL 5, RHEL 6, RHEL 7, RHEL 8, RHEL 9 & 10.

Causing kernel panic in OOM

Red Hat Enterprise Linux 4/5/6/7/8/9/10
  • If a kernel panic occurs in an out of memory condtion, check if panic_on_oom is set.

    • /proc/sys/vm/panic_on_oom

      # cat /proc/sys/vm/panic_on_oom
1
# echo 0 > /proc/sys/vm/panic_on_oom
# cat /proc/sys/vm/panic_on_oom
0

    • /etc/sysctl.conf

      vm.panic_on_oom = 1

Preventing OOM-KILLER from killing specific processes.

Red Hat Enterprise Linux 5

  • It is also possible to prioritize which processes get killed by adjusting the oom-killer score. In /proc/PID/ there are two tools labelled oom_adj and oom_score. Valid scores for oom_adj are in the range -16 to +15. This value is used to calculate the 'badness' of the process using an algorithm that also takes into account how long the process has been running, amongst other factors. To see the current oom_killer score, view the oom_score for the process. oom_killer will kill processes with the highest scores first.

  • This example adjusts the oom_score of a process with a PID of 12465 to make it less likely that oom_killer will kill it.

    # cat /proc/12465/oom_score 
79872
# echo -5 > /proc/12465/oom_adj           
# cat /proc/12465/oom_score
78

  • There is also a special value of -17, which disables oom-killer for that process. In the example below, oom_score returns a value of O, indicating that this process would not be killed.

    # cat /proc/12465/oom_score
78           
# echo -17 > /proc/12465/oom_adj           
# cat /proc/12465/oom_score
0
Red Hat Enterprise Linux 6/7/8/9/10

  • oom_score_adj is a newer parameter that replaces oom_adj.

  • oom_score_adj can range from -1000 to 1000.

  • For oom_score_adj -1000 is a special value that prevents oom-killer from killing the process.

    # echo '-1000' > /proc/<pid>/oom_score_adj

Additional Information

  • Setting "overcommit_memory" to 2, allows you to be precise about memory overcommitting. This instructs the kernel never to commit address space larger than swap space plus a fraction "overcommit_ratio" of the physical memory.

    RHEL5:
    Commit = swap + (overcommit_ratio/100) * physical memory

    RHEL6, 7:
    Commit = swap + (overcommit_ratio/100) * (physical memory - hugepage tlb)

  • When memory usage hits this number no more allocations should be possible. Committed_AS in /proc/meminfo shows the currently committed amount of memory on the system.

  • Out of Memory (OOM) refers to a computing state where all available memory, including swap space, has been allocated. Normally this will cause the system to panic and stop.

  • The vm.oom-kill for RHEL4 is provided by linux-2.6.9-proc-disable-oom.patch. That patch is not available for Red Hat Enterprise Linux 5 or above.

SBR
Product(s)
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.