How to use OS utilities to track down application memory leaks

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux 4
  • Red Hat Enterprise Linux 5
  • Red Hat Enterprise Linux 6
  • Red Hat Enterprise Linux 7
  • Red Hat Enterprise Linux 8
  • Red Hat Enterprise Linux 9
  • valgrind
  • c++filt

Issue

  • How to use OS utilities such as ps, valgrind and c++filt to track down application memory leaks.

Resolution

  • To determine what applications and code are using the most memory and to analyse those programs for leaks and issues can be a complex process. Programs such as ps, top, valgrind and pmap can be used to identify and analyse memory leaks. The complete use of these tools is beyond the scope of this article.

  • The valgrind program comes from the valgrind package and can be installed from RHN by running up2date or yum install valgrind. The valgrind manual in pdf and html format are provided within the package. See /usr/share/doc/valgrind-*version*/valgrind_manual.pdf for more information.

  • c++filt program comes from the binutils package and can be installed from RHN by running yum install binutils. See What is c++filt used for and how do I use it? for more information.

  • Additionally, from RHEL8 and 9 we can use eBPF tool to trace the memory leakage using the memleak script. For more details refer: How to trace memory leaks using eBPF/BCC script?

Diagnostic Steps

A simple 3 step approach may used in troubleshooting many memory problems.

  1. Identify suspect processes with ps or top.
  2. Use valgrind to go collect memory information about the suspect process.
  3. In the case of a C++ binary Use c++filt to further analyse the valgrind output.

This technique is has been greatly simplified for this article. Please refer to the man pages for each program for additional details.

1. Use a utility such as ps to identify suspect process.

The following ps command will sort the top 20 processes by memory usage.

# ps auwwx|gawk '!/%MEM/ {print $4,$11}'|sort -nr|head -n20

This will list additional process which may have a high thread count and may be of interest.

# ps auwwx|gawk '{count[$NF]++}END{for(j in count) print ""count[j]":",j}'|sort -nr|head -n20

2. Once a list of suspect applications has been created, use tools such as valgrind to collect further information from each application. (See valgrind 1 documentation for additional usage instructions.)

An example command to look for memory leaks explicitly (Vlagrind does far more than just this task) might look like this.

# valgrind --trace-children=yes --show-reachable=yes --track-origins=yes --read-var-info=yes --tool=memcheck --leak-check=full --num-callers=50 -v --time-stamp=yes --log-file=leaky.log leaky-app

Where leaky-app is the application that is leaking.

Use the valgrind output log to look for leaked memory. The following example summary from a valgrind output log identifies at least 4 lost records.

==1737== LEAK SUMMARY:
==1737==    definitely lost: 2,051 bytes in 7 blocks
==1737==    indirectly lost: 0 bytes in 0 blocks
==1737==      possibly lost: 2,080 bytes in 30 blocks
==1737==    still reachable: 35,572 bytes in 241 blocks
==1737==         suppressed: 0 bytes in 0 blocks

Use this data to track down possible issues by looking for the word 'definitely'.

$ grep definitely valgrind.log
==1737==    definitely lost: 0 bytes in 0 blocks
==1737== 0 bytes in 4 blocks are definitely lost in loss record 1 of 179
==1737== 2 bytes in 1 blocks are definitely lost in loss record 2 of 179
==1737== 1,024 bytes in 1 blocks are definitely lost in loss record 173 of 179
==1737== 1,025 bytes in 1 blocks are definitely lost in loss record 174 of 179
==1737==    definitely lost: 2,051 bytes in 7 blocks

From this output it can be seen that the totals match (1,024 + 1,025 + 2 equals 2,051 and 4 + 1 + 1 + 1 equals 7.)

3. Use programs such as c++filt to investigate these four entries further. Note that c++filt is only necessary in the case of a C++ binary as it demangles the symbols that valgrind displays.

The following is an edited example on how to feed the valgrind output log to the c++filt program and is an example of where to begin looking for the 4 suspect entries identified above. Use the stanzas below the identifying headers (in this example the suspect code was edited and replaced with [removed]) to look for potential issues in these locations within the applications' code.

$ c++filt< valgrind.log|less

==1737== 0 bytes in 4 blocks are definitely lost in loss record 1 of 179
[removed]
==1737== 2 bytes in 1 blocks are definitely lost in loss record 2 of 179
[removed]
==1737== 1,024 bytes in 1 blocks are definitely lost in loss record 173 of 179
[removed]
==1737== 1,025 bytes in 1 blocks are definitely lost in loss record 174 of 179
[removed]

Note: The ability to identify in a processes where a memory leak is suspect to occur can be difficult and is typically performed during the application's development.

SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.