How to collect a coredump of a running process without killing the process or without restarting an application?

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux

Issue

  • How to obtain a core file without restarting an application?

  • How an application core can be generated when a process is not crashing?

  • I want to collect a coredump of a running process without killing the process

  • I use gcore to collect the coredump but buildids and libraries are not available through executing eu-unstrip command

        # gcore <pid>
    [...]
    Saved corefile core.<pid>

    # eu-unstrip -n --core=core.<pid> | grep -c "/lib64/"
    0

Resolution

  • The gcore command can be used to obtain the core file without causing the application to abort using the following procedure:

    • Ensure that the gdb package is installed.

    • Run the application.

    • Execute the gcore command to generate a core file (like core.pid) from the application in the current directory:

      # gcore (pid of the process)

  • Some old versions of gcore commands in gdb package generates coredump files which are not suitable for analyzing on remote systems. Use following packages and commands.

    • RHEL 7: devtoolset-9-gdb from rhel-server-rhscl-7-rpms repository, command is /opt/rh/devtoolset-9/root/usr/bin/gcore
    • RHEL 8: gcc-toolset-9-gdb, command is /opt/rh/gcc-toolset-9/root/usr/bin/gcore
    • RHEL 9: gcc-toolset-13-gdb, command is /opt/rh/gcc-toolset-13/root/usr/bin/gcore

  • The core file will be outputted to the present working directory.

On RHEL 7

Use the gcore utility shipped with devtoolset-9-gdb, the package delivered with Red Hat Software Collections.
If this is not possible, then follow the instructions in section Fallback in case only standard gcore utility is available.

  1. Enable the Red Hat Software Collections

    # subscription-manager repos --enable rhel-server-rhscl-7-rpms

    For details, please refer to How to use Red Hat Software Collections (RHSCL) or Red Hat Developer Toolset (DTS)?.

  2. Install the devtoolset-9-gdb package

    # yum -y install devtoolset-9-gdb

  3. Should the install fail with a missing dependency as seen below:

        Error: Package: devtoolset-9-gdb-8.3-3.el7.x86_64 (rhel-server-rhscl-7-rpms)
           Requires: libsource-highlight.so.4()(64bit)

    Enable the rhel-7-server-optional-rpms repository so that source-highlight RPM can be installed:

    # subscription-manager repos --enable rhel-7-server-optional-rpms

    And repeat step 2 again.

  4. Produce the coredump using gcore

    # /opt/rh/devtoolset-9/root/usr/bin/gcore <pid>
[...]
Saved corefile core.<pid>

  5. Verify that eu-unstrip shows now the buildids and libraries

    # yum -y install elfutils
# eu-unstrip -n --core=core.<pid> | grep -c "/lib64/"
19

On RHEL 8

Use the gcore utility shipped with gcc-toolset-9-gdb, the package delivered with AppStream repository.

  1. Install the gcc-toolset-9-gdb package

    # yum -y install gcc-toolset-9-gdb

  2. Produce the coredump using gcore

    # /opt/rh/gcc-toolset-9/root/usr/bin/gcore <pid>
[...]
Saved corefile core.<pid>

    Note: if you get a Operation not permitted error message, verify that fapolicyd service is enabled, and stop it temporarily (this issue is tracked by This content is not included.BZ 1817969 - Cannot execute gcore from gcc-toolset-9 when fapolicyd is enabled):

        # /opt/rh/gcc-toolset-9/root/usr/bin/gcore <pid>
    /opt/rh/gcc-toolset-9/root/usr/bin/gcore: line 100: /opt/rh/gcc-toolset-9/root/usr/bin/gdb: Operation not permitted

    # systemctl is-active fapolicyd
    active

    # systemctl stop fapolicyd

    # /opt/rh/gcc-toolset-9/root/usr/bin/gcore <pid>
    [...]
    Saved corefile core.<pid>

    # systemctl start fapolicyd

  3. Verify that eu-unstrip shows now the buildids and libraries

    # yum -y install elfutils
# eu-unstrip -n --core=core.<pid> | grep -c "/lib64/"
19

Fallback in case only standard gcore utility is available

  1. Collect the coredump using the standard utility

    # yum -y install gdb
# gcore <pid>

  2. Collect the DSO list

    # yum -y install abrt-addon-ccpp
# abrt-action-list-dsos -m /proc/<pid>/maps -o dso_list

On RHEL 9

Use the gcore utility shipped with gcc-toolset-13-gdb, the package delivered with AppStream repository.

  1. Install the gcc-toolset-13-gdb package

    # yum -y install gcc-toolset-13-gdb

  2. Produce the coredump using gcore

    # /opt/rh/gcc-toolset-13/root/usr/bin/gcore <pid>
[...]
Saved corefile core.<pid>

Note :

  • The gcore command runs gdb and attaches to the process, causing the process to enter T status. However, the process should try to continue to run after gdb finishes collecting a core.

  • One point to remember is that with gcore we can't get a coredump of a kernel process. Because kernel processes are immune to system call tracing, ptrace which the gcore command uses. When we issue a gcore to a pid, we are using PTRACE_ATTACH . When a parent process traces a child process, it uses PTRACE_TRACEME. See the man page of ptrace for more details.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.