How to use gdb to generate a readable backtrace from an application core

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux

Issue

  • How to use gdb to generate a readable backtrace from an application core
  • How to use GDB to debug an application core
  • An application core was generated, how can this core be viewed?
  • How to troubleshoot an application that crashed
  • How to debug an app core
  • How to debug a crashed program

Resolution

Install gdb

  • To start off gdb will need to be installed:
 # yum -y install gdb

  • Next the machine must be associated with the 'Debug' channel which provides the repository which contains the debuginfo packages that will be needed to analyze the application core file for any specific application. Note that Red Hat provides all of the debuginfo packages only for packages that Red Hat ships in it's standard repositories. If a third-party application needs to be analyzed then the debuginfo will need to be obtained from the third-party.

  • The instructions for adding the debuginfo channel differ depending on how the machine is subscribed, the steps for the two subscription methods are found below:

Red Hat Network (RHN Classic)

  • Login to RHN and subscribe the machine to the Debug channel with the following steps:
  1. Click on the registered systems hostname or IP address
  2. Click 'Alter Channel Subscriptions' in the bottom left
  3. Check 'RHEL [Server/Desktop] Debuginfo (v.X for x86_64)' and click Change Subscriptions, where X is the version of RHEL and [Server/Desktop] is the type of product being utilized.

Red Hat Subscription Manager (RHSM)

Enable the repository for use by utilizing the yum-config-manager and subscription-manager commands. In the below example the RHEL6 server debug repositories are subscribed to:

 # yum-config-manager --enable rhel-6-server-debug-rpms
 # subscription-manager repos --enable rhel-6-server-debug-rpms

For a full list of debug repositories available for Red Hat Enterprise Linux 6 or 7 run the command:

 # subscription-manager repos | grep debug

Generating a back-trace

  • Now that the repository has been enabled install the debuginfo for the package(s) that the core file was created for. In the following example the autofs service segfaulted, so the debuginfo packages will need to be installed for autofs: 
 # debuginfo-install autofs
  • The debuginfo-install command is not available on any versions of Red Hat Enterprise Linux previous to 6, so a script like this may help to install relevant dependency debuginfos: 
# yum -y install autofs-debuginfo ;
# yum deplist autofs | grep 'provider' | awk '{print $2}' | sort | uniq | sed -e s/.x86_64/-debuginfo/ | xargs yum -y install
  • Hopefully, that should be enough information to generate what is needed from the core file, open the core in gdb:
 # gdb -c <core file>
  • Then produce a back trace to a file or, skip the logging to a file and produce the full back trace to console: 
 (gdb) set logging file bt.out
 (gdb) set logging on
 (gdb) thread apply all bt
 (gdb) quit

What if my backtrace contains little info?

  • When the backtrace does not contain line numbers nor function arguments like:
Program received signal SIGSEGV, Segmentation fault.
0x0027455e in fread () from /lib/libc.so.6
(gdb) where
#0  0x0027455e in fread () from /lib/libc.so.6

It typically means that gdb was unable to load debuginfo packages for the corresponding library. Make sure that:

  1. The debuginfo package is the exact same version of the installed package
  2. Verify that the architecture is the same: if your application is 32-bit then you will need the corresponding 32-bit debuginfo packages. This is true for x86_64 architectures as well.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.