Gathering a CRI-O Core Dump on Red Hat CoreOS in OCP 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Red Hat Enterprise Linux Core OS (RHCOS)
  • 4
• CRI-O

Issue

• How to generate a CRI-O Core Dump in RHCOS?

Resolution

1. First make sure Prior to note the exact version of CRI-O in use in the RHOCP node (if oc debug node command is failing, it will be needed to access with ssh):

   # rpm -qa cri-o
   cri-o-1.14.11-6.dev.rhaos4.2.git627b85c.el8
   

2. To begin gathering a Core Dump, start the support-tools container:

   # podman login registry.redhat.io
   # podman pull registry.redhat.io/rhel8/support-tools
   # podman container runlabel RUN registry.redhat.io/rhel8/support-tools
   

3. Install gcore tools in the container:

   # yum install gdb procps-ng -y
   

4. Produce the core using gcore which is provided by gdb:

   # gcore -o crio $(pidof crio)
   

5. Copy the resulting file labeled with core.$PID where $PID is the process number for the core to another system, and upload it to the support case. You could use /host/tmp for this. This will put the file from the container to the /tmp dir on the host. From the container run the following:

   cp /<filename> /host/tmp
   

6. It is also prudent to immediately collect an sosreport of a node after collecting the core, for lsof information and any output of crictl commands that can be obtained. While still in the container, run:

       # sosreport
       #### If the sosreport hangs, you may re-run the sosreport command without the `crio` plugin like so:
   
       # sosreport -n crio
   

Root Cause

In OpenShift 4, the CRI-O container runtime is used for container operations. This runtime is made to talk directly to the kubelet, to be directly utilized by Kubernetes and OpenShift, and is not intended for human interaction, unlike the previous container runtime Docker. However, certain scenarios like unresponsive and frozen runtime require a CRI-O Core Dump for troubleshooting and analysis.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.