Docker is not responding what data is needed?

Solution Verified - Updated

Environment

Red Hat Enterprise 7
Red Hat Enterprise Linux Atomic Host 7
Docker 1.12+

Issue

  • Docker is hung how do I capture a coredump and go-routine dump.
  • I run docker ps and it hangs forever
  • curl --unix-socket /var/run/docker.sock http://v1.26/containers/json?all=1 times out.

Resolution

When docker is hung capture the following and open a support case. For RHEL Atomic Host, refer the respective section for additional steps.

  1. Goroutine dumped to journal.

    # sudo kill -SIGUSR1 $(pidof dockerd-current)
# sudo kill -SIGUSR1 $(pidof docker-containerd-current)

 - After the above gather the logs and upload to case.
  
 # journalctl --since "2 hours ago" > $(hostname).logs

  2. Capture a gdb gcore from docker and containerd. To capture this you would need the 'gdb' package which will include the gcore command. The 'gcore' command captures a 'snapshot' of the process memory and the process
    remains running.

     # lsof -p $(pidof dockerd-current)
 # gcore -o /example_path_name/dockerd_core $(pidof dockerd-current)
 # lsof -p $(pidof docker-containerd-current)
 # gcore -o /example_path_name/containerd_core $(pidof docker-containerd-current)

  3. A sosreport with out the docker plug-in 

    # sosreport -n docker

For RHEL Atomic Host

1. Start the RHEL Tools container

# atomic run  rhel7/rhel-tools
Note: rhel-tools container image should match the atomic host version. Example if the RHEL AH is 7.6, then the corresponding rhel-tools image would be registry.access.redhat.com/rhel7/rhel-tools:7.6-6. For all the available tags, refer This content is not included.This content is not included.https://access.redhat.com/containers/?tab=tags#/registry.access.redhat.com/rhel7/rhel-tools

2. Install the required packages in the rhel-tools container

# yum install gdb sysvinit-tools lsof

3. Copy the binaries to the appropriate location

# cp /host/usr/bin/dockerd-current /usr/bin/dockerd-current
# cp /host//usr/bin/docker-containerd-current /usr/bin/docker-containerd-current

4. Collecting the gcore files

#   gcore -o /host/<path>/dockerd_core $(pidof dockerd-current)
#   gcore -o /host/<path>/containerd_core $(pidof docker-containerd-current)
  • Note: In the above command replace the <path> value. For using /var/tmp of the host filesystem, use gcore -o /host/var/tmp/dockerd_core $(pidof dockerd-current)

5. Collect the list of open files

#  lsof -p $(pidof dockerd-current) >> /host/<path>/dockerd_lsof.txt 
#  lsof -p $(pidof docker-containerd-current)  >> /host/<path>/containerd_lsof.txt
  • Note: In the above command replace the <path> value. For using /var/tmp of the host filesystem, use `lsof -p $(pidof docker-containerd-current) >> /host/var/tmp/containerd_lsof.txt

6. Capture sosreport of the node

# sosreport -n docker
SBR
Product(s)
Components
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.