Troubleshooting OpenShift Container Platform 4.x: Installation

Environment

• Red Hat OpenShift Container Platform
  • 4
• New installation

Issue

• I need to understand what data to collect in order for Support to help me diagnose and resolve the issue.

Resolution

• The installer terraform logs from .openshift_install.log
• Service Logs from the bootkube.service, service on your bootstrap host, if it is still around.
• Container Logs from the bootstrap host, if it is still around.
• Service Logs from the kubelet.service and crio.service, services on your master and worker hosts.
• Container Logs from the master and worker hosts.

Root Cause

• You may want to increase the verbosity of the installer as it can provide more information needed for debugging purposes.

Diagnostic Steps

• More information on watching an install progress can be found at KB 4139361

1. Find the information needed to debug an install.

   • Determine the IP or Hostname of your bootstrap node
   • Determine the IP or Hostname of your master and worker nodes

2. You can find the .openshift_install.log file in the directory (current working directory) from where you ran the installer.

   • This is useful in determining where you are in the install process for both UPI and IPI installs.

3. Use the provided installation debugging utilities.

   • If you have a bootstrap system running, run the installer-gather.sh script and collect the necessary debugging information from this process.

4. Use the manual debugging steps to gather information by hand.

   • If SSH access to your host[s] is available.

     • You can collect the bootkube.service, service logs from the bootstrap host, using the journalctl command by running:

       $ journalctl -b -f -u bootkube.service
       

     • You can collect the bootstrap host's container logs, using the podman logs or using the tail command by running:

       • Note: This is shown as a loop to get you all of the container logs from the host.

       $ for pod in $(sudo podman ps -a -q); do sudo podman logs $pod; done
       ### OR
       $ sudo tail -f /var/lib/containers/storage/overlay-containers/*/userdata/ctr.log
       

     • You can collect the kubelet.service and crio.service, service logs from the master and worker host[s], using the journalctl command by running:

       $ journalctl -b -f -u kubelet.service -u crio.service
       

     • You can collect the master and worker host[s]'s container logs, using the tail command by running:

       $ sudo tail -f /var/log/containers/*
       

   • If SSH access is _not available consider using the journal kublet shim to get the journal logs from the host.

     • Note: this requires an IPI install as SSL certificates (in your install directory) are needed to access these endpoints.

• In some situations, you might want to verify the version of the OS you are running on any of the hosts.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.