How to import an image for toolbox manually to a node on OpenShift 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Node networking impacted but host nodes are still reachable via virtual console or secondary address.
• Cannot pull support-tools image directly to node or from local repository

Issue

• It may be necessary to collect a SOSreport for one or more nodes to aide in debugging node issue.
• If the node does not already have the support-tools image available locally from a previous toolbox command run on that host, it may be problematic/difficult to run a sosreport if it cannot source the image due to networking concerns.
• How to copy a container image between nodes, or how to import a container image for local execution?

Resolution

A workaround for being able to get the image is to pull the image in a different machine (a node without issues or a machine out from the OpenShift cluster like a bastion host):

• Pull the image from a working node:

      ##### Connect to a working node
      $ oc debug node/[working-node-name]
      sh-5.1# chroot /host bash
  
      ##### pull the support-tools image with toolbox command
      [root@working-node-name /]# toolbox
      [...]
      toolbox-root
      Container started successfully. To exit, type 'exit'.
  
      ##### exit from the toolbox shell once it starts
      [root@working-node-name /]# exit
      [root@working-node-name /]#
  

• If the above is not possible, another option is to pull the image from a different machine not part of the OpenShift cluster (like a bastion host with SSH access to the failing node):

  # podman pull registry.redhat.io/rhel9/support-tools:latest
  

  For OCP 4.13 and newer, the image is registry.redhat.io/rhel9/support-tools:latest. For OCP 4.12 and older, the image is registry.redhat.io/rhel8/support-tools:latest)

When the image is pulled, either in a working node or in a different machine:

• Confirm that the image was pulled and is available:

  # podman images | grep support-tools
  registry.redhat.io/rhel9/support-tools         latest      abcdefg12345  12 days ago   374 MB
  

• Export the pulled image to tarball:

  # podman image save registry.redhat.io/rhel9/support-tools:latest -o /tmp/support-tools.tar
  

  Change the path if the pulled image is the rhel8 one for OCP 4.12 and older.

• Copy the image to the affected host, for example using scp:

  $ scp [path_to_the_image]/support-tools.tar core@<affected-node>:/tmp/
  

• Access to the affected node via SSH and restore the copied image with podman load:

      $ ssh core@[affected-node]
      [...]
      [core@affected-node ~]$ sudo -i
      [root@affected-node ~]# podman image load -i /tmp/support-tools.tar
      [...]
  
      ##### check the image is available after loading it
      [root@affected-node ~]# podman images | grep support-tools
      registry.redhat.io/rhel9/support-tools         latest      abcdefg12345  12 days ago   374 MB
  

• When the image is present in the node, start the toolbox container on the affected node with toolbox command:

  [root@affected-node ~]# toolbox
  

Root Cause

The toolbox tool must be sourced using the latest version of support-tools image; this is one method to ensure that the image can be found when toolbox is run. Refer to the documentation about toolbox for additional information.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.