Must-gather copy strategies

Introduction

One of the most common request from Red Hat Support in a Case opened towards OpenShift product is to collect the data from the cluster using the command oc adm must-gather which uses by default the image stream openshift/must-gather. The output of the command is a folder containing the collected cluster data which by default is store in the current working directory.

Along this article it is assumed that the command oc adm must-gather is being executing in an external server to the OpenShift cluster which will be known as local machine.

During the oc adm must-gather execution the copy of the data to the local disk can be achieved in several ways which will be explained in this article.

Gathering data process

When running oc adm must-gather command, a pod with two containers is created in a new project (openshift-must-gather-xxxxx) in the cluster:

• An init container (named gather) which collects the data and stores it in the mounted /must-gather/ directory.
• A container (named copy) which accesses the mounted /must-gather/ directory to copy the gathered data into the working directory, using oc rsync command.

The main focus of this article is the 3 different strategies that the oc rsync command can use to make such copy: rsync, rsync-daemon and tar:

1. rsync strategy

Firstly the oc rsync command tries to use the rsync binary over ssh.

oc rsync uses the function copy_rsync.go which defines a variable rsyncStrategy that implements the rsync copy strategy. This strategy calls the local rsync binary over ssh directly and passes the OpenShift oc rsh command as the remote shell for rsync. It requires that:

• the rsync binary is present in both, the local machine and the remote container. In the remote container the rsync binary is always present because it is shipped by Red Hat.
• the sshd service runs on the local machine.
  More details on code: Content from github.com is not included.copy_rsync.go code.

If the rsync binary is present on the local machine but the sshd service is not running, the oc rsync fails at copying the data using the rsync command (there is no console logs to identify this fail) and the oc rsync command jumps to execute the copy by using the second strategy.

If the rsync binary is not installed in the local machine, the oc rsync fails at copying the data using the rsync command giving the warning log:

WARNING: rsync command not found in path. Please use your package manager to install it.

and the oc rsync command jumps to execute the copy by using the third strategy.

2. rsync-daemon strategy

In case the rsync binary is installed in the local machine but the sshd service is not running the oc rsync command will copy the data using the rsync command in the container in daemon mode. More information about the daemon mode of rsync command in the Red Hat Solution Configure rsync as a daemon and the detailed code can be found on Content from github.com is not included.copy_rsyncd.go.

3. tar strategy

In case rsync is not installed in local machine, the oc rsync copy mechanism has a fallback option using the tar command from the copy container.

The tar binary is always present in the copy container, because it is shipped by Red Hat.

The presence of the tar binary in the local machine is not needed because the mechanism used by oc adm must-gather command to download logs from the remote container to the local machine uses the tar library in Go to download logs (which is imported by the oc command). The used function exec_remote.go can run commands in a container, so basically exec_remote.go directly executes the tar command inside the container. More details on code: Content from github.com is not included.exec_remote.go code.

If the third strategy is the one used, the logs in the output execution of the oc adm must-gather command (with --log-level 6) are:

WARNING: rsync command not found in path. Please use your package manager to install it.
I0409 13:55:08.657628   29457 copy_tar.go:119] Copying files with tar
I0409 13:55:08.657688   29457 copy_tar.go:147] Creating local tar file /tmp/rsync2523744157 from remote path /must-gather/
I0409 13:55:08.657703   29457 copy_tar.go:203] Tarring /must-gather/ remotely
I0409 13:55:08.657708   29457 copy_tar.go:227] Remote tar command: tar -C /must-gather/ -c .
I0409 13:55:08.657711   29457 exec_remote.go:29] Remote executor running command: tar -C /must-gather/ -c .

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.