The etcd backup script fails to generate a backup

Solution Verified - Updated

Environment

  • OpenShift 4.x
  • Disconnected cluster
  • Cluster with restricted networking

Issue

When attempting to create a backup snapshot of the etcd database, the /usr/local/bin/etcd-snapshot-backup.sh script finishes after only a few lines of output without showing any error:

sudo -E /usr/local/bin/etcd-snapshot-backup.sh ./assets/backup/snapshot.db
Creating asset directory ./assets
Downloading etcdctl binary..

Furthermore, the generated ./assets/backup directory is empty.

$ ls
assets  root-supporting-mandiri.crt
$ ls -ltr assets/backup/
total 0

The expected output of the script should be longer (example below):

sh-4.4# sudo -E /usr/local/bin/etcd-snapshot-backup.sh ./assets/backup/snapshot.db
Creating asset directory ./assets
Downloading etcdctl binary..
etcdctl version: 3.3.10
API version: 3.3
Trying to backup etcd client certs..
etcd client certs found in /etc/kubernetes/static-pod-resources/kube-apiserver-pod-2 backing up to ./assets/backup/
Backing up /etc/kubernetes/manifests/etcd-member.yaml to ./assets/backup/
Snapshot saved at ./assets/backup/snapshot.db

Resolution

If there is a cluster wide proxy in use, ensure that the HTTP_PROXY, HTTPS_PROXY and NO_PROXY environment variables have been exported in the ssh shell the script is being run from.

Root Cause

The etcd-snapshot-backup.sh downloads an archive of tools it needs to run the backup from an external URL. When accessing a node via ssh, the cluster proxy environment variables are not necessarily exported in the session, and may prevent the bundle from being downloaded.

Diagnostic Steps

Ensure whether a proxy is required for the cluster to access the internet, and then verify that the cluster proxy has been correctly set:

$ oc get proxy cluster -o yaml | grep -E "httpProxy|httpsProxy|noProxy"

Ensure whether the proxy environment variables are correctly set in the ssh session the backup script was run from:

set | grep -E "HTTP_PROXY|HTTPS_PROXY|NO_PROXY"
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.