Using etcdctl to investigate Objects in etcd (with OpenShift Container Platform)

Content from github.com is not included.Etcd is a distributed key-value store that serves as the backbone of OpenShift cluster coordination and state management. Having the ability to observe the state of etcd and how it is changing allows you to Content from github.com is not included.react to changes when events occur, or debug the platform interacting with the key-value store.

• Note: With some installs of openshift you may need to install the etcd package, as the etcdctl command is bundled with this package, but your install may be using an embedded etcd binary, instead of the binary provided by the etcd package.

Be sure to source the information needed to connect to your etcd instance, or be prepared to provide it manually.

• PRO TIP: Sourcing these variables on cluster connection are the simplest way to get connection details to your etcd cluster.

  # source /etc/etcd/etcd.conf
  

SYNTAX (on OpenShift Container Platform):

 # export ETCDCTL_API=2   ### the default
source /etc/etcd/etcd.conf 
etcdctl --endpoints=$ETCD_LISTEN_CLIENT_URLS --cert-file=$ETCD_PEER_CERT_FILE --key-file=$ETCD_PEER_KEY_FILE --ca-file=$ETCD_TRUSTED_CA_FILE COMMAND PATH

OR

# export ETCDCTL_API=3 
source /etc/etcd/etcd.conf 
etcdctl3 --cert=$ETCD_PEER_CERT_FILE --key=$ETCD_PEER_KEY_FILE --cacert=$ETCD_TRUSTED_CA_FILE --endpoints=$ETCD_LISTEN_CLIENT_URLS COMMAND OPTIONS

ALTERNATIVE SYNTAX (on OpenShift Container Platform, if the above fails)

• PRO TIP: Install a yaml parsing tool, to provide you quick access to yaml key, value stored information, so you could pull this directly from OCP configurations.

  • Note: This is not needed if you are sourcing /etc/etcd/etcd.conf

  # easy_install shyaml
  

• Environment Variables: (needed if you don't source in /etc/etcd/etcd.conf)

  • Used for v2(endpoints, cert-file, key-file, and ca-file), or v3(endpoints, cert, key, and cacert) settings on the etcdctl commandline

  etcd_endpoint="$(cat /etc/origin/master/master-config.yaml | shyaml get-value etcdClientInfo.urls.0)"
  cert_file='/etc/origin/master/master.etcd-client.crt'
  key_file='/etc/origin/master/master.etcd-client.key'
  ca_file="/etc/origin/master/$(cat /etc/origin/master/master-config.yaml | shyaml get-value etcdClientInfo.ca)"
  

  • Note: We uses commands / tools provided in PRO TIP that may not be supported or provided directly with Red Hat Products.

• If you use the variables above, and use direct option on etcdctl, commands will look like

  (v2)# etcdctl --endpoints ${etcd_endpoint} --cert-file ${cert_file} --key-file ${key_file} --ca-file ${ca_file} COMMAND PATH
  (v3)# etcdctl --endpoints=${etcd_endpoint} --cert ${cert_file} --key ${key_file} --cacert ${ca_file} COMMAND OPTIONS
  

Examples v3:

• Note: etcdctl, may or may not be installed with openshift, it can be installed from the etcd package provided with RHEL.

• Get a list of keys

  # etcdctl3 --cert=$ETCD_PEER_CERT_FILE --key=$ETCD_PEER_KEY_FILE --cacert=$ETCD_TRUSTED_CA_FILE --endpoints=$ETCD_LISTEN_CLIENT_URLS get /openshift.io --prefix  --keys-only  
  

• Get the value of a key

  # etcdctl3 --cert=$ETCD_PEER_CERT_FILE --key=$ETCD_PEER_KEY_FILE --cacert=$ETCD_TRUSTED_CA_FILE --endpoints=$ETCD_LISTEN_CLIENT_URLS get /openshift.io/deploymentconfigs/default/router
  

Examples v2:

• Note: etcdctl, may or may not be installed with openshift, it can be installed from the etcd package provided with RHEL.

• Popular Commands

  • ls
  • get
  • cluster-health
  • watch

With these commands you can monitor for changes to the etcd cluster, by watching a path or the entire server:

etcdctl --endpoints ${etcd_endpoint} --cert-file ${cert_file} --key-file ${key_file} --ca-file ${ca_file} watch --recursive --forever /

You can also get information from specific paths, with

etcdctl --endpoints ${etcd_endpoint} --cert-file ${cert_file} --key-file ${key_file} --ca-file ${ca_file} get /kubernetes.io/services/endpoints/default/router

{"kind":"Endpoints","apiVersion":"v1","metadata":{"name":"router","namespace":"default","selfLink":"/api/v1/namespaces/default/endpoints/router","uid":"f88a9d75-31ae-11e6-9587-fa163e71e8b9","creationTimestamp":"2016-06-13T21:37:04Z","labels":{"router":"router"}},"subsets":[{"addresses":[{"ip":"192.168.120.16","targetRef":{"kind":"Pod","namespace":"default","name":"router-2-tbjf1","uid":"30811fef-31b0-11e6-9587-fa163e71e8b9","resourceVersion":"2574691"}}],"ports":[{"name":"80-tcp","port":80,"protocol":"TCP"},{"name":"443-tcp","port":443,"protocol":"TCP"},{"name":"1936-tcp","port":1936,"protocol":"TCP"}]}]}

and parset the output with python tools.

etcdctl --endpoints ${etcd_endpoint} --cert-file ${cert_file} --key-file ${key_file} --ca-file ${ca_file} get /kubernetes.io/services/endpoints/default/router | python -m json.tool

{
    "apiVersion": "v1",
    "kind": "Endpoints",
    "metadata": {
        "creationTimestamp": "2016-06-13T21:37:04Z",
        "labels": {
            "router": "router"
        },
        "name": "router",
        "namespace": "default",
        "selfLink": "/api/v1/namespaces/default/endpoints/router",
        "uid": "f88a9d75-31ae-11e6-9587-fa163e71e8b9"
    },
    "subsets": [
        {
            "addresses": [
                {
                    "ip": "192.168.120.16",
                    "targetRef": {
                        "kind": "Pod",
                        "name": "router-2-tbjf1",
                        "namespace": "default",
                        "resourceVersion": "2574691",
                        "uid": "30811fef-31b0-11e6-9587-fa163e71e8b9"
                    }
                }
            ],
            "ports": [
                {
                    "name": "80-tcp",
                    "port": 80,
                    "protocol": "TCP"
                },
                {
                    "name": "443-tcp",
                    "port": 443,
                    "protocol": "TCP"
                },
                {
                    "name": "1936-tcp",
                    "port": 1936,
                    "protocol": "TCP"
                }
            ]
        }
    ]
}