Manually adding a new etcd host to the cluster for OpenShift Container Platform

3.7 and Higher

Starting in 3.7, this is possible using an ansible playbook as described in the documentation

3.6 and earlier

1. Install etcd on the "NEW_ETCD"

• Do not start the etcd service

• Version of etcd must be etcd-2.3.7-4.el7.x86_64 or greater for the back up steps to work.

  # yum install etcd
  

1. Add iptable rules on the "NEW_ETCD"

  # systemctl enable iptables.service --now

  # iptables -N OS_FIREWALL_ALLOW
  # iptables -t filter -I INPUT -j OS_FIREWALL_ALLOW
  # iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2379 -j ACCEPT
  # iptables -A OS_FIREWALL_ALLOW -p tcp -m state --state NEW -m tcp --dport 2380 -j ACCEPT

1. Then save the rules to the iptables configuration file following steps here: What is the correct method for adding a persistent rule to iptables on a OpenShift Master or Node?

2. Remotely login over to the any etcd host that was already a member of a cluster. We will call this the "ETCD_CA_HOST"

• This host will have the directory /etc/etcd/ca/, this directory will contain the openssl configuration needed to sign the new etcd certificate which we will be creating with the etcd CA.

1. Generate certificates from "ETCD_CA_HOST"

• This steps have been taken from the Ansible Role Content from github.com is not included.etcd_server_certificates

  • Set variables and working directory

    # cd /etc/etcd
    # pwd
    /etc/etcd
  
    # export NEW_ETCD="etcd-1.openshift.com"
  
    # host $NEW_ETCD
    etcd-1.openshift.com has address 192.168.0.15
  
    # export CN=$NEW_ETCD
    # export SAN="IP:192.168.0.15,DNS:$NEW_ETCD"
    # export PREFIX="./generated_certs/etcd-$CN/"
  

  • Create server.csr and server.crt

    # openssl req -new -keyout ${PREFIX}server.key \
      -config ca/openssl.cnf \
      -out ${PREFIX}server.csr \
      -reqexts etcd_v3_req -batch -nodes \
      -subj /CN=$CN
  
    # openssl ca -name etcd_ca -config ca/openssl.cnf \
      -out ${PREFIX}server.crt \
      -in ${PREFIX}server.csr \
      -extensions etcd_v3_ca_server -batch
  

  • Create peer.csr and peer.crt

    # openssl req -new -keyout ${PREFIX}peer.key \
      -config ca/openssl.cnf \
      -out ${PREFIX}peer.csr \
      -reqexts etcd_v3_req -batch -nodes \
      -subj /CN=$CN
  
    # openssl ca -name etcd_ca -config ca/openssl.cnf \
      -out ${PREFIX}peer.crt \
      -in ${PREFIX}peer.csr \
      -extensions etcd_v3_ca_peer -batch
  

  • Copy ca.crt and etcd.conf and archive the contents in the directory

  # cp ca.crt ${PREFIX}
  # cp etcd.conf ${PREFIX}
  # tar -czvf ${PREFIX}${CN}.tgz -C ${PREFIX} .
  

  • Transfer files over to the new etcd member which we want to add in our cluster.

  # scp ${PREFIX}${CN}.tgz  $CN:/etc/etcd/
  

1. While still on "ETCD_CA_HOST" we will add "NEW_ETCD" to the etcd cluster.
   • Add "NEW_ETCD" to the cluster
     • The ID and ETCD_* values outputted will be used in later steps

  #export ETCD_CA_HOST="master-1.openshift.com"
  #export NEW_ETCD="etcd-1.openshift.com"                                    //hostname of new-etcd host which we want to add

  # host $NEW_ETCD
  etcd-1.openshift.com has address 192.168.0.15
  # export NEW_ETCD_IP="192.168.0.15"

  # etcdctl -C https://${ETCD_CA_HOST}:2379 --ca-file=/etc/etcd/ca.crt     --cert-file=/etc/etcd/peer.crt     --key-file=/etc/etcd/peer.key member add ${NEW_ETCD} https://${NEW_ETCD_IP}:2380

  Added member named etcd-1.openshift.com with ID 328c30c625727e8 to cluster

  ETCD_NAME="etcd-1.openshift.com"
  ETCD_INITIAL_CLUSTER="etcd-1.openshift.com=https://192.168.0.15:2380,master-2.openshift.com=https://192.168.0.8:2380,master-0.openshift.com=https://192.168.0.9:2380,master-1.openshift.com=https://192.168.0.6:2380"
  ETCD_INITIAL_CLUSTER_STATE="existing"

- Copy this output and save we will use this later.    

1. Remotely login to "NEW_ETCD" and extract data, set permissions.

   • Extract data and set permissions

  # ls  /etc/etcd/
  etcd-1.openshift.com.tgz  etcd.conf

  # tar -xf /etc/etcd/etcd-1.openshift.com.tgz -C /etc/etcd/ --overwrite
  # chown etcd:etcd /etc/etcd/*    

1. Make changes to "NEW_ETCD" etcd.conf file.
   • Make change to etcd.conf file.
     • Using the values generated in the previous step when the member was added to the cluster . Replace the following values:

- ETCD_NAME
- ETCD_INITIAL_CLUSTER
- ETCD_INITIAL_CLUSTER_STATE

    - Replace IP addresses with "NEW_ETCD" ip address for:

- ETCD_LISTEN_PEER_URLS
- ETCD_LISTEN_CLIENT_URLS
- ETCD_INITIAL_ADVERTISE_PEER_URLS
- ETCD_ADVERTISE_CLIENT_URLS

1. Start etcd on "NEW_ETCD"

systemctl enable etcd --now
systemctl restart etcd 

1. Verify the cluster is healthy.

etcdctl -C https://${ETCD_CA_HOST}:2379 --ca-file=/etc/etcd/ca.crt     --cert-file=/etc/etcd/peer.crt     --key-file=/etc/etcd/peer.key cluster-health

1. Add the member to the master-config.yaml on all the masters under "etcdClientInfo.urls" and restart master services.