How to check if etcd needs defragmentation in OpenShift?

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
  • 3.11
• etcd

Issue

• How to check if etcd needs defragmentation.
• How much fragmentation is acceptable or unacceptable for etcd cluster?

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

According to how to defrag etcd to decrease DB size in OpenShift 4, user should defrag etcd when the dbSize and the dbSizeInUse differ in more than 40-50%.

On the other hand, the automatic etcd defragmentation in OpenShift 4 is done when the difference is more than 45%, according to the Content from github.com is not included.defragcontroller.go source code:

maxFragmentedPercentage float64 = 45

[...]

func isEndpointBackendFragmented(member *etcdserverpb.Member, endpointStatus *clientv3.StatusResponse) bool {
	if endpointStatus == nil {
		klog.Errorf("endpoint status validation failed: %v", endpointStatus)
		return false
	}
	fragmentedPercentage := checkFragmentationPercentage(endpointStatus.DbSize, endpointStatus.DbSizeInUse)
	if fragmentedPercentage > 0.00 {
		klog.Infof("etcd member %q backend store fragmented: %.2f %%, dbSize: %d", member.Name, fragmentedPercentage, endpointStatus.DbSize)
	}
	return fragmentedPercentage >= maxFragmentedPercentage && endpointStatus.DbSize >= minDefragBytes
}

[...]

func checkFragmentationPercentage(ondisk, inuse int64) float64 {
	diff := float64(ondisk - inuse)
	fragmentedPercentage := (diff / float64(ondisk)) * 100
	return math.Round(fragmentedPercentage*100) / 100
}

Check if the etcd needs manual defragmentation

• To check if the etcd needs manual defragmentation, it is possible to get the dbSize and dbSizeInUse by:

  $ export ETCD_POD_NAME=$(oc get pods -n openshift-etcd -l app=etcd --field-selector="status.phase==Running" -o jsonpath="{.items[0].metadata.name}")
  
  $ oc exec ${ETCD_POD_NAME} -c etcd -n openshift-etcd -- etcdctl endpoint status --cluster -w json | jq
  

• It is also possible to use jq to calculate the the fragment percentage:

  $ export ETCD_POD_NAME=$(oc get pods -n openshift-etcd -l app=etcd --field-selector="status.phase==Running" -o jsonpath="{.items[0].metadata.name}")
  
  $ oc exec $ETCD_POD_NAME -c etcd -n openshift-etcd -- etcdctl endpoint status --cluster -w json | jq '.[].Status|"dbSize: " + (.dbSize|tostring) + ", dbSizeInUse: " + (.dbSizeInUse|tostring) + ", (dbSize-dbSizeInUse)/dbSize => " + ((.dbSize - .dbSizeInUse)/.dbSize*100|tostring)+"%"'
  

  If [ (dbSize - dbSizeInUse) / dbSize ] % > 45%, it should be considered to perform etcd defragmentation. Then, refer to how to compact and defrag etcd to decrease database size in OpenShift 4 or how to defrag etcd to decrease DB size in Red Hat OpenShift 3.

Root Cause

The documentation of OpenShift Container Platform does not provide a clear standard on when should perform etcd defragmentation.

Diagnostic Steps

Get endpoint status from any etcd pod:

$ oc get pods -n openshift-etcd -l app=etcd
[...]
$ oc exec [etcd-pod] -c etcd -n openshift-etcd -- etcdctl endpoint status --cluster -w json | jq
  {
    "Endpoint": "https://10.0.95.12:2379",
    "Status": {
      "header": {
        "cluster_id": 9162435150823152000,
        "member_id": 7025450842823798000,
        "revision": 5641109,
        "raft_term": 30
      },
      "version": "3.4.9",
      "dbSize": 419430400,     
      "leader": 8648392815028961000,
      "raftIndex": 6939245,
      "raftTerm": 30,
      "raftAppliedIndex": 6939245,
      "dbSizeInUse": 209715200     
    }
[...]

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.