OpenShift automatic etcd defragmentation detection cycle

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform 4.10+

Issue

  • How to check automatic etcd defragmentation detection cycle and triggering conditions?

Resolution

Diagnostic Steps

  • Check the etcd-operator pod logs in the openshift-etcd-operator namespace and view the etcd defragmentation messages.
$ oc -n openshift-etcd-operator logs etcd-operator-5bfcff6cbf-xpn7q  |grep defrag
I0913 06:26:33.207338       1 defragcontroller.go:289] etcd member "master-0.ocp4.example.com" backend store fragmented: 50.89 %, dbSize: 259145728
I0913 06:26:33.207845       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"f91afb98-c193-4714-a064-1a4b96be2b3b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DefragControllerDefragmentAttempt' Attempting defrag on member: master-0.ocp4.example.com, memberID: 2f26cf76cfcb05fb, dbSize: 259145728, dbInUse: 127275008, leader ID: 14104827831415598805
I0913 06:26:34.696398       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"f91afb98-c193-4714-a064-1a4b96be2b3b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DefragControllerDefragmentSuccess' etcd member has been defragmented: master-0.ocp4.example.com, memberID: 3397631078096897531
I0913 06:27:12.735709       1 defragcontroller.go:289] etcd member "master-2.ocp4.example.com" backend store fragmented: 50.74 %, dbSize: 258572288
I0913 06:27:12.736277       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"f91afb98-c193-4714-a064-1a4b96be2b3b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DefragControllerDefragmentAttempt' Attempting defrag on member: master-2.ocp4.example.com, memberID: 9d8e4b6e28caec1a, dbSize: 258572288, dbInUse: 127373312, leader ID: 14104827831415598805
I0913 06:27:15.283363       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"f91afb98-c193-4714-a064-1a4b96be2b3b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DefragControllerDefragmentSuccess' etcd member has been defragmented: master-2.ocp4.example.com, memberID: 11353094647150472218
I0913 06:27:53.319572       1 defragcontroller.go:289] etcd member "master-1.ocp4.example.com" backend store fragmented: 50.60 %, dbSize: 257544192
I0913 06:27:53.320504       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"f91afb98-c193-4714-a064-1a4b96be2b3b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DefragControllerDefragmentAttempt' Attempting defrag on member: master-1.ocp4.example.com, memberID: c3be6a2e9d91eed5, dbSize: 257544192, dbInUse: 127229952, leader ID: 14104827831415598805
I0913 06:27:54.868521       1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"f91afb98-c193-4714-a064-1a4b96be2b3b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DefragControllerDefragmentSuccess' etcd member has been defragmented: master-1.ocp4.example.com, memberID: 14104827831415598805
I0913 06:37:33.205796       1 defragcontroller.go:289] etcd member "master-0.ocp4.example.com" backend store fragmented: 11.44 %, dbSize: 149323776
I0913 06:37:33.205823       1 defragcontroller.go:289] etcd member "master-2.ocp4.example.com" backend store fragmented: 10.69 %, dbSize: 148103168
I0913 06:37:33.205829       1 defragcontroller.go:289] etcd member "master-1.ocp4.example.com" backend store fragmented: 10.66 %, dbSize: 148066304
I0913 06:48:33.206092       1 defragcontroller.go:289] etcd member "master-0.ocp4.example.com" backend store fragmented: 10.22 %, dbSize: 149323776
I0913 06:48:33.206187       1 defragcontroller.go:289] etcd member "master-2.ocp4.example.com" backend store fragmented: 9.42 %, dbSize: 148103168
I0913 06:48:33.206226       1 defragcontroller.go:289] etcd member "master-1.ocp4.example.com" backend store fragmented: 10.65 %, dbSize: 150134784
SBR
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.