OpenShift automatic etcd defragmentation detection cycle
Environment
- Red Hat OpenShift Container Platform 4.10+
Issue
- How to check automatic
etcddefragmentation detection cycle and triggering conditions?
Resolution
-
Content from github.com is not included.By default, the defrag controller checks if a defragmentation is needed every 11 minutes (the 10 minutes of Content from github.com is not included.compaction interval plus Content from github.com is not included.1 additional minute).
-
It Content from github.com is not included.checks whether
[(DbSize - DbSizeInUse) / DbSize]is greater thanmaxFragmentedPercentage, which is Content from github.com is not included.hardcoded as 45%, andDbSizeis greater thanminDefragBytes, which is Content from github.com is not included.hardcoded as 100MB. If both these conditions are met, defrag is triggered.
Diagnostic Steps
- Check the
etcd-operatorpod logs in theopenshift-etcd-operatornamespace and view theetcddefragmentation messages.
$ oc -n openshift-etcd-operator logs etcd-operator-5bfcff6cbf-xpn7q |grep defrag
I0913 06:26:33.207338 1 defragcontroller.go:289] etcd member "master-0.ocp4.example.com" backend store fragmented: 50.89 %, dbSize: 259145728
I0913 06:26:33.207845 1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"f91afb98-c193-4714-a064-1a4b96be2b3b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DefragControllerDefragmentAttempt' Attempting defrag on member: master-0.ocp4.example.com, memberID: 2f26cf76cfcb05fb, dbSize: 259145728, dbInUse: 127275008, leader ID: 14104827831415598805
I0913 06:26:34.696398 1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"f91afb98-c193-4714-a064-1a4b96be2b3b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DefragControllerDefragmentSuccess' etcd member has been defragmented: master-0.ocp4.example.com, memberID: 3397631078096897531
I0913 06:27:12.735709 1 defragcontroller.go:289] etcd member "master-2.ocp4.example.com" backend store fragmented: 50.74 %, dbSize: 258572288
I0913 06:27:12.736277 1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"f91afb98-c193-4714-a064-1a4b96be2b3b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DefragControllerDefragmentAttempt' Attempting defrag on member: master-2.ocp4.example.com, memberID: 9d8e4b6e28caec1a, dbSize: 258572288, dbInUse: 127373312, leader ID: 14104827831415598805
I0913 06:27:15.283363 1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"f91afb98-c193-4714-a064-1a4b96be2b3b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DefragControllerDefragmentSuccess' etcd member has been defragmented: master-2.ocp4.example.com, memberID: 11353094647150472218
I0913 06:27:53.319572 1 defragcontroller.go:289] etcd member "master-1.ocp4.example.com" backend store fragmented: 50.60 %, dbSize: 257544192
I0913 06:27:53.320504 1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"f91afb98-c193-4714-a064-1a4b96be2b3b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DefragControllerDefragmentAttempt' Attempting defrag on member: master-1.ocp4.example.com, memberID: c3be6a2e9d91eed5, dbSize: 257544192, dbInUse: 127229952, leader ID: 14104827831415598805
I0913 06:27:54.868521 1 event.go:285] Event(v1.ObjectReference{Kind:"Deployment", Namespace:"openshift-etcd-operator", Name:"etcd-operator", UID:"f91afb98-c193-4714-a064-1a4b96be2b3b", APIVersion:"apps/v1", ResourceVersion:"", FieldPath:""}): type: 'Normal' reason: 'DefragControllerDefragmentSuccess' etcd member has been defragmented: master-1.ocp4.example.com, memberID: 14104827831415598805
I0913 06:37:33.205796 1 defragcontroller.go:289] etcd member "master-0.ocp4.example.com" backend store fragmented: 11.44 %, dbSize: 149323776
I0913 06:37:33.205823 1 defragcontroller.go:289] etcd member "master-2.ocp4.example.com" backend store fragmented: 10.69 %, dbSize: 148103168
I0913 06:37:33.205829 1 defragcontroller.go:289] etcd member "master-1.ocp4.example.com" backend store fragmented: 10.66 %, dbSize: 148066304
I0913 06:48:33.206092 1 defragcontroller.go:289] etcd member "master-0.ocp4.example.com" backend store fragmented: 10.22 %, dbSize: 149323776
I0913 06:48:33.206187 1 defragcontroller.go:289] etcd member "master-2.ocp4.example.com" backend store fragmented: 9.42 %, dbSize: 148103168
I0913 06:48:33.206226 1 defragcontroller.go:289] etcd member "master-1.ocp4.example.com" backend store fragmented: 10.65 %, dbSize: 150134784
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.