ETCD performance troubleshooting guide for OpenShift Container Platform

Updated 13 Nov 2025

ISSUE

The following problems could be observed in a generic way, among the following examples:

Content from github.com is not included.ETCD alerts from etcd-cluster-operator like:

etcdHighFsyncDurations
etcdInsufficientMembers
etcdMembersDown
etcdNoLeader
etcdBackendQuotaLowSpace
etcdGRPCRequestsSlow

Random timeouts in the cluster.
The cluster doesn't look stable.
oc login doesn't work every time.
ETCD members where RAFT indexes mismatch (which could mean some members are not fast enough or have connection issues).
Problem collecting must-gather (RHOCP 4) due to timeouts.

ENVIRONMENT

Red Hat OpenShift Container Platform (RHOCP)
- 3.11
- 4

RESOLUTION

The best performance can be achieved by:

Following only official Red Hat documentation as not everything suggested in upstream is supported.
Masters (where ETCD runs) using dedicated disk (not shared with other nodes/workers, using dedicated SSD or LUN).
- For cloud environments, a "fast" disk, i.e: "IOPS provisioned" should be used. For AWS the storage type io1, io2 or io2-block-express should be used and IOPS set to 2000. On the VMWare platform, snapshots of VMs should be disabled as this can have a serious impact on I/O.
- For bare metal environments with local devices, it is highly recommended that you use storage devices that handle serial writes (fsync) quickly, such as NVMe or SSD designed for write intensive workloads. Be aware that we have different types of SSD (SATA/SAS) drives in the field. These categories are constantly evolving, but we can generically distinguish SSD drives with the following categories:
  - Single-Level Cell (SLC) SSDs:
    - Fastest type of SSD, ideal for write intensive workloads
    - Reliable
    - Higher Price per GB, but smaller drives are enough for etcd proposes
  - Multi-Level Cell (MLC) SSDs:
    - Slower than SLCs
    - Less Reliable than SLCs
    - Affordable Price per GB
  - Triple-Layer Cell (TLC) SSDs:
    - More capacity
    - Less Reliable than MLCs
    - Most common type of SSD, due to good cost vs benefit relation
    - Affordable Price per GB
Storage should have generally
- Low latency for quick reads
- High bandwidth write for faster compactions/defrag
- High bandwidth read for quicker recovery on failure
Spinning disks and network drives are highly discouraged (mainly due to high or too variable latency)
NFS is highly discouraged (for masters only)
Masters don’t share with io-heavy write workload like log files
see also Storage recommendations for optimal etcd performance
For CEPH check the following article: Should I use Ceph or ODF to back etcd for my Openshift Cluster?
Masters have enough resources (CPU, RAM).
For Single-node (SNO) or Three-Node OpenShift Compact Clusters, make sure that you have dedicated NVMe or SSD drives for the control plane and separated Drives for application and another infrastructure stacks. Eventually, according to current workload in place, dedicated SSD drives for etcd (/var/lib/etcd) must be also taken in consideration.
Network can handle enough IO and bandwidth between masters is fast enough (ps: having masters in different DCs is not recommended as rather This content is not included.RHACM should be used, but in case there's no other option, network latency should be below 2ms KCS).
Following best practices like defragmentation, cleaning up unused projects, secrets, deployments (or any CRDs) and evaluating risks of installing new operators.

ROOT CAUSE

Slow disk

Fast disks are the most critical factor for ETCD deployment performance and stability. A slow disk will increase ETCD request latency and potentially hurt cluster stability. Since ETCD’s consensus protocol depends on persistently storing metadata to a log, a majority of ETCD cluster members must write every request down to disk.

Several indicators of issues can be found within ETCD logs:

mvcc: finished scheduled compaction at 241674166 (took 112.969552ms)

mvcc: finished scheduled compaction at 12281724800 (took 3m21.718193754s)
mvcc: finished scheduled compaction at 12282047743 (took 3.516775221s)
mvcc: finished scheduled compaction at 12282165959 (took 3.491617904s)

Since ETCD keeps an exact history of its keyspace, this history should be periodically compacted to avoid performance degradation and eventual storage space exhaustion. Compacting the keyspace history drops all information about keys superseded prior to a given keyspace revision. The space used by these keys then becomes available for additional writes to the keyspace.

IMPORTANT:
Compaction should be ideally around 200ms+ on smaller cluster but no more than 900ms to 2s on large clusters (20 to 100+ workers). These numbers might vary by 200ms plus minus and should NOT be taken literally.
Any compaction value above mentioned threshold could mean performance issues but it should be collerated with other metrics and not taken out of context.

Note that a higher value may not directly impair your cluster performance, most of the time you may only see slightly higher memory usage on etcd.
Compaction is running in batches of 1000 revisions at a time, each is finished by a database transaction to persist the change to disk. To write this transaction, a lock needs to be acquired and is blocking active writes. This is measured using the following PromQL query:

histogram_quantile(0.99, sum by(le, instance) (rate(etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket[5m])))

This directly affects the tail latency of incoming write requests, and may also result in "apply entries took too long" logs, as explained below.

apply entries took too long

After a majority of ETCD members agree to commit a request, each ETCD server applies the request to its data store and persists the result to disk. Even with a slow mechanical disk or a virtualized network disk, such as Amazon’s EBS or Google’s PD, applying a request should normally take fewer than 50 milliseconds (and around 5ms for fast SSD/NVMe disk). If the average applies duration exceeds 100 milliseconds, ETCD will warn that entries are taking too long to apply. There should be minimum of 'took too long' messages on the healthy cluster, ideally below few hundreds of them.

IMPORTANT: When looking for the took too long messages, it's needed to check the date and the amount of time in the message. For example, it's not the same ~1000 messages in 5 days for 100ms as 30k messages in just one day for 500ms or even more.

failed to send out heartbeat on time
server is likely overloaded

ETCD uses a leader-based consensus protocol for consistent data replication and log execution. Cluster members elect a single leader, all other members become followers. The elected leader must periodically send heartbeats to its followers to maintain its leadership. Followers infer leader failure if no heartbeats are received within an election interval and trigger an election. If a leader doesn’t send its heartbeats in time but is still running, the election is spurious and likely caused by insufficient resources. To catch these soft failures, if the leader skips two heartbeat intervals, ETCD will warn it failed to send a heartbeat on time.
Usually this issue is caused by a slow disk. Before the leader sends heartbeats attached with metadata, it may need to persist the metadata to disk. The disk could be experiencing contention among ETCD and other applications, or the disk is too simply slow

IMPORTANT:
server is likely overloaded is quite severe warning related either to slow network or slow disk. These warnings shouldn't be taken lightly as they might indicate serious performance problems.

CPU starvation

If monitoring of the machine’s CPU usage shows heavy utilization, there may not be enough compute capacity for etcd. Moving ETCD to a dedicated machine, increasing process resource isolation with cgroups, or renicing the ETCD server process into a higher priority can usually solve the problem.

This is a usual problem on virtualized platforms (like VMware) where the customer allows overcommitment on master's CPU or on compact clusters, where CPU intensive workload is contending with etcd or other control plane components.

The overcommitment on master's CPU is visible with CPU steal in the Virtual Machine. If CPU Steal is present, this means that the Hypervisor is overcommitted. For verifying it, run the top command and review in the %Cpu row, the field st that it should have the value "0.0":

%Cpu(s):  0.0 us,  1.6 sy,  2.7 ni, 95.3 id,  0.2 wa,  0.1 hi,  0.1 si,  0.0 st

Slow network

can also cause this issue. If network metrics among the ETCD machines shows long latencies or high drop rate, there may not be enough network capacity for ETCD. Moving etcd members to a less congested network will typically solve the problem.
However, if the etcd cluster is deployed across data centers, long latency between members is expected.

WARNING Tuning the heartbeat-interval as noted in Content from etcd.io is not included.3.4 tuning documentation is not supported and latency between DCs should be ideally less than 2ms.

Diagnostic Steps

OpenShift Container Platform 3.11: ETCD is located in kube-system project
OpenShift Container Platform 4.x: ETCDis located in openshift-etcd project.

oc get pod -n openshift-etcd
oc logs etcd-XYZ-master-0 -c etcd -n openshift-etcd
oc rsh -n openshift-etcd <etcd pod>
(From inside container run below commands)
etcdctl member list -w table
etcdctl endpoint health --cluster
etcdctl endpoint status -w table

in case oc command doesn't work, connect with ssh to node and run

$ crictl logs $(crictl ps -aql --label  "io.kubernetes.container.name=etcd-member")
$ crictl logs  --since 48h $(crictl ps -aql --label  "io.kubernetes.container.name=etcd-member")

Important: just because etcdctl endpoint health --cluster says all members are healthy, it doesn't mean they perform well. Always check full ETCD logs.

METRICS

Metrics are the most important values as they show how performance behaves over several days.

IMPORTANT:The tests like fio or etcdctl perf check are short tests executed at a specific moment and will not give an overall view of how performance behaves over several days. Storage latency can be at a specific time OK, but there could be peaks over the day when performance is not fit for ETCD. Also, be aware that running a performance benchmark can add additional load on cluster and degrade its performance.

Please see How to graph etcd metrics using Prometheus to gauge Etcd performance in OpenShift for more details.

Most important metric: etcd_disk_wal_fsync_duration 99th and 99.9th

should be lower than 10ms

>2ms = superb, probably NVMe on baremetal or AWS with io1 disk and 2000 IOPS set.
>5ms = great, usually well performing virtualized platform
5-7ms = OK
8-10ms = close to threshold, NOT GOOD if any peaks occur

Some versions may suggest that threshold is 20ms. Still, check docs and evaluate how many percents close to threshold value is and asses performance risk. Values above 15ms are also not good as they are close to threshold of 20ms.
Usually when 99th is close to threshold, we will see 99.9th going above threshold, which means storage can barely provide required performance (for ETCD) and it's really better when 99.0th is below 10ms.

Optional ETCD-Perf/FIO test

You can use standard etcd-perf tool (for ETCD benchmark) or upstream openshift-etcd-suite (for overall disk benchmark).

For OpenShift 4 connect to a Master node usingoc debug node/<master_node> and run a container with the image using podman:

  $ oc debug node/<master_node>
  [...]
  sh-4.4# chroot /host bash
  podman run --privileged --volume /var/lib/etcd:/test quay.io/peterducai/openshift-etcd-suite:latest fio

For OpenShift 3.11 connect to a Master node using ssh and run a container with the image:

  docker run --privileged --volume /var/lib/etcd:/test quay.io/peterducai/openshift-etcd-suite:latest fio

The following is an example output:

podman run --volume /$(pwd):/test:Z quay.io/peterducai/openshift-etcd-suite:latest fio
Trying to pull quay.io/peterducai/openshift-etcd-suite:latest...
Getting image source signatures
Copying blob fba03863e493 done  
Copying blob 75f075168a24 skipped: already exists  
Copying config 34d94812e9 done  
Writing manifest to image destination
Storing signatures
FIO SUITE version 0.1.27
 
WARNING: this test can run for several minutes without any progress! Please wait until it finish!
 

[ RANDOM IOPS TEST ]

[ RANDOM IOPS TEST ] - REQUEST OVERHEAD AND SEEK TIMES] ---
This job is a latency-sensitive workload that stresses per-request overhead and seek times. Random reads.
 

1GB file transfer:
  read: IOPS=55.1k, BW=215MiB/s (226MB/s)(1024MiB/4757msec)
--------------------------
RANDOM IOPS: 55000
--------------------------

200MB file transfer:
  read: IOPS=55.1k, BW=215MiB/s (226MB/s)(200MiB/929msec)
--------------------------
RANDOM IOPS: 55000
--------------------------

[ SEQUENTIAL IOPS TEST ]

[ SEQUENTIAL IOPS TEST ] - [ ETCD-like FSYNC WRITE with fsync engine ]

the 99th percentile of this metric should be less than 10ms

cleanfsynctest: (g=0): rw=write, bs=(R) 2300B-2300B, (W) 2300B-2300B, (T) 2300B-2300B, ioengine=sync, iodepth=1
fio-3.29
Starting 1 process
cleanfsynctest: Laying out IO file (1 file / 22MiB)

cleanfsynctest: (groupid=0, jobs=1): err= 0: pid=89: Tue Sep 27 16:39:22 2022
  write: IOPS=230, BW=517KiB/s (529kB/s)(22.0MiB/43595msec); 0 zone resets
    clat (usec): min=4, max=37506, avg=63.37, stdev=393.00
     lat (usec): min=4, max=37508, avg=64.45, stdev=393.12
    clat percentiles (usec):
     |  1.00th=[    7],  5.00th=[   16], 10.00th=[   18], 20.00th=[   20],
     | 30.00th=[   25], 40.00th=[   27], 50.00th=[   31], 60.00th=[   42],
     | 70.00th=[   63], 80.00th=[   88], 90.00th=[  122], 95.00th=[  143],
     | 99.00th=[  334], 99.50th=[  717], 99.90th=[ 1369], 99.95th=[ 1516],
     | 99.99th=[ 6652]
   bw (  KiB/s): min=   49, max= 1105, per=99.86%, avg=516.54, stdev=283.00, samples=87
   iops        : min=   22, max=  492, avg=230.16, stdev=125.97, samples=87
  lat (usec)   : 10=2.22%, 20=19.09%, 50=43.13%, 100=20.00%, 250=14.21%
  lat (usec)   : 500=0.59%, 750=0.28%, 1000=0.20%
  lat (msec)   : 2=0.24%, 10=0.02%, 50=0.01%
  fsync/fdatasync/sync_file_range:
    sync (usec): min=1245, max=293908, avg=4270.40, stdev=6256.20
    sync percentiles (usec):
     |  1.00th=[ 1532],  5.00th=[ 1811], 10.00th=[ 1926], 20.00th=[ 2180],
     | 30.00th=[ 2704], 40.00th=[ 3130], 50.00th=[ 3294], 60.00th=[ 3490],
     | 70.00th=[ 3785], 80.00th=[ 4359], 90.00th=[ 5538], 95.00th=[ 6456],
     | 99.00th=[38011], 99.50th=[43254], 99.90th=[62653], 99.95th=[65799],             <--- 99.0th and 99.9th percentile that should be below 10k
     | 99.99th=[73925]
  cpu          : usr=0.43%, sys=18.91%, ctx=52895, majf=0, minf=15
  IO depths    : 1=200.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,10029,0,0 short=10029,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

Run status group 0 (all jobs):
  WRITE: bw=517KiB/s (529kB/s), 517KiB/s-517KiB/s (529kB/s-529kB/s), io=22.0MiB (23.1MB), run=43595-43595msec

--------------------------
SEQUENTIAL IOPS: IOPS=230
BAD.. 99th fsync is higher than 10ms (10k).  38011
--------------------------

[ SEQUENTIAL IOPS TEST ] - [ libaio engine SINGLE JOB, 70% read, 30% write]
 
--------------------------
1GB file transfer:
  read: IOPS=10.3k, BW=40.3MiB/s (42.2MB/s)(471MiB/11683msec)
  write: IOPS=4444, BW=17.4MiB/s (18.2MB/s)(203MiB/11683msec); 0 zone resets
SEQUENTIAL WRITE IOPS: 4444
SEQUENTIAL READ IOPS: 10000
--------------------------
--------------------------
200MB file transfer:
  read: IOPS=13.8k, BW=53.7MiB/s (56.3MB/s)(140MiB/2608msec)
  write: IOPS=5881, BW=23.0MiB/s (24.1MB/s)(59.9MiB/2608msec); 0 zone resets
SEQUENTIAL WRITE IOPS: 5881
SEQUENTIAL READ IOPS: 13000
--------------------------
 
-- [ libaio engine SINGLE JOB, 30% read, 70% write] --
 
--------------------------
200MB file transfer:
  read: IOPS=6517, BW=25.5MiB/s (26.7MB/s)(60.2MiB/2366msec)
  write: IOPS=15.1k, BW=59.1MiB/s (61.9MB/s)(140MiB/2366msec); 0 zone resets
SEQUENTIAL WRITE IOPS: 15000
SEQUENTIAL READ IOPS: 6517
--------------------------
 
--------------------------
1GB file transfer:
  read: IOPS=5893, BW=23.0MiB/s (24.1MB/s)(68.7MiB/2986msec)
  write: IOPS=13.7k, BW=53.7MiB/s (56.3MB/s)(160MiB/2986msec); 0 zone resets
SEQUENTIAL WRITE IOPS: 13000
SEQUENTIAL READ IOPS: 5893
--------------------------
 
- END -----------------------------------------

Most important is FSYNC IOPS value as libiao is different kind of sequential write and doesn't reflect how ETCD writes data.

required fsync sequential IOPS:

50 - minimum, local development
300 - small to medium cluster with average load
500 - medium or large cluster with heavy load
800+ - large cluster with heavy load

Be aware that IOPS is just one reference number and equally important is also fsync latency itself (0.99 and 0.999 percentile which should be always below 10ms!), network latency and CPU power. Do not focus on IOPS, but rather on whole set of metrics and data!

To run etcd-perf, run

    $ oc debug node/<master_node>
    [...]
    sh-4.4# chroot /host bash
    [root@<master_node> /]# podman run --volume /var/lib/etcd:/var/lib/etcd:Z quay.io/openshift-scale/etcd-perf

For OpenShift 3.11 connect to a Master node using ssh and run a container with the image:

    $ sudo docker run --volume /var/lib/etcd:/var/lib/etcd:Z quay.io/openshift-scale/etcd-perf

The container will be pulled by the client and run a specially tailored version of fio. The results provide the 99th percentile of fsync and if it is in the recommended threshold to host etcd or not.

The following is an example output:

---------------------------------------------------------------- Running fio ---------------------------------------------------------------------------
{
  "fio version" : "fio-3.7",
  "timestamp" : 1608484646,
  "timestamp_ms" : 1608484646648,
  "time" : "Sun Dec 20 17:17:26 2020",
  "global options" : {
    "rw" : "write",
    "ioengine" : "sync",
    "fdatasync" : "1",
    "directory" : "/var/lib/etcd",
    "size" : "22m",
    "bs" : "2300"
  },
[...]
99th percentile of fsync is 5865472 ns
99th percentile of the fsync is within the recommended threshold - 10 ms, the disk can be used to host etcd

and the message if the disk is not fast enough:

99th percentile of fsync is 5865472 ns
99th percentile of the fsync is greater than the recommended value which is 10 ms, faster disks are recommended to host etcd for better performance.

IMPORTANT:

A very common scenario is where ETCD metrics jump up and down and 50% of the time you get 'the disk can be used to host etcd' despite there are heavy performance issues.

Optional DD test

Please take this test only as a reference that should indicate if the disk is slow or fast enough. You could omit oflag or change other parameters to make test faster, but that's not the point of test itself.

Make sure there is enough free space, this test will create a ~400 MB file to test the IO.

$ df -h /var/lib/etcd

Check the performance with 4K block size (it will take time, depending on the disk performance):

$ dd if=/dev/zero of=/var/lib/etcd/dd-zero.test bs=4k count=100000 oflag=direct

Check the performance with 1K block size (it will take time, depending on the disk performance):

$ dd if=/dev/zero of=/var/lib/etcd/dd-zero.test bs=1k count=400000 oflag=direct

Record the outputs, attach it to case and remove the test file:

$ rm /var/lib/etcd/dd-zero.test

For comparison here are numbers from different platforms

LAPTOP with fast NVMe (and CRC running):
$ dd if=/dev/zero of=$HOME/dd-zero.test bs=4k count=100000 oflag=direct
409600000 bytes (410 MB, 391 MiB) copied, 5.47258 s, 74.8 MB/s

$ dd if=/dev/zero of=$HOME/dd-zero.test bs=1k count=400000 oflag=direct
409600000 bytes (410 MB, 391 MiB) copied, 121.619 s, 3.4 MB/s

ps: the same laptop without CRC running gives  'copied, 0.329733 s, 1.2 GB/s'

TEST LAB:
$ dd if=/dev/zero of=$HOME/dd-zero.test bs=4k count=100000 oflag=direct
409600000 bytes (410 MB, 391 MiB) copied, 8.97905 s, 45.6 MB/s

$ dd if=/dev/zero of=$HOME/dd-zero.test bs=1k count=400000 oflag=direct
409600000 bytes (410 MB, 391 MiB) copied, 72.314 s, 2.1 MB/s 

BAREMETAL CLUSTER:
$ sudo dd if=/dev/zero of=$HOME/dd-zero.test bs=4k count=100000
409600000 bytes (410 MB, 391 MiB) copied, 3.66545 s, 112 MB/s
$ sudo dd if=/dev/zero of=$HOME/dd-zero.test bs=1k count=400000 oflag=direct
409600000 bytes (410 MB, 391 MiB) copied, 43.8072 s, 9.4 MB/s

and here is example of AWS cluster where performance is degraded

$ dd if=/dev/zero of=$HOME/dd-zero.test bs=1k count=400000 oflag=direct
409600000 bytes (410 MB, 391 MiB) copied, 240.198 s, 1.7 MB/s
$ dd if=/dev/zero of=$HOME/dd-zero.test bs=4k count=100000 oflag=direct
409600000 bytes (410 MB, 391 MiB) copied, 52.0834 s, 7.9 MB/s

Network latency

Overall network metrics can be checked, but also simple ping or curl test can be executed from masters with

curl -k https://api.<OCP URL>.com -w "%{time_connect}\n"

where time_connect is expected below 2ms (0.002 in output) but usually anything above 6-8ms (0.006-0.008) means performance issues. Ping can have a different value than connection to API and if the connection time is much higher (for example 100ms) than normal latency, then there's a possibility API is not performing well and could be overcommitted.

PS: there's ongoing discussion if a higher value for latency should be allowed

Dropped packets or RX/TX errors can be viewed with

$ ip -s link show
$ ifstat <NIC>
$ ifstat -d 10     (run ifstat on all NICs every 10 seconds)

ETCD cleanup

As ETCD and OCP is dynamic and live platform, is good to follow best practices and watch for bottlenecks. Here are to examples that could cause performance problems with ETCD.

Any number of CRDs (secrets, deployments, etc..) above 8k could cause performance issues on storage with not enough IOPS. Also check Control plane node sizing

$ oc project openshift-etcd
oc get pods
oc rsh <etcd pod>
> etcdctl get / --prefix --keys-only | sed '/^$/d' | cut -d/ -f3 | sort | uniq -c | sort -rn

Any non-openshift namespace with 20+ secrets should be cleaned up (unless there's specific customer need for so many secrets). ps: same could be done also with deployments, secrets and so on.

oc get secrets -A --no-headers | awk '{ns[$1]++}END{for (i in ns) print i,ns[i]}'

OTHER LINKS

Chinese translation of this article is at This content is not included.etcd

Azure

OCP 4 - Performance degradation on AZURE
Azure Disk performance by region
TLS handshake fails due to large packets discarded for OpenShift 4 on Azure

Additional

Recover ETCD quorum guard pod after a failing OpenShift 4 update
Is it possible to scale master / etcd nodes in OCP 4?
Can Etcd defrag process be automated?
How to list number of objects in etcd? 4.7
The etcd backup script fails to generate a backup
Prometheus not able to read etcd metrics after upgrade in OCP 3.11
EtcdCertSignerControllerDegraded error on etcd operator
3.11 How do I remove and add back an existing etcd member for the OpenShift cluster?
Storage recommendations for optimal etcd performance
ETCD pod is restarting frequently due to NTP out of sync

ETCD upstream

Content from etcd.io is not included.FAQ
Content from etcd.io is not included.Metrics
Content from etcd.io is not included.Tuning

SBR

Shift

Product(s)

Red Hat OpenShift Container Platform

Category

Troubleshoot

Components

Tags

Article Type

General