Storage recommendations for optimal etcd performance

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 3.11
  • 4

Issue

• OpenShiftofficial documentation does not provide clear recommendations for etcd backend storage.
• The requirements for etcd performance are not clearly defined.
• Receiving warning in RHOCP 4 for 99th percentile commit durations on etcd instance.

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products, or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

• etcd's performance depends strongly on the performance of the storage backing etcd.

• As per Content from etcd.io is not included.etcd's documentation the 99th percentile of the metric etcd_disk_wal_fsync_duration_seconds_bucket should be less than 10ms for storage to be considered fast enough for a production etcd cluster.

• Additional information is provided in the below-mentioned KCS solutions -

  • etcd backend performance requirements for OpenShift.

• Due to these requirements, Red Hat makes the following recommendation:

  • It is highly recommended to use etcd with storage that quickly handles small read/write operations, such as NVMe or SSD-based storage solutions. Ceph (and ODF), NFS, RAID configurations, other network-based storage, and spinning disks are not recommended.

• For more information, please refer to this This page is not included, but the link has been rewritten to point to the nearest parent document.Red Hat documentation.

Root Cause

• Clustered etcd is extremely sensitive to storage and network backend performance, and can be easily disrupted by any underlying bottlenecks.

Diagnostic Steps

• Check etcd backend performance requirements for OpenShift with additional information about metrics, fio tests, and other performance requirements.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.