Guidance for Red Hat OpenShift Container Platform Clusters - Deployments Spanning Multiple Sites(Data Centers/Regions)

Updated

Note: This article covers cluster deployments that ‘span’ multiple data centers, where control plane nodes are distributed across multiple logical or physical locations, or even connected in the same data center over non-ideal network conditions. This applies to regional boundaries, multiple data centers, multiple platform providers, and multiple availability zones as defined by an organization.

Overview

Applicable Environments

  • Red Hat OpenShift Container Platform 4.x

Introduction

Red Hat strongly recommends a deployment model where OpenShift clusters are deployed within a data center, but also acknowledges that there may be scenarios where a provider may use a deployment model where a cluster may span across data centers. This document outlines considerations when exploring the use of cluster deployments that ‘span’ multiple data centers and important metrics that affect the supportability of such deployments. The design of such deployments should adhere to these guidelines for the product to function optimally and ensure the highest quality of support from Red Hat with the appropriate product support subscriptions.

WARNING: A cluster deployment that ‘spans’ multiple data centers extends the cluster as a single failure domain across locations and should not be considered a replacement for a disaster recovery plan.

General Guidance

Clusters with cluster deployments that ‘span’ multiple data centers are bound by standard Red Hat OpenShift Container Platform support guidance:

Red Hat does not recommend deploying an OpenShift Container Platform Cluster that spans multiple sites and that if you need to be in multiple data centers or regions that, you deploy multiple clusters (one per region/site) and use tools like This content is not included.Advanced Cluster Management (ACM) to manage these clusters and the deployments on them, however this guide is here to outline the best practices and guidance you need to keep in mind if you are exploring a stretched deployment.
Some OpenShift platforms have This page is not included, but the link has been rewritten to point to the nearest parent document.specific support for multi-datacenter deployments. Check the platform specific product documentation and release notes for details. Other platforms can span data centers, depending on the quality of the network connectivity between nodes. See guidance in this article for additional details and the specific information for etcd in Understanding etcd and the tunables/conditions affecting performance. When implementing a cluster deployment that ‘spans’ multiple data centers, you should strive to implement the Red Hat OpenShift Container Platform High Availability, and Recommended Practices. An alternative to multi-site deployments is to deploy an OCP cluster per site, managed by Advanced Cluster Management (ACM).

Caveats

The guidance provided in this document focuses on general aspects of cluster deployment that ‘spans’ multiple data centers. Some caveats to remember:

  • While deployments that ‘span’ multiple data center designs are not bound by any special support requirements, these clusters do have additional inherent complexities that may require additional consideration or support involvement (time to identify, remediate and resolve issues) when compared to a standard single-site cluster.
  • Applications might not work well or not work at all in clusters with high Kube API latency or low transaction rates.
  • Layered products (e.g., storage providers) have lower latency requirements. In those cases, the latency limits are dictated by the architectures that are supported by the layered product.
  • The failure scenarios are amplified with stretched control planes, and the way they are affected is specific to the deployment. Because of this, before using cluster a deployment that ‘spans’ multiple data centers on a production environment, the organization should test and document the behavior of the cluster during disruptions such as:
    • When there is a network partition leaving one, two, or all control plane nodes isolated
    • When there are MTU mismatches on the transport network among the control plane nodes
    • When there is a sustained spike in latency as a Day-2 event towards one or more of the control plane nodes
    • When there is a considerable change in jitter due to network congestion, misconfiguration or lack of QoS, an intermediate network device causing packet errors, and others
  • Clusters deployed across multiple sites, network infrastructures, storage infrastructures, or other components inherently have a higher number of points of failure. Network disruptions or splits become a larger threat to such clusters especially, putting the nodes at risk of losing contact with each other. These multi-site clusters must be designed with the potential for such failures in mind. Organizations deploying multi-site clusters should extensively test failure scenarios, and should consider whether the cluster has protection from all points of failure. Consult with Red Hat Support for assistance in considering the important aspects of a resilient High Availability cluster design.
  • In some cases GEO awareness is a requirement or issue that must be solved, to minimize latency, so a proper implementation of a Global Service Load Balancing (GSLB) method must be available.

Infrastructure as a Service (IaaS) or Cloud Provider

For the purpose of this document, this guidance applies to any infrastructure provider for which OpenShift control plane nodes are supported by the UPI installer (platform=none) or the agent-based installer (platform=metal) using the ”User Managed Network” option. IPI installers are not covered by these guidelines, however where possible, IPI deployments will span zones/availability zones on cloud/Iaas providers if possible (following these/similar guidelines). This means infrastructure provider-specific integrations will not be available (e.g., integration with Cloud provider's services like storage services and load balancers). Provider-specific services might still be used as external services.

Using different infrastructure platform providers for control plane nodes is discouraged (e.g., mixing nodes across IaaS, cloud, and bare-metal as control plane nodes). Consider the following guidelines when such combinations are needed:

  • The minimum effective MTU across the infrastructure should be the maximum MTU used for the deployment (using a lower MTU is okay).
    See the MTU discovery and validation section for more information.
  • The combined disk and network latency and jitter must maintain an etcd peer round trip time of less than 100ms. This is NOT the same as the network round trip time. See the ETCD timers in OpenShift section below.
    Layered products (e.g., storage providers) may have lower latency requirements. In those cases, the latency limits are dictated by the requirements of the architecture supported by the layered product. For example, OpenShift cluster deployments that ‘span’ multiple data centers with Red Hat OpenShift Data Foundation must have a latency requirement of less than 10ms RTT. For those cases, follow the specific product guidance.

OpenShift Container Platform 4.x we only support (exactly) three control plane members.

  • Note: One node control places are also supported with single node OpenShift (however given the context of this article these deployments don't apply to spanning clusters.
  • Note: Starting with 4.17, and later versions of OCP the number of control plane nodes (or etcd members) is a bit more configurable, please see your releases docs on the specific considerations needed to deploy clusters with more or less control plane nodes.

Assuming each site gets one control plane member, you theoretically define three sites, which is what Red Hat recommends. This allows for one data center to go into an inactive state and the cluster still maintains quorum and operational consistency. Please see Content from etcd.io is not included.etcd guidance on fault tolerance for more information on this guidance.

When this assumption is not met, attention should be given to the desired and actual fault tolerance state of the cluster, as it will often outline or dictate the operational capabilities (uptime and stability) of the deployment.

etcd Requirements

There is a large list of factors and considerations that go into planning an etcd cluster deployment. Thus when planning an OpenShift cluster that ‘spans’ multiple data centers, you need to plan for situations that will likely stress or push etcd to the edge of its operational limits.

See KCS 7010406 for more details on how to maintain operational capabilities and reduce service-affecting events and instability of the cluster.

Network Requirements

The chosen network topology must yield direct IP (eg L3) connectivity between nodes. The minimum effective MTU across the infrastructure should be the maximum MTU used for the deployment (using a lower MTU is okay).

See the MTU discovery and validation section for more information. The latency needs are ultimately defined by the services using the network so please see sections related to etcd and storage for more details on requirements.

In addition to the base networking requirements, you need to think about how applications will be accessed (ingress into the cluster). Some top level Global Service Load Balancing (GSLB) method will be needed (outside of openshift) to enable external traffic to connect to the OpenShift control plane services and ingress controllers.

Storage Requirements

When considering an OpenShift cluster deployment that ‘spans’ multiple data centers, special consideration needs to be given to the selected storage integration, to ensure that it also meets "multi-site" requirements, as it pertains to accessibility (from all sites), fault tolerance, high availability, etc.

An "object storage" solutions should be used for the registry, and this storage solution needs to be in addition to any PV storage integration used (for application volumes/workloads). This "object storage" solution should also have the same special considerations given to accessibility (from all sites), fault tolerance, high availability, etc.

Since Disk IO is a critical factor in the health of etcd database, It is required that they are deployed on on high speed low latency media, see etcd guidance on etcd peer round trip time and etcd database size for more details on the exact requirements to meet.

Workload Placement Considerations

Due to the way workloads are places/scheduled on a 'cluster', with clusters that span multiple sites, administrators / developers have to take into account special considerations to ensure that critical workloads (routers, registries, applications, etc) are scheduled/placed based on the proper hardware/hosts within the topology of the cluster to ensure that the applications/services are Highly Available and Fault Tolerant based on the topology of the clusters deployment.

Without considering this, it is possible for OpenShift/Kubernetes to schedule workloads on hosts within the cluster so that a Single Point of Failure (SPoF) is created for OpenShift infrastructure services as well as other application services if there is a datacenter outage.

Product(s)
Category
Article Type