Distributed Inference with llm-d

Red Hat AI Inference 3.5

Architecture, components, and deployment of Distributed Inference with llm-d for scalable LLM serving on Kubernetes

Abstract

Learn about Distributed Inference with llm-d, a Kubernetes-native framework for serving large language models at scale.

Preface

Distributed Inference with llm-d is a Kubernetes-native framework for serving large language models at scale. You can deploy Distributed Inference with llm-d on OpenShift Container Platform or managed Kubernetes platforms such as Azure Kubernetes Service (AKS), CoreWeave Kubernetes Service, and Amazon Elastic Kubernetes Service (EKS).

Chapter 1. Distributed Inference with llm-d

The Distributed Inference with llm-d framework provides enterprise-grade large language model (LLM) inference serving on OpenShift Container Platform and managed Kubernetes clusters on public clouds such as Azure Kubernetes Service (AKS) and CoreWeave Kubernetes Service (CKS).

Enterprise platform engineering and infrastructure teams can use Distributed Inference with llm-d to build generative AI model services for internal and external use cases. Cloud service providers can also use it to build Models-as-a-Service (MaaS) offerings. Common use cases include:

Enterprise-wide Models-as-a-Service (MaaS) for generative AI: A central platform team provides generative AI and LLM capabilities as a managed service to business units across the organization. Rather than each team provisioning its own inference infrastructure, the platform team uses Distributed Inference with llm-d to offer standardized model serving with consistent performance, cost control, and security.
Production-ready inference at scale: An organization deploys a generative AI application in a limited production environment, such as A/B testing with a small user group or a soft launch. The deployment must be production-ready with reliable performance and security, while remaining flexible enough to scale to provider-grade inference as demand grows.

Chapter 2. Distributed Inference with llm-d core components

Distributed Inference with llm-d deploys a distributed inference stack through Helm charts packaged as OCI container images, integrating industry-standard Kubernetes components to provide intelligent model serving, secure communication, and scalable resource management.

The Distributed Inference with llm-d inference stack consists of the following core components:

Red Hat AI Inference: High-throughput, low-latency LLM inference engine that runs in worker pods and processes model inference requests.
KServe LLMISvc Controller: Kubernetes-native model serving control plane that defines the LLMInferenceService custom resource and manages model lifecycle, deployment orchestration, and operational governance. KServe reconciles the required serving state declared in the custom resource and coordinates component deployment. The vLLM model server exposes OpenAI-compatible APIs. Request routing decisions are handled by the llm-d router, also known as the Endpoint Picker (EPP).
llm-d router (Endpoint Picker): Scheduling service that makes per-request routing decisions by integrating with the gateway through the ext-proc protocol. The scheduler uses a pluggable system of filters, scorers, and profile handlers to evaluate KV cache locality, queue depth, session affinity, and load distribution when selecting optimal model-serving pods.
Routing sidecar: Sidecar service deployed alongside each decode model-serving pod that orchestrates disaggregated inference execution. The routing sidecar coordinates prefill and decode stages, manages KV cache transfers between pods, and provides SSRF protection. The routing sidecar is optional and is only required for prefill-decode disaggregation deployments.
Gateway API with Red Hat Connectivity Link: Kubernetes standard API for traffic routing and load balancing, with the llm-d inference scheduling specification for intelligent request routing. Red Hat Connectivity Link provides the Gateway API implementation.
Istio/Sail Operator: Service mesh that provides mutual TLS (mTLS) encryption, traffic routing, and observability across the inference stack.
cert-manager: Automates Transport Layer Security (TLS) certificate management and provides a Certificate Authority for mutual TLS (mTLS) communication between inference components.
LeaderWorkerSet: Kubernetes workload API for coordinating multi-node distributed inference deployments across multiple AI accelerator nodes. LeaderWorkerSet manages pod topology and lifecycle for workloads that span multiple GPU nodes, such as Mixture of Experts (MoE) models with expert parallelism.
Note
The LeaderWorkerSet Operator is required for multi-node deployments with Distributed Inference with llm-d. Prefill-decode disaggregation uses the routing sidecar and EPP, not LeaderWorkerSet.

Chapter 3. Inference scheduling and caching capabilities

Distributed Inference with llm-d provides intelligent scheduling, caching, and resource management capabilities for distributed inference. These features optimize GPU usage, reduce inference latency, and enable cost-effective scaling of large language models.

Intelligent inference scheduling: Provides prefix-cache aware routing that directs each request to the replica most likely to have relevant KV cache entries already populated, maximizing GPU KV cache reuse. The inference scheduler evaluates KV cache utilization, queue depth, cache residency, service level agreement (SLA) constraints, and load distribution across nodes to select the optimal replica for each request.
KV cache management: Manages key-value cache efficiently across distributed inference servers, reducing memory requirements and enabling longer context windows. Routing requests to replicas with warm KV cache entries avoids redundant prompt processing, which improves both throughput and time-to-first-token.
Prefill-decode disaggregation: Separates the compute-intensive prefill phase from the latency-sensitive decode phase, allowing you to assign each phase to appropriately optimized resources and scale them independently. The prefill phase processes the full input prompt in parallel and is assigned to compute-optimized resources. The decode phase generates tokens incrementally and is assigned to latency-optimized resources. This phase-aware architecture increases GPU utilization, reduces tail latency, and lowers cost per token.
Note
Prefill-decode disaggregated serving is generally available (GA) for CKS deployments.
Important
Prefill-decode disaggregation on OpenShift Container Platform and AKS is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
Batch inferencing: Enables asynchronous, fire-and-forget job submission through the OpenAI-compatible /v1/batches API. You can submit large volumes of inference requests as batch jobs that run at lower priority than interactive traffic, filling GPU capacity during low-utilization periods without affecting real-time SLOs. The batch processor uses adaptive concurrency control and prefix-cache-aware request ordering to maximize throughput.
Important
Batch inference for Distributed Inference with llm-d is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
Wide expert parallelism: Supports efficient distributed inference of mixture of experts (MoE) models across many GPU nodes, enabling cost-effective scaling of large models.
Important
Wide expert parallelism (WideEP) is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.
DP-aware load balancing: Enables the Endpoint Picker to route requests to individual data-parallel ranks within WideEP pods for prefix-cache-aware scheduling. A single vllm serve command in external multi-port DP mode provides one serving port per local DP rank for that routing, while the DP Supervisor starts the DP ranks and reports aggregated health on a separate admin port for Kubernetes probes. This feature delivers up to 26-30% higher throughput compared to hybrid load balancing for WideEP deployments.
Important
DP-aware load balancing is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.
For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

Chapter 4. Intelligent inference scheduling architecture

You can use intelligent inference scheduling in Distributed Inference with llm-d to route inference requests to the optimal model server replica based on real-time metrics such as queue depth, KV cache utilization, and prefix cache state. Intelligent scheduling reduces inference latency by up to 2x and increases throughput by up to 3x compared to round-robin load balancing by making cache-aware, load-aware routing decisions.

The Distributed Inference with llm-d inference stack uses a four-layer architecture that separates concerns between networking, scheduling intelligence, model serving, and observability.

Gateway layer

The Gateway layer handles ingress traffic using the Kubernetes Gateway API with a supported data plane such as Istio. The Gateway API Inference Extension integrates the Endpoint Picker (EPP) into the request path by using the Envoy external processing (ext-proc) protocol. When a request arrives at the inference gateway, Envoy forwards it to the Endpoint Picker for intelligent routing before delivering it to the selected model server.

The Gateway layer provides:

TLS termination and certificate management, optionally through cert-manager
HTTP and gRPC protocol support for inference requests, where gRPC is used for the ext-proc communication between the gateway and the EPP
Integration with the Kubernetes Gateway API Inference Extension specification

Scheduling layer

The Endpoint Picker (EPP) is a request-path component that makes workload placement decisions for every incoming inference request. For each incoming inference request, the EPP evaluates all available model server replicas and selects the optimal target by using a configurable set of scheduling plugins.

The EPP uses a scheduling lifecycle that includes filter, score, and pick plugins:

Filter plugins eliminate unsuitable replicas.
Score plugins assign weighted scores to remaining replicas based on metrics such as queue depth, prefix cache affinity, and predicted latency.
Picker plugins select the final target from scored replicas.

You can customize the scheduling behavior by configuring scheduling profiles that define which plugins are active and how their scores are weighted. The EPP supports automatic plugin activation, which activates a default set of plugins when no explicit configuration is provided.

Data plane layer

The Data Plane layer consists of vLLM model server pods that run inference requests. Each vLLM instance exposes Prometheus metrics on port 8000 at the /metrics endpoint, providing real-time data about queue depth, KV cache utilization, prefix cache state, and token throughput.

The EPP continuously monitors these metrics to maintain an up-to-date view of each replica’s state, enabling informed scheduling decisions.

In multi-node deployments, LeaderWorkerSet (LWS) manages the lifecycle of distributed inference workloads that span multiple GPU nodes, such as Mixture of Experts (MoE) models with expert parallelism.

Observability layer

The Observability layer collects and exposes Prometheus-compatible metrics from all inference components. On OpenShift, the controller automatically creates PodMonitor and ServiceMonitor resources to scrape metrics from vLLM engines and the EPP. On other Kubernetes platforms, you must configure Prometheus scrape targets manually.

You can use these metrics to build dashboards, configure alerts, verify scheduling effectiveness, and plan fleet capacity.

Chapter 5. Distributed inference workloads

Distributed Inference with llm-d uses a layered architecture to distribute inference workloads across a Kubernetes cluster.

Client requests flow through an inference gateway that routes the requests to optimized model replicas, with intelligent scheduling and autoscaling to manage resources efficiently. The gateway provides OpenAI-compatible APIs with streaming token responses. Body-based routing selects the InferencePool from the model name specified in the API request.

The inference pool contains model-serving variants, such as Prefill and Decode, each running Red Hat AI Inference instances. Variants handle different phases of the inference process. Shared prefix caching enables key-value (KV) cache sharing between model-serving variants.

Figure 5.1. Distributed Inference with llm-d processing architecture

Distributed Inference with llm-d system architecture

Body-based routing selects the InferencePool from the model name in the OpenAI-compatible API spec.

The inference scheduler is an extensible framework of filters, scorers, and profile handlers that provides load, KV, and prefill/decode-aware routing. The scheduler evaluates GPU utilization, queue depth, cache residency, and service level agreement (SLA) constraints to direct each request to the optimal replica. Routing favors replicas with warm KV cache entries for the request’s prefix, maximizing cache reuse and avoiding redundant computation.

Each decode pod asynchronously reports KV cache availability and load metrics to the inference scheduler.

Chapter 6. Separation of control plane and data plane

The Distributed Inference with llm-d architecture separates the model serving control plane from the inference data plane.

KServe manages model lifecycle, scaling, and API exposure. The llm-d inference scheduler handles runtime-aware scheduling, cache locality optimization, and intelligent request distribution across pods and nodes. This separation enables platform teams to swap runtimes or schedulers independently and integrate future innovations without redesigning the stack.

You deploy the inference stack by using Helm charts distributed as OCI container images, making it compatible with OpenShift Container Platform 4.19 or later and any Cloud Native Computing Foundation (CNCF) certified managed Kubernetes 1.33 or later cluster. On OpenShift Container Platform, the chart integrates with Operator Lifecycle Manager (OLM) to install required Operators automatically. On managed Kubernetes, the chart installs all dependencies directly.

Chapter 7. Disaggregated and aggregated inference serving modes

You can deploy models in aggregated mode or disaggregated mode, depending on your workload characteristics and performance requirements. Understanding the architectural difference between these modes helps you choose the deployment that best fits your latency, throughput, and resource utilization goals.

Aggregated serving mode

In aggregated mode, each model-serving pod handles both inference phases: prefill and decode. The prefill phase processes the full input prompt in parallel to populate the key-value (KV) cache. The decode phase generates output tokens one at a time by using the cached data.

All pods in the deployment share the same resource profile, and the inference scheduler routes each request to a single pod that performs both phases sequentially.

Aggregated mode is simpler to operate and works well for workloads with uniform resource demands, small models, or short input sequences where the prefill phase does not dominate GPU utilization.

Disaggregated serving mode

In disaggregated mode, the prefill and decode phases run on independent pod pools with separate replica counts and resource allocations. Prefill pods process incoming prompts and transfer the resulting KV cache data to decode pods over RDMA-capable networking by using the NIXL connector. Decode pods then generate output tokens from the transferred KV cache without repeating the prefill computation.

Disaggregated mode creates two separate Kubernetes Deployment resources: one for prefill pods and one for decode pods.

Disaggregated mode enables independent scaling of each phase. You can configure different replica counts and tensor parallelism degrees for prefill and decode pods to match their resource profiles. For example, you can run more prefill replicas at lower tensor parallelism for compute throughput, and fewer decode replicas at higher tensor parallelism for memory bandwidth. Both pod pools must use the same GPU type because KV cache layout must be compatible between prefill and decode pods.

By eliminating interference between the compute-intensive prefill phase and the memory-bandwidth-intensive decode phase, disaggregated mode improves GPU utilization and reduces inter-token latency (ITL). For workloads with latency-sensitive service level objectives, disaggregated mode can significantly reduce tail latency. However, because fewer total resources are dedicated to prefill processing, time-to-first-token (TTFT) typically increases compared to aggregated mode.

NIXL KV cache transfer

In disaggregated mode, prefill pods transfer computed KV cache data to decode pods by using the NIXL connector, which manages high-speed GPU memory transfers. The NIXL connector uses RDMA-capable networking to move KV cache tensors directly between GPU memory on different nodes, avoiding CPU copies and minimizing transfer latency.

Switching between modes

You can switch between aggregated and disaggregated mode by updating the LLMInferenceService CR. To enable disaggregated mode, add the spec.prefill block to provision separate prefill pods and configure --kv_transfer_config in your container arguments. Removing the spec.prefill block returns the service to aggregated mode, terminating dedicated prefill pods and routing all execution phases to unified pods.

Chapter 8. Workload characteristics for disaggregated inference

Evaluate whether your workload benefits from prefill/decode disaggregation by analyzing model size, input/output ratios, traffic patterns, and latency requirements. Disaggregated serving is most effective for large models with mixed workloads, but can degrade performance for workloads that do not match the right profile.

Prefill and decode have fundamentally different resource profiles. The prefill phase processes the entire input prompt in parallel and is compute-bound. The decode phase generates tokens one at a time and is memory-bandwidth-bound. In aggregated mode, a single pod must handle both profiles, leading to resource contention and suboptimal utilization of either phase.

Note

When evaluating disaggregated serving for your workload, consider the prefill-to-decode GPU ratio. With asymmetric configurations, such as 4 prefill replicas with tensor parallelism of 2 and 2 decode replicas with tensor parallelism of 4, you can match resource allocation to phase demands. Monitor vllm:prompt_tokens_by_source_total metrics to verify that decode pods receive the majority of their prompt tokens from external KV transfer rather than local recomputation.

Table 8.1. Workload characteristics that benefit from disaggregation

Workload characteristic	Description
Large models with 30 billion or more parameters	Large models amplify the resource profile difference between prefill and decode. The compute demands of prefill scale with model size, making phase separation more valuable as model size increases.
Long input sequences	Workloads with long input prompts, such as document summarization, code review, or retrieval-augmented generation, consume significant GPU time in the prefill phase. You can dedicate more replicas to the prefill pool to handle the compute load.
Mixed workloads with variable input/output ratios	When your traffic includes a mix of long-prompt and short-prompt requests, aggregated pods cannot optimize for both patterns simultaneously. Disaggregation enables the scheduler to route each request type to the appropriate phase-optimized pool.
High concurrency with strict latency service level objectives	Under high concurrency, aggregated pods experience interference between prefill and decode operations competing for the same GPU resources. Disaggregation eliminates this contention, reducing inter-token latency, tail latency, and improving time-to-first-token consistency.

Table 8.2. Workload characteristics that favor aggregated serving

Workload characteristic	Description
Small models with fewer than 30 billion parameters	Smaller models have low prefill compute demands, reducing the benefit of phase separation. The additional latency from NIXL KV cache transfer can negate any performance gains.
Short input prompts with long output sequences	When prefill is a small fraction of total inference time, the additional latency introduced by KV cache transfer outweighs the benefits of disaggregation.
Environments without RDMA-capable networking	RDMA-capable networking such as RoCE or InfiniBand is recommended for optimal NIXL KV cache transfer performance. If RDMA is not available, NIXL falls back to TCP transport with reduced transfer performance. Without RDMA, transfer latency increases and can become a bottleneck for latency-sensitive workloads.
Low-traffic deployments	Disaggregated mode introduces operational complexity with two separate pod pools to manage. For low-traffic or development environments, aggregated mode is simpler to operate and monitor.

Chapter 9. Distributed Inference with llm-d deployment with Helm

The Distributed Inference with llm-d Helm chart deploys a complete inference stack on OpenShift Container Platform or managed Kubernetes. On OpenShift Container Platform, the chart uses Operator Lifecycle Manager (OLM) to install and configure required Operators automatically. On managed Kubernetes, the chart installs all dependencies directly.

The Helm chart uses a three-tier deployment model:

Red Hat OpenShift AI Operator: The chart installs the Red Hat OpenShift AI Operator through an OLM Subscription. The chart controls the DataScienceCluster and DSCInitialization custom resources that manage the inference stack lifecycle.
Helm chart components: The Helm chart includes the rhaii profile, which provides an inference-focused deployment. For distributed inference, the primary component is KServe, which provides the LLMInferenceService custom resource (CR) for deploying and managing inference services.
Operator dependencies: Each component declares the Operators it requires. KServe depends on cert-manager, LeaderWorkerSet, and Red Hat Connectivity Link. The Helm chart resolves these dependencies, including transitive dependencies, and installs each Operator through OLM automatically.

Figure 9.1. Deploying Distributed Inference with llm-d with Helm charts

Helm chart deployment architecture showing the RHAI Operator and RHAI cloud controller manager components

The Red Hat AI Inference (RHAII) Helm chart packages, deploys, and configures the Red Hat AI (RHAI) Operator and the RHAI cloud controller manager. The RHAI Operator handles KServe and model serving, while the RHAI cloud controller manager handles the underlying cluster infrastructure.

The RHAI Operator manages the KServe Controller and reconciles custom resource definitions (CRDs).

The RHAI cloud controller manager manages Helm-based infrastructure components, including cert-manager, Gateway API, Istio, and LeaderWorkerSet.

The RHAI cloud controller manager configures the managed Kubernetes or OpenShift Container Platform cluster.

Chapter 10. Distributed Inference with llm-d deployment patterns

Distributed Inference with llm-d supports several deployment patterns that range from a single model on one GPU to disaggregated prefill/decode across many nodes.

10.1. Inference workload deployment topology patterns

You can deploy large language models by using validated topology patterns that balance resource allocation, latency, and throughput based on your model size and hardware constraints. Understanding these patterns helps you select the appropriate deployment strategy for your inference workloads.

Single-node topology

The single-node topology deploys all model components on a single GPU or set of GPUs within one Kubernetes pod. This pattern is ideal for development, testing, and production deployments of smaller models that fit within the memory and compute capacity of available accelerators.

Use single-node topology when:

Your model fits within the memory of the available GPUs on a single node
You are developing or testing model deployments before scaling
Your inference workload requires simple resource management
You need the fastest deployment path with minimal configuration

Single-node deployments support hardware profile selection, allowing you to specify the exact accelerator resources required for your model.

Multi-node data-parallel topology

The multi-node data-parallel topology distributes inference requests across many worker nodes, each running a complete copy of the model. This pattern provides horizontal scaling for high-throughput workloads by processing many requests in parallel.

Use multi-node data-parallel topology when:

You need to handle high request volumes with parallel processing
Your model fits on a single GPU but you require higher throughput than one instance can give
You want to distribute inference load across many workers for reliability
Your workload benefits from request-level parallelism

This topology requires administrator-created configuration templates to define the worker pool and resource allocation.

Single-node disaggregated prefill/decode topology

The single-node disaggregated prefill/decode topology separates the prefill phase, which is processing input prompts, from the decode phase, which is generating output tokens, on different GPUs within a single node. This pattern optimizes latency by specializing compute resources for each phase of LLM inference.

Use single-node disaggregated prefill/decode topology when:

You have at least 2 GPUs and RDMA-capable networking available on a single node
Your model has 30 billion or more parameters
Your workload is suitable for disaggregated inference

The scheduler component coordinates request routing between prefill and decode pods, enabling KV cache-aware scheduling.

Multi-node disaggregated prefill/decode topology

The multi-node prefill/decode disaggregate topology distributes prefill and decode workloads across many nodes, combining disaggregation with horizontal scaling. This pattern supports the largest models and highest throughput requirements by separating phases and parallelizing within each phase. Use multi-node prefill/decode disaggregate topology when:

You need to deploy models that require more GPUs than a single node provides
You have RDMA-capable networking such as InfiniBand or RoCE v2 across nodes
Your workload is suitable for disaggregated inference.

Wide expert parallelism (WideEP) topology

The WideEP topology distributes Mixture of Experts (MoE) model experts across multiple GPU nodes by using all-to-all communication backends such as DeepEP. This pattern enables cost-effective inference for very large MoE models such as DeepSeek-R1 by splitting expert layers across nodes rather than replicating the full model.

Important

Wide expert parallelism (WideEP) is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

Use WideEP topology when:

Your model uses a Mixture of Experts architecture
The model size exceeds what multi-node data-parallel or prefill/decode disaggregate topologies can handle efficiently
You have RDMA-capable networking such as RoCE or InfiniBand for expert parallelism communication
You need all GPUs on each node allocated to a single pod for NVLink intranode communication

WideEP deployments use a one-pod-per-node topology where each pod requests all GPUs on the node. The LeaderWorkerSet Operator manages multi-node coordination across pods.

10.2. LLMInferenceService CR for single-node deployment

The simplest deployment runs one model on a single GPU node behind an Inference Gateway. The Endpoint Picker (EPP) scheduler is not required for this deployment. This pattern suits development workloads and low-traffic production services.

The following LLMInferenceService custom resource (CR) deploys a single model replica on one GPU behind an Inference Gateway with the Endpoint Picker (EPP) scheduler enabled. This is the simplest deployment pattern for Distributed Inference with llm-d and suits development workloads and low-traffic production services.

The CR configures the following components:

A single replica serving the Qwen/Qwen3-0.6B model from Hugging Face.
The EPP scheduler for intelligent request routing.
Resource requests and limits for the model server container, including one NVIDIA GPU.
An HTTPS liveness probe on port 8000 to monitor model server health.

Note

Replace the spec.model.uri and spec.model.name values with your model. Adjust the CPU, memory, and GPU resource limits to match your model’s requirements and available infrastructure.

Single-node LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: single-gpu
spec:
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  replicas: 1
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        resources:
          limits:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '2'
            memory: 16Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 120
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5

where:

spec.model.uri: Specifies the model location. The hf:// prefix indicates a Hugging Face repository.
spec.model.name: Specifies the model identifier used by the serving runtime.
spec.replicas: Specifies the number of model server replicas.
spec.router.scheduler: Specifies the EPP scheduler configuration. The scheduler routes requests to the optimal replica based on queue depth and cache utilization.
spec.router.route: Specifies the route configuration. An empty object uses default settings.
spec.router.gateway: Specifies the gateway configuration. An empty object uses default settings.
spec.template.containers[].resources: Specifies the CPU, memory, and GPU resource requests and limits for the model server container.

10.3. LLMInferenceService CR for multi-pool deployment

When you need to serve multiple models from shared infrastructure, a multi-model deployment places each model behind its own LLMInferenceService resource while sharing the same GPU nodes.

A multi-pool deployment creates multiple independent model-serving pools, each with its own Endpoint Picker (EPP) scheduler, route, and gateway. You can use this pattern to isolate workloads across separate pools or to serve different models on shared cluster infrastructure. This deployment pattern for Distributed Inference with llm-d suits workload isolation and multi-model serving use cases.

The CRs configure the following components:

Two separate LLMInferenceService resources: multi-pool-a and multi-pool-b, each operating as an independent serving pool.
One replica per pool serving the Qwen/Qwen3-0.6B model from Hugging Face.
An EPP scheduler, route, and gateway per pool for independent request routing.
Resource requests and limits for each model server container, including one NVIDIA GPU per pool.

Multi-pool LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: multi-pool-a
spec:
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  replicas: 1
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        resources:
          limits:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"
---
apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: multi-pool-b
spec:
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  replicas: 1
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        resources:
          limits:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"

where:

metadata.name: Specifies a unique name for each LLMInferenceService resource. Each pool requires a distinct name to create independent serving endpoints.
spec.model.uri: Specifies the model location. The hf:// prefix indicates a Hugging Face repository.
spec.model.name: Specifies the model identifier used by the serving runtime.
spec.replicas: Specifies the number of model server replicas per pool.
spec.router.scheduler: Specifies the EPP scheduler configuration for each pool.
spec.router.route: Specifies the route configuration.
spec.router.gateway: Specifies the gateway configuration. Each pool registers with the Inference Gateway independently.
spec.template.containers[].resources: Specifies the CPU, memory, and GPU resource requests and limits for the model server container in each pool.

10.4. LLMInferenceService CR for prefix cache-aware routing

The following LLMInferenceService custom resource (CR) deploys a model with prefix cache-aware routing enabled. This deployment pattern uses the precise-prefix-cache-producer and prefix-cache-scorer EPP plugins to route requests with similar prompt prefixes to the same replica, increasing KV cache hit rates and reducing time-to-first-token latency.

The CR configures the following components:

Two replicas serving the Qwen/Qwen3-0.6B model from Hugging Face.
An inline EndpointPickerConfig with four scheduling plugins: precise-prefix-cache-producer, prefix-cache-scorer, queue-scorer, and kv-cache-utilization-scorer.
A scheduling profile that weights prefix cache scoring higher than queue and KV cache utilization scoring.
vLLM arguments that enable prefix caching with SHA-256 hashing and KV cache event publishing over ZMQ.

Note

Replace the spec.model.uri and spec.model.name values with your model. Update the kv-events-config topic string in VLLM_ADDITIONAL_ARGS to match your model name. Adjust the CPU, memory, and GPU resource limits to match your model’s requirements and available infrastructure.

Prefix cache-aware routing LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: cache-aware
spec:
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  replicas: 2
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
      config:
        inline:
          apiVersion: llm-d.ai/v1alpha1
          kind: EndpointPickerConfig
          plugins:
            - type: precise-prefix-cache-producer
              parameters:
                kvEventsConfig:
                  zmqEndpoint: "tcp://*:5557"
                indexerConfig:
                  kvBlockIndexConfig:
                    enableMetrics: true
                    metricsLoggingInterval: 60000000000
            - type: prefix-cache-scorer
              parameters:
                prefixMatchInfoProducerName: precise-prefix-cache-producer
            - type: queue-scorer
            - type: kv-cache-utilization-scorer
          schedulingProfiles:
            - name: default
              plugins:
                - pluginRef: queue-scorer
                  weight: 2
                - pluginRef: kv-cache-utilization-scorer
                  weight: 2
                - pluginRef: prefix-cache-scorer
                  weight: 3
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        env:
          - name: VLLM_ADDITIONAL_ARGS
            value: "--prefix-caching-hash-algo sha256 --block-size 16 --kv-events-config '{\"enable_kv_cache_events\":true,\"publisher\":\"zmq\",\"endpoint\":\"tcp://{{ ChildName .ObjectMeta.Name `-epp-service` }}:5557\",\"topic\":\"kv@${POD_IP}@Qwen/Qwen3-0.6B\"}'"
          - name: POD_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.podIP
          - name: PYTHONHASHSEED
            value: "42"
        resources:
          limits:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '2'
            memory: 16Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 120
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5

where:

spec.replicas: Specifies the number of model server replicas. Prefix cache-aware routing requires at least 2 replicas to demonstrate routing benefits.
spec.router.scheduler.config.inline: Specifies an inline EndpointPickerConfig resource that configures the EPP scheduling plugins and profiles.
precise-prefix-cache-producer: Specifies the plugin that tracks KV cache block state across replicas by consuming cache events published by vLLM over ZMQ.
prefix-cache-scorer: Specifies the plugin that scores replicas based on how much of the request’s prompt prefix is already cached in each replica’s KV cache.
schedulingProfiles: Specifies the scoring weights for each plugin. In this example, prefix-cache-scorer has a weight of 3, while queue-scorer and kv-cache-utilization-scorer each have a weight of 2.
VLLM_ADDITIONAL_ARGS: Specifies additional vLLM command-line arguments. The --prefix-caching-hash-algo, --block-size, and --kv-events-config flags enable prefix caching and KV cache event publishing.
PYTHONHASHSEED: Specifies a fixed Python hash seed to ensure consistent prefix hashing across replicas.

10.5. LLMInferenceService CR for flow control

Flow control allows the endpoint picker (EPP) to queue requests when all model server replicas are saturated, rather than dropping or overloading them. When a replica becomes available, the EPP dequeues and forwards the next request. You can enable flow control on any deployment pattern by setting the flowControl feature gate in the EndpointPickerConfig resource.

The following LLMInferenceService custom resource (CR) deploys a model with flow control enabled.

The CR configures the following components:

A single replica serving the Qwen/Qwen3-0.6B model from Hugging Face.
The flowControl feature gate enabled in the inline EndpointPickerConfig.
A default request time-to-live (TTL) of 30 seconds. Requests that remain queued longer than this TTL are rejected.
Scheduling plugins for queue scoring, KV cache utilization scoring, and prefix cache scoring.

Note

Replace the spec.model.uri and spec.model.name values with your model. Adjust the defaultRequestTTL value based on your latency requirements. A shorter TTL rejects requests faster under load; a longer TTL allows more time for replicas to become available. Adjust the CPU, memory, and GPU resource limits to match your model’s requirements and available infrastructure.

Flow control LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: flow-control
spec:
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  replicas: 1
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
      config:
        inline:
          apiVersion: llm-d.ai/v1alpha1
          kind: EndpointPickerConfig
          featureGates:
          - flowControl
          plugins:
            - type: queue-scorer
            - type: kv-cache-utilization-scorer
            - type: prefix-cache-scorer
          flowControl:
            defaultRequestTTL: 30s
          schedulingProfiles:
            - name: default
              plugins:
                - pluginRef: queue-scorer
                  weight: 2
                - pluginRef: kv-cache-utilization-scorer
                  weight: 2
                - pluginRef: prefix-cache-scorer
                  weight: 3
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        env:
          - name: VLLM_ADDITIONAL_ARGS
            value: "--max-num-seqs 2"
        resources:
          limits:
            cpu: '2'
            memory: 8Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '1'
            memory: 4Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5

where:

spec.router.scheduler.config.inline.featureGates: Specifies the feature gates to enable. The flowControl gate activates request queuing in the EPP.
spec.router.scheduler.config.inline.flowControl.defaultRequestTTL: Specifies the maximum time a request can remain in the queue before the EPP rejects it.
VLLM_ADDITIONAL_ARGS: Specifies additional vLLM command-line arguments. The --max-num-seqs 2 flag limits the maximum number of concurrent sequences per replica, which makes it easier to trigger flow control queuing for testing purposes.

10.6. LLMInferenceService CR for token-based flow control

Token-based flow control extends the default flow control behavior by detecting replica saturation based on the number of in-flight tokens rather than the number of concurrent connections. This approach prevents overloading replicas that are processing long sequences, because a single request with many tokens consumes more capacity than a short request.

The following LLMInferenceService custom resource (CR) deploys a model with token-based flow control enabled.

Token-based flow control LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: flow-control-tokens
spec:
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  replicas: 1
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
      config:
        inline:
          apiVersion: llm-d.ai/v1alpha1
          kind: EndpointPickerConfig
          featureGates:
          - flowControl
          plugins:
            - type: queue-scorer
            - type: kv-cache-utilization-scorer
            - type: prefix-cache-scorer
            - name: concurrency-detector
              type: concurrency-detector
              parameters:
                concurrencyMode: "tokens"
                maxTokenConcurrency: 200
            - name: inflight-load-producer
              type: inflight-load-producer
              parameters:
                addEstimatedOutputTokens: true
          flowControl:
            defaultRequestTTL: 30s
            saturationDetector:
              pluginRef: concurrency-detector
          schedulingProfiles:
            - name: default
              plugins:
                - pluginRef: queue-scorer
                  weight: 2
                - pluginRef: kv-cache-utilization-scorer
                  weight: 2
                - pluginRef: prefix-cache-scorer
                  weight: 3
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        env:
          - name: VLLM_ADDITIONAL_ARGS
            value: "--max-num-seqs 2"
        resources:
          limits:
            cpu: '2'
            memory: 8Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '1'
            memory: 4Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5

where:

spec.router.scheduler.config.inline.featureGates: Specifies the feature gates to enable. The flowControl gate activates request queuing in the EPP.
concurrency-detector.parameters.concurrencyMode: Specifies the concurrency detection mode.
concurrency-detector.parameters.maxTokenConcurrency: Specifies the maximum number of concurrent tokens per replica before the EPP considers the replica saturated and begins queuing requests.
inflight-load-producer.parameters.addEstimatedOutputTokens: Specifies whether to include estimated output tokens when calculating in-flight load.
flowControl.saturationDetector.pluginRef: Specifies the plugin that determines when a replica is saturated. The concurrency-detector reference connects token-based detection to the flow control system.
flowControl.defaultRequestTTL: Specifies the maximum time a request can remain in the queue before the EPP rejects it.
VLLM_ADDITIONAL_ARGS: Specifies additional vLLM command-line arguments. The --max-num-seqs 2 flag limits the maximum number of concurrent sequences per replica, which makes it easier to trigger flow control queuing for testing purposes.

10.7. LLMInferenceService CR for LoRA adapter serving

Low-Rank Adaptation (LoRA) adapter serving lets you load lightweight fine-tuned model layers on top of a base model without duplicating the full model weights. You can serve multiple specialized model variants from a single base model deployment, reducing GPU memory consumption compared to deploying each variant as a separate model.

Important

LoRA adapter serving is a Developer Preview feature only. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to upcoming product features in advance of their possible inclusion in a Red Hat product offering, enabling customers to test functionality and provide feedback during the development process. These features might not have any documentation, are subject to change or removal at any time, and testing is limited. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

The following LLMInferenceService custom resource (CR) examples deploy a base model with one or more Low-Rank Adaptation (LoRA) adapters.

Single LoRA adapter

The following CR deploys the facebook/opt-125m base model with a single LoRA adapter for SQL generation.

Note

Replace the spec.model.uri and spec.model.name values with your base model. Replace the adapter name and uri values with your LoRA adapter. Adjust the CPU, memory, and GPU resource limits to match your model’s requirements and available infrastructure.

LoRA adapter serving LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: lora-single
spec:
  replicas: 1
  model:
    uri: hf://facebook/opt-125m
    name: facebook/opt-125m
    lora:
      adapters:
        - name: sql-adapter
          uri: hf://edbeeching/opt-125m-lora
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        resources:
          limits:
            cpu: '2'
            memory: 8Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '1'
            memory: 4Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5

where:

spec.model.uri: Specifies the base model location. The hf:// prefix indicates a Hugging Face repository.
spec.model.lora.adapters: Specifies the list of LoRA adapters to load on top of the base model.
spec.model.lora.adapters[].name: Specifies the name of the adapter. Clients use this name to select the adapter in inference requests.
spec.model.lora.adapters[].uri: Specifies the adapter location in Hugging Face.

Multiple LoRA adapters

The following CR deploys the same base model with two LoRA adapters and sets limits on the maximum number of adapters and the maximum LoRA rank.

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: lora-multi
spec:
  replicas: 1
  model:
    uri: hf://facebook/opt-125m
    name: facebook/opt-125m
    lora:
      adapters:
        - name: sql-adapter
          uri: hf://edbeeching/opt-125m-lora
        - name: code-adapter
          uri: hf://velaa/opt-125m-lora-finetuned-mnli
      maxAdapters: 2
      maxRank: 64
  router:
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
    route: {}
    gateway: {}
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        resources:
          limits:
            cpu: '2'
            memory: 8Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '1'
            memory: 4Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 60
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5

where:

spec.model.lora.adapters: Specifies the list of LoRA adapters. In this example, two adapters are configured: sql-adapter and code-adapter.
spec.model.lora.maxAdapters: Specifies the maximum number of LoRA adapters that can be loaded simultaneously. Set this value based on your available GPU memory.
spec.model.lora.maxRank: Specifies the maximum LoRA rank allowed for adapters. Higher ranks use more GPU memory but can capture more complex fine-tuning patterns.

10.8. LLMInferenceService CR for prefill/decode disaggregation

The following LLMInferenceService custom resource (CR) deploys a model with prefill/decode disaggregation on GPU nodes with network connectivity between prefill and decode pods. Single-node and multi-node deployments require RDMA-capable network interfaces such as RoCE v2 or InfiniBand for high-throughput, low-latency KV cache transfer between prefill and decode pods.

Important

Prefill/decode disaggregation is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

The CR configures the following components:

One decode replica and two prefill replicas serving the Qwen/Qwen3-0.6B model from Hugging Face.
NIXL for KV cache transfer between prefill and decode workers over TCP.
UCX transport configuration for inter-worker communication.
The EPP scheduler for routing requests to decode replicas.

Prefill/decode disaggregation LLMInferenceService CR

apiVersion: serving.kserve.io/v1alpha2
kind: LLMInferenceService
metadata:
  name: pd
spec:
  replicas: 1
  model:
    uri: hf://Qwen/Qwen3-0.6B
    name: Qwen/Qwen3-0.6B
  router:
    route: {}
    gateway: {}
    scheduler:
      template:
        imagePullSecrets:
        - name: rhai-pull-secret
        containers:
        - name: main
  template:
    imagePullSecrets:
    - name: rhai-pull-secret
    containers:
      - name: main
        env:
          - name: VLLM_NIXL_SIDE_CHANNEL_HOST
            valueFrom:
              fieldRef:
                fieldPath: status.podIP
          - name: VLLM_ADDITIONAL_ARGS
            value: "--kv_transfer_config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\"}'"
          - name: UCX_TLS
            value: "tcp,sm,self,cuda_copy,cuda_ipc"
        resources:
          limits:
            cpu: '4'
            memory: 32Gi
            nvidia.com/gpu: "1"
          requests:
            cpu: '2'
            memory: 16Gi
            nvidia.com/gpu: "1"
        livenessProbe:
          httpGet:
            path: /health
            port: 8000
            scheme: HTTPS
          initialDelaySeconds: 120
          periodSeconds: 30
          timeoutSeconds: 30
          failureThreshold: 5
  prefill:
    replicas: 2
    template:
      imagePullSecrets:
      - name: rhai-pull-secret
      containers:
        - name: main
          env:
            - name: VLLM_NIXL_SIDE_CHANNEL_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: VLLM_ADDITIONAL_ARGS
              value: "--kv_transfer_config '{\"kv_connector\":\"NixlConnector\",\"kv_role\":\"kv_both\"}'"
            - name: UCX_TLS
              value: "tcp,sm,self,cuda_copy,cuda_ipc"
          resources:
            limits:
              cpu: '4'
              memory: 32Gi
              nvidia.com/gpu: "1"
            requests:
              cpu: '2'
              memory: 16Gi
              nvidia.com/gpu: "1"
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
              scheme: HTTPS
            initialDelaySeconds: 120
            periodSeconds: 30
            timeoutSeconds: 30
            failureThreshold: 5

where:

spec.replicas: Specifies the number of decode replicas. Decode replicas generate output tokens one at a time while reading from the KV cache.
spec.prefill: Specifies the prefill pool configuration. The prefill pool is separate from the main decode pool.
spec.prefill.replicas: Specifies the number of prefill replicas. Prefill replicas process the full input context in parallel.
VLLM_NIXL_SIDE_CHANNEL_HOST: Specifies the pod IP address for NIXL side-channel communication. This value is populated automatically from the pod’s status.podIP field.
VLLM_ADDITIONAL_ARGS: Specifies additional vLLM command-line arguments. The --kv_transfer_config flag configures the NIXL connector for KV cache transfer between prefill and decode workers.
UCX_TLS: Specifies the UCX transport layers for inter-worker communication. The value tcp,sm,self,cuda_copy,cuda_ipc enables TCP transport alongside shared memory and CUDA transports.

Legal Notice

Except as otherwise noted below, the text of and illustrations in this documentation are licensed by Red Hat under the Creative Commons Attribution–Share Alike 3.0 Unported license . If you distribute this document or an adaptation of it, you must provide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, the Red Hat logo, JBoss, Hibernate, and RHCE are trademarks or registered trademarks of Red Hat, LLC. or its subsidiaries in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

XFS is a trademark or registered trademark of Hewlett Packard Enterprise Development LP or its subsidiaries in the United States and other countries.

The OpenStack® Word Mark and OpenStack logo are trademarks or registered trademarks of the Linux Foundation, used under license.

All other trademarks are the property of their respective owners.