Release Notes

Red Hat OpenShift AI Self-Managed 3.4

Features, enhancements, resolved issues, and known issues associated with this release

Abstract

These release notes provide an overview of new features, enhancements, resolved issues, and known issues in version 3.4.1 of Red Hat OpenShift AI.

Chapter 1. Overview of OpenShift AI

Red Hat OpenShift AI is a platform for data scientists and developers of artificial intelligence and machine learning (AI/ML) applications.

OpenShift AI provides an environment to develop, train, serve, test, and monitor AI/ML models and applications on-premise or in the cloud.

For data scientists, OpenShift AI includes Jupyter and a collection of default workbench images optimized with the tools and libraries required for model development, and the TensorFlow and PyTorch frameworks. Deploy and host your models, integrate models into external applications, and export models to host them in any hybrid cloud environment. You can enhance your projects on OpenShift AI by building portable machine learning (ML) workflows with AI pipelines by using Docker containers. You can also accelerate your data science experiments through the use of graphics processing units (GPUs) and Intel Gaudi AI accelerators.

For administrators, OpenShift AI enables data science workloads in an existing Red Hat OpenShift or ROSA environment. Manage users with your existing OpenShift identity provider, and manage the resources available to workbenches to ensure data scientists have what they require to create, train, and host models. Use accelerators to reduce costs and allow your data scientists to enhance the performance of their end-to-end data science workflows using graphics processing units (GPUs) and Intel Gaudi AI accelerators.

OpenShift AI has a Self-managed software deployment option that you can install on-premise or in the cloud. You can install OpenShift AI Self-Managed in a self-managed environment such as OpenShift Container Platform, or in Red Hat-managed cloud environments such as Red Hat OpenShift Dedicated (with a Customer Cloud Subscription for AWS or GCP), Red Hat OpenShift Service on Amazon Web Services (ROSA classic or ROSA HCP), or Microsoft Azure Red Hat OpenShift.

For information about OpenShift AI supported software platforms, components, and dependencies, see the Supported Configurations for 3.x Knowledgebase article.

For a detailed view of the 3.4 release lifecycle, including the full support phase window, see the Red Hat OpenShift AI Self-Managed Life Cycle Knowledgebase article.

Chapter 2. New features and enhancements

This section describes new features and enhancements in Red Hat OpenShift AI 3.4 GA.

2.1. New features

2.1.1. 3.4 GA new features

Support for direct authentication with an OIDC identity provider

Direct authentication with an OpenID Connect (OIDC) identity provider is now available as a GA feature.

This enhancement centralizes OpenShift AI service authentication through the Gateway API, providing a secure, scalable, and manageable authentication model. You can configure the Gateway API with your external OIDC provider by using the GatewayConfig custom resource.

Migration guide available for transitioning from vLLM-based InferenceService to LLMInferenceService: Platform Operators deploying Distributed Inference with llm-d on OpenShift AI can follow a step-by-step guide covering LLMInferenceServiceConfig, YAML examples, and migration from vLLM-based InferenceService deployments. The guide explains how LLMInferenceServiceConfig replaces custom ServingRuntime definitions. For more information, see https://access.redhat.com/articles/7141739.

Prometheus metrics for Distributed Inference with llm-d: You can now monitor llm-d distributed inference deployments using documented Prometheus metrics and PromQL query examples. These metrics are exported by all llm-d components, including the Endpoint Picker (EPP), vLLM engine pods, and prefix cache, along with example queries for building custom dashboards and configuring ServiceMonitors for OpenShift User Workload Monitoring.

MLflow is fully supported: MLflow, previously available as a Technology Preview and Developer Preview feature, is fully supported with this release. OpenShift AI includes a Red Hat-built MLflow deployment managed through the MLflow Operator. For more information, see Working with MLflow.

MLFlow SDK pre-installed in workbench and runtime images: The MLFlow SDK is now pre-installed and included in the datascience, tensorflow (CUDA & ROCm), pytorch (CUDA & ROCm), and codeserver workbench and runtime images.

MLflow Operator is now a managed component in the DataScienceCluster CR: Starting with Red Hat OpenShift AI 3.4, the MLflow Operator is a managed component in the DataScienceCluster custom resource (CR). You can enable the mlflowoperator component by setting managementState to Managed in the DataScienceCluster CR. MLflow availability in the dashboard is now determined by the component state, and the deprecated mlflow dashboard feature flag is no longer required. For more information, see Enable the MLflow Operator component.

NeMo Guardrails to enable AI safety

NeMo Guardrails, introduced in Red Hat OpenShift AI 3.3 as a Technology Preview, is fully supported with this release. You can add guardrails and safety controls to your deployed models. NeMo Guardrails provides a framework for controlling conversations with large language models, enabling you to define a variety of rails, such as sensitive data detection, content filtering, or custom validation rules. NeMo Guardrails introduces the following capabilities:

/v1/guardrails/checks endpoint for standalone querying of guardrail policies
Full OpenAI compatibility, with a new /v1/models/ endpoint
New regex rails for regex-based guardrail logic
Support for multiple replicas of the NeMo Guardrails pod to improve scalability
Out-of-the-box OpenTelemetry support
Automatic redeployment after configuration changes, for zero-downtime guardrail tuning

Models-as-a-Service now Generally Available

Models-as-a-Service (MaaS) is now Generally Available in Red Hat OpenShift AI 3.4. Previously introduced as a Technology Preview feature in version 3.3, MaaS provides an enterprise platform for centralized governance, consumption tracking, and self-service access to large language models. MaaS addresses resource consumption and governance challenges by exposing models through managed API endpoints with subscription-based controls. Administrators can define subscriptions that grant groups access to specific models with configurable token limits, while users can generate and manage their own API keys for programmatic access.

Key capabilities include:

Subscription-based model access with configurable token quotas and rate limiting
Self-service API key generation and management
Centralized authentication and authorization policies (external OIDC authentication available as Technology Preview)
Support for distributed inference with llm-d (vLLM runtime support available as Technology Preview)
Usage tracking and showback reporting through an observability dashboard (Technology Preview)
Routing to external model providers such as OpenAI or Anthropic (Technology Preview)

For more information, see Governing LLM access with Models-as-a-Service.

The Models-as-a-Service subscription model redesign

The Models-as-a-Service (MaaS) subscription model has been redesigned to replace the tier-based model introduced in version 3.3. The new subscription model provides administrators with flexible access control and resource management for large language models, while enabling data scientists to select from available subscriptions when accessing models.

Key enhancements include:

Priority-based subscription assignment: As a cluster administrator, you can assign priority levels to subscriptions. As a user, when you belong to multiple groups with different subscriptions, the system automatically assigns you to the highest-priority subscription by default, while allowing you to manually select from your available subscriptions when generating API keys.
Group-based access control: As a cluster administrator, you can define subscriptions that grant groups access to specific models. Subscriptions support integration with OpenShift groups, OIDC group claims, and API key group snapshots.
Configurable token quotas: As a cluster administrator, you can define distinct token limits per model for each subscription, enabling cost control and resource allocation aligned with organizational policies. As a user, your token consumption is tracked against your active subscription’s limits.
Authorization policy integration: Subscriptions work in combination with authorization policies to control API gateway access and token consumption limits.

The subscription model redesign enables organizations to enforce consumption policies across teams while maintaining self-service access for data scientists and developers.

For more information, see Governing LLM access with Models-as-a-Service.

Self-service API key management for Models-as-a-Service

You can now create and manage your own API keys for programmatic access to large language models through Models-as-a-Service. This self-service capability streamlines access to large language models while maintaining centralized governance through subscriptions and authorization policies.

Key capabilities include:

User-managed API key lifecycle: Create permanent API keys with configurable expiration dates, view active keys, and revoke keys when no longer needed
Temporary API key generation: Generate short-lived API keys directly from the model endpoint dialog for quick testing and prototyping
Subscription-scoped authentication: API keys are scoped to specific subscriptions, inheriting the subscription’s model access and token limits
Group membership snapshot: User group membership is captured at API key creation time, ensuring consistent access control even when group assignments change

Self-service API key management empowers you to access models programmatically through OpenAI-compatible APIs without administrative bottlenecks, while administrators retain control through subscription-based governance.

For more information, see Governing LLM access with Models-as-a-Service.

Support for Llama Stack and KubeRay on IBM Power: Red Hat OpenShift AI 3.4 GA introduces official support for both Llama Stack and KubeRay on the IBM Power architecture.

OCI-compliant storage layer for model registry

You can now use the OpenShift AI dashboard to register a model from an S3-compatible source or URI, transform it into an OCI ModelCar image, and store it in an OCI registry. The ModelCar target format enables fast deployment with KServe.

The model transfer job runs as a background Kubernetes Job that you can monitor from the dashboard. This feature provides the following capabilities:

Register and store models from object storage (S3, MinIO) or URLs in a single operation.
Models are automatically converted to ModelCar OCI images, ensuring compatibility with KServe ModelCar for model serving.
Track model transfer jobs in real-time with detailed status information, Kubernetes event logs, and automatic polling.
Retry failed jobs or delete completed jobs directly from the dashboard.
ConfigMaps and Secrets are automatically garbage-collected when jobs are deleted.

MLServer ServingRuntime for KServe is now generally available

The MLServer ServingRuntime for KServe is now generally available in Red Hat OpenShift AI. You can use this runtime to deploy models trained on structured data, such as classical machine learning models. You can deploy models directly in their native format, simplifying the deployment process.

Supported model frameworks include: * Scikit-learn * XGBoost * LightGBM * ONNX

For models with well-known file names, MLServer automatically configures all required environment variables during deployment through the Deploy a model wizard.

For more information, see Deploying models using the MLServer runtime.

2.1.2. 3.4 EA2 new features

MLflow operator promoted to a managed component: Starting with Red Hat OpenShift AI 3.4, the MLflow Operator is a managed component in the DataScienceCluster custom resource (CR). You can enable the mlflowoperator component by setting managementState to Managed in the DataScienceCluster CR. MLflow availability in the dashboard is now determined by the component state, and the deprecated mlflow dashboard feature flag is no longer required. For more information, see Enable the MLflow Operator component.

Automatic MLflow SDK configuration for workbenches: When the MLflow Operator is enabled, you can annotate workbench notebook resources with opendatahub.io/mlflow-instance to automatically configure the MLflow SDK. The platform injects the tracking URI, authentication, and Kubernetes integration environment variables, and provisions the required RBAC RoleBinding resource. This removes the need to manually configure MLflow connectivity in workbench notebooks. For more information, see Enable MLflow integration for a workbench.

Granular RBAC for workbenches

Project administrators can now define custom, fine-grained roles for workbench resources by using the oc CLI or by importing YAML in the OpenShift web console. Custom roles are standard Kubernetes Role objects with the opendatahub.io/dashboard: 'true' label. After creating a custom role, administrators can assign it to users and groups from the Permissions tab in the OpenShift AI dashboard. Multiple custom roles can be assigned simultaneously, and roles labeled for the dashboard are displayed with an AI role label in the Role type column to distinguish them from default OpenShift roles.

Multi-architecture support for the model catalog

The model catalog includes support for IBM Power (ppc64le) architecture. With this enhancement, you can discover and deploy models directly from the dashboard. Support is available for the following validated models:

registry.redhat.io/rhai/modelcar-granite-3-3-8b-instruct
registry.redhat.io/rhai/modelcar-granite-4-0-h-small:3.0
registry.redhat.io/rhai/modelcar-granite-4-0-h-tiny:3.0

Multi-architecture support for the model catalog

The model catalog now includes support for the IBM Z architecture. This enhancement enables users on IBM Z platforms to discover and deploy models directly from the OpenShift AI dashboard. Currently, support is available for the following model:

registry.redhat.io/rhai/modelcar-granite-3-3-8b-instruct

Just-In-Time Checkpointing and S3 Storage for Kubeflow Trainer

Kubeflow Trainer now provides Just-In-Time (JIT) and periodic checkpointing for distributed training jobs on OpenShift AI. This enhancement automatically saves the training state, including model weights, optimizer state, and training step at regular intervals and immediately before interruptions such as preemption, eviction, or maintenance. Interrupted jobs automatically resume from the latest valid checkpoint, significantly reducing wasted GPU compute and improving overall training efficiency.

Checkpoints can be stored on PersistentVolumeClaims (PVCs) or S3-compatible object storage. With S3, checkpoints are uploaded in the background without pausing training, enabling low-overhead, continuous protection of progress. S3-backed storage also provides a cost-efficient, portable alternative to PVCs, allowing checkpoints to be retained, shared, and reused across clusters.

2.1.3. 3.4 EA1 new features

Model deployments are not visible under the model registry deployments tab on IBM Power (ppc64le) in RHOAI 3.4-EA1.

Workbench and runtime images default to Red Hat Python index

Workbench and runtime images default to the Red Hat Python index. When you install or update Python packages, the following packages are pulled from the Red Hat Python index rather than PyPI:

Minimal (CPU,CUDA,ROCm) 3.12
Codeserver 3.12
Jupyter Data science 3.12
Jupyter PyTorch 3.12

Garak evaluation provider available in Llama Stack distribution: The Garak evaluation provider is available in the Llama Stack distribution. Garak provides security scanning capabilities for large language models to help identify potential vulnerabilities and safety issues. The provider is available in two versions: an inline version that runs scans in the same process as the Llama Stack server, and a remote version that runs scans by using Kubeflow Pipelines.

PostgreSQL database support for Model Registry: You can configure a PostgreSQL database as the backend for Model Registry from the OpenShift AI dashboard.

Default database solution for Model Registry: Model Registry includes a default database solution for testing. Use this solution to start using Model Registry without configuring an external database.
Note
The default database is not intended for production workloads.

2.2. Enhancements

2.2.1. 3.4 GA enhancements

vLLM uvicorn access logs are disabled by default in Distributed Inference with llm-d: vLLM uvicorn access logs are disabled by default in LLMInferenceServiceConfig, including logs generated by router-scheduler /metrics endpoint polling. This reduces excessive logging caused by the EndpointPicker scraping metrics scraping every 200 milliseconds. Operators who need access logs for debugging can re-enable them explicitly.

2.2.2. 3.4 EA2 enhancements

Simplified configuration for Distributed Inference with llm-d scheduler settings: Configure Distributed Inference with llm-d scheduler settings using the endpointPickerConfig field in the LLMInferenceService specification. You can specify the configuration inline or reference a ConfigMap. This approach replaces the previous method that required making extensive specifications in the EndpointPicker configuration in the scheduler’s --configText argument.

Configure vLLM runtime arguments using Kubernetes container args field: You can configure vLLM runtime arguments using the standard Kubernetes container args field in LLMInferenceService resources. User-specified arguments are merged with system defaults, allowing you to add new arguments or override specific defaults without replacing the entire argument list.

The previous VLLM_ADDITIONAL_ARGS environment variable method continues to work for backward compatibility.

2.2.3. 3.4 EA1 enhancements

Hybrid search support for Qdrant remote vector database provider: Vector Store Search supports hybrid and keyword search for the Qdrant Vector IO provider.

Chapter 3. Technology Preview features

Important

This section describes Technology Preview features in Red Hat OpenShift AI 3.4 GA. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

3.1. 3.4 GA Technology Preview features

Workload variant autoscaler for Distributed Inference with llm-d model deployments: Platform Operators can enable autoscaling for Distributed Inference with llm-d model deployments based on incoming request volume as a Technical Preview. Scaling decisions use inference request metrics as the signal, so replicas respond to actual demand.

Priority-based flow control for mixed llm-d workloads: Platform Operators can assign priority tiers to workload classes and configure queuing policies so that latency-sensitive requests are served ahead of throughput-oriented batch traffic. Multiple workload profiles can share a single distributed inference with llm-d deployment without dedicated GPU pools per priority level.

Gateway discovery for llm-d deployments

You can discover and select an existing Kubernetes Gateway resource from the model serving UI when deploying LLMInferenceService models. This Technology Preview feature enables self-service Gateway management, supports multitenant namespace-scoped network isolation, and provides programmatic access through the Gateway discovery REST API.

Gateway discovery and creation is disabled by default and must be explicitly enabled by a cluster administrator through the dashboard configuration.

vLLM runtime support for Models-as-a-Service

You can now deploy models using the vLLM runtime through Models-as-a-Service (MaaS). This Technology Preview feature enables you to serve large language models with vLLM’s high-performance inference capabilities while benefiting from MaaS governance and subscription-based controls.

Key capabilities include:

vLLM deployment from the dashboard: As a cluster administrator, you can enable vLLM runtime support for MaaS by setting the vLLMDeploymentOnMaaS feature flag in the OdhDashboardConfig custom resource. Once enabled, you can deploy models using vLLM runtime directly from the Models-as-a-Service interface.
Performance optimization: vLLM provides optimized inference performance for large language models through features such as continuous batching, PagedAttention, and efficient memory management, improving throughput and reducing latency for model serving workloads.
Subscription-based governance: As a cluster administrator, you can apply the same subscription-based access controls, token quotas, and authorization policies to vLLM-served models as you do for other MaaS models, maintaining consistent governance across different serving runtimes.
Unified user experience: As a user, you access vLLM-served models through the same OpenAI-compatible API endpoints and authentication methods as other MaaS models, providing a seamless experience regardless of the underlying serving runtime.

vLLM runtime support for Models-as-a-Service enables organizations to leverage high-performance inference capabilities while maintaining centralized governance and cost control.

For more information, see Governing LLM access with Models-as-a-Service.

External OIDC authentication for Models-as-a-Service

You can now configure Models-as-a-Service to authenticate users with an external OpenID Connect (OIDC) identity provider. This Technology Preview feature enables enterprise-wide access to large language models without requiring OpenShift accounts for every user.

Key capabilities include:

External identity provider integration: As a cluster administrator, you can integrate MaaS with existing identity providers, leveraging your organization’s existing authentication infrastructure.
OIDC group mapping: As a cluster administrator, you can map external user groups from OIDC group claims to MaaS subscriptions, enabling centralized access control and quota enforcement based on your organization’s existing group structure.
Seamless user experience: As a user, you authenticate using your existing enterprise credentials and receive API keys scoped to your OIDC group memberships, providing consistent access across your organization’s AI platform.
Multiple authentication methods: MaaS supports authentication through OpenShift groups and OIDC token claims, allowing you to choose the authentication method that best fits your deployment model.

External OIDC authentication enables organizations to integrate MaaS into existing identity and access management workflows while maintaining self-service access for data scientists and developers.

For more information, see Governing LLM access with Models-as-a-Service.

Models-as-a-Service observability dashboard

As a cluster administrator, you can now monitor platform-wide Models-as-a-Service (MaaS) usage through a dedicated observability dashboard embedded in the OpenShift AI console. This Technology Preview feature provides comprehensive usage metrics for cost attribution and showback reporting to finance teams.

Key capabilities include:

Subscription-level metrics: View total tokens consumed, total requests, error counts, and success rates across all subscriptions, or drill down to specific subscriptions for detailed analysis
Token consumption tracking: Track token usage by user, subscription, and model, with detailed tables showing request counts and rate limit violations for cost allocation and capacity planning
Filtering and time ranges: Filter metrics by user, subscription, and model to analyze specific usage patterns. View metrics for configurable time periods ranging from the last 5 minutes to the last 14 days, or specify custom date ranges
CSV export: Export usage data in CSV format for integration with external metering, billing, and financial reporting systems
Prometheus metrics integration: The dashboard queries Prometheus metrics collected from Kuadrant and MaaS components, providing real-time visibility into platform health and resource consumption

For more information, see Governing LLM access with Models-as-a-Service.

External model egress via inference gateway

You can now route model inference requests to external model providers through the Models-as-a-Service (MaaS) inference gateway. This Technology Preview feature enables you to apply MaaS governance policies, token tracking, and rate limiting to third-party LLM services such as OpenAI, Anthropic, or other external providers.

Key capabilities include:

External model configuration: As a cluster administrator, you can define external LLM provider configurations using the ExternalModel custom resource, specifying the provider endpoint, authentication method, and model identifiers.
Unified governance: As a cluster administrator, you can apply the same subscription-based access controls, authorization policies, and token quotas to external models as you do for models served within the cluster.
Centralized usage tracking: As a cluster administrator, you can monitor token consumption and request metrics for external models alongside internally-served models in the observability dashboard, providing a unified view of LLM usage across your organization.
Consistent API interface: As a user, you access external models through the same OpenAI-compatible API endpoints and authentication methods as internally-served models, providing a seamless experience regardless of where models are hosted.

External model egress enables organizations to maintain centralized governance and cost visibility when using external LLM providers while preserving flexibility for teams to leverage best-in-class models from multiple sources.

For more information, see Governing LLM access with Models-as-a-Service.

Llama Stack Responses API parity with OpenAI: The Llama Stack Responses API in OpenShift AI 3.4 introduces systematic alignment with OpenAI’s Responses API. Previously, the Llama Stack Responses API diverged from OpenAI’s Responses API specifications. This update allows you to use OpenAI compatible SDKs and clients with Llama Stack without error and removes differences in specifications including: field names, unsupported parameters, and response shapes. This creates a consistent experience of API specifications between Llama Stack and OpenAI.
Responses API on Llama Stack: The Responses API on Llama Stack is now available on OpenShift AI as a Technology Preview, previously available as a Developer Preview. This allows for a high-level orchestration layer for building agentic applications and provides closer OpenAI-compatibility in Llama Stack.
Automate machine learning model training with AutoML: AutoML is available as a Technology Preview feature in Red Hat OpenShift AI 3.4. You can use AutoML to train and compare machine learning models for your data. AutoML provides a dashboard UI to configure optimization runs, compare models on a leaderboard, generate notebooks for running models, and register a model for deployment. For more information, see Working with AutoML.

Automate RAG optimization with AutoRAG: AutoRAG is available as a Technology Preview feature in Red Hat OpenShift AI 3.4. You can use AutoRAG to find the best retrieval-augmented generation (RAG) configuration for your documents and use case. AutoRAG provides a dashboard UI to configure optimization runs, evaluate RAG patterns on a leaderboard, and generate notebooks to run patterns. For more information, see Working with AutoRAG.

Recommended vLLM runtime configurations in model catalog

Red Hat OpenShift AI now provides recommended vLLM runtime configurations for select high-demand models. These configurations are designed to help users achieve maximum performance on specific hardware profiles, matching or exceeding industry benchmarks for low-latency and high-throughput serving.

Previously, users had to manually tune vLLM arguments to find the sweet spot for performance. Now, validated YAML templates are embedded directly in the model card Readme section as markdown, serving as a reference for manual configuration.

Key features of this update include:

Optimal performance recipes: Validated configurations optimized for specific hardware (initially NVIDIA H200) and workload profiles (8K prefill / 1K generation).
Structured YAML metadata: Recommended settings include specific environment variables and CLI arguments verified by the Red Hat Performance and Scale (PSAP) team.
Model card integration: No UI changes are required. Optimized recipes are appended to the model’s Readme field for easy discovery.
Performance parity: These configurations ensure OpenShift AI performance remains competitive with alternative serving stacks by leveraging the latest vLLM optimizations.

Initial targeted models:

GPT-OSS 120B: Optimized for major enterprise usage.
Llama 3-70B: Validated for standard high-performance deployments.
DeepSeek R1: Provided as a showcase for advanced reasoning and Red Hat Summit demonstrations.

To find the optimal runtime settings for a supported model:

In the OpenShift AI dashboard, navigate to AI hub → Catalog.
Select one of the supported models, for example, GPT-OSS 120B.
On the model card, scroll to the readme section.
Locate the Recommended Configurations YAML block.

During this Technical Preview phase, you can manually apply these recommendations during the model deployment workflow:

Copy the CLI arguments and environment variables from the model card YAML template.
Initiate the Deploy workflow for the model.
In the Serving Runtime configuration, add the copied CLI arguments to the Additional Service Arguments field.
Add the recommended environment variables to the deployment configuration.

Technical details and scope: * Hardware profile: The initial set of recipes is specifically tuned for NVIDIA H200 GPUs. * Workload profile: Configurations are optimized for Online Serving (balancing latency and throughput) with a single profile of 8K prefill / 1K generation. * Validation: Every configuration has undergone accuracy and performance testing by the Red Hat Model Validation and PSAP teams.

Note

This Technical Preview feature is intended for feedback collection. Future releases will include UI enhancements such as one-click application of these presets and automated hardware detection.

Artifact signing and verification for model registry

Model registry now provides cryptographic signing and verification of AI artifacts by using the Python client API. This Technology Preview feature enables you to sign artifacts to establish authenticity and verify signatures to confirm integrity. Artifact signing provides foundational capabilities for model serving verification and AI asset provenance tracking in future releases.

For more information, see the Content from github.com is not included.Kubeflow Model Registry Python client documentation and Content from github.com is not included.Kubeflow async upload job documentation.

EvalHub client SDK and CLI: OpenShift AI includes a Technology Preview of the EvalHub client SDK and command-line interface (CLI). The EvalHub SDK provides a Python client library and CLI for integrating evaluation workflows into development environments, automation pipelines, and notebook-based experimentation.

This Technology Preview introduces the following capabilities:

Python client library (eval-hub-sdk) for programmatic interaction with the EvalHub REST API, including job submission, provider discovery, and result retrieval
CLI utilities through the evalhub command for submitting evaluation jobs, checking job status, retrieving results, and managing evaluation collections from the terminal
Support for both synchronous and asynchronous evaluation workflows
Authentication support for API keys and service accounts
Adapter SDK for writing custom evaluation framework adapters

Install the client library with the pip install "eval-hub-sdk[client]" command, to include CLI support with pip install"eval-hub-sdk[cli]" and use "eval-hub-sdk" for installing only the minimum required for writing an EvalHub adapter using the Adapter SDK.

Evaluation Stack user interface: Red Hat OpenShift AI includes a Technology Preview of the Evaluation Stack user interface. The Evaluation Stack UI provides a guided workflow in the OpenShift AI dashboard for running model evaluations without requiring command-line tools or notebook workflows.

This Technology Preview introduces the following capabilities:

Browse and select evaluation tasks from registered providers, including LM Evaluation Harness, RAGAS, Garak, and GuideLLM
Run built-in evaluation collections that group benchmarks across providers into reusable suites
Configure evaluation parameters such as sample count and maximum tokens per response
Evaluate models or AI applications against standardized benchmarks or custom evaluation datasets
Monitor evaluation progress in real time and cancel running evaluations
View summarized evaluation scores and metrics

The Evaluation Stack UI requires EvalHub to be deployed on the cluster. For information about deploying and configuring EvalHub, see Evaluate LLMs with EvalHub.

3.2. 3.4 EA2 Technology Preview features

Create ability to sign and verify AI Artifacts in Registry: For more information about signing and verifying models in the Model Registry see Content from github.com is not included.https://github.com/kubeflow/model-registry/tree/main/clients/python#signing-and-verifying-models. For async upload job documentation, see Content from github.com is not included.https://github.com/kubeflow/model-registry/tree/main/jobs/async-upload#signing

TLS and proxy configuration for all Llama Stack remote inference providers: Red Hat OpenShift AI 3.4 EA2 introduces a standardized network configuration block for all Llama Stack remote inference providers. This generalizes TLS and proxy settings, previously exclusive to the remote::vllm provider, to ensure compatibility across private network architectures.

Llama Stack versions in Red Hat OpenShift AI 3.4 EA2: Red Hat OpenShift AI 3.4 EA2 includes Open Data Hub Llama Stack version 0.6.0.1+rhai0, which is based on upstream Llama Stack version 0.6.0.

MLflow integration: MLflow is no longer a Technology Preview feature. Starting with Red Hat OpenShift AI 3.4, the MLflow Operator is a managed component in the DataScienceCluster. For more information, see Track experiments with MLflow SDK.

Support for text embedding models in the Model Catalog

Red Hat OpenShift AI includes a dedicated suite of text embedding models within the Model Catalog that can be deployed on vLLM. This Technology Preview feature allows data scientists and AI engineers to discover and deploy models designed specifically for vector generation, a critical component for Retrieval-Augmented Generation (RAG) and semantic search workflows. By separating these models from standard generative large language models (LLMs), OpenShift AI provides a streamlined experience for users building vector-generation engines.

Key features of this update include:

Categorized discovery: Embedding models are available under the Other Models category in the Model Catalog.
Distinct labeling: Each model card is labeled with a text-embedding tag to clarify the model’s function (outputting numerical vectors) compared to text-generation models.
Ready-to-deploy images: The models are provided as OCI-compliant ModelCar images, ensuring they are ready for immediate instantiation as running services on an OpenShift cluster.
Supported models include:
Granite Embedding English R2
Embedding Gemma 300M
Nomic Embed Text v1.5
Qwen3 Embedding 8B
All-MiniLM-L6-v2

To view embedding models in the Catalog, navigate to AI hub → Models → Catalog in the OpenShift AI dashboard, select the Other models category, and look for the text-embedding tag on model cards. You can deploy a model directly from its card by following the deployment wizard to configure the serving runtime and hardware profile. Once deployed, the model provides an API endpoint that generates numerical embeddings for text inputs for use in downstream vector databases or RAG pipelines.

Note

Some models, such as Nomic Embed Text v1.5, require extra vLLM parameters to be set, such as --trust-remote-code. Refer to upstream Content from huggingface.co is not included.Hugging Face model cards for specific details.

Workbench and runtime images default to the Red Hat Python index: Access and use Red Hat built and supported Python packages. Installing or updating Python packages will pull from the Red Hat Python index rather than PyPI.

YAML viewer for Distributed Inference with llm-d model deployments

The model deployment wizard includes a YAML viewer that provides real-time YAML preview and manual editing capabilities for LLMInferenceService deployments. You can toggle between Form and YAML views to see the generated LLMInferenceService YAML as you fill out the deployment form. The preview updates automatically as form fields change, enabling you to verify configuration before deployment. You can also enter Manual Edit Mode to edit YAML directly for advanced Distributed Inference with llm-d parameters not exposed in the form, such as autoscaling, disaggregated serving, LoRA adapters, and custom runtime configurations.

This Technology Preview feature introduces the following capabilities:

Real-time YAML preview with automatic form-to-YAML synchronization
Copy to clipboard and download functionality
Manual Edit Mode for direct YAML editing of advanced configurations
Automatic fallback to YAML editing when the wizard cannot parse deployment configurations (for example, deployments created via CLI or GitOps)
Client-side YAML syntax validation before deployment

Note

Manual Edit Mode is a one-way operation. Once you enter manual YAML editing mode, you cannot return to the guided form view for that deployment session. Changes made in YAML edit mode bypass form validation.

3.3. 3.4 EA1 Technology Preview features

Llama Stack versions in OpenShift AI 3.4 EA1: OpenShift AI 3.4 EA1 includes Open Data Hub Llama Stack version 0.5.0+rhai0, which is based on upstream Llama Stack version 0.5.0.

Gen AI Playground interface redesign: The Gen AI Playground interface provides a prompt-lab-style experience with improved prompt-driven experimentation, rapid iteration capabilities, and clear visual feedback. This redesign aligns the Playground experience with industry-standard patterns for GenAI tools.

Multi-instance chat comparison in Playground

You can compare results across multiple configurations in the Playground by using multiple chat panes side-by-side. Each pane supports unique configurations for models, prompts, MCP servers, guardrails, and knowledge sources. You can run synchronized prompts across all panes or use independent prompts for each pane.

Key capabilities include:

Side-by-side output comparison
Toggle individual chat panes for A/B-style testing
Runtime information display including latency and token counts

Basic guardrails available in Gen AI Playground

The Gen AI Playground provides access to basic safety guardrails from Llama Stack. You can enable or disable guardrails in the playground interface to filter unsafe content and detect prompt injection attempts.

The following guardrails are available:

Content Safety (Llama Guard): Filters categories such as hate, violence, sexual content, self-harm, criminal activity, and privacy violations.
Prompt Injection / Jailbreak Detection (Prompt Guard): Detects user attempts to override system or tool behavior.
Privacy Awareness: Flags possible PII in inputs and outputs.
Note
Guardrail enforcement applies to llm_input and llm_output touchpoints only. Tool-level guardrails are not included in this release.

Llama Stack Connectors: Llama Stack Connectors provide a high-level abstraction for AI registries such as MCP. Platform Engineers can register connectors by using a connector_id, and AI Engineers can consume pre-registered connectors without managing complex configurations. This feature simplifies the workflow for AI engineers by abstracting away infrastructure concerns.

Conversations API: The Conversations API enables multi-turn, context-aware chats by managing message history, tool outputs, and conversation state. Developers can use this API to build AI applications with memory, moving beyond simple stateless requests to create persistent, intelligent interactions.

OpenAI-compatible annotations for search and responses in Llama Stack

Starting with OpenShift AI 3.3, Llama Stack provides OpenAI-compatible grounding and citation annotations for search-backed responses as a Technology Preview feature.

This enhancement enables retrieval-augmented generation (RAG) applications to trace generated responses back to source documents by using the same annotation schemas returned by OpenAI Search and Responses APIs. The feature supports document source attribution and preserves citation metadata in API responses, allowing existing OpenAI client applications to consume citation information without code changes.

This capability improves transparency, auditability, and explainability for enterprise RAG workloads, and serves as a foundation for future advanced tracing and observability features in Llama Stack. For more information, see OpenAI-compatible file citation annotations.

The Llama Stack Operator available on multi-architecture clusters: The Llama Stack Operator is deployable on multi-architecture clusters in OpenShift AI version 3.3 and is available by default.

Llama Stack versions in OpenShift AI 3.3: OpenShift AI 3.3.0 includes Open Data Hub Llama Stack version Content from github.com is not included.0.4.2.1+rhai0, which is based on upstream Llama Stack version Content from github.com is not included.0.4.2.

The Llama Stack Operator with ConfigMap driven image updates

The Llama Stack Operator in OpenShift AI 3.3 now offers ConfigMap driven image updates for LlamaStackDistribution resources. This allows you to patch security or bug fixes without new operator versions. To enable this feature, update your ConfigMap with the following parameters:

 image-overrides: |
    starter-gpu: registry.redhat.io/rhoai/odh-llama-stack-core-rhel9:v3.3
    starter: registry.redhat.io/rhoai/odh-llama-stack-core-rhel9:v3.3

Using the starter-gpu and starter distributions names as the key allows the operator to apply these overrides automatically.

To update the Llama Stack Distributions image for all starter distributions, run the following command:

$ kubectl patch configmap llama-stack-operator-config -n llama-stack-k8s-operator-system --type merge -p '{"data":{"image-overrides":"starter: quay.io/opendatahub/llama-stack:latest"}}'

This allows the LlamaStackDistribution resources to restart with the new image.

Model-as-a-Service (MaaS) integration

This feature is available as a Technology Preview.

OpenShift AI now includes Model-as-a-Service (MaaS) to address resource consumption and governance challenges associated with serving large language models (LLMs).

MaaS provides centralized control over model access and resource usage by exposing models through managed API endpoints, allowing administrators to enforce consumption policies across teams.

This Technology Preview introduces the following capabilities:

Policy and quota management
Authentication and authorization
Usage tracking
User management
Zero-Touch setup through Red Hat OpenShift AI Operator
For more information, see Governing LLM access with models-as-a-service.

MLServer ServingRuntime for KServe

The MLServer serving runtime for KServe is now available as a technology preview feature in Red Hat OpenShift AI. You can use this runtime to deploy models trained on structured data, such as classical machine learning models. You can deploy models directly without converting them to ONNX format, which simplifies the deployment process and improves performance.

This feature provides support for the following common machine learning frameworks:

scikit-learn
XGBoost
LightGBM
For more information, see Deploying models using the mlserver runtime and Supported configurations.

pgvector support as a remote vector store provider in Llama Stack

Starting with OpenShift AI 3.2, you can use PostgreSQL with the pgvector extension as a remote vector store provider for the Llama Stack vector_store endpoint as a Technology Preview feature.

This enhancement enables vector storage backed by PostgreSQL, providing durable and transactional persistence for vector embeddings. For more information, see Llama Stack API provider support and Deploying a PostgreSQL instance with pgvector.

Llama Stack versions in OpenShift AI 3.2: OpenShift AI 3.2.0 uses the Open Data Hub Llama Stack version 0.3.5+rhai0 in the Llama Stack Distribution, which is based on the upstream Llama Stack version 0.3.5.

Llama Stack servers now require installation of the PostgreSQL Operator: In OpenShift AI 3.2, the PostgreSQL Operator is now required to deploy a Llama Stack server. For more information, see the Deploying a Llama Stack server documentation.

Enabling high availability on Llama Stack: Llama Stack servers can be configured to remain operational in the event of a single point of failure as a Technology Preview feature. You can enable PostgreSQL high-availability settings in your LlamaStackDistribution custom resource. For more information, see the Enabling high availability on Llama Stack (Optional) documentation.

Custom embeddings on Llama Stack

OpenShift AI 3.2 allows you to customize your embedding models as a Technology Preview feature. In the version of Llama Stack shipped in OpenShift AI 3.2, vLLM controls embeddings by default. You can update the VLLM_EMBEDDING_URL environment variable in your LlamaStackDistribution custom resource to enable embeddings, or you can use custom embeddings providers. For example:

  - name: ENABLE_SENTENCE_TRANSFORMERS
    value: "true"
  - name: EMBEDDING_PROVIDER
    value: "sentence-transformers"

NVIDIA NeMo Guardrails: You can use NVIDIA NeMo Guardrails as a Technology Preview feature to add guardrails and safety controls to your deployed models in Red Hat OpenShift AI. NeMo Guardrails provides a framework for controlling conversations with large language models, enabling you to define a variety of rails, such as sensitive data detection, content filtering, or custom validation rules.

Stop button for chatbot in Generative AI Studio: You can interrupt the chatbot as it is composing a response to a prompt. In the Playground, after you send a prompt, the Send button in the chat input field changes to a Stop button. Click it if you want to interrupt the model’s response, for example, when the response takes longer than you anticipated or if you notice that you made an error in your prompt. The chatbot posts "You stopped this message" to confirm your stop request.

Kubeflow Trainer v2

Kubeflow Trainer v2 is now available as a Technology Preview feature in OpenShift AI 3.2.

Kubeflow Trainer v2 is the next generation of distributed training for OpenShift AI, replacing the Kubeflow Training Operator v1 (KFTOv1). This Kubernetes-native solution simplifies how data scientists and ML engineers run PyTorch training workloads at scale using a unified TrainJob API and Python SDK.

This Technology Preview release introduces the following capabilities:

Simplified job definitions using TrainJob and TrainingRuntime resources
Python SDK for programmatic job creation and management
A new web-based user interface for inspecting and interacting with training jobs
Real-time progress tracking with visibility into training steps, epochs, and metrics
Smart checkpoint management with automatic preservation during pod preemption or termination
Pausing and resuming train jobs
Resource-aware scheduling via native integration with Red Hat build of Kueue
Users of the deprecated Kubeflow Training Operator v1 (KFTOv1) should migrate their workloads to Kubeflow Trainer v2 before KFTOv1 is removed. For guidance and more details, see the migration guide.
For more information about Kubeflow Trainer v2 features and usage, see the Kubeflow Trainer v2 documentation.

TrustyAI–Llama Stack integration for safety, guardrails, and evaluation

You can now use the Guardrails Orchestrator from TrustyAI with Llama Stack as a Technology Preview feature.

This integration enables built-in detection and evaluation workflows to support AI safety and content moderation. When TrustyAI is enabled and the FMS Orchestrator and detectors are configured, no manual setup is required.

To activate this feature, set the following field in the DataScienceCluster custom resource for the OpenShift AI Operator: spec.llamastackoperator.managementState: Managed

For more information, see the TrustyAI FMS Provider on GitHub: Content from github.com is not included.TrustyAI FMS Provider.

AI Available Assets page for deployed models and MCP servers

A new AI Available Assets page enables AI engineers and application developers to view and consume deployed AI resources within their projects.

This enhancement introduces a filterable UI that lists available models and Model Context Protocol (MCP) servers in the selected project, allowing users with appropriate permissions to identify accessible endpoints and integrate them directly into the AI Playground or other applications.

Generative AI Playground for model testing and evaluation

The Generative AI (GenAI) Playground introduces a unified, interactive experience within the OpenShift AI dashboard for experimenting with foundation and custom models.

Users can test prompts, compare models, and evaluate Retrieval-Augmented Generation (RAG) workflows by uploading documents and chatting with their content. The GenAI Playground also supports integration with approved Model Context Protocol (MCP) servers and enables export of prompts and agent configurations as runnable code for continued iteration in local IDEs.

Chat context is preserved within each session, providing a suitable environment for prompt engineering and model experimentation.

Support for air-gapped Llama Stack deployments

You can now install and operate Llama Stack and RAG/Agentic components in fully disconnected (air-gapped) OpenShift AI environments.

This enhancement enables secure deployment of Llama Stack features without internet access, allowing organizations to use AI capabilities while maintaining compliance with strict network security policies.

Feature Store integration with Workbenches and new user access capabilities

This feature is available as a Technology Preview.

The Feature Store is now integrated with OpenShift AI, data science projects, and workbenches. This integration also introduces centrally managed, role-based access control (RBAC) capabilities for improved governance.

These enhancements provide two key capabilities:

Feature development within the workbench environment.
Administrator-controlled user access.
This update simplifies and accelerates feature discovery and consumption for data scientists while allowing platform teams to maintain full control over infrastructure and feature access.

Feature Store user interface

The Feature Store component now includes a web-based user interface (UI).

You can use the UI to view registered Feature Store objects and their relationships, such as features, data sources, entities, and feature services.

To enable the UI, edit your FeatureStore custom resource (CR) instance. When you save the change, the Feature Store Operator starts the UI container and creates an OpenShift route for access.

For more information, see Setting up the Feature Store user interface for initial use.

IBM Spyre AI Accelerator model serving support on x86 platforms: Model serving with the IBM Spyre AI Accelerator is now available as a Technology Preview feature for x86 platforms. The IBM Spyre Operator automates installation and integrates the device plugin, secondary scheduler, and monitoring. For more information, see the This content is not included.IBM Spyre Operator catalog entry.

Build Generative AI Apps with Llama Stack on OpenShift AI

With this release, the Llama Stack Technology Preview feature enables Retrieval-Augmented Generation (RAG) and agentic workflows for building next-generation generative AI applications. It supports remote inference, built-in embeddings, and vector database operations. It also integrates with providers like TrustyAI’s provider for safety and Trusty AI’s LM-Eval provider for evaluation.

This preview includes tools, components, and guidance for enabling the Llama Stack Operator, interacting with the RAG Tool, and automating PDF ingestion and keyword search capabilities to enhance document discovery.

Centralized platform observability

Centralized platform observability, including metrics, traces, and built-in alerts, is available as a Technology Preview feature. This solution introduces a dedicated, pre-configured observability stack for OpenShift AI that allows cluster administrators to perform the following actions:

View platform metrics (Prometheus) and distributed traces (Tempo) for OpenShift AI components and workloads.
Manage a set of built-in alerts (alertmanager) that cover critical component health and performance issues.
Export platform and workload metrics to external 3rd party observability tools by editing the DataScienceClusterInitialization (DSCI) custom resource.
You can enable this feature by integrating with the Cluster Observability Operator, Red Hat build of OpenTelemetry, and Tempo Operator. For more information, see Monitoring and observability. For more information, see Managing observability.

Support for Llama Stack Distribution version 0.3.0

The Llama Stack Distribution now includes version 0.3.0 as a Technology Preview feature.

This update introduces several enhancements, including expanded support for retrieval-augmented generation (RAG) pipelines, improved evaluation provider integration, and updated APIs for agent and vector store management. It also provides compatibility updates aligned with recent OpenAI API extensions and infrastructure optimizations for distributed inference.

The previously supported version was 0.2.22.

Support for Kubernetes Event-driven Autoscaling (KEDA)

OpenShift AI now supports Kubernetes Event-driven Autoscaling (KEDA) in its KServe RawDeployment mode. This Technology Preview feature enables metrics-based autoscaling for inference services, allowing for more efficient management of accelerator resources, reduced operational costs, and improved performance for your inference services.

To set up autoscaling for your inference service in KServe RawDeployment mode, you need to install and configure the OpenShift Custom Metrics Autoscaler (CMA), which is based on KEDA.

For more information about this feature, see: Configuring metrics-based autoscaling.

LM-Eval model evaluation UI feature: TrustyAI now offers a user-friendly UI for LM-Eval model evaluations as Technology Preview. This feature allows you to input evaluation parameters for a given model and returns an evaluation-results page, all from the UI.

Support for creating and managing Ray Jobs with the CodeFlare SDK

You can now create and manage Ray Jobs on Ray Clusters directly through the CodeFlare SDK.

This enhancement aligns the CodeFlare SDK workflow with the KubernetesFlow Training Operator (KFTO) model, where a job is created, run, and completed automatically. This enhancement simplifies manual cluster management by preventing Ray Clusters from remaining active after job completion.

Custom flow estimator for Synthetic Data Generation pipelines

You can now use a custom flow estimator for synthetic data generation (SDG) pipelines.

For supported and compatible tagged SDG teacher models, the estimator helps you evaluate a chosen teacher model, custom flow, and supported hardware on a sample dataset before running full workloads.

Llama Stack support and optimization for single node OpenShift (SNO)

Llama Stack core can now deploy and run efficiently on single node OpenShift (SNO).

This enhancement optimizes component startup and resource usage so that Llama Stack can operate reliably in single-node cluster environments.

FAISS vector storage integration

You can now use the FAISS (Facebook AI Similarity Search) library as an inline vector store in OpenShift AI.

FAISS is an open-source framework for high-performance vector search and clustering, optimized for dense numerical embeddings with both CPU and GPU support. When enabled with an embedded SQLite backend in the Llama Stack Distribution, FAISS stores embeddings locally within the container, removing the need for an external vector database service.

New Feature Store component

You can now install and manage Feature Store as a configurable component in OpenShift AI. Based on the open-source Content from feast.dev is not included.Feast project, Feature Store acts as a bridge between ML models and data, enabling consistent and scalable feature management across the ML lifecycle.

This Technology Preview release introduces the following capabilities:

Centralized feature repository for consistent feature reuse
Python SDK and CLI for programmatic and command-line interactions to define, manage, and retrieve features for ML models
Feature definition and management
Support for a wide range of data sources
Data ingestion via feature materialization
Feature retrieval for both online model inference and offline model training
Role-Based Access Control (RBAC) to protect sensitive features
Extensibility and integration with third-party data and compute providers
Scalability to meet enterprise ML needs
Searchable feature catalog
Data lineage tracking for enhanced observability
For configuration details, see Configuring Feature Store.

FIPS support for Llama Stack and RAG deployments

You can now deploy Llama Stack and RAG or agentic solutions in regulated environments that require FIPS compliance.

This enhancement provides FIPS-certified and compatible deployment patterns to help organizations meet strict regulatory and certification requirements for AI workloads.

Validated sdg-hub notebooks for Red Hat AI Platform

Validated sdg_hub example notebooks are now available to provide a notebook-driven user experience in OpenShift AI 3.0.

These notebooks support multiple Red Hat platforms and enable customization through SDG pipelines. They include examples for the following use cases:

Knowledge and skills tuning, including annotated examples for fine-tuning models.
Synthetic data generation with reasoning traces to customize reasoning models.
Custom SDG pipelines that demonstrate using default blocks and creating new blocks for specialized workflows.

RAGAS evaluation provider for Llama Stack (inline and remote)

You can now use the Retrieval-Augmented Generation Assessment (RAGAS) evaluation provider to measure the quality and reliability of RAG systems in OpenShift AI.

RAGAS provides metrics for retrieval quality, answer relevance, and factual consistency, helping you identify issues and optimize RAG pipeline configurations.

The integration with the Llama Stack evaluation API supports two deployment modes:

Inline provider: Runs RAGAS evaluation directly within the Llama Stack server process.
Remote provider: Runs RAGAS evaluation as distributed jobs using OpenShift AI pipelines.
The RAGAS evaluation provider is now included in the Llama Stack distribution.

Enable targeted deployment of workbenches to specific worker nodes in Red Hat OpenShift AI Dashboard using node selectors

The hardware profiles feature enables users to target specific worker nodes for workbenches or model-serving workloads. It allows users to target specific accelerator types or CPU-only nodes.

This feature replaces the current accelerator profiles feature and container size selector field, offering a broader set of capabilities for targeting different hardware configurations. While accelerator profiles, taints, and tolerations provide some capabilities for matching workloads to hardware, they do not ensure that workloads land on specific nodes, especially if some nodes lack the appropriate taints.

The hardware profiles feature supports both accelerator and CPU-only configurations, along with node selectors, to enhance targeting capabilities for specific worker nodes. Administrators can configure hardware profiles in the settings menu. Users can select the enabled profiles using the UI for workbenches, model serving, and AI pipelines where applicable.

RStudio Server workbench image

With the RStudio Server workbench image, you can access the RStudio IDE, an integrated development environment for R. The R programming language is used for statistical computing and graphics to support data analysis and predictions.

To use the RStudio Server workbench image, you must first build it by creating a secret and triggering the BuildConfig, and then enable it in the OpenShift AI UI by editing the rstudio-rhel9 image stream. For more information, see Building the RStudio Server workbench images.

Important

Disclaimer: Red Hat supports managing workbenches in OpenShift AI. However, Red Hat does not provide support for the RStudio software. RStudio Server is available through Content from rstudio.org is not included.rstudio.org and is subject to their licensing terms. You should review their licensing terms before you use this sample workbench.

CUDA - RStudio Server workbench image

With the CUDA - RStudio Server workbench image, you can access the RStudio IDE and NVIDIA CUDA Toolkit. The RStudio IDE is an integrated development environment for the R programming language for statistical computing and graphics. With the NVIDIA CUDA toolkit, you can enhance your work by using GPU-accelerated libraries and optimization tools.

To use the CUDA - RStudio Server workbench image, you must first build it by creating a secret and triggering the BuildConfig, and then enable it in the OpenShift AI UI by editing the rstudio-rhel9 image stream. For more information, see Building the RStudio Server workbench images.

Important

The CUDA - RStudio Server workbench image contains NVIDIA CUDA technology. CUDA licensing information is available in the Content from docs.nvidia.com is not included.CUDA Toolkit documentation. You should review their licensing terms before you use this sample workbench.

Support for multinode deployment of very large models

Serving models over multiple graphical processing unit (GPU) nodes when using a single-model serving runtime is now available as a Technology Preview feature. Deploy your models across multiple GPU nodes to improve efficiency when deploying large models such as large language models (LLMs). For more information, see Deploying models by using multiple GPU nodes.

Chapter 4. Developer Preview features

Important

This section describes Developer Preview features in Red Hat OpenShift AI 3.4. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to functionality in advance of possible inclusion in a Red Hat product offering. Customers can use these features to test functionality and provide feedback during the development process. Developer Preview features might not have any documentation, are subject to change or removal at any time, and have received limited testing. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

For more information about the support scope of Red Hat Developer Preview features, see This content is not included.Developer Preview Support Scope.

4.1. 3.4 GA Developer Preview features

AgentCard support for post-deployment agent discovery: You can now discover deployed agents and their capabilities through the AgentCard custom resource. When you deploy an agent as a Kubernetes Deployment or StatefulSet with the kagenti.io/type: agent label and a protocol label such as protocol.kagenti.io/a2a, the platform automatically creates an AgentCard that advertises the agent’s capabilities, endpoints, and supported protocols. This enables machine-readable agent-to-agent discovery for multi-agent workflows. For more information, see Content from github.com is not included.Kagenti Operator Repository.

Agent deploy and runtime management: You can now manage the runtime concerns of deployed agents using the AgentRuntime custom resource. Create an AgentRuntime that references your agent’s Deployment or StatefulSet via spec.targetRef, and the operator automatically injects authentication and identity sidecars into agent pods. This includes an Envoy-based AuthBridge proxy for inbound JWT validation and outbound token exchange, SPIFFE-based workload identity via a spiffe-helper sidecar, and per-agent OpenTelemetry trace configuration. Agents labeled with kagenti.io/type: agent are injected by default, but you can opt out with kagenti.io/inject: disabled. The AgentRuntime also supports per-workload overrides for SPIFFE trust domains, OTEL collector endpoints, and trace sampling rates. For more information, see Content from github.com is not included.Kagenti Operator Repository.

Distributed Tracing for Distributed Inference with llm-d: Platform Operators can trace distributed inference with llm-d requests end-to-end across service boundaries using OpenTelemetry-compatible distributed tracing. Traces correlate latency and errors across the full request path, from gateway through router-scheduler to model servers.
Batch inference compatible with OpenAI batch APIs in Distributed Inference with llm-d: Platform Operators can submit large request volumes asynchronously through the OpenAI-compatible /v1/batches API and retrieve results without maintaining an active connection. Batch workloads run at lower priority than real-time traffic and are scheduled during periods of low cluster activity. For more information, see Content from github.com is not included.Batch Gateway and Content from github.com is not included.Deploy Batch Gateway on Kubernetes.

Existing vector stores available as RAG knowledge sources in Gen AI Studio Playground

You can surface previously-created vector stores as retrieval-augmented generation (RAG) knowledge sources in the Gen AI Studio Playground. Platform engineers define vector stores through ConfigMaps, and AI engineers can select them as knowledge sources when chatting with models in the Playground. This enables rapid RAG experimentation without writing code or re-ingesting data.

This release includes the following capabilities:

Platform engineers can declare vector stores through ConfigMaps with connection details, collection names, and optional metadata
Available vector stores appear under RAG / Knowledge Sources in the Playground
Users can enable or disable a vector store per chat session
Queries are routed through Llama Stack RAG primitives

Note

This feature does not include document upload, ingestion, chunking, indexing, or vector store lifecycle management. Only read-only query and retrieval of pre-existing vector data is supported.

Interact with Red Hat OpenShift AI using MCP clients

Red Hat OpenShift AI provides an MCP (Model Context Protocol) server that enables MCP-compatible clients to interact with your environment through natural-language conversations. When you describe your goals, the MCP server recommends the optimal model from your model registry by matching your intent against benchmarks such as MMLU and HumanEval, with cost comparisons. You can also manage data science projects, create workbenches, and monitor pipeline runs.

When you are ready to deploy, the MCP server generates three production-ready Kubernetes manifests for model serving, auto-scaling, and observability. The MCP server works with AI coding assistants such as Claude Code, OpenCode, and Gemini CLI. For more information, see Content from github.com is not included.RHOAI MCP server.

Tool calling metadata on model cards in the model catalog

Red Hat OpenShift AI now displays tool callin, also known as function calling, configuration metadata directly in the model catalog. This update ensures that users have immediate access to the specific CLI arguments and chat templates required to successfully deploy models for agentic or tool-use workflows.

Previously, users had to manually identify which arguments were necessary for a model to interact with external APIs or functions. Validated models in the catalog now include this technical metadata as a single source of truth, accessible directly on the model card.

Key features of this update include:

Metadata visibility: Tool-calling commands and configuration requirements are now rendered in the YAML frontmatter of the model card. Users can view and copy these parameters directly from the UI to ensure deployment accuracy.
New Tool calling filter: A Tool calling task filter has been added to the left-hand navigation pane in the model catalog. Users can quickly filter the catalog for models specifically validated by the Red Hat Model Validation team for tool-calling capabilities.
Validated parameters: For supported models, the catalog now provides specific CLI requirements, such as --enable-auto-tool-choice, --tool-call-parser, and the correct chat_template in the metadata on the tool calling section of the model card in the model catalog.

To identify models that support external tool integration:

In the OpenShift AI dashboard, from the left navigation menu, click AI hub → Catalog.
In the left filter pane, under the Task section, select Tool calling.
The catalog refreshes to show only models that have been validated for tool-use compatibility.

To view and copy the specific arguments required for a tool-calling deployment:

Select a model from the Tool calling filtered list, for example, Ministral-3-14B-Instruct-2512.
Click the model to view its model card and scroll to the model details frontmatter section to locate the tool-calling metadata.
Copy the validated CLI arguments for use in your model-serving configuration in the additional arguments text box.

MCP Catalog for enterprise management of MCP servers

The MCP Catalog provides a centralized experience for discovering, deploying, and experimenting with Model Context Protocol (MCP) servers in Red Hat OpenShift AI. AI Operators and Platform Engineers can browse available MCP servers in a catalog UI, view descriptive metadata about each server’s capabilities and tools, and deploy MCP servers into their namespace directly from the catalog. After deployment, a platform engineer can register deployed MCP servers in the gen AI studio configuration, making them available in the playground for interactive experimentation.

The MCP Catalog ships pre-loaded with MCP servers from Red Hat, technology partners, and the open source community. These servers can be deployed directly from the catalog without sourcing container images or configuring transport manually. Some additional prerequisite steps, such as mirroring images may be required for disconnected deployments or configuring server-specific credentials.

Red Hat MCP servers:
- Red Hat OpenShift - Cluster management and troubleshooting
- Red Hat Ansible Automation Platform - Playbook execution and configuration orchestration
- Red Hat Insights - platform intelligence and remediation recommendations.
Technology partner MCP servers:
- Confluent Cloud - Kafka and Flink streaming
- EDB Postgres AI - database queries and schema management
- HashiCorp Terraform - infrastructure as code
- Microsoft Azure - cloud resource management
- Dynatrace - performance monitoring and troubleshooting.
Other MCP servers: MongoDB (document collections and RAG workflows) and MariaDB (relational database connectivity).

The MCP Catalog relies on the following pre-requisite upstream community components:

mcp-lifecycle-operator: A Kubernetes operator that provides a declarative API to deploy, manage, and roll out MCP servers. When an MCPServer custom resource is created, the operator automatically provisions the required Deployments, Services, and cluster-internal URLs for service discovery. The MCP lifecycle operator must be installed on the cluster before deploying MCP servers from the catalog. For more information, see kubernetes-sigs/mcp-lifecycle-operator on GitHub.
mcp-gateway: A Kubernetes-native gateway that provides a unified runtime endpoint for accessing deployed MCP servers. The MCP Gateway aggregates tools from multiple registered MCP servers behind a single endpoint, enabling centralized access control and routing. For more information, see the mcp-gateway/docs/guides/quick-start.md on GitHub.

The deploy action in the catalog UI is gated on the presence of the MCP lifecycle operator. If the operator is not installed, the deployment option is not available.

4.2. 3.4 EA2 Developer Preview features

Automate machine learning model training with AutoML: AutoML is available as a Developer Preview feature in Red Hat OpenShift AI 3.4. You can use AutoML to automatically train and compare machine learning models for your tabular data. AutoML evaluates multiple models and ranks the results in a leaderboard. For the top-performing models, AutoML generates trained model artifacts and a notebook that you can use to deploy the best model. For more information, see Content from github.com is not included.AutoML on GitHub.

Automate RAG optimization with AutoRAG: AutoRAG is available as a Developer Preview feature in Red Hat OpenShift AI 3.4. You can use AutoRAG to find optimal RAG configurations for your documents and use cases. AutoRAG evaluates multiple configurations and ranks the results in a leaderboard. For the top-performing configurations, AutoRAG generates Jupyter notebooks that you can use to deploy your RAG application. For more information, see Content from github.com is not included.AutoRAG on GitHub.

Core Evaluation Stack control plane

The Evaluation Stack control plane provides an API REST routing and orchestration layer for AI evaluation, benchmarking, and profiling backends on OpenShift AI. AI engineers can deploy and manage a comprehensive evaluation platform that supports multiple frameworks and execution modes through a unified interface.

The Evaluation Stack control plane includes the following capabilities:

REST API endpoints for programmatic evaluation triggering from web UI interfaces and other components
Built-in support for LM Evaluation Harness, RAGAS, Garak, and GuideLLM evaluation frameworks
Custom framework integration by using container images or Python package specifications for Kubeflow Pipelines
Evaluation results tracked in MLflow with a standardized schema
Evaluation job monitoring and progress tracking through Kubernetes constructs
Concurrent evaluation jobs with resource isolation
Support for air-gapped and disconnected environments
Installable Python package for local development

4.3. 3.4 EA1 Developer Preview features

Automatic MLflow experiment creation in EvalHub: The EvalHub service automatically creates an MLflow experiment when you specify experiment.name in the evaluation job request. If the experiment creation fails due to missing MLflow configuration, authentication issues, or other problems, the job request returns an error.

Kubeflow Spark Operator for distributed data processing

The Kubeflow Spark Operator is now available in OpenShift AI as a Developer Preview. This feature provides Apache Spark integration with OpenShift AI, enabling the full lifecycle of Spark applications on Kubernetes. This enables unified orchestration and monitoring of large-scale data processing and preparation Spark jobs alongside ML training and inference workflows.

To enable the Kubeflow Spark Operator, navigate to your dsc.yaml CR and update the kubeflowsparkoperator parameter to the Managed state.

This feature introduces the following capabilities:

Integration with the OpenShift AI distributed workloads ecosystem.
SparkApplication custom resources (CRs) for defining Spark jobs.
Automatic submission, monitoring and restart of Spark applications with configuration retry policies.
Pod customization via mutating webhooks, supporting ConfigMaps, volumes, and affinity rules.

Run evaluations for TrustyAI-Llama Stack using LM-Eval

You can now run evaluations using LM-Eval on Llama Stack with TrustyAI as a Developer Preview feature, using the built-in LM-Eval component and advanced content moderation tools. To use this feature, ensure TrustyAI is enabled, the FMS Orchestrator and detectors are set up, and KServe RawDeployment mode is in use for full compatibility if needed. There is no manual set up required.

Then, in the DataScienceCluster custom resource for the Red Hat OpenShift AI Operator, set the spec.llamastackoperator.managementState field to Managed.

For more information, see the following resources on GitHub:

LLM Compressor integration

LLM Compressor capabilities are now available in Red Hat OpenShift AI as a Developer Preview feature. A new workbench image with the llm-compressor library and a corresponding data science pipelines runtime image make it easier to compress and optimize your large language models (LLMs) for efficient deployment with vLLM. For more information, see Content from github.com is not included.llm-compressor in GitHub.

You can use LLM Compressor capabilities in two ways:

Use a Jupyter notebook with the workbench image available at Content from quay.io is not included.Red Hat Quay.io: opendatahub / llmcompressor-workbench.
For an example Jupyter notebook, see Content from github.com is not included.examples/llmcompressor/workbench_example.ipynb in the red-hat-ai-examples repository.
Run a data science pipeline that executes model compression as a batch process with the runtime image available at Content from quay.io is not included.Red Hat Quay.io: opendatahub / llmcompressor-pipeline-runtime.
For an example pipeline, see Content from github.com is not included.examples/llmcompressor/oneshot_pipeline.py in the red-hat-ai-examples repository.

MLflow integration: MLflow is no longer a Developer Preview feature. Starting with Red Hat OpenShift AI 3.4, the MLflow Operator is a managed component in the DataScienceCluster with automatic workbench SDK configuration. For more information, see Track experiments with MLflow SDK.

AI Available Assets integration with Model-as-a-Service (MaaS)

This feature is available as a Developer Preview.

You can now access and consume Model-as-a-Service (MaaS) models directly from the AI Available Assets page in the GenAI Studio.

Administrators can configure a MaaS by enabling the toggle in the Model Deployments page. When a model is marked as a service, it becomes global and visible across all projects in the cluster.

Additional fields added to Model Deployments for AI Available Assets integration

This feature is available as a Developer Preview.

Administrators can now add metadata to models during deployment so that they are automatically listed on the AI Available Assets page.

The following table describes the new metadata fields that streamline the process of making models discoverable and consumable by other teams:

Field name	Field type	Description
Use Case	Free-form text	Describes the model’s primary purpose, for example, "Customer Churn Prediction" or "Image Classification for Product Catalog."
Description	Free-form text	Provides more detailed context and functionality notes for the model.
Add to AI Assets	Checkbox	When enabled, automatically publishes the model and its metadata to the AI Available Assets page.

Compatibility of Llama Stack remote providers and SDK with MCP HTTP streaming protocol

This feature is available as a Developer Preview.

Llama Stack remote providers and the SDK are now compatible with the Model Context Protocol (MCP) HTTP streaming protocol.

This enhancement enables developers to build fully stateless MCP servers, simplify deployment on standard Llama Stack infrastructure (including serverless environments), and improve scalability. It also prepares for future enhancements such as connection resumption and provides a smooth transition away from Server-Sent Events (SSE).

Packaging of ITS Hub dependencies to the Red Hat–maintained Python index

This feature is available as a Developer Preview.

All Inference Time Scaling (ITS) runtime dependencies are now packaged in the Red Hat-maintained Python index, allowing Red Hat AI and OpenShift AI customers to install its_hub and its dependencies directly by using pip.

This enhancement enables users to build custom inference images with ITS algorithms focused on improving model accuracy at inference time without requiring model retraining, such as:

Particle filtering
Best-of-N
Beam search
Self-consistency
Verifier or PRM-guided search
For more information, see the Content from github.com is not included.ITS Hub on GitHub.

Dynamic hardware-aware continual training strategy

Static hardware profile support is now available to help users select training methods, models, and hyperparameters based on VRAM requirements and reference benchmarks. This approach ensures predictable and reliable training workflows without dynamic hardware discovery.

The following components are included:

API Memory Estimator: Accepts model, training method, dataset metadata, and assumed hyperparameters as input and returns an estimated VRAM requirement for the training job. Delivered as an API within Training Hub.
Reference Profiles and Benchmarks: Provides end-to-end training time benchmarks for OpenShift AI Innovation (OSFT) and Performance Team (LAB SFT) baselines, delivered as static tables and documentation in Training Hub.
Hyperparameter Guidance: Publishes safe starting ranges for key hyperparameters such as learning rate, batch size, epochs, and LoRA rank. Integrated into example notebooks maintained by the AI Innovation team.
Important
Hardware discovery is not included in this release. Only static reference tables and guidance are provided; automated GPU or CPU detection is not yet supported.

Human-in-the-Loop (HIL) functionality in the Llama Stack agent

Human-in-the-Loop (HIL) functionality has been added to the Llama Stack agent to allow users to approve unread tool calls before execution.

This enhancement includes the following capabilities:

Users can approve or reject unread tool calls through the responses API.
Configuration options specify which tool calls require HIL approval.
Tool calls pause until user approval is received for HIL-enabled tools.
Tool calls that do not require HIL continue to run without interruption.

Chapter 5. Support removals

This section describes major changes in support for user-facing features in Red Hat OpenShift AI. For information about OpenShift AI supported software platforms, components, and dependencies, see the Supported Configurations for 3.x Knowledgebase article.

5.1. Deprecated

Deprecated default group creation for model registry: Starting with OpenShift AI 3.4, the default group creation performed by the OpenShift AI Operator when a model registry is created is deprecated. This default group will be removed in a future release of OpenShift AI. The OpenShift administrator will then be responsible for creating this group after the model registry is created, if needed in their workflow.

5.1.1. Ray-based multi-node vLLM template

In Red Hat OpenShift AI 3.3, the Ray-based multi-node vLLM template remains available as a Technology Preview. Starting with Red Hat OpenShift AI 3.4, Ray will be removed from the vLLM multi-node ServingRuntime and multi-node inference will rely on native vLLM multiprocessing support.

Customers can continue using the Ray-based multi-node template in 3.3 (Tech Preview).

5.1.2. Training images and ClusterTrainingRuntimes for Kubeflow Training Operator v1

The Kubeflow Training Operator (v1) is deprecated starting OpenShift AI 2.25 and is scheduled to be removed. This deprecation is part of our transition to Kubeflow Trainer v2, which delivers enhanced capabilities and improved functionality.

New and/or updated container images and associated ClusterTrainingRuntimes will be released for Kubeflow Trainer v2 in Red Hat OpenShift AI 3.4, and the existing runtimes and container images will be deprecated. Guidance for updating to the new runtimes and images will be provided in the Red Hat OpenShift AI 3.4 release.

The list of images being deprecated in Red Hat OpenShift AI 3.4 is:

registry.redhat.io/rhoai/odh-training-cuda121-torch24-py311-rhel9
registry.redhat.io/rhoai/odh-training-cuda124-torch25-py311-rhel9
registry.redhat.io/rhoai/odh-training-cuda128-torch28-py312-rhel9
registry.redhat.io/rhoai/odh-training-cuda128-torch29-py312-rhel9
registry.redhat.io/rhoai/odh-training-rocm62-torch24-py311-rhel9
registry.redhat.io/rhoai/odh-training-rocm62-torch25-py311-rhel9
registry.redhat.io/rhoai/odh-training-rocm64-torch28-py312-rhel9
registry.redhat.io/rhoai/odh-training-rocm64-torch29-py312-rhel9

The list of ClusterTrainingRuntimes being deprecated in Red Hat OpenShift AI 3.4 is:

training-hub-th05-cuda128-torch29-py312
torch-distributed-cuda128-torch29-py312
torch-distributed-rocm64-torch29-py312
For the list of supported configuration see Red Hat OpenShift AI: Supported Configurations for 3.x .

5.1.3. Deprecated SQLite as a production metadata store for Llama Stack

Starting with OpenShift AI 3.2, SQLite is deprecated for use as a metadata store in production Llama Stack deployments. PostgreSQL is required for production-grade environments to ensure adequate performance, concurrency, and scalability. SQLite remains available for local development and testing only and must be explicitly configured. This includes configurations that define SQLite backends such as kv-sqlite or sql-sqlite in the Llama Stack storage configuration. SQLite is not intended for production workloads.

5.1.4. Deprecated annotation format for Connection Secrets::

Starting with OpenShift AI 3.0, the opendatahub.io/connection-type-ref annotation format for creating Connection Secrets is deprecated.

For all new Connection Secrets, use the opendatahub.io/connection-type-protocol annotation instead. While both formats are currently supported, connection-type-protocol takes precedence and should be used for future compatibility.

5.1.5. Deprecated Kubeflow Training operator v1

The Kubeflow Training Operator (v1) is deprecated starting OpenShift AI 2.25 and is planned to be removed in a future release. This deprecation is part of our transition to Kubeflow Trainer v2, which delivers enhanced capabilities and improved functionality.

5.1.6. Deprecated TrustyAI service CRD v1alpha1

Starting with OpenShift AI 2.25, the v1apha1 version is deprecated and planned for removal in an upcoming release. You must update the TrustyAI Operator to version v1 to receive future Operator updates.

5.1.7. Deprecated KServe Serverless deployment mode

Starting with OpenShift AI 2.25, The KServe Serverless deployment mode is deprecated. You can continue to deploy models by migrating to the KServe RawDeployment mode. If you are upgrading to Red Hat OpenShift AI 3.0, all workloads that use the retired Serverless or ModelMesh modes must be migrated before upgrading.

5.1.8. Deprecated model registry API v1alpha1

Starting with OpenShift AI 2.24, the model registry API version v1alpha1 is deprecated and will be removed in a future release of OpenShift AI. The latest model registry API version is v1beta1.

5.1.9. Multi-model serving platform (ModelMesh)

Starting with OpenShift AI version 2.19, the multi-model serving platform based on ModelMesh is deprecated. You can continue to deploy models on the multi-model serving platform, but it is recommended that you migrate to the single-model serving platform.

For more information or for help on using the single-model serving platform, contact your account manager.

5.1.10. Accelerator Profiles and legacy Container Size selector deprecated

Starting with OpenShift AI 3.0, Accelerator Profiles and the Container Size selector for workbenches are deprecated.

+ These features are replaced by the more flexible and unified Hardware Profiles capability.

5.1.11. Deprecated OpenVINO Model Server (OVMS) plugin

The CUDA plugin for the OpenVINO Model Server (OVMS) is now deprecated and will no longer be available in future releases of OpenShift AI.

5.1.12. OpenShift AI dashboard user management moved from `OdhDashboardConfig` to `Auth` resource

Previously, cluster administrators used the groupsConfig option in the OdhDashboardConfig resource to manage the OpenShift groups (both administrators and non-administrators) that can access the OpenShift AI dashboard. Starting with OpenShift AI 2.17, this functionality has moved to the Auth resource. If you have workflows (such as GitOps workflows) that interact with OdhDashboardConfig, you must update them to reference the Auth resource instead.

Table 5.1. Updated configurations

Resource	2.16 and earlier	2.17 and later versions
`apiVersion`	`opendatahub.io/v1alpha`	`services.platform.opendatahub.io/v1alpha1`
`kind`	`OdhDashboardConfig`	`Auth`
`name`	`odh-dashboard-config`	`auth`
Admin groups	`spec.groupsConfig.adminGroups`	`spec.adminGroups`
User groups	`spec.groupsConfig.allowedGroups`	`spec.allowedGroups`

5.1.13. Deprecated cluster configuration parameters

When using the CodeFlare SDK to run distributed workloads in Red Hat OpenShift AI, the following parameters in the Ray cluster configuration are now deprecated and should be replaced with the new parameters as indicated.

Deprecated parameter	Replaced by
`head_cpus`	`head_cpu_requests`, `head_cpu_limits`
`head_memory`	`head_memory_requests`, `head_memory_limits`
`min_cpus`	`worker_cpu_requests`
`max_cpus`	`worker_cpu_limits`
`min_memory`	`worker_memory_requests`
`max_memory`	`worker_memory_limits`
`head_gpus`	`head_extended_resource_requests`
`num_gpus`	`worker_extended_resource_requests`

You can also use the new extended_resource_mapping and overwrite_default_resource_mapping parameters, as appropriate. For more information about these new parameters, see the Content from github.com is not included.CodeFlare SDK documentation (external).

5.2. Removed functionality

tf2onnx package removed from TensorFlow images

The tf2onnx package has been removed from TensorFlow workbench and runtime images. This package, which converts TensorFlow models to ONNX format, was incompatible with Keras 3 (used in TensorFlow 2.16+) and had irreconcilable dependency conflicts with protobuf versions required by onnx, tensorflow, and feast. The upstream project has been abandoned since January 2024.

If you require TensorFlow to ONNX conversion, see This content is not included.RHAIENG-1632 for alternative approaches.

Caikit-NLP component removed

The caikit-nlp component has been formally deprecated and removed from OpenShift AI 3.0.

This runtime is no longer included or supported in OpenShift AI. Users should migrate any dependent workloads to supported model serving runtimes.

TGIS component removed

The TGIS component, which was deprecated in OpenShift AI 2.19, has been removed in OpenShift AI 3.0.

TGIS continued to be supported through the OpenShift AI 2.16 Extended Update Support (EUS) lifecycle, which ended in June 2025.

Starting with this release, TGIS is no longer available or supported. Users should migrate their model serving workloads to supported runtimes such as Caikit or Caikit-TGIS.

AppWrapper Controller removed

The AppWrapper controller has been removed from OpenShift AI as part of the broader CodeFlare Operator removal process.

This change eliminates redundant functionality and reduces maintenance overhead and architectural complexity.

5.2.1. CodeFlare Operator removed

Starting with OpenShift AI 3.0, the CodeFlare Operator has been removed.

+ The functionality previously provided by the CodeFlare Operator is now included in the KubeRay Operator, which provides equivalent capabilities such as mTLS, network isolation, and authentication.

LAB-tuning feature removed

Starting with OpenShift AI 3.0, the LAB-tuning feature has been removed.

Users who previously relied on LAB-tuning for large language model customization should migrate to alternative fine-tuning or model customization methods.

Embedded Kueue component removed

The embedded Kueue component, which was deprecated in OpenShift AI 2.24, has been removed in OpenShift AI 3.0.

OpenShift AI now uses the Red Hat Build of the Kueue Operator to provide enhanced workload scheduling across distributed training, workbench, and model serving workloads.

The embedded Kueue component is not supported in any Extended Update Support (EUS) release.

Removal of DataSciencePipelinesApplication v1alpha1 API version

The v1alpha1 API version of the DataSciencePipelinesApplication custom resource (datasciencepipelinesapplications.opendatahub.io/v1alpha1) has been removed.

OpenShift AI now uses the stable v1 API version (datasciencepipelinesapplications.opendatahub.io/v1).

You must update any existing manifests or automation to reference the v1 API version to ensure compatibility with OpenShift AI 3.0 and later.

5.2.2. Microsoft SQL Server command-line tool removal

Starting with OpenShift AI 2.24, the Microsoft SQL Server command-line tools (sqlcmd, bcp) have been removed from workbenches. You can no longer manage Microsoft SQL Server using the preinstalled command-line client.

5.2.3. Model registry ML Metadata (MLMD) server removal

Starting with OpenShift AI 2.23, the ML Metadata (MLMD) server has been removed from the model registry component. The model registry now interacts directly with the underlying database by using the existing model registry API and database schema. This change simplifies the overall architecture and ensures the long-term maintainability and efficiency of the model registry by transitioning from the ml-metadata component to direct database access within the model registry itself.

If you see the following error for your model registry deployment, this means that your database schema migration has failed:

error: error connecting to datastore: Dirty database version {version}. Fix and force version.

You can fix this issue by manually changing the database from a dirty state to 0 before traffic can be routed to the pod. Perform the following steps:

Find the name of your model registry database pod as follows:
kubectl get pods -n <your-namespace> | grep model-registry-db
Replace <your-namespace> with the namespace where your model registry is deployed.
Use kubectl exec to run the query on the model registry database pod as follows:
kubectl exec -n <your-namespace> <your-db-pod-name> -c mysql -- mysql -u root -p"$MYSQL_ROOT_PASSWORD" -e "USE <your-db-name>; UPDATE schema_migrations SET dirty = 0;"
Replace <your-namespace> with your model registry namespace and <your-db-pod-name> with the pod name that you found in the previous step. Replace <your-db-name> with your model registry database name.
This will reset the dirty state in the database, allowing the model registry to start correctly.

5.2.4. Embedded subscription channel not used in some versions

For OpenShift AI 2.8 to 2.20 and 2.22 to 3.4, the embedded subscription channel is not used. You cannot select the embedded channel for a new installation of the Operator for those versions. For more information about subscription channels, see Installing the Red Hat OpenShift AI Operator.

5.2.5. Anaconda removal

Anaconda is an open source distribution of the Python and R programming languages. Starting with OpenShift AI version 2.18, Anaconda is no longer included in OpenShift AI, and Anaconda resources are no longer supported or managed by OpenShift AI.

If you previously installed Anaconda from OpenShift AI, a cluster administrator must complete the following steps from the OpenShift command-line interface to remove the Anaconda-related artifacts:

Remove the secret that contains your Anaconda password:
oc delete secret -n redhat-ods-applications anaconda-ce-access
Remove the ConfigMap for the Anaconda validation cronjob:
oc delete configmap -n redhat-ods-applications anaconda-ce-validation-result
Remove the Anaconda image stream:
oc delete imagestream -n redhat-ods-applications s2i-minimal-notebook-anaconda
Remove the Anaconda job that validated the downloading of images:
oc delete job -n redhat-ods-applications anaconda-ce-periodic-validator-job-custom-run
Remove any pods related to Anaconda cronjob runs:
oc get pods n redhat-ods-applications --no-headers=true | awk '/anaconda-ce-periodic-validator-job-custom-run*/'

5.2.6. Pipeline logs for Python scripts running in Elyra pipelines are no longer stored in S3

Logs are no longer stored in S3-compatible storage for Python scripts which are running in Elyra pipelines. From OpenShift AI version 2.11, you can view these logs in the pipeline log viewer in the OpenShift AI dashboard.

Note

For this change to take effect, you must use the Elyra runtime images provided in workbench images at version 2024.1 or later.

If you have an older workbench image version, update the Version selection field to a compatible workbench image version, for example, 2024.1, as described in Updating a project workbench.

Updating your workbench image version will clear any existing runtime image selections for your pipeline. After you have updated your workbench version, open your workbench IDE and update the properties of your pipeline to select a runtime image.

5.2.7. Beta subscription channel no longer used

Starting with OpenShift AI 2.5, the beta subscription channel is no longer used. You can no longer select the beta channel for a new installation of the Operator. For more information about subscription channels, see Installing the Red Hat OpenShift AI Operator.

5.2.8. HabanaAI workbench image removal

Support for the HabanaAI 1.10 workbench image has been removed. New installations of OpenShift AI from version 2.14 do not include the HabanaAI workbench image. However, if you upgrade OpenShift AI from a previous version, the HabanaAI workbench image remains available, and existing HabanaAI workbench images continue to function.

Chapter 6. Resolved issues

The following notable issues are resolved in Red Hat OpenShift AI 3.4.1. Security updates, bug fixes, and enhancements for Red Hat OpenShift AI 3.4.1 are released as asynchronous errata. All OpenShift AI errata advisories are published on the This content is not included.Red Hat Customer Portal.

6.1. Security updates in Red Hat OpenShift AI 3.4.1 (June 2026)

This release provides security updates. For a complete list of updates, see the associated errata advisory on the This content is not included.Red Hat Customer Portal.

6.2. Issues resolved in Red Hat OpenShift AI 3.4 EA1

This content is not included.RHOAIENG-46420 - Inference failures with vLLM when using the temperature parameter on IBM Z

Before this update, on IBM Z platforms, vLLM inference failed when the temperature parameter was explicitly set in inference requests. Any request that included the temperature field caused the vLLM process to terminate, and the serving pod entered a CrashLoopBackOff state. This issue occurred only when the temperature parameter was present in the request.

This issue is resolved. You can use the temperature parameter in vLLM inference requests on IBM Z platforms.

6.3. Issues resolved in Red Hat OpenShift AI 3.3

This content is not included.RHOAIENG-24545 - Runtime images are not present in workbench after first start

Before this update, the list of runtime images was not properly populated for the first running workbench instance in the namespace. As a result, no image was shown for selection in the Elyra pipeline editor and required a workaround to populate the runtime image list.

The list of runtime images is now properly populated even for the first running workbench instance in the namespace without needing any extra workaround. Elyra now contains the required runtime image list in the editor as expected from the first workbench start. This issue has been resolved.

6.4. Issues resolved in Red Hat OpenShift AI 3.2

This content is not included.RHOAIENG-31071 - LM-Eval evaluations using Parquet datasets fail on IBM Z (s390x)

Before this update, Apache Arrow’s Parquet implementation contained endianness-specific code that was incompatible with big-endian IBM Z (s390x) architecture, causing byte-order mismatches when reading Parquet-formatted datasets. This resulted in LM-Eval evaluation tasks using datasets in Parquet format failing on s390x systems with parsing errors. A workaround applied compatibility patches to Apache Arrow and built a custom version specifically for s390x to support proper Parquet encoding/decoding.

This content is not included.RHOAIENG-38579 - Cannot stop models served with the Distributed Inference Server runtime

Before this update, you could not stop models served with the Distributed Inference Server with llm-d runtime from the OpenShift AI dashboard. This issue has been resolved.

This content is not included.RHOAIENG-38180 - Unable to send requests to Feature Store using the Feast SDK from workbench

Before this update, Feast was missing certificates and a service when running the default configuration, which prevented you from sending requests to your Feature Store by using the Feast SDK.

This issue has been resolved.

This content is not included.RHOAIENG-41588 - Standard openshift-container-platform route support added for dashboard access

Before this update, the transition to using Gateway API for Red Hat OpenShift AI version 3.0 required load balancer configuration. This configuration requirement caused usability issues and led to deployment delays for users of baremetal and cloud infrastructures. This issue has been resolved. The Gateway API now supports Cluster IP mode and standard openshift-container-platform route configuration in addition to the load balancer configuration option, simplifying dashboard access for the users.

For more information, see Configurable Ingress Mode for RHOAI 3.2 on Bare Metal, OpenStack and Private Clouds.

This content is not included.RHOAIENG-44616 - Inferencing with granite-3b model fails on IBM Power

Before this update, inference services for the granite-3b-code-instruct-2k model were created successfully. However, when a chat completion request was sent, it failed with an Internal server error. This issue is now resolved.

This content is not included.RHOAIENG-37686 - Metrics not displayed on the Dashboard due to image name mismatch in runtime detection logic

Previously, metrics were not displayed on the OpenShift AI dashboard because digest-based image names were not correctly recognized by the runtime detection system. This issue affected all InferenceService deployments in OpenShift AI 2.25 and later. This issue has been resolved.

This content is not included.RHOAIENG-37492 - Dashboard console link not accessible on IBM Power in 3.0.0

Previously, on private cloud deployments running on IBM Power, the OpenShift AI dashboard link was not visible in the OpenShift console when the dashboard was enabled in the DataScienceCluster configuration. As a result, users could not access the dashboard through the console without manually creating a route. This issue has been resolved.

This content is not included.RHOAIENG-1152 - Basic workbench creation process fails for users who have never logged in to the dashboard

This issue is now obsolete as of OpenShift AI 3.0. The basic workbench creation process has been updated, and this behavior no longer occurs.

This content is not included.RHOAIENG-9418 - Elyra raises error when you use parameters in uppercase

Previously, Elyra raised an error when you tried to run a pipeline that used parameters in uppercase. This issue is now resolved.

This content is not included.RHOAIENG-30493 - Error creating a workbench in a Kueue-enabled project

Previously, when using the dashboard to create a workbench in a Kueue-enabled project, the creation failed if Kueue was disabled on the cluster or if the selected hardware profile was not associated with a LocalQueue. In this case, the required LocalQueue could not be referenced, the admission webhook validation failed, and an error message was shown. This issue has been resolved.

This content is not included.RHOAIENG-32942 - Elyra requires unsupported filters on the REST API when pipeline store is Kubernetes

Before this update, when the pipeline store was configured to use Kubernetes, Elyra required equality (eq) filters that were not supported by the REST API. Only substring filters were supported in this mode. As a result, pipelines created and submitted through Elyra from a workbench could not run successfully. This issue has been resolved.

This content is not included.RHOAIENG-32897 - Pipelines defined with the Kubernetes API and invalid platformSpec do not appear in the UI or run

Before this update, when a pipeline version defined with the Kubernetes API included an empty or invalid spec.platformSpec field (for example, {} or missing the kubernetes key), the system misidentified the field as the pipeline specification. As a result, the REST API omitted the pipelineSpec, which prevented the pipeline version from being displayed in the UI and from running. This issue is now resolved.

This content is not included.RHOAIENG-31386 - Error deploying an Inference Service with authenticationRef

Before this update, when deploying an InferenceService with authenticationRef under external metrics, the authenticationRef field was removed. This issue is now resolved.

This content is not included.RHOAIENG-33914 - LM-Eval Tier2 task test failures

Previously, there could be failures with LM-Eval Tier2 task tests because the Massive Multitask Language Understanding Symbol Replacement (MMLUSR) tasks were broken. This issue is resolved witih the latest version of the trustyai-service-operator.

This content is not included.RHOAIENG-35532 - Unable to deploy models with HardwareProfiles and GPU

Before this update, the HardwareProfile to use GPU for model deployment had stopped working. The issue is now resolved.

This content is not included.RHOAIENG-4570 - Existing Argo Workflows installation conflicts with install or upgrade

Previously, installing or upgrading OpenShift AI on a cluster that already included an existing Argo Workflows instance could cause conflicts with the embedded Argo components deployed by Data Science Pipelines. This issue has been resolved. You can now configure OpenShift AI to use an existing Argo Workflows instance, enabling clusters that already run Argo Workflows to integrate with Data Science Pipelines without conflicts.

This content is not included.RHOAIENG-35623 - Model deployment fails when using hardware profiles

Previously, model deployments that used hardware profiles failed because the Red Hat OpenShift AI Operator did not inject the tolerations, nodeSelector, or identifiers from the hardware profile into the underlying InferenceService when manually creating InferenceService resources. As a result, the model deployment pods could not be scheduled to suitable nodes and the deployment fails to enter a ready state. This issue is now resolved.

Chapter 7. Known issues

This section describes known issues in Red Hat OpenShift AI 3.4.1, and any known methods of working around these issues.

7.1. Issues discovered at version 3.4 GA

Content from redhat.atlassian.net is not included.RHOAIENG-37916 - Deployed llm-d model shows failed in UI

Models deployed using the "{llm-d}" will initially show a Status of Failed in the OpenShift AI Dashboard. To get more information on actual status, use the OpenShift Console to follow the status of pods in the project. When the model is fully ready, the OpenShift AI Dashboard will display a status of Started". Workaround:: Wait until the status changes, or consult the pod statuses.

Content from redhat.atlassian.net is not included.RHOAIENG-60940 - BUG: NemoGuardrails RBAC ClusterRoles missing Kubernetes aggregation labels cause 403 for non-cluster-admin users

When testing NemoGuardrails with a non-cluster-admin user, a 403 Forbidden error is returned: bq. User "X" cannot get resource "nemoguardrails" in API group trustyai.opendatahub.io in namespace "Y".

The nemoguardrail-viewer-role and nemoguardrail-editor-role ClusterRoles on the operator side are missing the standard Kubernetes RBAC aggregation labels (aggregate-to-view, aggregate-to-edit, aggregate-to-admin). Without these labels, regular namespace users do not inherit the NemoGuardrails permissions through the default aggregated ClusterRoles, resulting in access denied errors.

Workaround

Cluster admins can create ClusterRoleBindings that apply the nemoguardrail-editor-role or nemoguardrail-viewer-role to users that require edit or view permissions to NeMo Guardrail resources:

$ oc create clusterrolebinding <binding-name> \ --clusterrole=nemoguardrail-editor-role \ --user=<username>

$ oc create clusterrolebinding <binding-name> \ --clusterrole=nemoguardrail-viewer-role \ --user=<username>

Content from redhat.atlassian.net is not included.RHOAIENG-60855 - Upgrade error: Llama Stack Operator produces invalid Deployment when storage is configured

When upgrading OpenShift AI from 3.3 to 3.4, the Llama Stack Operator can fail to reconcile an existing LlamaStackDistribution custom resource that includes a storage specification, for example storage.size: 2Gi. Due to an upgrade-strategy change, the operator may generate an invalid Deployment that specifies both spec.strategy.type: Recreate and spec.strategy.rollingUpdate, which Kubernetes rejects with an error similar to: Deployment.apps "llama-stack-distribution-upgrade" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy 'type' is 'Recreate'

Workaround

Delete the affected Deployment so that the operator recreates it with a valid strategy:

oc delete deployment <cr-name> -n <namespace>

Replace <cr-name> with the name of the LlamaStackDistribution custom resource and <namespace> with its namespace. Llama stack operator will recreate deployment and new pod will work as expected.

Content from redhat.atlassian.net is not included.RHOAIENG-60292 - MLflow does not automatically run database migrations after upgrade

There are database migration changes that are automatically applied after upgrade.

Workaround: It is recommended to bring down the MLflow replicas to 1 on the MLflow CR, the spec.replicas parameter, during upgrade.

Content from redhat.atlassian.net is not included.RHOAIENG-58969 - precise-prefix-cache-scorer returns score of zero due to PodIdentifier key format mismatch

In OpenShift AI 3.4, the precise prefix cache scorer was updated to identify routing endpoints using a combination of the IP address and port. Previous versions relied solely on the IP address. If the vLLM configuration is not updated to provide the port, the scorer cannot match cache entries to the correct endpoints. As a result, traffic routing ignores prefix cache locality entirely and falls back to standard load balancing, which can reduce performance efficiency.

Workaround: Update the vLLM arguments in your configuration to include the port number in the KV events topic. Modify the topic format from the older format (kv@${POD_IP}@<model>) to include the port. For example, kv@${POD_IP}:8000@<model>. The precise scorer will correctly identify cache locations, successfully restoring prefix cache locality and routing requests to the pods with the most relevant cached data.

Content from redhat.atlassian.net is not included.RHOAIENG-59950 - Search space preparation fails when too many models are provided

User starts AutoRAG experiment with more than 3 embedding models or more than 2 generation models. Consequence: search_space_preparation component runs models pre-selection and produces incorrect search_space_prep_report that cannot be properly parsed in the next component: rag_templates_optimization. User sees:

AutoRAG fails with: SearchSpaceValueError: Missing required parameters in the search space: {'embedding_model', 'foundation_model'}

Workaround: User may start the experiment with up to 2 foundation models and up to 3 embedding models.

Content from redhat.atlassian.net is not included.RHOAIENG-59340 - Installing the kfp_components module fails in disconnected environments

In disconnected environments, installing the kfp_components Python module in a workbench notebook fails because pip install attempts to clone the Content from github.com is not included.pipelines-components repository from GitHub, which is not reachable. As a consequence, you cannot use kfp_components to build or run AutoML and AutoRAG pipelines programmatically.

Workaround

Mirror the Content from github.com is not included.pipelines-components repository to a Git host accessible from your disconnected cluster.
Add the following environment variables to your workbench:

PIP_INDEX_URL

Specifies the PyPI mirror URL on your bastion node, for example \https://<bastion-hostname>:9443/root/pypi/+simple/.

PIP_TRUSTED_HOST

Specifies the hostname of your bastion node, for example <bastion-hostname>.

GIT_SSL_NO_VERIFY

Set to true to allow Git operations over self-signed TLS certificates.

Install kfp_components from the mirrored repository:

$ pip install -U --no-cache-dir git+https://<bastion-hostname>/pipelines-components.git@rhoai-3.4

Content from redhat.atlassian.net is not included.RHOAIENG-64768 - AutoML and AutoRAG pipeline runs fail with image pull errors

The default pipeline definitions shipped with OpenShift AI reference container image digests that are not available in the production registry. As a consequence, AutoML and AutoRAG experiment runs remain in progress indefinitely, and pipeline task pods log ImagePullBackOff or ErrImagePull errors with messages such as manifest unknown.

Workaround

Download the updated pipeline definition for your experiment type from the rhoai-3.4-fixed branch of the Content from github.com is not included.red-hat-data-services/pipelines-components repository on GitHub:

Content from github.com is not included.AutoML Tabular
Content from github.com is not included.AutoML Time Series
Content from github.com is not included.AutoRAG
If you have already imported the pipeline, upload the updated file as a new version and re-run the experiment as a new run. For more information, see Uploading a pipeline version.
If you have not yet imported the pipeline, import the updated file. For more information, see Importing a pipeline.
After you upload the updated pipeline definition, experiment runs pull the correct images and complete successfully.

7.2. Issues discovered at version 3.4 EA2

This content is not included.RHOAIENG-60236 - Workbench images and external accelerator runtimes are not part of Designed for FIPS

Red Hat OpenShift AI workbench images and external accelerator runtimes are not part of Designed for FIPS. While the core platform components support FIPS-enabled clusters, workbench images and external accelerator runtimes have not been designed or tested for FIPS compliance.

Workaround: None.

This content is not included.RHOAIENG-58765 - Distributed Inference with llm-d prefill and decode disaggregation fails on FIPS-enabled clusters

Using Distributed Inference with llm-d prefill and decode disaggregation for LLM deployments on FIPS-enabled clusters causes the routing sidecar pod to enter a crash loop, preventing the LLM deployment from functioning. This issue is caused by a runtime image introduced in the 3.4 EA2 release that is not FIPS-compatible.

Workaround: Do not use prefill and decode disaggregation with Distributed Inference with llm-d in Red Hat OpenShift AI 3.4 EA2 on FIPS-enabled clusters. Other features continue to work correctly on FIPS-enabled clusters.

Content from redhat.atlassian.net is not included.RHOAIENG-3816 - Encrypted PDF uploads to Llama Stack vector stores fail on FIPS-enabled clusters

On FIPS-enabled clusters, registering certain encrypted PDF files into Llama Stack vector stores fails due to a limitation in the underlying PDF parsing library. The library uses an MD5-based digest as part of its encryption handling, which is not allowed in FIPS mode and triggers an UnsupportedDigestmodError: [digital envelope routines] unsupported error during file processing. Due to this, on FIPS-enabled clusters, affected encrypted PDFs cannot be uploaded into Llama Stack vector stores. As a result, these documents are not indexed and are unavailable to retrieval-augmented generation (RAG) workflows, which may lead to incomplete or missing answers when those documents are expected to be part of the context.

Workaround: Use non‑encrypted PDF files when ingesting documents into Llama Stack vector stores on FIPS-enabled clusters, or re-export or convert the original document to a non‑encrypted PDF before uploading it to the vector store. Until the underlying PDF library is updated to use FIPS-compliant cryptographic primitives, encrypted PDFs that rely on disallowed digests in FIPS mode will continue to fail to upload to Llama Stack vector stores on FIPS-enabled clusters.

Content from redhat.atlassian.net is not included.RHOAIENG-57224 - ROCm universal image training produces NaN on MI300X due to torch aotriton 0.11.1 regression

ROCm universal training image (th06) produces NaN values on MI300X due to aotriton 0.11.1 regression in AIPCC-built PyTorch wheel.

Workaround: Use th05 image or set attn_implementation="flash_attention_2".

Content from redhat.atlassian.net is not included.RHOIAIENG-57427 - RAG in Gen AI Playground doesn’t work with default system prompt and model Qwen/Qwen3-14B-AWQ

In Gen AI Playground RAG, the default system prompt might not reliably trigger the knowledge search/tool-calling behavior for some models, so document retrieval is not performed. Due to this, questions about uploaded documents can return answers without using the vector store, resulting in incomplete/incorrect responses unless the prompt is adjusted.

Workaround: Manually edit the system prompt to explicitly instruct the model to use the knowledge search tool first for document-based/factual questions (as documented in the Gen AI Playground RAG documentation). As a result, after updating the system prompt, RAG retrieval works and the model can answer based on the uploaded document content.

Content from redhat.atlassian.net is not included.RHOAIENG-54005 - Generate MaaS Token Endpoint Removed - breaks Gen AI Studio Playground

The /v1/token API was removed and this endpoint was merged in with the new post creation of /v1/api-keys. As a result, Gen AI Playground cannot generate a token on the fly for MaaS and cannot talk to MaaS Models in 3.4 EA2.

Workaround: There is no existing workaround for this known issue. As a result, there is no access to MaaS and Playground in 3.4 EA2.

Content from redhat.atlassian.net is not included.RHOAIENG-48753 - Pipeline Name must be DNS-compliant to use “Store pipeline definitions in Kubernetes”

Elyra does not convert the pipeline name to a DNS-compliant name when using the default Kubernetes storage. As a consequence, if you don’t use a DNS-compliant name when you start an Elyra pipeline, it gives a cryptic error "[TIP: did you mean to set Content from ds-pipeline-dspa-robert-tests.apps.test.rhoai.rh-aiservices-bu.com is not included.https://ds-pipeline-dspa-robert-tests.apps.test.rhoai.rh-aiservices-bu.com/pipeline as the endpoint, take care not to include s at end]".

Workaround: Use DNS-compliant naming when running Elyra pipelines.

7.3. Issues discovered at version 3.4 EA1

This content is not included.RHOAIENG-54101 - Deployments not listed in Model Registry on IBM Z

When you deploy a model from the Model Registry on IBM Z, the deployment does not appear under the Deployments tab in the Model Registry.

Workaround: Access and manage the deployment from the global Deployments page in the OpenShift AI dashboard.

This content is not included.RHOAIENG-53206 - Spark driver pods fail to communicate due to RpcTimeoutException

After installing the Spark Operator, Spark executor pods cannot communicate with the driver pod because the redhat-ods-applications namespace defaults to a "deny-all" traffic rule. SparkApplication pods hang and fail with an RpcTimeoutException.

Workaround

Create a NetworkPolicy in the redhat-ods-applications namespace to allow communication between the pods created by the SparkApplication controller:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: spark-operator-allow-internal
spec:
  podSelector:
    matchLabels:
      sparkoperator.k8s.io/launched-by-spark-operator: "true"
  policyTypes:
    - Ingress
  ingress:
    - ports:
        - port: 7078
          protocol: TCP
        - port: 7079
          protocol: TCP
        - port: 4040
          protocol: TCP
      from:
        - podSelector: {}
        - namespaceSelector:
            matchLabels:
              network.openshift.io/policy-group: ingress

This content is not included.RHOAIENG-52130 - Workbenches with Feast integration fail to start due to missing ConfigMap

Workbenches with Feast integration enabled fail to start in OpenShift AI 3.4 EA1. Pods remain stuck in ContainerCreating state with the following error:

[FailedMount] [Warning] MountVolume.SetUp failed for volume "odh-feast-config"
  configmap "jupyter-nb-kube-3aadmin-feast-config" not found

Workaround

Restart the Feast Operator after DSC deployment completes:

$ kubectl rollout restart deployment/feast-operator-controller-manager -n redhat-ods-applications

This content is not included.RHOAIENG-53239 - Custom ServingRuntime required for IBM Z (s390x) vLLM Spyre deployments

When deploying models using the vLLM Spyre runtime on IBM Z (s390x) systems, the default ServingRuntime cannot be used directly for KServe-based deployments. Model deployment fails if the runtime is used without modification.

Workaround

Create a custom ServingRuntime by duplicating the vllm-spyre-s390x-runtime ServingRuntime and removing the command section from the container specification. Keep all other configuration, including environment variables, ports, and volume mounts, unchanged.

The following example shows only the affected section. Your complete ServingRuntime must include all other fields from the original template:

apiVersion: serving.kserve.io/v1alpha1
kind: ServingRuntime
metadata:
  name: vllm-spyre-s390x-runtime-copy
spec:
  containers:
    - name: kserve-container
      image: <image>
      # Remove the 'command' section that appears here in the original
      args:
        - --model=/mnt/models
        - --port=8000
        - --served-model-name={{.Name}}
      # ... keep all env, ports, volumeMounts from original ...

This content is not included.RHOAIENG-50523 - Unable to upload RAG documents in Gen AI Playground on disconnected clusters

On disconnected clusters, uploading documents in the Gen AI Playground RAG section fails. The progress bar never exceeds 50% because Llama Stack attempts to download the ibm-granite/granite-embedding-125m-english embedding model from HuggingFace, even though the model is already included in the Llama Stack Distribution image in OpenShift AI 3.3.

Workaround

Modify the LlamaStackDistribution custom resource to include the following environment variables:

export MY_PROJECT=my-project

oc patch llamastackdistribution lsd-genai-playground \
  -n $MY_PROJECT \
  --type='json' \
  -p='[
    {
      "op": "add",
      "path": "/spec/server/containerSpec/env/-",
      "value": {
        "name": "SENTENCE_TRANSFORMERS_HOME",
        "value": "/opt/app-root/src/.cache/huggingface/hub"
      }
    },
    {
      "op": "add",
      "path": "/spec/server/containerSpec/env/-",
      "value": {
        "name": "HF_HUB_OFFLINE",
        "value": "1"
      }
    },
    {
      "op": "add",
      "path": "/spec/server/containerSpec/env/-",
      "value": {
        "name": "TRANSFORMERS_OFFLINE",
        "value": "1"
      }
    },
    {
      "op": "add",
      "path": "/spec/server/containerSpec/env/-",
      "value": {
        "name": "HF_DATASETS_OFFLINE",
        "value": "1"
      }
    }
  ]'

The Llama Stack pod restarts automatically after applying this configuration.

This content is not included.RHAIENG-2827 - Unsecured routes created by older CodeFlare SDK versions

Existing 2.x workbenches continue to use an older version of the CodeFlare SDK when used in OpenShift AI 3.x. The older version of the SDK creates unsecured OpenShift routes on behalf of the user.

Workaround: To resolve this issue, update your workbench to the latest image provided in OpenShift AI 3.x before using CodeFlare SDK.

This content is not included.RHOAIENG-48867 - TrainJob fails to resume after Red Hat OpenShift AI upgrade due to immutable JobSet spec

TrainJobs that are suspended (e.g., queued by Kueue) before a Red Hat OpenShift AI upgrade cannot resume after the upgrade completes. The Trainer controller fails to update the immutable JobSet spec.replicatedJobs field.

Workaround: To resolve this issue, delete and recreate the affected TrainJob after the upgrade.

This content is not included.RHOAIENG-45142 - Dashboard URLs return 404 errors after upgrading Red Hat OpenShift AI from 2.x to 3.x

The Red Hat OpenShift AI dashboard URL subdomain changed from rhods-dashboard-redhat-ods-applications.apps.<cluster>`to `data-science-gateway.apps.<cluster> due to the use of Gateways in OpenShift AI version 3.x. Existing bookmarks to the dashboard using the default rhods-dashboard-redhat-ods-applications.apps.<cluster> format will no longer function after you upgrade to OpenShift AI version 3.0 or later. It is recommended that you update your bookmarks and any internal documentation to use the new URL format: data-science-gateway.apps.<cluster>.

Workaround: To resolve this issue, deploy an nginx-based redirect solution that recreates the old route name and redirects traffic to the new gateway URL. For instructions, see Dashboard URLs return 404 errors after RHOAI upgrade from 2.x to 3.x

Note

Cluster administrators must provide the new dashboard URL to all Red Hat OpenShift AI administrators and users. In a future release, URL redirects may be supported.

This content is not included.RHOAIENG-43686 - Red Hat build of Kueue 1.2 installation or upgrade fails with Kueue CRD reconciliation error

Installing Red Hat build of Kueue 1.2 or upgrading from Red Hat build of Kueue 1.1 to 1.2 fails if legacy Kueue CustomResourceDefinitions (CRDs) remain in the cluster from a previous Red Hat OpenShift AI 2.x installation. As a result, when the legacy v1alpha1 CRDs are present, the Kueue operator cannot reconcile successfully and the Data Science Cluster (DSC) remains in a Not Ready state.

Workaround: To resolve this issue, delete the legacy Kueue CRDs, cohorts.kueue.x-k8s.io/v1alpha1 or topologies.kueue.x-k8s.io/v1alpha1 from the cluster. For detailed instructions, see Red Hat Build of Kueue 1.2 installation or upgrade fails with Kueue CRD reconciliation error.

This content is not included.RHOAIENG-49389 - Tier management unavailable after deleting all tiers

If you delete all service tiers from Settings > Tiers, the Create tier button is no longer displayed. You cannot create tiers through the dashboard until at least one tier exists. To avoid this issue, ensure at least one tier remains in the system at all times.

Workaround

Create a basic tier using the CLI, then configure its settings through the dashboard. You must have cluster administrator privileges for your OpenShift cluster to perform these steps:

Retrieve the tier-to-group-mapping ConfigMap:

$ oc get configmap tier-to-group-mapping redhat-ods-namespace -o yaml tier-config.yaml

Edit the ConfigMap to add a basic tier definition:

apiVersion: v1
  kind: ConfigMap
  metadata:
    name: tier-to-group-mapping
    namespace: redhat-ods-applications
  data:
    tiers.yaml: |
      - name: basic
        displayName: Basic Tier
        level: 0
        groups:
          - system:authenticated

Apply the updated ConfigMap:
```
$ oc apply -f tier-config.yaml
```
In the dashboard, navigate to Settings → Tiers to configure rate limits for the newly created tier.

This content is not included.RHOAIENG-47589 - Missing Kueue validation for TrainJob

A TrainJob creation without a defined Kueue LocalQueue passes without validation check, even when Kueue managed namespace is enabled. As a result, it is possible to create TrainJob not managed by Kueue in Kueue managed namespace.

Workaround: None.

This content is not included.RHOAIENG-49017 - Upgrade RAGAS provider to Llama Stack 0.4.z / 0.5.z

In order to use the Ragas provider in OpenShift AI 3.3, you must update your Llama Stack distribution to use llama-stack-provider-ragas==0.5.4, which works with Llama Stack >=0.4.2,<0.5.0. This version of the provider is a workaround release that is using the deprecated register endpoints as a workaround. See the full Content from github.com is not included.compatibility matrix for more information.

Workaround: None.

This content is not included.RHOAIENG-44516 - MLflow tracking server does not accept Kubernetes service account tokens

Red Hat OpenShift AI does not accept Kubernetes service accounts when you authenticate through the dashboard MLflow URL.

Workaround

Use the automatic MLflow workbench integration, which configures service account token authentication through the MLFLOW_TRACKING_AUTH environment variable. Annotate your workbench notebook with opendatahub.io/mlflow-instance and restart the workbench. For more information, see Enable MLflow integration for a workbench.

If you are not using a workbench, create an OpenShift Route directly to the MLflow service endpoints and use the Route URL as the MLFLOW_TRACKING_URI when you authenticate.

Create an OpenShift Route directly to the MLflow service endpoints.
Use the Route URL as the MLFLOW_TRACKING_URI when you authenticate.

This content is not included.RHOAIENG-36757 - "Existing cluster storage" option is missing during model deployment when no connections exist

When you create a model deployment in a project with no data connections defined, the Existing cluster storage option is not displayed, even if Persistent Volume Claims (PVCs) suitable for model storage exist in the project. This prevents you from selecting an existing PVC for the deployment.

Workaround: Create at least one connection of type URI in the project to display the Existing cluster storage option.

This content is not included.RHOAIENG-37743 - Progress tab does not display steps when starting a workbench

When you start a workbench, the Progress tab on the Workbench Status screen does not display the status of individual steps. Instead, a warning is issued that some steps might be repeated or occur in any order.

Workaround: To view the detailed status, check the Event Log tab, or connect to the OpenShift web console and view the pod details.

This content is not included.AIPCC-8019 - Jemalloc consumes more memory than glibc when deploying models on IBM Z systems

When a model is deployed using jemalloc as the memory allocator, memory consumption is significantly higher compared to glibc. A comparison shows more than a fifty percent memory increase when jemalloc is used.

Workaround: In RHAIIS 3.2.5, unassign the LD_PRELOAD environment variable to disable jemalloc and use glibc as the memory allocator.

This content is not included.AIPCC-8043 - RHAIIS fails to start on IBM Z systems with FIPS enabled when using the Spyre platform plugin

RHAIIS fails to start on IBM Z systems with FIPS enabled when using the Spyre platform plugin, preventing model deployment. Logs show that a _hashlib.UnsupportedDigestmodError occurs during the model startup. RHAIIS 3.2.5 Spyre on IBM Z uses vLLM 0.11.0.

Workaround: The issue is fixed in vLLM 0.11.1 and is planned for future versions of RHAIIS.

This content is not included.RHOAIENG-44557 - Offline Feast container fails to start when applying the default FeatureStore YAML

When applying the default FeatureStore YAML for Feast, the offline-store container in the resulting pod fails to start. While the feature-server, registry-server, and ui containers start successfully, the offline-store container crashes because the current image lacks PyArrow built with flight support, a mandatory requirement for that specific container.

Workaround: None.

This content is not included.RHOAIENG-44610-MaaS component resource is visible in OpenShift Operator UI

The ModelsAsService resource displays incorrectly in the Operator resource list. This resource is an internal one and should not display there.

Workaround: None.

This content is not included.RHOAIENG-44931-Downstream MLflow image now has Postgres support

The developer preview MLflow container image does not have the psycopg2 Python package preinstalled for connecting to Postgres.

Workaround: Use SQLite.

This content is not included.RHOAIENG-44516 - Kubernetes tokens not supported by the Red Hat OpenShift AI Gateway

Red Hat OpenShift AI does not accept Kubernetes service accounts when you authenticate through the dashboard MLflow URL. Workaround:: To authenticate with a service account token, complete the following steps:

Create an OpenShift Route directly to the MLflow service endpoints.
Use the Route URL as the MLFLOW_TRACKING_URI when you authenticate.

This content is not included.RHOAIENG-46420 - Inference failures with vLLM when using the temperature parameter on IBM Z

On IBM Z platforms, there are inference failures with vLLM when the temperature parameter is explicitly set in inference requests. Any request that includes the temperature field results in an inference failure. When this occurs, the vLLM process terminates, causing the serving pod to enter a CrashLoopBackOff state. The pod restarts automatically, but if the temperature parameter remains, subsequent inference requests continue to fail. This issue does not occur when the temperature parameter is omitted from the request.

Workaround: None.

This content is not included.RHOAIENG-46944 - OpenShift route update fails when GatewayConfig subdomain changes

When updating the GatewayConfig subdomain field, the Gateway controller fails to update the corresponding OpenShift Route due to missing update verb in the Role-Based Access Control (RBAC) permissions. As a result, users cannot change the Gateway subdomain after initial deployment and see the following "permission denied" error:

Route.route.openshift.io "data-science-gateway" is invalid: spec.host: Invalid value: "test.apps.rosa.p2r4t4z7a2o1m8i.c8l6.p3.openshiftapps.com": you do not have permission to set the host field of the route

Workaround: Use the default subdomain provided during initial deployment. Avoid adding or changing the GatewayConfig subdomain field as post-deployment modifications have limited support.

This content is not included.RHOAIENG-37228 - Manual DNS configuration required on OpenStack and private cloud environments

When deploying OpenShift AI 3.4 on OpenStack, CodeReady Containers (CRC), or other private cloud environments without integrated external DNS, external access to components such as the dashboard and workbenches might fail after installation. This occurs because the dynamically provisioned LoadBalancer Service does not automatically register its IP address in external DNS.

Workaround: To restore access, manually create the required A or CNAME records in your external DNS system. For instructions, see the Configuring External DNS for RHOAI 3.x on OpenStack and Private Clouds Knowledgebase article.

This content is not included.RHOAIENG-38658 - TrustyAI service issues during model inference with token authentication on IBM Z (s390x)

On IBM Z (s390x) architecture, the TrustyAI service encounters errors during model inference when token authentication is enabled. A JsonParseException displays while logging to the TrustyAI service logger, causing the bias monitoring process to fail or behave unexpectedly.

Workaround: Run the TrustyAI service without authentication. The issue occurs only when token authentication is enabled.

This content is not included.RHOAIENG-38333 - Code generated by the Generative AI Playground is invalid and required packages are missing from workbenches

Some code automatically generated by the Generative AI Playground might cause syntax errors when run in OpenShift AI workbenches. Additionally, the LlamaStackClient package is not currently included in standard workbench images.

Workaround

Install the missing package manually within your workbench environment before running the generated code:

$ pip install llamastack-client

This content is not included.RHOAIENG-38263 - Intermittent failures with Guardrails Detector model on Hugging Face runtime for IBM Z

On IBM Z platforms, the Guardrails Detector model running on the Hugging Face runtime might intermittently fail to process identical requests. In some cases, a request that previously returned valid results fails with a parse error similar to the following example:

Invalid numeric literal at line 1, column 20

This error can cause the serving pod to temporarily enter a CrashLoopBackOff state, although it typically recovers automatically.

Workaround: None. The pod restarts automatically and resumes normal operation.

This content is not included.RHOAIENG-38253 - Distributed Inference Server with llm-d not listed on the Serving Runtimes page

While Distributed Inference Server with llm-d appears as an available option when deploying a model, it is not listed on the Serving Runtimes page under the Settings section.

This occurs because Distributed Inference Server with llm-d is a composite deployment type that includes additional components beyond a standard serving runtime. It therefore does not appear in the list of serving runtimes visible to administrators and cannot currently be hidden from end users.

Workaround: None. The Distributed Inference Server with llm-d option can still be used for model deployments, but it cannot be managed or viewed from the Serving Runtimes page.

This content is not included.RHOAIENG-38252 - Model Registry Operator does not work with BYOIDC mode on OpenShift 4.20

On OpenShift 4.20 clusters configured with Bring Your Own Identity Provider (BYOIDC) mode, deploying the Model Registry Operator fails.

When you create a ModelRegistry custom resource, it does not reach the available: True state. Instead, the resource shows a status similar to the following example:

status:
  conditions:
  - lastTransitionTime: "2025-11-06T22:09:04Z"
    message: 'unexpected reconcile error: failed to get API group resources: unable to retrieve the complete list of server APIs: user.openshift.io/v1: the server could not find the requested resource'
    reason: DeploymentUnavailable
    status: "False"
    type: Available

Workaround: None.

You cannot create or deploy a Model Registry instance when using BYOIDC mode on OpenShift 4.20.

This content is not included.RHOAIENG-37882 - Custom workbench (AnythingLLM) fails to load

Deploying a custom workbench such as AnythingLLM 1.8.5 might fail to finish loading. Starting with OpenShift AI 3.0, all workbenches must be compatible with the Kubernetes Gateway API’s path-based routing. Custom workbench images that do not support this requirement fail to load correctly.

Workaround: Update your custom workbench image to support path-based routing by serving all content from the ${NB_PREFIX} path (for example, /notebook/<namespace>/<workbench-name>). Requests to paths outside this prefix (such as /index.html or /api/data) are not routed to the workbench container.

To fix existing workbenches:

Update your application to handle requests at ${NB_PREFIX}/... paths.
Configure the base path in your framework, for example: FastAPI(root_path=os.getenv('NB_PREFIX', ''))
Update nginx to preserve the prefix in redirects.
Implement health endpoints returning HTTP 200 at: ${NB_PREFIX}/api, ${NB_PREFIX}/api/kernels, and ${NB_PREFIX}/api/terminals.
Use relative URLs and remove any hardcoded absolute paths such as /menu.

For more information, see the migration guide: Content from github.com is not included.Gateway API migration guide.

This content is not included.RHOAIENG-37855 - Model deployment from Model Catalog fails due to name length limit

When deploying certain models from the Model Catalog, the deployment might fail silently and remain in the Starting state. This issue occurs because KServe cannot create a deployment from the InferenceService when the resulting object name exceeds the 63-character limit.

Example: Attempting to deploy the model RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic results in KServe trying to create a deployment named isvc.redhataimistral-small-31-24b-instruct-2503-fp8-dynamic-predictor, which has 69 characters and exceeds the maximum allowed length.
Workaround: Use shorter model names or rename the InferenceService to ensure the generated object name stays within the 63-character limit.

This content is not included.RHOAIENG-37842 - Ray workloads requiring ray.init() cannot be triggered outsid OpenShift AI

Ray workloads that require ray.init() cannot be triggered outside the OpenShift AI environment. These workloads must be submitted from within a workbench or pipeline running on OpenShift AI in OpenShift. Running these workloads externally is not supported and results in initialization failures.

Workaround: Run Ray workloads that call ray.init() only within an OpenShift AI workbench or pipeline context.

This content is not included.RHOAIENG-37743 - No progress bar displayed when starting workbenches

When starting a workbench, the Progress tab in the Workbench Status screen does not display step-by-step progress. Instead, it shows a generic message stating that “Steps may repeat or occur in a different order.”

Workaround: To view detailed progress information, open the Event Log tab or use the OpenShift console to view the pod details associated with the workbench.

This content is not included.RHOAIENG-37667 - Model-as-a-Service (MaaS) available only for Distributed Inference with llm-d runtime

Model-as-a-Service (MaaS) is currently supported only for models deployed with the Distributed Inference Server with llm-d runtime. Models deployed with the vLLM runtime cannot be served by MaaS at this time.

Workaround: None. Use the Distributed Inference with llm-d runtime for deployments that require Model-as-a-Service functionality.

This content is not included.RHOAIENG-37561 - Dashboard console link fails to access OpenShift AI on IBM Z clusters in 3.0.0

When attempting to access the OpenShift AI 3.0.0 dashboard using the console link on an IBM Z cluster, the connection fails.

Workaround: Create a route to the Gateway link by applying the following YAML file:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: data-science-gateway-data-science-gateway-class
  namespace: openshift-ingress
spec:
  host: data-science-gateway.apps.<baseurl>
  port:
    targetPort: https
  tls:
    termination: passthrough
  to:
    kind: Service
    name: data-science-gateway-data-science-gateway-class
    weight: 100
  wildcardPolicy: None

This content is not included.RHOAIENG-37259 - Elyra Pipelines not supported on IBM Z (s390x)

Elyra Pipelines depend on Data Science Pipelines (DSP) for orchestration and validation. Because DSP is not currently available on IBM Z, Elyra pipeline-related functionality and tests are skipped.

Workaround: None. Elyra Pipelines will function correctly once DSP support is enabled and validated on IBM Z.

This content is not included.RHOAIENG-37015 - TensorBoard reporting fails in PyTorch 2.8 training image

When using TensorBoard reporting for training jobs that use the SFTTrainer with the image registry.redhat.io/rhoai/odh-training-cuda128-torch28-py312-rhel9:rhoai-3.0, or when the report_to parameter is omitted from the training configuration, the training job fails with a JSON serialization error.

Workaround: Install the latest versions of the transformers and trl packages and update the torch_dtype parameter to dtype in the training configuration.

If you are using the Training Operator SDK, you can specify the packages to install by using the packages_to_install parameter in the create_job function:

packages_to_install=[
    "transformers==4.57.1",
    "trl==0.24.0"
]

This content is not included.RHOAIENG-36757 - Existing cluster storage option missing during model deployment when no connections exist

When creating a model deployment in a project that has no data connections defined, the Existing cluster storage option is not displayed, even if suitable Persistent Volume Claims (PVCs) exist in the project. This prevents you from selecting an existing PVC for model deployment.

Workaround: Create at least one connection of type URI in the project to make the Existing cluster storage option appear.

This content is not included.RHOAIENG-31071 - Parquet datasets not supported on IBM Z (s390x)

Some built-in evaluation tasks, such as arc_easy and arc_challenge, use datasets provided by Hugging Face in Parquet format. Parquet is not supported on IBM Z.

Workaround: None. To evaluate models on IBM Z, use datasets in a supported format instead of Parquet.

This content is not included.RHAIENG-1795 - CodeFlare with Ray does not work with Gateway

When running the following commands, the output indicates that the Ray cluster has been created and is running, but the cell never completes because the Gateway route does not respond correctly:

cluster.up()
cluster.wait_ready()

As a result, subsequent operations such as fetching the Ray cluster or obtaining the job client fail, preventing job submission to the cluster.

Workaround: None. The Ray Dashboard Gateway route does not function correctly when created through CodeFlare.

This content is not included.RHAIENG-1796 - Pipeline name must be DNS compliant when using Kubernetes pipeline storage

When using Kubernetes as the storage backend for pipelines, Elyra does not automatically convert pipeline names to DNS-compliant values. If a non-DNS-compliant name is used when starting an Elyra pipeline, an error similar to the following appears:

[TIP: did you mean to set 'https://ds-pipeline-dspa-robert-tests.apps.test.rhoai.rh-aiservices-bu.com/pipeline' as the endpoint, take care not to include 's' at end]

Workaround: Use DNS-compliant names when creating or running Elyra pipelines.

This content is not included.RHAIENG-1139 - Cannot deploy LlamaStackDistribution with the same name in multiple namespaces

If you create two LlamaStackDistribution resources with the same name in different namespaces, the ReplicaSet for the second resource fails to start the Llama Stack pod. The Llama Stack Operator does not correctly assign security constraints when duplicate names are used across namespaces.

Workaround: Use a unique name for each LlamaStackDistribution in every namespace. For example, include the project name or add a suffix such as llama-stack-distribution-209342.

This content is not included.RHAIENG-1624 - Embeddings API timeout on disconnected clusters

On disconnected clusters, calls to the embeddings API might time out when using the default embedding model (ibm-granite/granite-embedding-125m-english) included in the default Llama Stack distribution image.

Workaround

Add the following environment variables to the LlamaStackDistribution custom resource to use the embedded model offline:

- name: SENTENCE_TRANSFORMERS_HOME
  value: /opt/app-root/src/.cache/huggingface/hub
- name: HF_HUB_OFFLINE
  value: "1"
- name: TRANSFORMERS_OFFLINE
  value: "1"
- name: HF_DATASETS_OFFLINE
  value: "1"

This content is not included.RHOAIENG-34923 - Runtime configuration missing when running a pipeline from JupyterLab

The runtime configuration might not appear in the Elyra pipeline editor when you run a pipeline from the first active workbench in a project. This occurs because the configuration fails to populate for the initial workbench session.

Workaround: Restart the workbench. After restarting, the runtime configuration becomes available for pipeline execution.

This content is not included.RHAIENG-35055 - Model catalog fails to initialize after upgrading from OpenShift AI 2.24

After upgrading from OpenShift AI 2.24, the model catalog might fail to initialize and load. The OpenShift AI dashboard displays a Request access to model catalog error.

Workaround

Delete the existing model catalog ConfigMap and deployment by running the following commands:

$ oc delete configmap model-catalog-sources -n rhoai-model-registries --ignore-not-found
$ oc delete deployment model-catalog -n rhoai-model-registries --ignore-not-found

This content is not included.RHAIENG-35529 - Reconciliation issues in Data Science Pipelines Operator when using external Argo Workflows

If you enable the embedded Argo Workflows controllers (argoWorkflowsControllers: Managed) before deleting an existing external Argo Workflows installation, the workflow controller might fail to start and the Data Science Pipelines Operator (DSPO) might not reconcile its custom resources correctly.

Workaround: Before enabling the embedded Argo Workflows controllers, delete any existing external Argo Workflows instance from the cluster.

This content is not included.RHAIENG-36756 - Existing cluster storage option missing during model deployment when no connections exist

When creating a model deployment in a project with no defined data connections, the Existing cluster storage option does not appear, even if Persistent Volume Claims (PVCs) are available. As a result, you cannot select an existing PVC for model storage.

Workaround: Create at least one connection of type URI in the project. Afterward, the Existing cluster storage option becomes available.

This content is not included.RHOAIENG-36817 - Inference server fails when Model server size is set to small

When creating an inference service via the dashboard, selecting a small Model server size causes subsequent inferencing requests to fail. As a result, the deployment of the inference service itself succeeds, but the inferencing requests fail with a timeout error.

Workaround: To resolve this issue, select the Model server size as large from the dropdown.

This content is not included.RHOAIENG-9418 - Elyra raises error when you use parameters in uppercase

Elyra raises an error when you try to run a pipeline that uses parameters in uppercase.

Workaround: To resolve this issue, convert the parameter name to lowercase.

This content is not included.RHOAIENG-35623 - Model deployment fails when using hardware profiles

Model deployments that use hardware profiles fail because the Red Hat OpenShift AI Operator does not inject the tolerations, nodeSelector, or identifiers from the hardware profile into the underlying InferenceService when manually creating InferenceService resources. As a result, the model deployment pods cannot be scheduled to suitable nodes and the deployment fails to enter a ready state. Workbenches that use the same hardware profile continue to deploy successfully.

Workaround: Run a script to manually inject the tolerations, nodeSelector, or identifiers from the hardware profile into the underlying InferenceService as described in the Knowledgebase solution Workaround for model deployment failure when using hardware profiles.

This content is not included.RHOAIENG-33995 - Deployment of an inference service for Phi and Mistral models fails

The creation of an inference service for Phi and Mistral models using vLLM runtime on IBM Power cluster with openshift-container-platform 4.19 fails due to an error related to CPU backend. As a result, deployment of these models is affected, causing inference service creation failure.

Workaround: To resolve this issue, disable the sliding_window mechanism in the serving runtime if it is enabled for CPU and Phi models. Sliding window is not currently supported in V1.

This content is not included.RHOAIENG-33914 - LM-Eval Tier2 task test failures

There can be some failures with LM-Eval Tier2 task tests because the Massive Multitask Language Understanding Symbol Replacement (MMLUSR) tasks are broken, if you are using an older version of the trustyai-service-operator.

Workaround: Ensure that the latest version of the trustyai-service-operator is installed.

This content is not included.RHOAIENG-33795 - Manual Route creation needed for gRPC endpoint verification for Triton Inference Server on IBM Z

When verifying Triton Inference Server with gRPC endpoint, Route does not get created automatically. This happens because the Operator currently defaults to creating an edge-terminated route for REST only.

Workaround

To resolve this issue, manual Route creation is needed for gRPC endpoint verification for Triton Inference Server on IBM Z.

When the model deployment pod is up and running, define an edge-terminated Route object in a YAML file with the following contents:

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: <grpc-route-name>                  # e.g. triton-grpc
  namespace: <model-deployment-namespace>  # namespace where your model is deployed
  labels:
    inferenceservice-name: <inference-service-name>
  annotations:
    haproxy.router.openshift.io/timeout: 30s
spec:
  host: <custom-hostname>                  # e.g. triton-grpc.<apps-domain>
  to:
    kind: Service
    name: <service-name>                   # name of the predictor service (e.g. triton-predictor)
    weight: 100
  port:
    targetPort: grpc                       # must match the gRPC port exposed by the service
  tls:
    termination: edge
  wildcardPolicy: None

Create the Route object:
```
oc apply -f <route-file-name>.yaml
```

To send an inference request, enter the following command:

grpcurl -cacert <ca_cert_file>\ 1
  -protoset triton_desc.pb \
  -d '{
    "model_name": "<model_name>",
    "inputs": [
      {
        "name": "<input_tensor_name>",
        "shape": [<shape>],
        "datatype": "<data_type>",
        "contents": {
          "<datatype_specific_contents>": [<input_data_values>]
        }
      }
    ],
    "outputs": [
      {
        "name": "<output_tensor_name>"
      }
    ]
  }' \
  <grpc_route_host>:443 \
  inference.GRPCInferenceService/ModelInfer

1: <ca_cert_file> is the path to your cluster router CA cert (for example, router-ca.crt).

Note

<triton_protoset_file> is compiled as a protobuf descriptor file. You can generate it as protoc -I. --descriptor_set_out=triton_desc.pb --include_imports grpc_service.proto.

Download grpc_service.proto and model_config.proto files from the Content from github.com is not included.triton-inference-service GitHub page.

This content is not included.RHOAIENG-33697 - Unable to Edit or Delete models unless status is "Started"

When you deploy a model on the NVIDIA NIM or single-model serving platform, the Edit and Delete options in the action menu are not available for models in the Starting or Pending states. These options become available only after the model has been successfully deployed.

Workaround: Wait until the model is in the Started state to make any changes or to delete the model.

This content is not included.RHOAIENG-33645 - LM-Eval Tier1 test failures

There can be failures with LM-Eval Tier1 tests because confirm_run_unsafe_code is not passed as an argument when a job is run, if you are using an older version of the trustyai-service-operator.

Workaround: Ensure that you are using the latest version of the trustyai-service-operator and that AllowCodeExecution is enabled.

This content is not included.RHOAIENG-32897 - Pipelines defined with the Kubernetes API and invalid platformSpec do not appear in the UI or run

When a pipeline version defined with the Kubernetes API includes an empty or invalid spec.platformSpec field (for example, {} or missing the kubernetes key), the system misidentifies the field as the pipeline specification. As a result, the REST API omits the pipelineSpec, which prevents the pipeline version from being displayed in the UI and from running.

Workaround: Remove the spec.platformSpec field from the PipelineVersion object. After removing the field, the pipeline version is displayed correctly in the UI and the REST API returns the pipelineSpec as expected.

This content is not included.RHOAIENG-30493 - Error creating a workbench in a Kueue-enabled project

When using the dashboard to create a workbench in a Kueue-enabled project, the creation fails if Kueue is disabled on the cluster or if the selected hardware profile is not associated with a LocalQueue. In this case, the required LocalQueue cannot be referenced, the admission webhook validation fails, and the following error message is shown:

Error creating workbench
admission webhook "kubeflow-kueuelabels-validator.opendatahub.io" denied the request: Kueue label validation failed: missing required label "kueue.x-k8s.io/queue-name"

Workaround

Enable Kueue and hardware profiles on your cluster as a user with cluster-admin permissions:

Log in to your cluster by using the oc client.
Run the following command to patch the OdhDashboardConfig custom resource in the redhat-ods-applications namespace:

oc patch OdhDashboardConfig odh-dashboard-config -n redhat-ods-applications --type=merge -p '{"spec": {"dashboardConfig": {"disableKueue": false, "disableHardwareProfiles": false}}}'

This content is not included.RHOAIENG-29729 - Model registry Operator in a restart loop after upgrade

After upgrading from OpenShift AI version 2.22 or earlier to version 2.23 or later with the model registry component enabled, the model registry Operator might enter a restart loop. This is due to an insufficient memory limit for the manager container in the model-registry-operator-controller-manager pod.

Workaround

To resolve this issue, you must trigger a reconciliation for the model-registry-operator-controller-manager deployment. Adding the opendatahub.io/managed='true' annotation to the deployment will accomplish this and apply the correct memory limit. You can add the annotation by running the following command:

oc annotate deployment model-registry-operator-controller-manager -n redhat-ods-applications opendatahub.io/managed='true' --overwrite

Note

This command overwrites custom values in the model-registry-operator-controller-manager deployment. For more information about custom deployment values, see Customizing component deployment resources.

After the deployment updates and the memory limit increases from 128Mi to 256Mi, the container memory usage will stabilize and the restart loop will stop.

This content is not included.RHOAIENG-31248 - KServe http: TLS handshake error

The OpenShift CA auto-injection in the localmodelcache validation webhook configuration is missing the necessary annotation, leading to repeated TLS handshake errors.

Workaround

To resolve this issue, enter the following command:

oc patch validatingwebhookconfiguration localmodelcache.serving.kserve.io --type='merge' -p='{"metadata":{"annotations":{"service.beta.openshift.io/inject-cabundle":"true"}}}'
validatingwebhookconfiguration.admissionregistration.k8s.io/localmodelcache.serving.kserve.io patched

This command replaces the previously used cert-manager.io/inject-ca-from and adds the service.beta.openshift.io/inject-cabundle annotation to the localmodelcache.serving.kserve.io webhook configuration. This change ensures secure connections in KServe by eliminating TLS handshake errors.

This content is not included.RHOAIENG-31238 - New observability stack enabled when creating DSCInitialization

When you remove a DSCInitialization resource and create a new one using OpenShift AI console form view, it enables a Technology Preview observability stack. This results in the deployment of an unwanted observability stack when recreating a DSCInitialization resource.

Workaround

To resolve this issue, manually remove the "metrics" and "traces" fields when recreating the DSCInitiliazation resource using the form view.

This is not required if you want to use the Technology Preview observability stack.

This content is not included.RHAIENG-496 - Error creating LlamaStackDistribution as a non-administrator user

Non-administrator requests fail due to insufficient role-based access control (RBAC) as the deployed role definitions are outdated or incomplete for the current Llama Stack resources (for example, the LlamaStackDistribution CRD). When creating a LlamaStackDistribution as non-administrator user, the following error appears:

error (forbidden): error when retrieving current configuration of: Resource: "llamastack.io/v1alpha1, Resource=llamastackdistributions", GroupVersionKind: "llamastack.io/v1alpha1, Kind=LlamaStackDistribution" Name: "lsd-llama-milvus", Namespace: "my-project" from server for: "example-llamastackdistribution.yaml": llamastackdistributions.llamastack.io "lsd-llama-milvus" is forbidden: User "my-non-admin-user" cannot get resource "llamastackdistributions" in API group "llamastack.io" in the namespace "my-project"

Workaround

To resolve this issue, perform the following steps:

Deploy the upstream ClusterRole definition for OpenShift AI by using the following command:

oc apply -f https://raw.githubusercontent.com/llamastack/llama-stack-k8s-operator/main/config/rbac/llsd_editor_role.yaml

Apply the role to users that need permission to manage the LLamaStackDistribution:

oc create rolebinding llamastack-editor --clusterrole=llsd-editor-role --user=my-non-admin-user -n my-project

This content is not included.RHOAIENG-32145 - Llama Stack Operator deployment failures on OpenShift versions earlier than 4.17

When installing OpenShift AI on OpenShift clusters running versions earlier than 4.17, the integrated Llama Stack Operator (llamastackoperator) might fail to deploy.

The Llama Stack Operator requires Kubernetes version 1.32 or later, but OpenShift 4.15 uses Kubernetes 1.28. This version gap can cause schema validation failures when applying the LlamaStackDistribution custom resource definition (CRD), due to unsupported selectable fields introduced in Kubernetes 1.32.

Workaround: Install OpenShift AI on an OpenShift cluster running version 4.17 or later.

This content is not included.RHOAIENG-32242 - Failure on creating NetworkPolicies for OpenShift versions 4.15 and 4.16

When installing OpenShift AI on OpenShift clusters running versions 4.15 or 4.16, deployment of certain NetworkPolicy resources might fail. This can occur when the llamastackoperator or related components attempt to create a NetworkPolicy in a protected namespace, such as redhat-ods-applications. The request can be blocked by the admission webhook networkpolicies-validation.managed.openshift.io, which restricts modifications to certain namespaces and resources, even for cluster-admin users. This restriction can apply to both self-managed and Red Hat–managed OpenShift environments.

Workaround: Deploy OpenShift AI on an OpenShift cluster running version 4.17 or later. For clusters where the webhook restriction is enforced, contact your OpenShift administrator or Red Hat Support to determine an alternative deployment pattern or approved change to the affected namespace.

This content is not included.RHOAIENG-32599 - Inference service creation fails on IBM Z cluster

When you attempt to create an inference service using the vLLM runtime on an IBM Z cluster, it fails with the following error: ValueError: 'aimv2' is already used by a Transformers config, pick another name.

Workaround: None.

This content is not included.RHOAIENG-29731 - Inference service creation fails on IBM Power cluster with OpenShift 4.19

When you attempt to create an inference service by using the vLLM runtime on an IBM Power cluster on OpenShift Container Platform version 4.19, it fails due to an error related to Non-Uniform Memory Access (NUMA).

Workaround: When you create an inference service, set the environment variable VLLM_CPU_OMP_THREADS_BIND to all.

This content is not included.RHOAIENG-29352 - Missing Documentation and Support menu items

In the OpenShift AI top navigation bar, when you click the help icon ( ), the menu contains only the About menu item. The Documentation and Support menu items are missing.

Workaround: None.

This content is not included.RHOAIENG-29292 - vLLM logs permission errors on IBM Z due to usage stats directory access

When running vLLM on the IBM Z architecture, the inference service starts successfully, but logs an error in a background thread related to usage statistics reporting. This happens because the service tries to write usage data to a restricted location (/.config), which it does not have permission to access.

The following error appears in the logs:

Exception in thread Thread-2 (_report_usage_worker):
Traceback (most recent call last):
 ...
PermissionError: [Error 13] Permission denied: '/.config'

Workaround: To prevent this error and suppress the usage stats logging, set the VLLM_NO_USAGE_STATS=1 environment variable in the inference service deployment. This disables automatic usage reporting, avoiding permission issues when you write to system directories.

Obsolete

This content is not included.RHOAIENG-28910 - Unmanaged KServe resources are deleted after upgrading from 2.16 to 2.19 or later

During the upgrade from OpenShift AI 2.16 to 3.4, the FeatureTracker custom resource (CR) is deleted before its owner references are fully removed from associated KServe-related resources. As a result, resources that were originally created by the Red Hat OpenShift AI Operator with a Managed state and later changed to Unmanaged in the DataScienceCluster (DSC) custom resource (CR) might be unintentionally removed. This issue can disrupt model serving functionality until the resources are manually restored.

The following resources might be deleted in 3.4 if they were changed to Unmanaged in 2.16:

Kind	Namespace	Name
`KnativeServing`	`knative-serving`	`knative-serving`
`ServiceMeshMember`	`knative-serving`	`default`
`Gateway`	`istio-system`	`kserve-local-gateway`
`Gateway`	`knative-serving`	`knative-ingress-gateway`
`Gateway`	`knative-serving`	`knative-local-gateway`

Workaround

If you have already upgraded from OpenShift AI 2.16 to 3.4, perform one of the following actions:

If you have an existing backup, manually recreate the deleted resources without owner references to the FeatureTracker CR.
If you do not have an existing backup, you can use the Operator to recreate the deleted resources:
1. Back up any resources you have already recreated.
2. In the DSC, set spec.components.kserve.serving.managementState to Managed, and then save the change to allow the Operator to recreate the resources.
  Wait until the Operator has recreated the resources.
3. In the DSC, set spec.components.kserve.serving.managementState back to Unmanaged, and then save the change.
4. Reapply any previous custom changes to the recreated KnativeServing, ServiceMeshMember, and Gateway CRs resources.

If you have not yet upgraded, perform the following actions before upgrading to prevent this issue:

In the DSC, set spec.components.kserve.serving.managementState to Unmanaged.
For each of the affected KnativeServing, ServiceMeshMember, and Gateway resources listed in the above table, edit its CR by deleting the FeatureTracker owner reference. This edit removes the resource’s dependency on the FeatureTracker and prevents the deletion of the resource during the upgrade process.

This content is not included.NVPE-302, This content is not included.NVPE-303 - Missing storage classes for NIM models

When you try to deploy a NVIDIA Inference Microservice (NIM) model on the NVIDIA NIM model serving platform in a newly-installed OpenShift AI cluster, you might observe that the Storage class drop-down menu is not populated or is missing on the Model deployment page. This is because the storage classes are not loaded or cached in the user interface in new installations of OpenShift AI. As a result, you cannot configure storage for your deployment.

Workaround

From the OpenShift AI dashboard, click Settings → Storage classes. Do not make any changes.
Click Models → Model deployments to view your NIM model deployment.
Click Deploy model.
On the Model deployment page, the Storage class drop-down menu is visible and populated with the available storage class options.

This content is not included.RHOAIENG-25734 - Duplicate name issue with notebook images

When you delete a workbench after you have created a workbench, deployment, or model server and use the same name for both the product-scoped and global-scoped Imagrestreams, the workbench displays an incorrect name in the workbench table and in the Edit workbench form.

Workaround: Do not use the same name for your project-scoped and global-scoped Accelerator profiles.

This content is not included.RHOAIENG-25056 - Data science pipeline task fails when optional input parameters used in nested pipelines are not set

When a pipeline has optional input parameters, if values for those parameters are not provided and they are used in a nested pipeline, the tasks using them fail with the following error:

failed: failed to resolve inputs: resolving input parameter with spec component_input_parameter:"optional_input": parent DAG does not have input parameter optional_input

Workaround: Provide values for all optional parameters when using nested pipeline tasks.

This content is not included.RHOAIENG-24786 - Upgrading the Authorino Operator from Technical Preview to Stable fails in disconnected environments

In disconnected environments, upgrading the Red Hat Authorino Operator from Technical Preview to Stable fails with an error in the authconfig-migrator-qqttz pod.

Workaround

Update the Red Hat Authorino Operator to the latest version in the tech-preview-v1 update channel (v1.1.2).

Run the following script:

#!/bin/bash
set -xe

# delete the migrator job
oc delete job -n openshift-operators authconfig-migrator || true

# run the migrator script
authconfigs=$(oc get authconfigs -A -o custom-columns='NAMESPACE:.metadata.namespace,NAME:.metadata.name' --no-headers)

if [[ ! -z "${authconfigs}" ]]; then
    while IFS=" " read -r namespace name; do
        oc get authconfig "$name" -n "$namespace" -o yaml > "/tmp/${name}.${namespace}.authconfig.yaml"
        oc apply -f "/tmp/${name}.${namespace}.authconfig.yaml"
    done <<< "${authconfigs}"
fi

oc patch crds authconfigs.authorino.kuadrant.io --type=merge --subresource status --patch '{"status":{"storedVersions":["v1beta2"]}}'

Update the Red Hat Authorino Operator subscription to use the stable update channel.
Select the update option for Authorino 1.2.1.

This content is not included.RHOAIENG-24886 - Cannot deploy OCI model when Model URI field includes prefix

When deploying an OCI model, if you paste the complete URI in the Model URI field and then move the cursor to another field, the URL prefix (for example, http://) is removed from the Model URI field, but it is included in the storageUri value in the InferenceService resource. As a result, you cannot deploy the OCI model.

Workaround

To prevent the issue, manually remove the URI prefix from the Model URI field before you move the cursor from that field, or copy and paste the URI text excluding the prefix.

To recover from the issue, open the InferenceService resource and edit the storageUri value to change the prefix from http to oci.

This content is not included.RHOAIENG-24104 - KServe reconciler should only deploy certain resources when Authorino is installed

When Authorino is not installed, Red Hat OpenShift AI applies the AuthorizationPolicy and EnvoyFilter resources to the KServe serverless deployment mode. This could block some inference requests.

Workaround: Install Authorino, then restart the OpenShift AI operator, KServe controller, and odh-model-controller by deleting the existing pods.

This content is not included.RHOAIENG-23562 - TrustyAIService TLS handshake error in FIPS clusters

When using a FIPS cluster that uses an external route to send a request to the TrustyAIService, a TLS handshake error appears in the logs and the request is not processed.

Workaround: Use the UI to send a request to the TrustyAIService.

This content is not included.RHOAIENG-23169 - StorageInitializer fails to download models from Hugging Face repository

Deploying models from Hugging Face in a KServe environment using the hf:// protocol fails when the cluster lacks built-in support for this protocol. Additionally, the storage initializer InitContainer in KServe can encounter a PermissionError because of insufficient write permissions in the default cache directory (/.cache).

Workaround

Install a ClusterStorageContainer resource in the KServe cluster with the following values:

Enable support for the hf:// URI format in the supportedUriFormats section.
Set the HF_HOME environment variable in the storage-initializer container to a writable directory, such as /tmp/hf_home.

The following code sample shows a ClusterStorageContainer resource with these values set:

apiVersion: "serving.kserve.io/v1alpha1"
kind: ClusterStorageContainer
metadata:
  name: default
spec:
  container:
    name: storage-initializer
    env:
      - name: HF_HOME
        value: /tmp/hf_home
  supportedUriFormats:
    - prefix: hf://
    - prefix: gs://
    - prefix: s3://
    - prefix: hdfs://
    - prefix: webhdfs://
    - regex: "https://(.+?).blob.core.windows.net/(.+)"
    - regex: "https://(.+?).file.core.windows.net/(.+)"
    - regex: "https?://(.+)/(.+)"

This content is not included.RHOAIENG-22965 - Data science pipeline task fails when optional input parameters are not set

When a pipeline has optional input parameters, creating a pipeline run and tasks with unset parameters fails with the following error:

failed: failed to resolve inputs: resolving input parameter optional_input with spec component_input_parameter:"parameter_name": parent DAG does not have input parameter

This issue also affects the InstructLab pipeline.

Workaround: Provide values for all optional parameters.

This content is not included.RHOAIENG-22439 - cuda-rstudio-rhel9 cannot be built

When building the RStudio Server workbench images, building the cuda-rstudio-rhel9 fails with the following error:

Package supervisor-4.2.5-6.el9.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!
.M...U...    /run/supervisor
error: build error: building at STEP "RUN yum -y module enable nginx:$NGINX_VERSION &&     INSTALL_PKGS="nss_wrapper bind-utils gettext hostname nginx nginx-mod-stream nginx-mod-http-perl fcgiwrap initscripts chkconfig supervisor" &&     yum install -y --setopt=tsflags=nodocs $INSTALL_PKGS &&     rpm -V $INSTALL_PKGS &&     nginx -v 2>&1 | grep -qe "nginx/$NGINX_VERSION\." && echo "Found VERSION $NGINX_VERSION" &&     yum -y clean all --enablerepo='*'": while running runtime: exit status 1

Workaround

Run the following command to re-tag cuda-rstudio-rhel9:

oc tag --source=imagestreamtag redhat-ods-applications/cuda-rhel9:latest redhat-ods-applications/cuda-rstudio-rhel9:latest

Output similar to the following appears:

Tag redhat-ods-applications/cuda-rstudio-rhel9:latest set to redhat-ods-applications/cuda-rhel9@sha256:c03a3017364f311ad41f3a3677e0a532020cdd9f57fd98578288eb789cffbf4f.

This content is not included.RHOAIENG-21274 - Connection type changes back to S3 or URI when deploying a model with an OCI connection type

If you deploy a model that is using the S3 or URI connection type in a project with no matching connections, the Create new connection section is pre-populated using data from the model location of the S3 or URI storage location. If you change the connection type to OCI and enter a value in the Model URI field, the connection type changes back to S3 or URI.

Workaround: To deploy a model with OCI connection type, either register a model with OCI connections or use a model that has an OCI connection. Do not change the connection type while deploying from the model registry.

This content is not included.RHOAIENG-20209 - Warning message not displayed when requested resources exceed threshold

When you click Distributed workloads → Project metrics and view the Requested resources section, the charts show the requested resource values and the total shared quota value for each resource (CPU and Memory). However, when the Requested by all projects value exceeds the Warning threshold value for that resource, the expected warning message is not displayed.

Workaround: None.

This content is not included.RHOAIENG-19711 - Kueue-controller-manager uses old metrics port after upgrade from 2.16.0 to 2.17.0

After upgrade, the Kueue Operator continues to use the old port (8080) instead of the new port (8443) for metrics. As a result, the OpenShift console Observe > Targets page shows that the status of the Kueue Operator is Down . Workaround Restart the Kueue Operator Pod (kueue-controller-manager-*) or delete the kueue-controller-manager deployment. The Kueue Operator status changes to Up.

This content is not included.RHOAIENG-19564 - The deploymentMode annotation is not added to isvc if defaultDeploymentMode is serverless and isvc does not contain annotation

The serving.kserve.io/deploymentMode annotation is not added to the Inference Service resource if it is not explicitly set by the user, if defaultDeploymentMode is set to Serverless. As a result, another pod is created. The queue-proxy side-car container is missing this pod and the inference URL is changed.

Workaround: Add the annotation to the manifest before creating the resource. The inference service then works as expected.

This content is not included.RHOAIENG-19261 - The TrustyAI installation might fail due to missing custom resource definitions (CRDs)

Previously, when installing or upgrading OpenShift AI, the TrustyAI installation might have failed due to missing InferenceService and ServingRuntime CRDs. As a result, the Trusty AI controller went into the CrashLoopBackOff state. This issue is now resolved.

This content is not included.RHOAIENG-18884 - Enabling NIM account setup is incomplete

Trying to enable the NVIDIA NIM model serving platform results in the odh-model-controller deployment starting before the NIM account setup is complete. As a result, the NIM account setup is incomplete and the platform is not enabled. Workaround:: Restart odh-model-controller deployment. The NIM feature will then work.

This content is not included.RHOAIENG-18675 - Workbenches component fails after upgrading

When upgrading to OpenShift AI 3.4, the workbench component doesn’t upgrade correctly. Specifically, BuildConfigs and resources that follow it (such as RStudio BuildConfigs and ROCm imagestreams) are not updated, which causes the workbench component reconciliation in the DataScienceCluster to fail. This issue is now resolved. Workaround:: To update to 2.17, first delete buildconfig objects in the cluster before upgrading

This content is not included.SRVKS-1301 (previously documented as RHOAIENG-18590) - The KnativeServing resource fails after disabling and enabling KServe

After disabling and enabling the kserve component in the DataScienceCluster, the KnativeServing resource might fail.

Workaround

Delete all ValidatingWebhookConfiguration and MutatingWebhookConfiguration webhooks related to Knative:

Get the webhooks:

oc get ValidatingWebhookConfiguration,MutatingWebhookConfiguration | grep -i knative

Ensure KServe is disabled.

Get the webhooks:

oc get ValidatingWebhookConfiguration,MutatingWebhookConfiguration | grep -i knative

Delete the webhooks.
Enable KServe.
Verify that the KServe pod can successfully spawn, and that pods in the knative-serving namespace are active and operational.

This content is not included.RHOAIENG-16247 - Elyra pipeline run outputs are overwritten when runs are launched from OpenShift AI dashboard

When a pipeline is created and run from Elyra, outputs generated by the pipeline run are stored in the folder bucket-name/pipeline-name-timestamp of object storage.

When a pipeline is created from Elyra and the pipeline run is started from the OpenShift AI dashboard, the timestamp value is not updated. This can cause pipeline runs to overwrite files created by previous pipeline runs of the same pipeline.

This issue does not affect pipelines compiled and imported using the OpenShift AI dashboard because runid is always added to the folder used in object storage. For more information about storage locations used in AI pipelines, see Storing data with pipelines.

Workaround: When storing files in an Elyra pipeline, use different subfolder names on each pipeline run.

This content is not included.OCPBUGS-49422 - AMD GPUs and AMD ROCm workbench images are not supported in a disconnected environment

This release of OpenShift AI does not support AMD GPUs and AMD ROCm workbench images in a disconnected environment because installing the AMD GPU Operator requires internet access to fetch dependencies needed to compile GPU drivers.

Workaround: None.

This content is not included.RHOAIENG-16900 - Space-separated format in serving-runtime arguments can cause deployment failure

When deploying models, using a space-separated format to specify additional serving runtime arguments can cause unrecognized arguments errors, as shown in the following example:

api_server.py: error: unrecognized arguments: --chat-template /app/data/template/template-alpaca.jinja

Workaround

Use one of the following formats instead of the space-separated format:

Separate the argument and its value with an equal sign (=) instead of a space, as shown in the following example:
```
--<argument>=<value>
```
List the argument and its value on separate lines, as shown in the following example:
```
--<argument>
<value>
```

This content is not included.RHOAIENG-16484 - vLLM server engine for Gaudi accelerators fails after a period of inactivity

When using the vLLM ServingRuntime with Gaudi accelerators support for KServe model-serving runtime on a cluster equipped with Gaudi hardware, the vLLM server fails with a TimeoutError message after a period of inactivity where it is not processing continuous inference requests.

Workaround: None.

This content is not included.RHOAIENG-15982 - Replicas not created correctly in multinode deployment when pipelineParallelSize parameter updated

When you update the pipelineParallelSize parameter in a multinode deployment, new replicas are not created correctly. Instead, the existing ReplicaSet’s replicas are modified, causing the new deployment to malfunction.

Workaround: Remove the existing InferenceService CR and create an InferenceService CR with the updated pipelineParallelSize value.

This content is not included.RHOAIENG-15033 - Model registry instances do not restart or update after upgrade from OpenShift AI 2.14 to 2.15

When you upgrade from OpenShift AI 2.14.z to 2.15, existing instances of the model registry component are not updated, which causes the instance pods to use older images than the ones referenced by the operator pod.

Workaround

After upgrading to OpenShift AI 2.15, for each existing model registry, create a new model registry instance that uses the same database configuration, and delete the old model registry instance. Your new model registry instance will contain all existing registered models and their metadata. You can perform these steps from the OpenShift AI dashboard or from the OpenShift command-line interface (CLI). From the OpenShift AI dashboard:

For each existing model registry, create a new instance that uses the same database configuration.
1. From the OpenShift AI dashboard, click Settings → Model registry settings.
2. Check the database connection details for an existing registry by clicking the action menu (⋮) beside the model registry on the Model registry settings page, and then clicking View database configuration.
3. Create a new model registry with the same database connection details as the existing model registry instance.
4. In OpenShift, verify that the REST and GRPC images defined in your model registry instance pod match the ones in the model registry operator pod.
After creating the new model registry, delete the original registry:
1. On the Model registry settings page, click the action menu (⋮) beside the model registry, and then click Delete model registry.
2. In the Delete model registry? dialog that appears, enter the name of the model registry in the text field to confirm that you intend to delete it.
3. Click Delete model registry.

Alternatively, from the OpenShift command-line interface (CLI):

Run the following command, replacing <model_registry_name> with the name of your model registry:

oc patch -n rhoai-model-registries mr <model_registry_name> --type=merge -p='{"spec":{"grpc":{"image":""},"rest":{"image":""}}}'

In OpenShift, verify that the REST and GRPC images defined in your model registry instance pod match the ones in the model registry operator pod.

This content is not included.RHOAIENG-15008 - Error when creating a bias metric from the CLI without a request name

The user interface might display an error message when viewing bias metrics if the requestName parameter is not set. If you use the user interface to view bias metrics, but want to configure them through the CLI, you must specify a requestName parameter within your payload:

curl -sk -H "Authorization: Bearer ${TOKEN}" -X POST
      --location $TRUSTY_ROUTE/metrics/group/fairness/spd/request
      --header 'Content-Type: application/json'
      --data "{
                 \"modelId\": \"demo-loan-nn-onnx-alpha\",
                 \"protectedAttribute\": \"Is Male-Identifying?\",
                 \"privilegedAttribute\": 1.0,
                 \"unprivilegedAttribute\": 0.0,
                 \"outcomeName\": \"Will Default?\",
                 \"favorableOutcome\": 0,
                 \"batchSize\": 5000,
                 \"requestName\": \"My Request\"
      }

Workaround: Use the CLI to delete metric requests that have no specified requestName and then refresh the UI.

This content is not included.RHOAIENG-14986 - Incorrect package path causes copy_demo_nbs to fail

In OpenShift AI SDK version 0.22.0, the copy_demo_nbs() function of the CodeFlare SDK fails because of an incorrect path to the SDK package. Running this function results in a FileNotFound error.

Workaround

There are two options to work around the issue:

Manually create a demo-notebooks directory.
From a Python shell or a blank notebook, run the following Python script to create the demo-notebooks directory:

import codeflare_sdk
import os
import pathlib
import shutil

nb_dir = os.path.join(os.path.dirname(codeflare_sdk.__file__), "demo-notebooks")
def copy_nbs(dir: str = "./demo-notebooks", overwrite: bool = False):
    # does dir exist already?
    if overwrite is False and pathlib.Path(dir).exists():
        raise FileExistsError(
            f"Directory {dir} already exists. Please remove it or provide a different location."
        )

    shutil.copytree(nb_dir, dir, dirs_exist_ok=True)

copy_nbs()

This content is not included.RHOAIENG-14552 - Workbench or notebook OAuth proxy fails with FIPS on OpenShift Container Platform 4.16

When using OpenShift 4.16 or newer in a FIPS-enabled cluster, connecting to a running workbench fails because the connection between the internal component oauth-proxy and the OpenShift ingress fails with a TLS handshake error. When opening a workbench, the browser shows an "Application is not available" screen without any additional diagnostics.

Workaround: None.

This content is not included.RHOAIENG-14095 - The dashboard is temporarily unavailable after the installing OpenShift AI Operator

After you install the OpenShift AI Operator, the OpenShift AI dashboard is unavailable for approximately three minutes. As a result, a Cannot read properties of undefined page might appear.

Workaround: After you first access the OpenShift AI dashboard, wait for a few minutes and then refresh the page in your browser.

This content is not included.RHOAIENG-13633 - Cannot set a serving platform for a project without first deploying a model from outside of the model registry

You cannot set a serving platform for a project without first deploying a model from outside of the model registry.

Currently, you cannot deploy a model from a model registry to a project unless the project already has single-model or multi-model serving selected. The only way to select single-model or multi-model serving from the OpenShift AI UI is to first deploy a model or model server from outside the registry.

Workaround

Before deploying a model from a model registry to a new project, select a serving platform for the project from the OpenShift AI dashboard, or from the OpenShift console.

From the OpenShift AI dashboard:
1. From the OpenShift AI dashboard, click Data science projects.
2. Click the Models tab.
3. Deploy a model or create a model server, and then delete it.
From the OpenShift console:
1. In the OpenShift web console, select Home → Projects.
2. Click the name of the project.
3. Click the YAML tab.
4. In the metadata.labels section, add the modelmesh-enabled label.
  1. To choose multi-model serving, set the modelmesh-enabled value to true.
  2. To choose single-model serving, set the modelmesh-enabled value to false.
5. Click Save.

After you select a serving platform for a project, you can deploy models from a model registry to that project.

This content is not included.RHOAIENG-12516 - fast releases are available in unintended release channels

Due to a known issue with the stream image delivery process, fast releases are currently available on unintended streaming channels, for example, stable, and stable-x.y. For accurate release type, channel, and support lifecycle information, refer to the Life-cycle Dates table on the Red Hat OpenShift AI Self-Managed Life Cycle page.

Workaround: None.

This content is not included.RHOAIENG-11297 - Authentication failure after pipeline run

During the execution of a pipeline run, a connection error might occur due to a certificate authentication failure. This certificate authentication failure can be caused by the use of a multi-line string separator for customCABundle in the default-dsci object, which is not supported by data science pipelines.

Workaround

Log in to the OpenShift as a cluster administrator.
Click Operators → Installed Operators and then click the Red Hat OpenShift AI Operator.
Click the DSC Initialization tab.
Click the default-dsci object.
Click the YAML tab.
Change the spec section to the following:
```
spec:
trustedCABundle:
    customCABundle: |
```
Click Save.
Wait for a few minutes until all the dsp ca-bundle config maps are automatically updated for the existing pipeline server.
Run the failing pipeline again.

This content is not included.RHOAIENG-11232 - Distributed workloads: Kueue alerts do not provide runbook link

After a Kueue alert fires, the cluster administrator can click Observe → Alerting → Alerts and click the name of the alert to open its Alert details page. On the Alert details page, the Runbook section should provide a link to the appropriate runbook to help to diagnose and resolve the issues that triggered the alert. However, the runbook link is missing.

Workaround: You can access the Kueue alert runbooks at Content from github.com is not included.Kueue alert runbooks.

This content is not included.RHOAIENG-11024 - Resources entries get wiped out after removing opendatahub.io/managed annotation

Manually removing the opendatahub.io/managed annotation from any component deployment YAML file might cause resource entry values in the file to be erased.

Workaround

To remove the annotation from a deployment, use the following steps to delete the deployment.

The controller pod for the component will redeploy automatically with default values.

Log in to the OpenShift console as a cluster administrator.
In the Administrator perspective, click Workloads > Deployments.
From the Project list, select redhat-ods-applications.
Click the options menu beside the deployment for which you want to remove the annotation.
Click Delete Deployment.

This content is not included.RHOAIENG-10665 - Unable to query Speculating with a draft model for granite model

Currently, you cannot use speculative decoding on the granite-7b model and granite-7b-accelerator draft model. When querying these models, the queries fail with an internal error.

 [id="known-issues_RHOAIENG-2974_{context}"]
*https://issues.redhat.com/browse/RHOAIENG-2974[RHOAIENG-2974^] - Data science cluster cannot be deleted without its associated initialization object*

You cannot delete a DataScienceCluster (DSC) object, if its associated DSCInitialization object (DSCI) does not exist.

When you uninstall the OpenShift AI Operator , before you delete your DataScienceCluster (DSC) object, it is good practice to first remove its associated DSCInitialization object (DSCI). Due to this issue, this practice is not currently possible.

Workaround: None.

This content is not included.RHOAIENG-9670 - vLLM container intermittently crashes while processing requests

If you have deployed a model by using the vLLM ServingRuntime for KServe runtime on the single-model serving platform and have also configured tensor-parallel-size, depending on the hardware platform you have used, the kserve-container container intermittently crashes while processing requests.

Workaround: None. Requests are processed successfully after the container restarts.

This content is not included.RHOAIENG-9498 - Pipeline run execution status does not update

Executions from completed pipeline runs appear in the UI with the Running status.

Workaround: None.

This content is not included.RHOAIENG-9481 - Pipeline runs menu glitches when clicking action menu

When you click the action menu (⋮) next to a pipeline run on the Experiments > Experiments and runs page, the menu that appears is not fully visible, and you have to scroll to see all of the menu items.

Workaround: None.

This content is not included.RHOAIENG-8819 - ibm-granite/granite-3b-code-instruct model fails to deploy on single-model serving platform

If you try to deploy the ibm-granite/granite-3b-code-instruct model on the single-model serving platform by using the vLLM ServingRuntime for KServe runtime, the model deployment fails with the error shown.

ValueError: Head size 80 is not supported by FlashAttention. Supported head sizes are: [32, 64, 96, 128, 160, 192, 224, 256]

Workaround

Add the following parameter to the env section of the ServingRuntime specification for the vLLM ServingRuntime for KServe runtime.

VLLM_ATTENTION_BACKEND=XFORMERS

This content is not included.RHOAIENG-8553 - Workbench created with custom image shows !Deleted flag

If you disable the internal image registry on your OpenShift cluster and then create a workbench with a custom image that is imported by using the image tag, for example: quay.io/my-wb-images/my-image:tag, a !Deleted flag is shown in the Notebook image column on the Workbenches tab of the Data science projects page. If you stop the workbench, you cannot restart it.

Workaround

Import the custom image using the SHA digest, for example quay.io/my-repo/my-image@sha256:xxxxxxxxxxxxx, and then create the workbench using the custom image.

Note

An OpenShift cluster admin can confirm if the internal image registry is enabled on your cluster.
An OpenShift AI admin user can confirm if a custom image was imported by using the tag notation.

This content is not included.RHOAIENG-8294 - CodeFlare error when upgrading OpenShift AI 2.8 to version 2.10 or later

If you try to upgrade OpenShift AI 2.8 to version 2.10 or later, the following error message is shown for the CodeFlare component, due to a mismatch with the AppWrapper custom resource definition (CRD) version.

ReconcileCompletedWithComponentErrors DataScienceCluster resource reconciled with component errors: 1 error occurred: * CustomResourceDefinition.apiextensions.k8s.io "appwrappers.workload.codeflare.dev" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions

Workaround

Delete the existing AppWrapper CRD:
```
$ oc delete crd appwrappers.workload.codeflare.dev
```

Wait for about 20 seconds, and then ensure that a new AppWrapper CRD is automatically applied, as shown in the following example:

$ oc get crd appwrappers.workload.codeflare.dev
NAME                                 CREATED AT
appwrappers.workload.codeflare.dev   2024-11-22T18:35:04Z

This content is not included.RHOAIENG-8218 - Cannot log in to a workbench created on an OpenShift 4.15 cluster without OpenShift Container Platform internal image registry

When you create a workbench on an OpenShift 4.15 cluster that does not have the OpenShift Container Platform internal image registry enabled, the workbench starts successfully, but you cannot log in to it.

A The authorization server encountered an unexpected condition that prevented it from fulfilling the request error is displayed.

Workaround

Manually create a service account token secret in OpenShift Container Platform for every workbench that you create.

From the OpenShift CLI, use the following command to create a service account token secret. Replace <workbench-name> with the name of your workbench, and <project-name> with the name of your data science project.

—---
cat <<'EOF' | oc apply -f -
---
kind: Secret
apiVersion: v1
metadata:
  name: <workbench-name>-token
  namespace: <project-name>
  annotations:
    kubernetes.io/service-account.name: <workbench-name>
type: kubernetes.io/service-account-token
EOF
—---

Refresh the page in your browser and log in to your workbench.

For information about creating a service account token secret using the web console, see Creating a service account token secret.

This content is not included.RHOAIENG-7887 - Kueue fails to monitor RayCluster or PyTorchJob resources

When you create a DataScienceCluster CR with all components enabled, the Kueue component is installed before the Ray component and the Training Operator component. As a result, the Kueue component does not monitor RayCluster or PyTorchJob resources.

Workaround

Perform one of the following actions:

After installing the Ray component and the Training Operator component, restart the Kueue controller pod in the redhat-ods-applications namespace.
Alternatively, edit the DataScienceCluster CR to mark the kueue component as Removed, wait until Kueue is uninstalled, and then mark the kueue component as Managed again.

This content is not included.RHOAIENG-7716 - Pipeline condition group status does not update

When you run a pipeline that has loops (dsl.ParallelFor) or condition groups (dsl.lf), the UI displays a Running status for the loops and groups, even after the pipeline execution is complete.

Workaround

You can confirm if a pipeline is still running by checking that no child tasks remain active.

From the OpenShift AI dashboard, click Develop & train → Pipelines → Runs.
From the Project list, click your data science project.
From the Runs tab, click the pipeline run that you want to check the status of.
Expand the condition group and click a child task.
A panel that contains information about the child task is displayed
On the panel, click the Task details tab.
The Status field displays the correct status for the child task.

This content is not included.RHOAIENG-7346 - Distributed workloads no longer run from existing pipelines after upgrade

After upgrading to OpenShift AI 2.10, a distributed workload no longer runs from an existing pipeline if the cluster is created only inside the pipeline. The pipeline stops working, and the Ray cluster does not start.

Workaround: In the Python code for each existing pipeline for a distributed workload, replace cluster.wait_ready with time.sleep(180) and recompile the code. Import the pipeline and schedule the pipeline run.

This content is not included.RHOAIENG-1197 - Cannot create pipeline due to the End date picker in the pipeline run creation page defaulting to NaN values when using Firefox on Linux

When you try to create a pipeline with a scheduled recurring run using Firefox on Linux, enabling the End Date parameter results in Not a Number (NaN) values for both the date and time, and you cannot create the pipeline.

Workaround: On Linux, use a different supported browser to create scheduled pipeline runs. For more information about the browsers that OpenShift AI supports, see the Supported Configurations for 3.x Knowledgebase article.

This content is not included.RHOAIENG-7312 - Model serving fails during query with token authentication in KServe

If you have enabled both the ModelMesh and KServe components in your DataScienceCluster object and you have also added Authorino as an authorization provider, a race condition might occur that results in the odh-model-controller pods being rolled out in a state that is appropriate for ModelMesh, but not for KServe and Authorino. In this situation, when you make an inference request to a running model that was deployed using KServe, you see a 404 - Not Found error. In addition, the logs for the odh-model-controller deployment object show a Reconciler error message.

Workaround: In OpenShift, restart the odh-model-controller deployment object.

This content is not included.RHOAIENG-7079 - Pipeline task status and logs sometimes not shown in OpenShift AI dashboard

Sometimes, especially when running pipelines by using Elyra, the OpenShift AI dashboard does not show the pipeline task status and logs, even when the related pods have not been pruned and the information is still available in the OpenShift Console.

Workaround: Follow the steps in the Data Science Pipelines workaround for how to view pipeline run pod logs in the Red Hat OpenShift AI dashboard Knowledgebase solution.

This content is not included.RHOAIENG-6853 - Cannot set pod toleration in Elyra pipeline pods

When you set a pod toleration for an Elyra pipeline pod, the toleration does not take effect.

Workaround: None.

This content is not included.RHOAIENG-7209 - Error displays when setting the default pipeline root

When you set the default pipeline root using the data science pipelines SDK or the OpenShift AI user interface, an error similar to the following appears:

F0513 18:25:25.155794 28 main.go:49] failed to execute component: Failed to open bucket "mlpipeline": Failed to retrieve credentials for bucket mlpipeline: Failed to get Bucket credentials from secret name="" namespace="dspa1": resource name may not be empty44

Workaround: None.

This content is not included.RHOAIENG-6711 - ODH-model-controller overwrites the spec.memberSelectors field in ServiceMeshMemberRoll objects

If you add a project or namespace to a ServiceMeshMemberRoll resource using the spec.memberSelectors field of the ServiceMeshMemberRoll resource, the ODH-model-controller overwrites the field.

Workaround

Explicitly add namespaces to the ServiceMeshMemberRoll resource using the spec.members field as shown in the example:

spec:
 members:
 - <namespace 1>
 - <namespace 2>

This content is not included.RHOAIENG-6709 - Jupyter notebook creation might fail when different environment variables specified

If you start and then stop a Jupyter notebook, and edit its environment variables in an OpenShift AI workbench, the notebook fails to restart.

Workaround

Delete the Jupyter tile resources (Notebook, PersistentVolumeClaims, ConfigMap resources, and Secrets) that start with jupyterhub-singleuser-profile.
Restart the notebook.

This content is not included.RHOAIENG-6701 - Users without cluster administrator privileges cannot access the job submission endpoint of the Ray dashboard

Users of the distributed workloads feature who do not have cluster administrator privileges for OpenShift might not be able to access or use the job submission endpoint of the Ray dashboard.

This content is not included.RHOAIENG-6649 - An error is displayed when viewing a model on a model server that has no external route defined

If you use the dashboard to deploy a model on a model server that does not have external routes enabled, a t.components is undefined error message might be shown while the model creation is in progress.

Workaround: None.

This content is not included.RHOAIENG-6646 - An error is displayed when viewing the Model Serving page during an upgrade

If you try to use the dashboard to deploy a model while an upgrade of OpenShift AI is in progress, a t.status is undefined error message might be shown.

Workaround: Wait until the upgraded OpenShift AI Operator is ready and then refresh the page in your browser.

This content is not included.RHOAIENG-6578 - Models that show authorization tokens do not actually require them after deletion of Authorino Operator

If you added Authorino as an authorization provider for the single-model serving platform, but later delete the Authorino Operator, the OpenShift AI dashboard might still show tokens for models that had token authorization enabled. However, authorization is no longer enabled, and you are able to make successful inference requests to the models without specifying the tokens in your requests.

This content is not included.RHOAIENG-6505 - Disconnected environments: Extra image needed for RayCluster TLS certificate creation

In OpenShift AI 3.4, the default image (quay.io/project-codeflare/ray:latest-py39-cu118) is used for TLS certificate creation for Ray Clusters when implementing mutual Transport Layer Security (mTLS). If you use distributed workloads in a disconnected environment, you must add this image to your mirror registry.

Workaround: You can provide your own image by editing the kuberay:certGeneratorImage value in the CodeFlare Operator config map, as described in Running distributed data science workloads in a disconnected environment.

This content is not included.RHOAIENG-6409 - Cannot save parameter errors appear in pipeline logs for successful runs

When you run a pipeline more than once, Cannot save parameter errors appear in the pipeline logs for successful pipeline runs. You can safely ignore these errors.

Workaround: None.

This content is not included.RHOAIENG-6376 - Pipeline run creation fails after setting pip_index_urls in a pipeline component to a URL that contains a port number and path

When you create a pipeline and set the pip_index_urls value for a component to a URL that contains a port number and path, compiling the pipeline code and then creating a pipeline run results in the following error:

ValueError: Invalid IPv6 URL

Workaround

Create a new pip server using only protocol://hostname, and update the pip_index_urls value for the component with the new server.
Recompile the pipeline code.
Create a new pipeline run.

This content is not included.RHOAIENG-6317 - An error is displayed when viewing pipeline run pod logs in the dashboard

When using the log viewer in the OpenShift AI dashboard to view pipeline run pod logs, a pods not found error message might be shown.

Workaround: Follow the steps in the Data Science Pipelines workaround for how to view pipeline run pod logs in the Red Hat OpenShift AI dashboard Knowledgebase solution.
Note
This issue is partially fixed in OpenShift AI 2.9.1; see This content is not included.RHOAIENG-7079 for details of the remaining work.

This content is not included.RHOAIENG-5314 - Data science pipeline server fails to deploy in fresh cluster due to network policies

When you create a data science pipeline server on a fresh cluster, the user interface remains in a loading state and the pipeline server does not start. A Pipeline server failed error message might be displayed.

Workaround

Log in to the OpenShift web console as a cluster administrator.
Click Networking > NetworkPolicies.
Click the Project list and select your project.
Click the Create NetworkPolicy button.

For Configure via, select YAML view and define the network policy as shown:

kind: NetworkPolicy
apiVersion: networking.k8s.io/v1
metadata:
  name: allow-from-redhat-ods-app-to-mariadb
spec:
  podSelector:
    matchLabels:
      app: mariadb-pipelines-definition
  ingress:
    - ports:
        - protocol: TCP
          port: 3306
      from:
        - podSelector:
            matchLabels:
              app.kubernetes.io/name: data-science-pipelines-operator
          namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: redhat-ods-applications
  policyTypes:
    - Ingress

Click Create.

This content is not included.RHOAIENG-4570 - Existing Argo Workflows installation conflicts with install or upgrade

Data science pipelines 2.0 contains an installation of Argo Workflows. Red Hat does not support direct customer use of this instance of Argo Workflows. To install or upgrade OpenShift AI with data science pipelines 2.0, ensure that there is no existing installation of Argo Workflows on your cluster.

Workaround: Remove the existing Argo Workflows installation or set datasciencepipelines to Removed, and then proceed with the installation or upgrade.

This content is not included.RHOAIENG-10129 - Notebook and Ray cluster with matching names causes secret resolution failure

If you create a notebook and a Ray cluster that have matching names in the same namespace, one controller fails to resolve its secret because the secret already has an owner.

Workaround: Change the name of the Ray cluster so that its name differs from the corresponding notebook name in the namespace.

This content is not included.RHOAIENG-5067 - Model server metrics page does not load for a model server based on the ModelMesh component

Data science project names that contain capital letters or spaces can cause issues on the model server metrics page for model servers based on the ModelMesh component. The metrics page might not receive data correctly, resulting in a 400 Bad Request error and preventing the page from loading.

Workaround: In OpenShift, change the display names of your data science projects to meet Kubernetes resource name standards: use only lowercase alphanumeric characters and hyphens.

This content is not included.RHOAIENG-5025 - Self-signed certificates do not apply to the first created workbench

After self-signed certificates are configured centrally, the certificates do not apply to the first workbench created in a data science project.

Workaround: For each data science project that contains a workbench, delete the first workbench that was created after configuring self-signed certificates, and then create a new workbench. The self-signed certificates work as expected with the new workbench.

This content is not included.RHOAIENG-4966 - Self-signed certificates in a custom CA bundle might be missing from the odh-trusted-ca-bundle configuration map

Sometimes after self-signed certificates are configured in a custom CA bundle, the custom certificate is missing from the odh-trusted-ca-bundle ConfigMap, or the non-reserved namespaces do not contain the odh-trusted-ca-bundle ConfigMap when the ConfigMap is set to managed. These issues rarely occur.

Workaround: Restart the Red Hat OpenShift AI Operator pod.

This content is not included.RHOAIENG-4572 - Unable to run data science pipelines after install and upgrade in certain circumstances

You are unable to run data science pipelines after installing or upgrading OpenShift AI in the following circumstances:

You have installed OpenShift AI and you have a valid CA certificate. Within the default-dsci object, you have changed the managementState field for the trustedCABundle field to Removed post-installation.
You have upgraded OpenShift AI from version 2.6 to version 2.8 and you have a valid CA certificate.
You have upgraded OpenShift AI from version 2.7 to version 2.8 and you have a valid CA certificate.
Workaround
As a workaround, perform the following steps:
In the OpenShift web console, click Operators → Installed Operators and then click the Red Hat OpenShift AI Operator.
Click the DSC Initialization tab.
Click the default-dsci object.
Click the YAML tab.
In the spec section, change the value of the managementState field for trustedCABundle to Managed, as shown:
spec: trustedCABundle: managementState: Managed
Note
If you upgraded from OpenShift AI version 2.6 or 2.7 to version 3.4, you must manually add the trustedCABundle field and the managementState field as they are not present in the YAML code. In addition, you do not need to enter a value in the customCABundle field.
Click Save.
Restart the dashboard replicaset.
In the OpenShift web console, switch to the Administrator perspective.
Click Workloads → Deployments.
Set the Project to All Projects or redhat-ods-applications to ensure you can see the appropriate deployment.
Search for the rhods-dashboard deployment.
Click the action menu (⋮) and select Restart Rollout from the list.
Wait until the Status column indicates that all pods in the rollout have fully restarted.

This content is not included.RHOAIENG-4524 - BuildConfig definitions for RStudio images contain occurrences of incorrect branch

The BuildConfig definitions for the RStudio and CUDA - RStudio workbench images point to the wrong branch in OpenShift AI. The BuildConfig definitions incorrectly point to the main branch instead of the rhoai-2.8 branch.

Workaround: To use the RStudio and CUDA - RStudio workbench images in OpenShift AI, follow the steps in the Branch workaround for RStudio image BuildConfig definition Knowledgebase solution.

This content is not included.RHOAIENG-4497 - Models on the multi-model serving platform with self-signed certificates stop working after upgrading to 2.8

In previous versions, if you wanted to use a self-signed certificate when serving models on the multi-model serving platform, you had to manually configure the storage-config secret used by your data connection to specify a certificate authority (CA) bundle.

If you upgrade a previous version of OpenShift AI that used that workaround to the latest version, the multi-model serving platform can no longer serve models.

Workaround: To use a self-signed certificate with both the multi- and single-model serving platforms, follow the steps in Adding a CA bundle.

This content is not included.RHOAIENG-4430 - CA Bundle does not work for KServe without a data connection

If you have installed a certificate authority (CA) bundle on your OpenShift cluster to use self-signed certificates and then use the OpenShift AI dashboard to create a data connection to serve a model, OpenShift AI automatically stores the certificate in a secret called storage-config. However, if you bypass the OpenShift AI dashboard and configure the underlying InferenceService resource to specify a different secret name or a service account, OpenShift AI fails to validate SSL connections to the model and the model status includes [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate.

Workaround: Use the OpenShift AI dashboard to create the data connection for your model. Do not manually modify the InferenceService resource to specify a different secret name or a service account.

This content is not included.RHOAIENG-4327 - Workbenches do not use the self-signed certificates from centrally configured bundle automatically

There are two bundle options to include self-signed certificates in OpenShift AI, ca-bundle.crt and odh-ca-bundle.crt. Self-signed certificates should apply to workbenches that you create after configuring self-signed certificates centrally. Workbenches do not use the self-signed certificates from the centrally configured bundle automatically.

Workaround

After configuring self-signed certificates centrally, they apply to any new workbenches and are available at /etc/pki/tls/certs/ with the custom prefix. You can force the tools in your workbench to use these certificates by setting a known environment variable that points to your certificate path.

If you used ca-bundle.crt when you configured certificates centrally, your path is /etc/pki/tls/certs/custom-ca-bundle.crt.
If you used odh-ca-bundle.crt when you configured certificates centrally, your path is /etc/pki/tls/certs/custom-odh-ca-bundle.crt.

Set a known environment variable:

From the OpenShift AI dashboard, go to Data science projects and select the name of the project containing your workbench.
Click the Workbenches tab.
Click the action menu (⋮) beside the workbench that you want to update, and click Edit workbench.
In the Environment variables section, click Add variable.
From the Select environment variable type list, select Config Map.
From the Select one list, select Key / value.
In the Key field, enter SSL_CERT_FILE.
In the Value field, enter the path to your certificate file. For example, /etc/pki/tls/certs/custom-ca-bundle.crt.
Click Update workbench.

For more information, see How to execute a pipeline from a Jupyter notebook in a disconnected environment.

This content is not included.RHOAIENG-4252 - Data science pipeline server deletion process fails to remove ScheduledWorkFlow resource

The pipeline server deletion process does not remove the ScheduledWorkFlow resource. As a result, new DataSciencePipelinesApplications (DSPAs) do not recognize the redundant ScheduledWorkFlow resource.

Workaround

Delete the pipeline server. For more information, see Deleting a pipeline server.
In the OpenShift command-line interface (CLI), log in to your cluster as a cluster administrator and perform the following command to delete the redundant ScheduledWorkFlow resource.
```
$ oc -n <data science project name> delete scheduledworkflows --all
```

This content is not included.RHOAIENG-4240 - Jobs fail to submit to Ray cluster in unsecured environment

When running distributed data science workloads from Jupyter notebooks in an unsecured OpenShift cluster, a ConnectionError: Failed to connect to Ray error message might be shown.

Workaround: In the ClusterConfiguration section of the notebook, set the openshift_oauth option to True.

This content is not included.RHOAIENG-3981 - In unsecured environment, the functionality to wait for Ray cluster to be ready gets stuck

When running distributed data science workloads from Jupyter notebooks in an unsecured OpenShift cluster, the functionality to wait for the Ray cluster to be ready before proceeding (cluster.wait_ready()) gets stuck even when the Ray cluster is ready.

Workaround

Perform one of the following actions:

In the ClusterConfiguration section of the notebook, set the openshift_oauth option to True.
Instead of using the cluster.wait_ready(), functionality, you can manually check the Ray cluster availability by opening the Ray cluster Route URL. When the Ray dashboard is available on the URL, then the cluster is ready.

This content is not included.RHOAIENG-3963 - Unnecessary managed resource warning

When you edit and save the OdhDashboardConfig custom resource for the redhat-ods-applications project, the system incorrectly displays the following Managed resource warning message.

This resource is managed by DSC default-doc and any modifications may be overwritten. Edit the managing resource to preserve changes.

You can safely ignore this message.

Workaround: Click Save to close the warning message and apply your edits.

This content is not included.RHOAIENG-3372 - Pipeline runs in disconnected environments fail due to image URL

Data science pipeline runs reference the registry.access.redhat.com/ubi8/ubi-micro image URL instead of the registry.redhat.io/ubi8/ubi-micro image URL. In disconnected environments, this discrepancy will cause pipeline run failures.

Workaround

Create a new custom resource (CR) containing the following code, where my-disconnected-registry.com:8443 is your image registry URL:

apiVersion: operator.openshift.io/v1alpha1
kind: ImageContentSourcePolicy
metadata:
  name: workaround-3372 # Enter a unique name
spec:
  repositoryDigestMirrors:
  - mirrors:
    - my-disconnected-registry.com:8443/ubi8 # Use your image registry URL
    source: registry.access.redhat.com/ubi8

Run the pipeline again.

This content is not included.RHOAIENG-1825 - After setting up self-signed certificates, executing pipelines might fail with workbenches that contain Elyra

After configuring self-signed certificates centrally, executing pipelines with workbenches that contain Elyra might fail.

Workaround

See the following Knowledgebase solutions for workaround steps:

This content is not included.RHOAIENG-3355 - OVMS on KServe does not use accelerators correctly

When you deploy a model using the single-model serving platform and select the OpenVINO Model Server serving runtime, if you request an accelerator to be attached to your model server, the accelerator hardware is detected but is not used by the model when responding to queries. The queries are computed by using the CPUs only.

Workaround: To configure OVMS to use accelerators in preference to CPUs, update your OVMS runtime template to add --target_device AUTO to the CLI options.

This content is not included.RHOAIENG-3134 - OVMS supports different model frameworks in single- and multi-model serving platforms

When you deploy a model using the single-model serving platform and select the OpenVINO Model Server runtime, you see additional frameworks in the Model framework (name - version) list.

Workaround: None.

This content is not included.RHOAIENG-3025 - OVMS expected directory layout conflicts with the KServe StoragePuller layout

When you use the OpenVINO Model Server (OVMS) runtime to deploy a model on the single-model serving platform (which uses KServe), there is a mismatch between the directory layout expected by OVMS and that of the model-pulling logic used by KServe. Specifically, OVMS requires the model files to be in the /<mnt>/models/1/ directory, while KServe places them in the /<mnt>/models/ directory.

Workaround

Perform the following actions:

In your S3-compatible storage bucket, place your model files in a directory called 1/, for example, /<s3_storage_bucket>/models/1/<model_files>.
To use the OVMS runtime to deploy a model on the single-model serving platform, choose one of the following options to specify the path to your model files:
- If you are using the OpenShift AI dashboard to deploy your model, in the Path field for your data connection, use the /<s3_storage_bucket>/models/ format to specify the path to your model files. Do not specify the 1/ directory as part of the path.
- If you are creating your own InferenceService custom resource to deploy your model, configure the value of the storageURI field as /<s3_storage_bucket>/models/. Do not specify the 1/ directory as part of the path.

KServe pulls model files from the subdirectory in the path that you specified. In this case, KServe correctly pulls model files from the /<s3_storage_bucket>/models/1/ directory in your S3-compatible storage.

This content is not included.RHOAIENG-3018 - OVMS on KServe does not expose the correct endpoint in the dashboard

When you use the OpenVINO Model Server (OVMS) runtime to deploy a model on the single-model serving platform, the URL shown in the Inference endpoint field for the deployed model is not complete.

Workaround: To send queries to the model, you must add the /v2/models/_<model-name>_/infer string to the end of the URL. Replace _<model-name>_ with the name of your deployed model.

This content is not included.RHOAIENG-2542 - Inference service pod does not always get an Istio sidecar

When you deploy a model using the single-model serving platform (which uses KServe), the istio-proxy container might be missing in the resulting pod, even if the inference service has the sidecar.istio.io/inject=true annotation.

In OpenShift AI 2.7, the missing istio-proxy container might not present a problem. However, if the pod experiences connectivity issues, they might be caused by the missing container.

Workaround: Delete the faulty pod. OpenShift AI automatically creates a new pod, which should have the missing container.

This content is not included.RHOAIENG-3378 - Internal Image Registry is an undeclared hard dependency for Jupyter notebooks spawn process

Before you can start OpenShift AI notebooks and workbenches, you must first enable the internal, integrated container image registry in OpenShift. Attempts to start notebooks or workbenches without first enabling the image registry will fail with an "InvalidImageName" error.

You can confirm whether the image registry is enabled for a cluster by using the following command:

$ oc get pods -n openshift-image-registry

Workaround: Enable the internal, integrated container image registry in OpenShift. See This page is not included, but the link has been rewritten to point to the nearest parent document.Image Registry Operator in OpenShift for more information about how to set up and configure the image registry.

This content is not included.RHOAIENG-2869 - Cannot edit existing model framework and model path in a multi-model project

When you try to edit a model in a multi-model project using the Deploy model dialog, the Model framework and Path values do not update.

Workaround: None.

This content is not included.RHOAIENG-2759 - Model deployment fails when both secured and regular model servers are present in a project

When you create a second model server in a project where one server is using token authentication, and the other server does not use authentication, the deployment of the second model might fail to start.

Workaround: None.

This content is not included.RHOAIENG-2724 - Model deployment fails because fields automatically reset in dialog

When you deploy a model or edit a deployed model, the Model servers and Model framework fields in the "Deploy model" dialog might reset to the default state. The Deploy button might remain enabled even though these mandatory fields no longer contain valid values.

If you click Deploy when the Model servers and Model framework fields are not set, the model deployment pods are not created.

Workaround: None.

This content is not included.RHOAIENG-2620 - Unable to create duplicate bias metrics from existing bias metrics

You can’t duplicate existing bias metrics.

Workaround

In the left menu of the OpenShift AI dashboard, click Model Serving.
On the Deployed models page, click the name of the model with the bias metric that you want to duplicate.
In the metrics page for the model, click the Model bias tab.
Click the action menu (⋮) next to the metric that you want to copy and then click Duplicate.
The Configure bias metrics dialog will open with prepopulated values for the bias configuration. For each of the Privileged value, Unprivileged value and Output value fields, cut the value and then paste it back in.
Note: Do not copy and paste these values.
Click Configure.

This content is not included.RHOAIENG-2602 - “Average response time" server metric graph shows multiple lines due to ModelMesh pod restart

The Average response time server metric graph shows multiple lines if the ModelMesh pod is restarted.

Workaround: None.

This content is not included.RHOAIENG-2585 - UI does not display an error/warning when UWM is not enabled in the cluster

Red Hat OpenShift AI does not correctly warn users if User Workload Monitoring (UWM) is disabled in the cluster. UWM is necessary for the correct functionality of model metrics.

Workaround: Manually ensure that UWM is enabled in your cluster, as described in Enabling monitoring for user-defined projects.

This content is not included.RHOAIENG-2555 - Model framework selector does not reset when changing Serving Runtime in form

When you use the Deploy model dialog to deploy a model on the single-model serving platform, if you select a runtime and a supported framework, but then switch to a different runtime, the existing framework selection is not reset. This means that it is possible to deploy the model with a framework that is not supported for the selected runtime.

Workaround: While deploying a model, if you change your selected runtime, click the Select a framework list again and select a supported framework.

This content is not included.https://issues.redhat.com/browse/RHOAIENG-2506 [RHOAIENG-2506^] - TrustyAI Target is down on Prometheus*

The Prometheus target for the TrustyAI controller manager is down due to a mismatch with the endpoint’s port. Alerts for TrustyAI will fire if the controller deployment pod is down.

Workaround: None.

This content is not included.RHOAIENG-2479 - ModelMesh monitoring stack are not deleted during upgrade from 2.4 or 2.5

If you upgrade the Red Hat OpenShift AI operator from version 2.4 to 2.5, and then update the operator to version 2.6, 2.7, or 2.8, all components related to hardware resource-consuming model monitoring are removed from the cluster. Some residual model-monitoring resources, which do not consume hardware resources, will still be present.

Workaround: To delete these resources, execute the following oc delete commands with cluster-admin privileges:

$ oc delete service rhods-model-monitoring -n redhat-ods-monitoring
$ oc delete service prometheus-operated -n redhat-ods-monitoring
$ oc delete sa prometheus-custom -n redhat-ods-monitoring
$ oc delete sa rhods-prometheus-operator -n redhat-ods-monitoring
$ oc delete prometheus rhods-model-monitoring -n redhat-ods-monitoring
$ oc delete route rhods-model-monitoring -n redhat-ods-monitoring

This content is not included.RHOAIENG-2468 - Services in the same project as KServe might become inaccessible in OpenShift

If you deploy a non-OpenShift AI service in a data science project that contains models deployed on the single-model serving platform (which uses KServe), the accessibility of the service might be affected by the network configuration of your OpenShift cluster. This is particularly likely if you are using the This content is not included.OVN-Kubernetes network plugin in combination with host network namespaces.

Workaround

Perform one of the following actions:

Deploy the service in another data science project that does not contain models deployed on the single-model serving platform. Or, deploy the service in another OpenShift project.
In the data science project where the service is, add a This content is not included.network policy to accept ingress traffic to your application pods, as shown in the following example:
```
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-to-myapp
spec:
  podSelector:
    matchLabels:
      app: myapp
  ingress:
     - {}
```

This content is not included.RHOAIENG-2312 - Importing numpy fails in code-server workbench

Importing numpy in your code-server workbench fails.

Workaround

In your code-server workbench, from the Activity bar, select the menu icon ( ) > View > Command Palette to open the Command Palette.
In Firefox, you can use the F1 keyboard shortcut to open the command palette.
Enter python: s.
From the drop-down list, select the Python: Select interpreter action.
In the Select Interpreter dialog, select Enter interpreter path….
Enter /opt/app-root/bin/python3 as the interpreter path and press Enter.
From the drop-down list, select the new Python interpreter.
Confirm that the new interpreter (app-root) appears on the Status bar. The selected interpreter persists if the workbench is stopped and started again, so the workaround should need to be performed only once for each workbench.

This content is not included.RHOAIENG-2270 - (Single-model) Users cannot update model deployment settings

You can’t edit the deployment settings (for example, the number of replicas) of a model you deployed with a single-model platform.

Workaround: None.

This content is not included.RHOAIENG-2269 - (Single-model) Dashboard fails to display the correct number of model replicas

On a single-model platform, the Models and model servers section of a data science project does not show the correct number of model replicas.

Workaround: Check the number of replicas using the following CLI command:

$ oc -n <project_resource_name> get pods --selector serving.kserve.io/inferenceservice=<model_resource_name>

You can find your <project_resource_name> and <model_resource_name> values in the OpenShift AI dashboard.

You can also check the number of model replicas from the OpenShift web console, under Workloads > Pods.

This content is not included.RHOAIENG-2228 - The performance metrics graph changes constantly when the interval is set to 15 seconds

On the Endpoint performance tab of the model metrics screen, if you set the Refresh interval to 15 seconds and the Time range to 1 hour, the graph results change continuously.

Workaround: None.

This content is not included.RHOAIENG-2183 - Endpoint performance graphs might show incorrect labels

In the Endpoint performance tab of the model metrics screen, the graph tooltip might show incorrect labels.

Workaround: None.

This content is not included.RHOAIENG-1919 - Model Serving page fails to fetch or report the model route URL soon after its deployment

When deploying a model from the OpenShift AI dashboard, the system displays the following warning message while the Status column of your model indicates success with an OK/green checkmark.

Failed to get endpoint for this deployed model. routes.rout.openshift.io"<model_name>" not found

Workaround: Refresh your browser page.

This content is not included.RHOAIENG-1467 - Serverless net-istio controller pod might hit OOM

The Knative net-istio-controller pod (which is a dependency for KServe) might continuously crash due to an out-of-memory (OOM) error.

Workaround: In the custom resource (CR) for your KnativeServing instance, add an ENABLE_SECRET_INFORMER_FILTERING_BY_CERT_UID=true annotation to inject an environment variable to the net-istio-controller pod. Injecting this environment variable reduces the number of secrets that the net-istio-controller watches and loads into memory.

For more information about this configuration, see Creating a Knative serving instance.

For more information about this configuration, see This content is not included.Creating a Knative Serving instance.

This content is not included.RHOAIENG-1452 - The Red Hat OpenShift AI Add-on gets stuck

The Red Hat OpenShift AI Add-on uninstall does not delete OpenShift AI components after being triggered via OCM APIs.

Workaround

Manually delete the remaining OpenShift AI resources as follows:

Delete the DataScienceCluster CR.
Wait until all pods are deleted from the redhat-ods-applications namespace.
If Serverless was set to Managed in the DataScienceCluster CR, wait until all pods are deleted from the knative-serving namespace.
Delete the DSCInitialization CR.
If Service Mesh was set to Managed in the DSCInitialization CR, wait until all pods are deleted from the istio-system namespace.
Uninstall the Red Hat OpenShift AI Operator.
Wait until all pods are deleted from the redhat-ods-operator namespace and the redhat-ods-monitoring namespace.

This content is not included.RHOAIENG-880 - Default pipelines service account is unable to create Ray clusters

You cannot create Ray clusters using the default pipelines Service Account.

Workaround

Authenticate using the CodeFlare SDK, by adding the following lines to the pipeline code:

from codeflare_sdk.cluster.auth import TokenAuthentication
   auth = TokenAuthentication(
       token=openshift_token, server=openshift_server
   )
   auth_return = auth.login()

Note

If your cluster uses self-signed certificates, include ca-cert-path=<path> in the TokenAuthentication parameter list.

This content is not included.RHOAIENG-535 - Metrics graph showing HTTP requests for deployed models is incorrect if there are no HTTP requests

If a deployed model does not receive at least one HTTP request for each of the two data types (success and failed), the graphs that show HTTP request performance metrics (for all models on the model server or for the specific model) render incorrectly, with a straight line that indicates a steadily increasing number of failed requests.

Workaround: After the deployed data model receives at least one HTTP request that is successful and one that is failed, the graphs show the HTTP request performance metrics correctly. The graphs work correctly as long as one HTTP request of each data type (success and failed) occur at any point in the history of the deployed model, regardless of the time range that you specify for the graphs.

This content is not included.RHOAIENG-404 - No Components Found page randomly appears instead of Enabled page in OpenShift AI dashboard

A No Components Found page might appear when you access the Red Hat OpenShift AI dashboard.

Workaround: Refresh the browser page.

This content is not included.RHOAIENG-234 - Unable to view .ipynb files in VSCode in Insecured cluster

When you use the code-server workbench image on Google Chrome in an insecure cluster, you cannot view .ipynb files.

Workaround: Use a different browser.

This content is not included.RHOAIENG-164 - "Number of model server replicas for Kserve is not applied correctly from the dashboard

When you set a number of model server replicas different from the default (1), the model (server) is still deployed with 1 replica. —-

This content is not included.RHOAIENG-2541 - KServe controller pod experiences OOM because of too many secrets in the cluster

If your OpenShift cluster has a large number of secrets, the KServe controller pod might continually crash due to an out-of-memory (OOM) error.

Workaround: Reduce the number of secrets in the OpenShift cluster until the KServe controller pod becomes stable.

This content is not included.RHOAIENG-1128 - Unclear error message displays when attempting to increase the size of a Persistent Volume (PV) that is not connected to a workbench

When attempting to increase the size of a Persistent Volume (PV) that is not connected to a workbench, an unclear error message is displayed.

Workaround: Verify that your PV is connected to a workbench before attempting to increase the size.

This content is not included.RHOAIENG-2184 - Cannot create Ray clusters or distributed workloads

Users cannot create Ray clusters or distributed workloads in namespaces where they have admin or edit permissions.

Workaround: To grant the appropriate permissions, create a ClusterRole for the resources created by the KubeRay Operator and CodeFlare Operator, and specify the admin and edit aggregation labels, as described in the Red Hat Knowledgebase solution How to grant permission to create Ray clusters and distributed workloads in RHOAI.

This content is not included.RHOAIENG-2099 - Data science pipeline server fails to deploy in fresh cluster

When you create a data science pipeline server on a fresh cluster, the user interface remains in a loading state and the pipeline server does not start. A “Pipeline server failed” error message might be displayed.

Workaround: Delete the pipeline server and create a new one.

If the problem persists, disable the database health check in the DSPA custom resource:

Use the following command to edit the custom resource:
```
$ oc edit dspa pipelines-definition -n my-project
```
Set the spec.database.disableHealthCheck value to true.
Save the change.

This content is not included.RHOAIENG-908 - Cannot use ModelMesh if KServe was previously enabled and then removed

When both ModelMesh and KServe are enabled in the DataScienceCluster object, and you subsequently remove KServe, you can no longer deploy new models with ModelMesh. You can continue to use models that were previously deployed with ModelMesh.

Example error message:

Error creating model serverInternal error occurred: failed calling webhook "inferenceservice.kserve-webhook-server.defaulter": failed to call webhook: Post "https://kserve-webhook-server-service.redhat-ods-applications.svc:443/mutate-serving-kserve-io-v1beta1-inferenceservice?timeout=10s": service "kserve-webhook-server-service" not found

Workaround

You can resolve this issue in either of the following ways:

Re-enable KServe.
Delete the KServe MutatingWebHook configuration by completing the following steps as a user with cluster-admin permissions:
1. Log in to your cluster by using the oc client.
2. Enter the following command:
```
oc delete mutatingwebhookconfigurations inferenceservice.serving.kserve.io
```

This content is not included.RHOAIENG-807 - Accelerator profile toleration removed when restarting a workbench

If you create a workbench that uses an accelerator profile that in turn includes a toleration, restarting the workbench removes the toleration information, which means that the restart cannot complete. A freshly created GPU-enabled workbench might start the first time, but never successfully restarts afterwards because the generated pod remains forever pending.

This content is not included.RHOAIENG-804 - Cannot deploy Large Language Models with KServe on FIPS-enabled clusters

Red Hat OpenShift AI is not yet fully designed for FIPS. You cannot deploy Large Language Models (LLMs) with KServe on FIPS-enabled clusters.

This content is not included.RHOAIENG-545 - Cannot specify a generic default node runtime image in JupyterLab pipeline editor

When you edit an Elyra pipeline in the JupyterLab IDE pipeline editor, and you click the PIPELINE PROPERTIES tab, and scroll to the Generic Node Defaults section and edit the Runtime Image field, your changes are not saved.

Workaround: Define the required runtime image explicitly for each node. Click the NODE PROPERTIES tab, and specify the required image in the Runtime Image field.

This content is not included.RHOAIENG-517 - User with edit permissions cannot see created models

A user with edit permissions cannot see any created models, unless they are the project owner or have admin permissions for the project.

Workaround: If the project owner or a user with admin permissions subsequently creates a model, the user with edit permissions can then see all models.

This content is not included.RHOAIENG-499 - Uninstalling Red Hat OpenShift AI Self Managed by using the CLI does not uninstall

If you uninstall Red Hat OpenShift AI by using the command-line interface, then the DataScienceCluster CR, the DSCInitialization CR, and the Red Hat OpenShift AI Operator are not removed.

Workaround

Manually delete the remaining OpenShift AI resources as follows:

Delete the DataScienceCluster CR.
Wait until all pods are deleted from the redhat-ods-applications namespace.
If Serverless was set to Managed in the DataScienceCluster CR, wait until all pods are deleted from the knative-serving namespace.
Delete the DSCInitialization CR.
If Service Mesh was set to Managed in the DSCInitialization CR, wait until all pods are deleted from the istio-system namespace.
Uninstall the Red Hat OpenShift AI Operator.
Wait until all pods are deleted from the redhat-ods-operator namespace and the redhat-ods-monitoring namespace.

This content is not included.RHOAIENG-497 - Removing DSCI Results In OpenShift Service Mesh CR Being Deleted Without User Notification

If you delete the DSCInitialization resource, the OpenShift Service Mesh CR is also deleted. A warning message is not shown.

Workaround: None.

This content is not included.RHOAIENG-343 - Manual configuration of OpenShift Service Mesh and OpenShift Serverless does not work for KServe

If you install OpenShift Serverless and OpenShift Service Mesh and then install Red Hat OpenShift AI with KServe enabled, KServe is not deployed.

Workaround

Edit the DSCInitialization resource: Set the managementState field of the serviceMesh component to Unmanaged.
Edit the DataScienceCluster resource: Within the kserve component, set the managementState field of the serving component to Unmanaged. For more information, see Installing KServe.

This content is not included.RHOAIENG-339 - KServe component images are not updated after upgrade to 2.5

Previously, the KServe component was a Limited Availability feature. If you enabled the kserve component and created models in an earlier version, then after you upgrade to Red Hat OpenShift AI 2.5, you must update some OpenShift AI resources as follows:

Update the DSCInitialization resource as follows:

$ oc patch $(oc get dsci -A -oname) --type='json' -p='[{"op": "replace", "path": "/spec/serviceMesh/managementState", "value":"Unmanaged"}]'

Update the DataScienceCluster resource as follows:

$ oc patch $(oc get dsc -A -oname) --type='json' -p='[{"op": "replace", "path": "/spec/components/kserve/serving/managementState", "value":"Unmanaged"}]'

Update the InferenceServices CRD as follows:

$ oc patch crd inferenceservices.serving.kserve.io --type=json -p='[{"op": "remove", "path": "/spec/conversion"}]'

Optionally, restart the Operator pod.

This content is not included.RHOAIENG-307 - Removing the DataScienceCluster deletes all OpenShift Serverless CRs

If you delete the DataScienceCluster custom resource (CR), all OpenShift Serverless CRs (including knative-serving, deployments, gateways, and pods) are also deleted. A warning message is not shown.

Workaround: None.

This content is not included.RHOAIENG-293 - Deprecated ModelMesh monitoring stack not deleted after upgrading from 2.4 to 2.5

In Red Hat OpenShift AI 2.5, the former ModelMesh monitoring stack is no longer deployed because it is replaced by user workload monitoring. However, the former monitoring stack is not deleted during an upgrade to OpenShift AI 2.5. Some components remain and use cluster resources.

This content is not included.RHOAIENG-288 - Recommended image version label for workbench is shown for two versions

Most of the workbench images that are available in OpenShift AI are provided in multiple versions. The only recommended version is the latest version. In the current release, the Recommended tag is erroneously shown for multiple versions of an image.

This content is not included.RHOAIENG-282 - Workload should not be dispatched if required resources are not available

Sometimes a workload is dispatched even though a single machine instance does not have sufficient resources to provision the RayCluster successfully. The AppWrapper CRD remains in a Running state and related pods are stuck in a Pending state indefinitely.

Workaround: Add extra resources to the cluster.

This content is not included.RHOAIENG-162 - Project remains selected after navigating to another page

When you select a project on the Data science projects page, the project remains selected, even after you navigate to another page. For example, if you subsequently open the Model Serving page, the page lists only the models for the previously selected project, instead of the models for all projects.

Workaround: From the Project list, select All projects.

This content is not included.RHOAIENG-131 - gRPC endpoint not responding properly after the InferenceService reports as Loaded

When numerous InferenceService instances are generated and directed requests, Service Mesh Control Plane (SMCP) becomes unresponsive. The status of the InferenceService instance is Loaded, but the call to the gRPC endpoint returns with errors.

Workaround: Edit the ServiceMeshControlPlane custom resource (CR) to increase the memory limit of the Istio egress and ingress pods.

This content is not included.RHOAIENG-130 - Synchronization issue when the model is just launched

When the status of the KServe container is Ready, a request is accepted even though the TGIS container is not ready.

Workaround: Wait a few seconds to ensure that all initialization has completed and the TGIS container is actually ready, and then review the request output.

This content is not included.RHOAIENG-88 - Cannot log in to Red Hat OpenShift AI dashboard

Sometimes, when you try to log in to Red Hat OpenShift AI, the 500 internal error error message is shown.

Workaround: Disable and re-enable the dashboard component in the DataScienceCluster object.

This content is not included.RHOAIENG-84 - Cannot use self-signed certificates with KServe

The single-model serving platform does not support self-signed certificates.

Workaround: To deploy a model from S3 storage, disable SSL authentication as described in the Red Hat Knowledgebase solution How to skip the validation of SSL for KServe.

This content is not included.RHOAIENG-66 - Ray dashboard route deployed by CodeFlare SDK exposes self-signed certs instead of cluster cert

When you deploy a Ray cluster by using the CodeFlare SDK with the openshift_oauth=True option, the resulting route for the Ray cluster is secured by using the passthrough method. As a result, the self-signed certificate used by the OAuth proxy is exposed.

Workaround

Use one of the following workarounds:

Set the openshift_oauth option to False.
Add the self-signed certificate used by the OAuth proxy to the client’s truststore.
Create a route manually, using a route configuration and certificate that is based on the needs of the client.

This content is not included.RHOAIENG-3115 - Model cannot be queried for a few seconds after it is shown as ready

Models deployed using the multi-model serving platform might be unresponsive to queries despite appearing as Ready in the dashboard. You might see an “Application is not available" response when querying the model endpoint.

Workaround: Wait 30-40 seconds and then refresh the page in your browser.

This content is not included.RHOAIENG-1619 (previously documented as DATA-SCIENCE-PIPELINES-165) - Poor error message when S3 bucket is not writable

When you set up a data connection and the S3 bucket is not writable, and you try to upload a pipeline, the error message Failed to store pipelines is not helpful.

Workaround: Verify that your data connection credentials are correct and that you have write access to the bucket you specified.

This content is not included.RHOAIENG-1207 (previously documented as Content from github.com is not included.ODH-DASHBOARD-1758) - Error duplicating OOTB custom serving runtimes several times

If you duplicate a model-serving runtime several times, the duplication fails with the Serving runtime name "<name>" already exists error message.

Workaround: Change the metadata.name field to a unique value.

This content is not included.RHOAIENG-1204 (previously documented as Content from github.com is not included.ODH-DASHBOARD-1771) - JavaScript error during Pipeline step initializing

Sometimes the pipeline Run details page stops working when the run starts.

Workaround: Refresh the page.

This content is not included.RHOAIENG-1203 (previously documented as Content from github.com is not included.ODH-DASHBOARD-1781) - Missing tooltip for Started Run status

Data science pipeline runs sometimes don’t show the tooltip text for the status icon shown.

Workaround: For more information, view the pipeline Run details page and see the run output.

This content is not included.RHOAIENG-1201 (previously documented as Content from github.com is not included.ODH-DASHBOARD-1908) - Cannot create workbench with an empty environment variable

When creating a workbench, if you click Add variable but do not select an environment variable type from the list, you cannot create the workbench. The field is not marked as required, and no error message is shown.

Workaround: None.

This content is not included.RHOAIENG-1196 (previously documented as Content from github.com is not included.ODH-DASHBOARD-2140) - Package versions displayed in dashboard do not match installed versions

The dashboard might display inaccurate version numbers for packages such as JupyterLab and Notebook. The package version number can differ in the image if the packages are manually updated.

Workaround

To find the true version number for a package, run the pip list command and search for the package name, as shown in the following examples:

$ pip list | grep jupyterlab
jupyterlab                        3.5.3
$ pip list | grep notebook
notebook                          6.5.3

This content is not included.RHOAIENG-582 (previously documented as Content from github.com is not included.ODH-DASHBOARD-1335) - Rename Edit permission to Contributor

The term Edit is not accurate:

For most resources, users with the Edit permission can not only edit the resource, they can also create and delete the resource.
Users with the Edit permission cannot edit the project.

The term Contributor more accurately describes the actions granted by this permission.

Workaround: None.

This content is not included.RHOAIENG-432 (previously documented as This content is not included.RHODS-12928) - Using unsupported characters can generate Kubernetes resource names with multiple dashes

When you create a resource and you specify unsupported characters in the name, then each space is replaced with a dash and other unsupported characters are removed, which can result in an invalid resource name.

Workaround: None.

This content is not included.RHOAIENG-226 (previously documented as This content is not included.RHODS-12432) - Deletion of the notebook-culler ConfigMap causes Permission Denied on dashboard

If you delete the notebook-controller-culler-config ConfigMap in the redhat-ods-applications namespace, you can no longer save changes to the Cluster Settings page on the OpenShift AI dashboard. The save operation fails with an HTTP request has failed error.

Workaround

Complete the following steps as a user with cluster-admin permissions:

Enter the following command to update the OdhDashboardConfig custom resource in the redhat-ods-applications application namespace:

$ oc patch OdhDashboardConfig odh-dashboard-config -n redhat-ods-applications --type=merge -p '{"spec": {"dashboardConfig": {"notebookController.enabled": true}}}'

This content is not included.RHOAIENG-133 - Existing workbench cannot run Elyra pipeline after workbench restart

If you use the Elyra JupyterLab extension to create and run pipelines within JupyterLab, and you configure the pipeline server after you created a workbench and specified a workbench image within the workbench, you cannot execute the pipeline, even after restarting the workbench.

Workaround

Stop the running workbench.
Edit the workbench to make a small modification. For example, add a new dummy environment variable, or delete an existing unnecessary environment variable. Save your changes.
Restart the workbench.
In the left sidebar of JupyterLab, click Runtimes.
Confirm that the default runtime is selected.

This content is not included.RHOAIENG-52 - Token authentication fails in clusters with self-signed certificates

If you use self-signed certificates, and you use the Python codeflare-sdk in a notebook or in a Python script as part of a pipeline, token authentication will fail.

Workaround: None.

This content is not included.RHODS-12986 - Potential reconciliation error after upgrade to Red Hat OpenShift AI 3.4

After you upgrade to Red Hat OpenShift AI 3.4, a reconciliation error might appear in the Red Hat OpenShift AI Operator pod logs and in the DataScienceCluster custom resource (CR) conditions.

Example error:

2023-11-23T09:45:37Z    ERROR    Reconciler error    {"controller": "datasciencecluster", "controllerGroup": "datasciencecluster.opendatahub.io", "controllerKind": "DataScienceCluster", "DataScienceCluster": {"name":"default-dsc"}, "namespace": "", "name": "default-dsc", "reconcileID": "0c1a32ca-7ffd-4310-8259-f6baabf3c868", "error": "1 error occurred:\n\t* Deployment.apps \"rhods-prometheus-operator\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/part-of\":\"model-mesh\", \"app.opendatahub.io/model-mesh\":\"true\", \"k8s-app\":\"rhods-prometheus-operator\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable\n\n"}

Workaround: Restart the Red Hat OpenShift AI Operator pod.

This content is not included.RHODS-12798 - Pods fail with "unable to init seccomp" error

Pods fail with CreateContainerError status or Pending status instead of Running status, because of a known kernel bug that introduced a seccomp memory leak. When you check the events on the namespace where the pod is failing, or run the oc describe pod command, the following error appears:

runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524

Workaround: Increase the value of net.core.bpf_jit_limit as described in the Red Hat Knowledgebase solution Pods failing with error loading seccomp filter into kernel: errno 524 in OpenShift 4.

Content from github.com is not included.KUBEFLOW-177 - Bearer token from application not forwarded by OAuth-proxy

You cannot use an application as a custom workbench image if its internal authentication mechanism is based on a bearer token. The OAuth-proxy configuration removes the bearer token from the headers, and the application cannot work properly.

Workaround: None.

This content is not included.RHOAIENG-1199 (previously documented as Content from github.com is not included.ODH-DASHBOARD-1928 - Custom serving runtime creation error message is unhelpful

When you try to create or edit a custom model-serving runtime and an error occurs, the error message does not indicate the cause of the error.

Example error message: Request failed with status code 422

Workaround: Check the YAML code for the serving runtime to identify the reason for the error.

Content from github.com is not included.ODH-DASHBOARD-1991 - ovms-gpu-ootb is missing recommended accelerator annotation

When you add a model server to your project, the Serving runtime list does not show the Recommended serving runtime label for the NVIDIA GPU.

Workaround: Make a copy of the model-server template and manually add the label.

This content is not included.RHODS-12717 - Pipeline server creation might fail on OpenShift with Open Virtual Network on OpenStack

When you try to create a pipeline server on OpenShift with Open Virtual Network on OpenStack, the creation might fail with a Pipeline server failed error. See This content is not included.OCPBUGS-22251.

This content is not included.RHODS-12899 - OpenVINO runtime missing annotation for NVIDIA GPUs

Red Hat OpenShift AI currently includes an out-of-the-box serving runtime that supports NVIDIA GPUs: OpenVINO model server (support GPUs). You can use the accelerator profile feature introduced in OpenShift AI 2.4 to select a specific accelerator in model serving, based on configured accelerator profiles. If the cluster had NVIDIA GPUs enabled in an earlier OpenShift AI release, the system automatically creates a default NVIDIA accelerator profile during upgrade to OpenShift AI 2.4. However, the OpenVINO model server (supports GPUs) runtime has not been annotated to indicate that it supports NVIDIA GPUs. Therefore, if a user selects the OpenVINO model server (supports GPUs) runtime and selects an NVIDIA GPU accelerator in the model server user interface, the system displays a warning that the selected accelerator is not compatible with the selected runtime. In this situation, you can ignore the warning.

This content is not included.RHOAIENG-642 (previously documented as RHODS-12903) - Successfully-submitted Elyra pipeline fails to run

If you use a private TLS certificate, and you successfully submit an Elyra-generated pipeline against the data science pipeline server, the pipeline steps fail to execute, and the following error messages are shown:

File "/opt/app-root/src/bootstrapper.py", line 747, in <module>
main()
File "/opt/app-root/src/bootstrapper.py", line 730, in main
Actions
...
WARNING: Retrying (Retry (total-4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection obj
In this situation, a new runtime image should be created, to include the correct CA bundle, as well as all the required pip packages.

Workaround: Contact Red Hat Support for detailed steps to resolve this issue.

This content is not included.RHOAIENG-637 (previously documented as RHODS-12904) - Pipeline submitted from Elyra might fail when using private certificate

If you use a private TLS certificate, and you submit a pipeline from Elyra, the pipeline might fail with a certificate verify failed error message. This issue might be caused by either or both of the following situations:

The object storage used for the pipeline server is using private TLS certificates.
The data science pipeline server API endpoint is using private TLS certificates.

Workaround: Provide the workbench with the correct Certificate Authority (CA) bundle, and set various environment variables so that the correct CA bundle is recognized. Contact Red Hat Support for detailed steps to resolve this issue.

This content is not included.RHODS-12906 - Cannot use ModelMesh with object storage that uses private certificates

Sometimes, when you store models in an object storage provider that uses a private TLS certificate, the model serving pods fail to pull files from the object storage, and the signed by unknown authority error message is shown.

Workaround: Manually update the secret created by the data connection so that the secret includes the correct CA bundle. Contact Red Hat Support for detailed steps to resolve this issue.

This content is not included.RHODS-12937 - Previously deployed model server might no longer work after upgrade in disconnected environment

In disconnected environments, after upgrade to Red Hat OpenShift AI 3.4, previously deployed model servers might no longer work. The model status might be incorrectly reported as OK on the dashboard.

Workaround

Update the inferenceservices resource to replace the storage section with the storageUri section. In the following instructions, replace <placeholders> with the values for your environment.

Remove the storage parameter section from the existing inferenceservices resource:

"storage":
      "key": "<your_key>",
      "path": "<your_path>"

Example:

"storage":
      "key": "aws-connection-minio-connection",
      "path": "mnist-8.onnx"

Add the storageUri section to the inferenceservices resource, with the specified format s3://bucket-name/path/to/object, as shown in the following example:
Example:
```
storageUri: 's3://bucket/mnist-8.onnx'
```

Capture the secret key name as follows:

secret_key=$(oc get secret -n <project_name> | grep -i aws-connection | awk '{print $1}')

Update the annotation as follows:

oc annotate $(oc get inferenceservices -n <project_name> -o name) -n <project_name> serving.kserve.io/secretKey="$secret_key"

This content is not included.RHOAIENG-673 (previously documented as RHODS-12946) - Cannot install from PyPI mirror in disconnected environment or when using private certificates

In disconnected environments, Red Hat OpenShift AI cannot connect to the public-facing PyPI repositories, so you must specify a repository inside your network. If you are using private TLS certificates, and a data science pipeline is configured to install Python packages, the pipeline run fails.

Workaround: Add the required environment variables and certificates to your pipeline, as described in the Red Hat Knowledgebase solution Install packages from PyPI Mirror fails on data science pipelines in disconnected installation.

This content is not included.RHOAIENG-12 - Cannot access Ray dashboard from some browsers

In some browsers, users of the distributed workloads feature might not be able to access the Ray dashboard, because the browser automatically changes the prefix of the dashboard URL from http to https. The distributed workloads feature is currently available in Red Hat OpenShift AI as a Technology Preview feature. See Technology Preview features.

Workaround: Change the URL prefix from https to http.

This content is not included.RHOAIENG-1666 (previously documented as Content from github.com is not included.DATA-SCIENCE-PIPELINES-OPERATOR-349) - The Import Pipeline button is prematurely accessible

When you import a pipeline to a workbench that belongs to a data science project, the Import Pipeline button is prematurely accessible before the pipeline server is fully available.

Workaround: Refresh your browser page and import the pipeline again.

This content is not included.RHOAIENG-5646 (previously documented as Content from github.com is not included.NOTEBOOKS-218) - Data science pipelines saved from the Elyra pipeline editor reference an incompatible runtime

When you save a pipeline in the Elyra pipeline editor with the format .pipeline in OpenShift AI version 1.31 or earlier, the pipeline references a runtime that is incompatible with OpenShift AI version 1.32 or later.

As a result, the pipeline fails to run after you upgrade OpenShift AI to version 1.32 or later.

Workaround: After you upgrade to OpenShift AI to version 1.32 or later, select the relevant runtime images again.

Content from github.com is not included.DATA-SCIENCE-PIPELINES-OPERATOR-362 - Pipeline server fails that uses object storage signed by an unknown authority

Data science pipeline servers fail if you use object storage signed by an unknown authority. As a result, you cannot currently use object storage with a self-signed certificate. This issue has been observed in a disconnected environment.

Workaround: Configure your system to use object storage with a self-signed certificate, as described in the Red Hat Knowledgebase solution Data Science Pipelines workaround for an object storage connection with a self-signed certificate.

This content is not included.RHOAIENG-548 (previously documented as Content from github.com is not included.ODH-DASHBOARD-1776) - Error messages when user does not have project administrator permission

If you do not have administrator permission for a project, you cannot access some features, and the error messages do not explain why. For example, when you create a model server in an environment where you only have access to a single namespace, an Error creating model server error message appears. However, the model server is still successfully created.

This content is not included.RHOAIENG-1210 (previously documented as Content from github.com is not included.ODH-DASHBOARD-1699) - Workbench does not automatically restart for all configuration changes

When you edit the configuration settings of a workbench, a warning message appears stating that the workbench will restart if you make any changes to its configuration settings. This warning is misleading because in the following cases, the workbench does not automatically restart:

Edit name
Edit description
Edit, add, or remove keys and values of existing environment variables

Workaround: Manually restart the workbench.

This content is not included.RHOAIENG-1208 (previously documented as Content from github.com is not included.ODH-DASHBOARD-1741) - Cannot create a workbench whose name begins with a number

If you try to create a workbench whose name begins with a number, the workbench does not start.

Workaround: Delete the workbench and create a new one with a name that begins with a letter.

This content is not included.RHOAIENG-1205 (previously documented as RHODS-11791) - Usage data collection is enabled after upgrade

If you previously had the Allow collection of usage data option deselected (that is, disabled), this option becomes selected (that is, enabled) when you upgrade OpenShift AI.

Workaround

Manually reset the Allow collection of usage data option. To do this, perform the following actions:

In the OpenShift AI dashboard, in the left menu, click Settings → Cluster settings.
The Cluster Settings page opens.
In the Usage data collection section, deselect Allow collection of usage data.
Click Save changes.

Content from github.com is not included.KUBEFLOW-157 - Logging out of JupyterLab does not work if you are already logged out of the OpenShift AI dashboard

If you log out of the OpenShift AI dashboard before you log out of JupyterLab, then logging out of JupyterLab is not successful. For example, if you know the URL for a Jupyter notebook, you are able to open this again in your browser.

Workaround: Log out of JupyterLab before you log out of the OpenShift AI dashboard.

Content from github.com is not included.DATA-SCIENCE-PIPELINES-OPERATOR-294 - Scheduled pipeline run that uses data-passing might fail to pass data between steps, or fail the step entirely

A scheduled pipeline run that uses an S3 object store to store the pipeline artifacts might fail with an error such as the following:

Bad value for --endpoint-url "cp": scheme is missing. Must be of the form http://<hostname>/ or https://<hostname>/

This issue occurs because the S3 object store endpoint is not successfully passed to the pods for the scheduled pipeline run.

Workaround

Depending on the size of the pipeline artifacts being passed, you can either partially or completely work around this issue by applying a custom artifact-passing script and then restarting the pipeline server. Specifically, this workaround results in the following behavior:

For pipeline artifacts smaller than 3 kilobytes, the pipeline run now successfully passes the artifacts into your S3 object store.
For pipeline artifacts larger than 3 kilobytes, the pipeline run still does not pass the artifacts into your S3 object store. However, the workaround ensures that the run continues to completion. Any smaller artifacts in the remainder of the pipeline run are successfully stored.

To apply this workaround, perform the following actions:

In a text editor, paste the following YAML-based artifact-passing script. The script defines a ConfigMap object.

apiVersion: v1
data:
  artifact_script: |-
  #!/usr/bin/env sh
    push_artifact() {
       workspace_dir=$(echo $(context.taskRun.name) | sed -e "s/$(context.pipeline.name)-//g")
        workspace_dest=/workspace/${workspace_dir}/artifacts/$(context.pipelineRun.name)/$(context.taskRun.name)
        artifact_name=$(basename $2)
        if [ -f "$workspace_dest/$artifact_name" ]; then
            echo sending to: ${workspace_dest}/${artifact_name}
            tar -cvzf $1.tgz -C ${workspace_dest} ${artifact_name}
            aws s3 --endpoint <Endpoint> cp $1.tgz s3://<Bucket>/artifacts/$PIPELINERUN/$PIPELINETASK/$1.tgz
        elif [ -f "$2" ]; then
            tar -cvzf $1.tgz -C $(dirname $2) ${artifact_name}
            aws s3 --endpoint <Endpoint> cp $1.tgz s3://<Bucket>/artifacts/$PIPELINERUN/$PIPELINETASK/$1.tgz
        else
            echo "$2 file does not exist. Skip artifact tracking for $1"
        fi
    }
    push_log() {
        cat /var/log/containers/$PODNAME*$NAMESPACE*step-main*.log > step-main.log
        push_artifact main-log step-main.log
    }
    strip_eof() {
        if [ -f "$2" ]; then
            awk 'NF' $2 | head -c -1 > $1_temp_save && cp $1_temp_save $2
        fi
    }
kind: ConfigMap
metadata:
  name: custom-script

In the script, replace any occurrences of <Endpoint> with your S3 endpoint (for example, Content from s3.amazonaws.com is not included.https://s3.amazonaws.com), and occurrences of <Bucket> with your S3 bucket name.
Save the YAML file for the ConfigMap object.
Apply the YAML file.
```
$ oc apply -f <configmap_file_name>.yaml
```

Restart the pipeline server.

$ oc project <project_name>
$ oc delete pod $(oc get pods  -l app=ds-pipeline-pipelines-definition --no-headers | awk {print $1})

This content is not included.RHODS-9789 - Pipeline servers fail to start if they contain a custom database that includes a dash in its database name or username field

When you create a pipeline server that uses a custom database, if the value that you set for the dbname field or username field includes a dash, the pipeline server fails to start.

Workaround: Edit the pipeline server to omit the dash from the affected fields.

This content is not included.RHODS-9764 - Data connection details get reset when editing a workbench

When you edit a workbench that has an existing data connection and then select the Create new data connection option, the edit page might revert to the Use existing data connection option before you have finished specifying the new connection details.

Workaround

To work around this issue, perform the following actions:

Select the Create new data connection option again.
Specify the new connection details and click Update workbench before the page reverts to the Use existing data connection option.

This content is not included.RHODS-9030 - Uninstall process for OpenShift AI might become stuck when removing kfdefs resources

The steps for uninstalling OpenShift AI self-managed are described in Uninstalling OpenShift AI self-managed.

However, even when you follow this guide, you might see that the uninstall process does not finish successfully. Instead, the process stays on the step of deleting kfdefs resources that are used by the Kubeflow Operator. As shown in the following example, kfdefs resources might exist in the redhat-ods-applications, redhat-ods-monitoring, and rhods-notebooks namespaces:

$ oc get kfdefs.kfdef.apps.kubeflow.org -A

NAMESPACE                  NAME                                   AGE
redhat-ods-applications    rhods-anaconda                         3h6m
redhat-ods-applications    rhods-dashboard                        3h6m
redhat-ods-applications    rhods-data-science-pipelines-operator  3h6m
redhat-ods-applications    rhods-model-mesh                       3h6m
redhat-ods-applications    rhods-nbc                              3h6m
redhat-ods-applications    rhods-osd-config                       3h6m
redhat-ods-monitoring      modelmesh-monitoring                   3h6m
redhat-ods-monitoring      monitoring                             3h6m
rhods-notebooks            rhods-notebooks                        3h6m
rhods-notebooks            rhods-osd-config                       3h5m

Failed removal of the kfdefs resources might also prevent later installation of a newer version of OpenShift AI.

Workaround: To manually delete the kfdefs resources so that you can complete the uninstall process, see the "Force individual object removal when it has finalizers" section of the Red Hat Knowledgebase solution This content is not included.Unable to Delete a Project or Namespace in OCP.

This content is not included.RHODS-8939 - For a Jupyter notebook created in a previous release, default shared memory might cause a runtime error

For a Jupyter notebook created in a release earlier than 1.31, the default shared memory for a Jupyter notebook is set to 64 MB and you cannot change this default value in the notebook configuration.

For example, PyTorch relies on shared memory and the default size of 64 MB is not enough for large use cases, such as training a model or performing heavy data manipulations. Jupyter reports a "no space left on device" message and /dev/smh is full.

Starting with release 1.31, this issue is fixed and any new notebook’s shared memory is set to the size of the node.

Workaround

For a Jupyter notebook created in a release earlier than 1.31, either recreate the Jupyter notebook or follow these steps:

In your data science project, create a workbench.
In the data science project page, in the Workbenches section, click the Status toggle for the workbench to change it from Running to Stopped.
Open your OpenShift Console and then select Administrator.
Select Home → API Explorer.
In the Filter by kind field, type notebook.
Select the kubeflow v1 notebook.
Select the Instances tab and then select the instance for the workbench that you created in Step 1.
Click the YAML tab and then select Actions → Edit Notebook.
Edit the YAML file to add the following information to the configuration:
- For the container that has the name of your Workbench notebook, add the following lines to the volumeMounts section:
```
- mountPath: /dev/shm
  name: shm
```
  For example, if your workbench name is myworkbench, update the YAML file as follows:
```
spec:
    containers:
      - env
        ...
        name: myworkbench
        ...
         volumeMounts:
         - mountPath: /dev/shm
           name: shm
```
- In the volumes section, add the lines shown in the following example:
```
     volumes:
       name: shm
       emptyDir:
         medium: Memory
```
  Note: Optionally, you can specify a limit to the amount of memory to use for the emptyDir.
Click Save.
In the data science dashboard, in the Workbenches section of the data science project, click the Status toggle for the workbench. The status changes from Stopped to Starting and then Running.
Restart the notebook.

Warning

If you later edit the notebook’s configuration through the Data Science dashboard UI, your workaround edit to the notebook configuration will be erased.

This content is not included.RHOAIENG-583 (previously documented as This content is not included.RHODS-8921 and This content is not included.RHODS-6373) - You cannot create a pipeline server or start a workbench when cumulative character limit is exceeded

When the cumulative character limit of a data science project name and a pipeline server name exceeds 62 characters, you are unable to successfully create a pipeline server. Similarly, when the cumulative character limit of a data science project name and a workbench name exceeds 62 characters, workbenches fail to start.

Workaround: Rename your data science project so that it does not exceed 30 characters.

This content is not included.RHODS-8865 - A pipeline server fails to start unless you specify an Amazon Web Services (AWS) Simple Storage Service (S3) bucket resource

When you create a data connection for a data science project, the AWS_S3_BUCKET field is not designated as a mandatory field. However, if you do not specify a value for this field, and you attempt to configure a pipeline server, the pipeline server fails to start successfully.

This content is not included.RHODS-7718 - User without dashboard permissions is able to continue using their running workbenches indefinitely

When a Red Hat OpenShift AI administrator revokes a user’s permissions, the user can continue to use their running workbenches indefinitely.

Workaround: When the OpenShift AI administrator revokes a user’s permissions, the administrator should also stop any running workbenches for that user.

This content is not included.RHODS-6907 - Attempting to increase the size of a Persistent Volume (PV) fails when it is not connected to a workbench

Attempting to increase the size of a Persistent Volume (PV) that is not connected to a workbench fails. When changing a data science project’s storage, users can still edit the size of the PV in the user interface, but this action does not have any effect.

This content is not included.RHODS-6950 - Unable to scale down a workbench’s GPUs when all GPUs in the cluster are being used

It is not possible to scale down a workbench’s GPUs if all GPUs in the cluster are being used. This issue applies to GPUs being used by one workbench, and GPUs being used by multiple workbenches.

Workaround

To workaround around this issue, perform the following steps:

Stop all active workbenches that are using GPUs.
Wait until the relevant GPUs are available again.
Edit the workbench and scale down the GPU instances.

This content is not included.RHODS-6539 - Anaconda Professional Edition cannot be validated and enabled in OpenShift AI

Anaconda Professional Edition cannot be enabled as the dashboard’s key validation for Anaconda Professional Edition is inoperable.

This content is not included.RHODS-6346 - Unclear error message displays when using invalid characters to create a data science project

When creating a data science project’s data connection, workbench, or storage connection using invalid special characters, the following error message is displayed:

the object provided is unrecognized (must be of type Secret): couldn't get version/kind; json parse error: unexpected end of JSON input ({"apiVersion":"v1","kind":"Sec ...)

The error message fails to clearly indicate the problem.

This content is not included.RHOAIENG-1157 (previously documented as This content is not included.RHODS-6955) - An error can occur when trying to edit a workbench

When editing a workbench, an error similar to the following can occur:

Error creating workbench
Operation cannot be fulfilled on notebooks.kubeflow.org "workbench-name": the object has been modified; please apply your changes to the latest version and try again

Workaround: None.

This content is not included.RHODS-6913 - When editing the configuration settings of a workbench, a misleading error message appears

When you edit the configuration settings of a workbench, a warning message appears stating the workbench will restart if you make any changes to its configuration settings. This warning is misleading, as if you change the values of its environment variables, the workbench does not automatically restart.

This content is not included.RHODS-6373 - Workbenches fail to start when cumulative character limit is exceeded

When the cumulative character limit of a data science project’s title and workbench title exceeds 62 characters, workbenches fail to start.

This content is not included.RHODS-6216 - The ModelMesh oauth-proxy container is intermittently unstable

ModelMesh pods do not deploy correctly due to a failure of the ModelMesh oauth-proxy container. This issue occurs intermittently and only if authentication is enabled in the ModelMesh runtime environment. It is more likely to occur when additional ModelMesh instances are deployed in different namespaces.

This content is not included.RHODS-5906 - The NVIDIA GPU Operator is incompatible with OpenShift 4.11.12

Provisioning a GPU node on a OpenShift 4.11.12 cluster results in the nvidia-driver-daemonset pod getting stuck in a CrashLoopBackOff state. The NVIDIA GPU Operator is compatible with OpenShift 4.11.9 and 4.11.13. In addition, the minimum version of OpenShift required for an installation of OpenShift AI is 4.19.

This content is not included.RHODS-5763 - Incorrect package version displayed during notebook selection

The Start a notebook server page displays an incorrect version number for the Anaconda notebook image.

Workaround: None.

This content is not included.RHODS-5543 - When using the NVIDIA GPU Operator, more nodes than needed are created by the Node Autoscaler

When a pod cannot be scheduled due to insufficient available resources, the Node Autoscaler creates a new node. There is a delay until the newly created node receives the relevant GPU workload. Consequently, the pod cannot be scheduled and the Node Autoscaler’s continuously creates additional new nodes until one of the nodes is ready to receive the GPU workload. For more information about this issue, see the Red Hat Knowledgebase solution When using the NVIDIA GPU Operator, more nodes than needed are created by the Node Autoscaler.

Workaround: Apply the cluster-api/accelerator label in machineset.spec.template.spec.metadata. This causes the autoscaler to consider those nodes as unready until the GPU driver has been deployed.

This content is not included.RHOAIENG-1149 (previously documented RHODS-5216) - The application launcher menu incorrectly displays a link to OpenShift Cluster Manager

Red Hat OpenShift AI incorrectly displays a link to the OpenShift Cluster Manager from the application launcher menu. Clicking this link results in a "Page Not Found" error because the URL is not valid.

Workaround: None.

This content is not included.RHOAIENG-1137 (previously documented as RHODS-5251) - Administration page for basic workbenches shows users who have lost permission access

If a user who previously started a basic workbench loses their permissions to do so (for example, if an OpenShift AI administrator changes the user’s group settings or removes the user from a permitted group), administrators continue to see the user’s basic workbench on the Administration page. As a consequence, an administrator is able to restart a basic workbench that belongs to a user whose permissions were revoked.

Workaround: None.

This content is not included.RHODS-4769 - GPUs on nodes with unsupported taints cannot be allocated to notebook servers

GPUs on nodes marked with any taint other than the supported nvidia.com/gpu taint cannot be selected when creating a notebook server. To avoid this issue, use only the nvidia.com/gpu taint on GPU nodes used with OpenShift AI.

This content is not included.RHODS-4799 - Tensorboard requires manual steps to view

When a user has TensorFlow or PyTorch workbench images and wants to use TensorBoard to display data, manual steps are necessary to include environment variables in the workbench environment, and to import those variables for use in your code.

Workaround

When you start your basic workbench, use the following code to set the value for the TENSORBOARD_PROXY_URL environment variable to use your OpenShift AI user ID.

import os
os.environ["TENSORBOARD_PROXY_URL"]= os.environ["NB_PREFIX"]+"/proxy/6006/"

This content is not included.RHODS-4718 - The Intel® oneAPI AI Analytics Toolkits quick start references nonexistent sample notebooks

The Intel® oneAPI AI Analytics Toolkits quick start, located on the Resources page on the dashboard, requires the user to load sample notebooks as part of the instruction steps, but refers to notebooks that do not exist in the associated repository.

Workaround: None.

This content is not included.RHODS-4627 - The CronJob responsible for validating Anaconda Professional Edition’s license is suspended and does not run daily

The CronJob responsible for validating Anaconda Professional Edition’s license is automatically suspended by the OpenShift AI operator. As a result, the CronJob does not run daily as scheduled. In addition, when Anaconda Professional Edition’s license expires, Anaconda Professional Edition is not indicated as disabled on the OpenShift AI dashboard.

Workaround: None.

This content is not included.RHOAIENG-1141 (previously documented as RHODS-4502) - The NVIDIA GPU Operator tile on the dashboard displays button unnecessarily

GPUs are automatically available in Jupyter after the NVIDIA GPU Operator is installed. The Enable button, located on the NVIDIA GPU Operator tile on the Explore page, is therefore redundant. In addition, clicking the Enable button moves the NVIDIA GPU Operator tile to the Enabled page, even if the Operator is not installed.

Workaround: None.

This content is not included.RHOAIENG-1135 (previously documented as RHODS-3985) - Dashboard does not display Enabled page content after ISV operator uninstall

After an ISV operator is uninstalled, no content is displayed on the Enabled page on the dashboard. Instead, the following error is displayed:

Error loading components
HTTP request failed

Workaround: Wait 30-40 seconds and then refresh the page in your browser.

This content is not included.RHODS-3984 - Incorrect package versions displayed during notebook selection

In the OpenShift AI interface, the Start a notebook server page displays incorrect version numbers for the JupyterLab and Notebook packages included in the oneAPI AI Analytics Toolkit notebook image. The page might also show an incorrect value for the Python version used by this image.

Workaround: When you start your oneAPI AI Analytics Toolkit notebook server, you can check which Python packages are installed on your notebook server and which version of the package you have by running the !pip list command in a notebook cell.

This content is not included.RHODS-2956 - Error can occur when creating a notebook instance

When creating a notebook instance in Jupyter, a Directory not found error appears intermittently. This error message can be ignored by clicking Dismiss.

Workaround: None.

This content is not included.RHOAING-1147 (previously documented as RHODS-2881) - Actions on dashboard not clearly visible

The dashboard actions to revalidate a disabled application license and to remove a disabled application tile are not clearly visible to the user. These actions appear when the user clicks on the application tile’s Disabled label. As a result, the intended workflows might not be clear to the user.

Workaround: None.

This content is not included.RHOAIENG-1134 (previously documented as RHODS-2879) - License revalidation action appears unnecessarily

The dashboard action to revalidate a disabled application license appears unnecessarily for applications that do not have a license validation or activation system. In addition, when a user attempts to revalidate a license that cannot be revalidated, feedback is not displayed to state why the action cannot be completed.

Workaround: None.

This content is not included.RHOAIENG-2305 (previously documented as RHODS-2650) - Error can occur during Pachyderm deployment

When creating an instance of the Pachyderm operator, a webhook error appears intermittently, preventing the creation process from starting successfully. The webhook error is indicative that, either the Pachyderm operator failed a health check, causing it to restart, or that the operator process exceeded its container’s allocated memory limit, triggering an Out of Memory (OOM) kill.

Workaround: Repeat the Pachyderm instance creation process until the error no longer appears.

This content is not included.RHODS-2096 - IBM Watson Studio not available in OpenShift AI

IBM Watson Studio is not available when OpenShift AI is installed on OpenShift Dedicated 4.9 or higher, because it is not compatible with these versions of OpenShift Dedicated.

Workaround: Contact the Red Hat Customer Portal for assistance with manually configuring Watson Studio on OpenShift Dedicated 4.9 and higher.

Chapter 8. Product features

Red Hat OpenShift AI provides a rich set of features for data scientists and cluster administrators. To learn more, see This content is not included.Introduction to Red Hat OpenShift AI.

Legal Notice

Except as otherwise noted below, the text of and illustrations in this documentation are licensed by Red Hat under the Creative Commons Attribution–Share Alike 3.0 Unported license . If you distribute this document or an adaptation of it, you must provide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, the Red Hat logo, JBoss, Hibernate, and RHCE are trademarks or registered trademarks of Red Hat, LLC. or its subsidiaries in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

XFS is a trademark or registered trademark of Hewlett Packard Enterprise Development LP or its subsidiaries in the United States and other countries.

The OpenStack® Word Mark and OpenStack logo are trademarks or registered trademarks of the Linux Foundation, used under license.

All other trademarks are the property of their respective owners.

Release Notes

Features, enhancements, resolved issues, and known issues associated with this release

Chapter 1. Overview of OpenShift AI

Chapter 2. New features and enhancements

2.1. New features

2.1.1. 3.4 GA new features

2.1.2. 3.4 EA2 new features

2.1.3. 3.4 EA1 new features

2.2. Enhancements

2.2.1. 3.4 GA enhancements

2.2.2. 3.4 EA2 enhancements

2.2.3. 3.4 EA1 enhancements

Chapter 3. Technology Preview features

3.1. 3.4 GA Technology Preview features

3.2. 3.4 EA2 Technology Preview features

3.3. 3.4 EA1 Technology Preview features

Chapter 4. Developer Preview features

4.1. 3.4 GA Developer Preview features

4.2. 3.4 EA2 Developer Preview features

4.3. 3.4 EA1 Developer Preview features

Chapter 5. Support removals

5.1. Deprecated

5.1.1. Ray-based multi-node vLLM template

5.1.2. Training images and ClusterTrainingRuntimes for Kubeflow Training Operator v1

5.1.3. Deprecated SQLite as a production metadata store for Llama Stack

5.1.4. Deprecated annotation format for Connection Secrets::

5.1.5. Deprecated Kubeflow Training operator v1

5.1.6. Deprecated TrustyAI service CRD v1alpha1

5.1.7. Deprecated KServe Serverless deployment mode

5.1.8. Deprecated model registry API v1alpha1

5.1.9. Multi-model serving platform (ModelMesh)

5.1.10. Accelerator Profiles and legacy Container Size selector deprecated

5.1.11. Deprecated OpenVINO Model Server (OVMS) plugin

5.1.12. OpenShift AI dashboard user management moved from OdhDashboardConfig to Auth resource

5.1.13. Deprecated cluster configuration parameters

5.2. Removed functionality

5.2.1. CodeFlare Operator removed

5.2.2. Microsoft SQL Server command-line tool removal

5.2.3. Model registry ML Metadata (MLMD) server removal

5.2.4. Embedded subscription channel not used in some versions

5.2.5. Anaconda removal

5.2.6. Pipeline logs for Python scripts running in Elyra pipelines are no longer stored in S3

5.2.7. Beta subscription channel no longer used

5.2.8. HabanaAI workbench image removal

Chapter 6. Resolved issues

6.1. Security updates in Red Hat OpenShift AI 3.4.1 (June 2026)

6.2. Issues resolved in Red Hat OpenShift AI 3.4 EA1

6.3. Issues resolved in Red Hat OpenShift AI 3.3

6.4. Issues resolved in Red Hat OpenShift AI 3.2

Chapter 7. Known issues

7.1. Issues discovered at version 3.4 GA

7.2. Issues discovered at version 3.4 EA2

7.3. Issues discovered at version 3.4 EA1

Chapter 8. Product features

Legal Notice

5.1.12. OpenShift AI dashboard user management moved from `OdhDashboardConfig` to `Auth` resource