AMD ROCm MI300X memory access faults during CUDA graph capture are now resolved.

Previously, when running models such as meta-llama/Llama-3.2-1B-Instruct or ibm-granite/granite-3.2-2b-instruct on AMD MI300X accelerators, GPU memory access faults occurred with the error "Memory access fault by GPU node-2". This issue is now resolved.

AMD Zen CPU deployments no longer crash due to zentorch plugin incompatibility.

Previously, the zentorch vLLM plugin supported vLLM versions 0.15.0 through 0.18.0, which was incompatible with the bundled vLLM version. The engine crashed during model loading with a segfault. This issue is now resolved.

Gemma 4 12B model now deploys successfully.

Previously, the google/gemma-4-12B-it model failed to deploy because the bundled Transformers version did not support the gemma4_unified architecture. This issue is now resolved.

Mistral Small FP8-dynamic now starts successfully.

Previously, the Mistral Small FP8-dynamic model failed to start with an error indicating that CachedMistralCommonBackend was missing the is_fast attribute. This issue is now resolved.

Granite 4.0-h-small model no longer crashes on first inference with ROCm.

Previously, when deploying the ibm-granite/granite-4.0-h-small model on AMD ROCm accelerators, the vLLM server crashed during the first inference request due to a Triton JIT compilation failure. This issue is now resolved.

Qwen 3.5 and 3.6 models now work on CPU deployments.

Previously, Qwen 3.5 and 3.6 models failed to run on CPU-only deployments due to an upstream vLLM issue with GDN attention operations. This issue is now resolved.

Prefix caching with models requiring transformers 5.x no longer crashes the inference scheduler.

Previously, deploying models whose architecture was introduced in transformers 5.0 or later with prefix caching enabled (precise-prefix-cache-scorer) caused the inference scheduler to crash. The UDS tokenizer sidecar is replaced by a vLLM render sidecar that stays in sync with the serving image’s transformers version. This issue is now resolved.

Gemma4 multimodal models produce incoherent output with FlashAttention-4 on H100/H200 GPUs.

When serving Gemma4 multimodal models on NVIDIA H100 or H200 GPUs, the model produces incoherent output when processing image inputs with contexts longer than 1024 tokens. This issue affects the 12B, 26B-A4B, and 31B model sizes. FlashAttention-4, the default attention backend for Gemma4 on NVIDIA SM90+ GPUs, has incorrect sliding-window mask handling for multimodal attention in vLLM v0.24.0.

To work around this issue, use the Triton attention backend:

--attention-backend TRITON_ATTN

Mamba models might crash on AMD ROCm due to a Triton compiler bug.

When running Mamba-based models on AMD ROCm accelerators, the Triton ROCm compiler might fail with an unrecoverable assertion error during JIT compilation of the _chunk_scan_fwd_kernel kernel. This issue is triggered by new tensor shapes during inference that were not warmed at startup.

Qwen2.5-VL models crash during encoder profiling on AMD ROCm.

The Qwen/Qwen2.5-VL-7B-Instruct and RedHatAI/Qwen2.5-VL-7B-Instruct-FP8-Dynamic models crash with a hipErrorInvalidConfiguration error during VL encoder profiling on ROCm 7.14.

To work around this issue, disable CUDA graph capture:

--enforce-eager true

Qwen3.6-FP8 produces empty streaming output on AMD ROCm.

The RedHatAI/Qwen3.6-35B-A3B-FP8-dynamic model produces empty streaming output when FP8 quantization is combined with CUDA graph capture on ROCm.

To work around this issue, disable CUDA graph capture:

--enforce-eager true

Gemma 4 MoE models crash on AMD ROCm due to unsupported activation function.

The google/gemma-4-26B-A4B-it and RedHatAI/gemma-4-26B-A4B-it-FP8-Dynamic models crash because the aiter MoE kernel does not support the GELU_TANH activation function on ROCm.

To work around this issue, use the Triton MoE backend:

--moe-backend triton

Mistral-Small-3.1 FP8-dynamic returns invalid output on image inputs.

The RedHatAI/Mistral-Small-3.1-24B-Instruct-2503-FP8-dynamic model returns NaN logits when processing image inputs on CUDA. Text, chat, tool calling, and structured output requests are not affected. The bf16 base model is also not affected.

To work around this issue, disable graph compilation:

--enforce-eager true

Kimi-K2.7-Code model fails to start due to non-writable tiktoken cache directory.

The RedHatAI/Kimi-K2.7-Code model fails to start with a PermissionError when the tokenizer attempts to write to the /opt/app-root/tiktoken-cache/ directory. This issue occurs in rootless container environments where the cache directory is not writable.

To work around this issue, set the TIKTOKEN_CACHE_DIR environment variable to a writable directory:

TIKTOKEN_CACHE_DIR=/tmp/tiktoken

Prefix cache hit rate is overcounted when batching requests on IBM Spyre.

When multiple requests are batched, the scheduler records prefix cache hits while checking scheduling constraints. This inflates the reported prefix cache hit rate. Inference results and scheduling behavior are not affected. No known workaround exists.

Qwen3.6-35B-A3B produces corrupted output after the first inference request.

When serving the Qwen3.6-35B-A3B model, only the first request returns valid output. All subsequent requests return nonsensical results. No known workaround exists.

Phi-4 Vision model fails to load with SigLIP2 compatibility error.

The RedHatAI/Phi-4-reasoning-vision-15B model requires a filter_out_non_signature_kwargs() function that is missing from the transformers library bundled with vLLM. The vLLM API server crashes at startup and cannot serve the model.

Prithvi TerraTorch model warmup fails on arm64 GH200 due to tensor size limit.

When deploying the ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11 model on arm64 NVIDIA GH200 accelerators, the vLLM engine core fails during warmup with the error "upsample_bilinear2d_nhwc only supports output tensors with less than INT_MAX elements". A PyTorch CUDA kernel limitation on arm64 causes this issue when the TerraTorch UPerNet decoder warmup tensor exceeds the signed 32-bit integer maximum.

To work around this issue, deploy the model with limited concurrency:

--max-num-seqs=8 --max-num-batched-tokens=8

This configuration does not affect performance for geospatial models because each request uses only one token.

DeepSeek V4 model variants require explicit KV cache data type configuration.

DeepSeek V4 model variants such as Flash and Pro fail to load with a KeyError for scale_fmt in the quantization configuration.

To work around this issue, set the --kv-cache-dtype flag to fp8 when serving the model:

vllm serve deepseek-ai/DeepSeek-V4-Flash --kv-cache-dtype fp8

Mistral Small 3.1 tool calling returns no tool calls with tool_choice=auto.

When using the Mistral Small 3.1 model with tool calling and tool_choice=auto, the model does not return tool calls in the response.

Qwen2.5-VL FP8 tool calling fails because the model does not support tools.

The Qwen2.5-VL FP8 model fails tool calling smoke tests because the model architecture does not support tool calling functionality.

Gemma 4 12B model evaluation crashes due to batch token limit.

When running LM Evaluation Harness on the Gemma 4 12B model, the evaluation crashes with an error indicating that max_num_batched_tokens is too small for the prefix-LM multimodal model.

To work around this issue, explicitly set --max-num-batched-tokens to any value greater than 2496:

--max-num-batched-tokens 4096

Intel Gaudi deployments fail due to habana-torch-plugin version rejection.

When deploying models on Intel Gaudi accelerators, the vllm-gaudi configuration validation rejects the habana-torch-plugin version because it does not conform to PEP 440 versioning requirements.

Structured output generation on IBM Z with llguidance backend now works correctly.

Previously, when using the llguidance structured decoding backend on IBM Z (s390x), JSON Schema-constrained generation produced invalid output or became stuck generating whitespace indefinitely. This issue is now resolved.

Sarvam-105B model now loads successfully.

Previously, the RedHatAI/sarvam-105b-FP8-Dynamic model failed to load due to a RoPE validation error caused by a Transformers v5 incompatibility. This issue is now resolved.

Prithvi TerraTorch model warmup fails on arm64 GH200 due to tensor size limit.

When deploying the ibm-nasa-geospatial/Prithvi-EO-2.0-300M-TL-Sen1Floods11 model on arm64 NVIDIA GH200 accelerators, the vLLM engine core fails during warmup with the error "upsample_bilinear2d_nhwc only supports output tensors with less than INT_MAX elements". A PyTorch CUDA kernel limitation on arm64 causes this issue when the TerraTorch UPerNet decoder warmup tensor exceeds the signed 32-bit integer maximum.

To work around this issue, deploy the model with limited concurrency:

--max-num-seqs=8 --max-num-batched-tokens=8

This configuration does not affect performance for geospatial models because each request uses only one token.

DeepSeek-V4-Flash-NVFP4-FP8 model crashes during initialization.

The RedHatAI/DeepSeek-V4-Flash-NVFP4-FP8 quantized model fails to load with a KeyError for scale_fmt in the quantization configuration. The DeepseekV4Attention initialization expects a scale_fmt key that is not present in the model configuration.

Mistral Small 3.1 tool calling returns no tool calls with tool_choice=auto.

When using the Mistral Small 3.1 model with tool calling and tool_choice=auto, the model does not return tool calls in the response.

Qwen2.5-VL FP8 tool calling fails because the model does not support tools.

The Qwen2.5-VL FP8 model fails tool calling smoke tests because the model architecture does not support tool calling functionality.

Mistral Small FP8-dynamic fails to start due to missing backend attribute.

The Mistral Small FP8-dynamic model fails to start with an error indicating that CachedMistralCommonBackend is missing the is_fast attribute.

Gemma 4 12B model evaluation crashes due to batch token limit.

When running LM Evaluation Harness on the Gemma 4 12B model, the evaluation crashes with an error indicating that max_num_batched_tokens is too small for the prefix-LM multimodal model.

To work around this issue, explicitly set --max-num-batched-tokens to any value greater than 2496:

--max-num-batched-tokens 4096

Granite 4.0-h-small model crashes on first inference with ROCm.

When deploying the ibm-granite/granite-4.0-h-small model on AMD ROCm accelerators, the vLLM server crashes during the first inference request with a RuntimeError: PassManager::run failed error. This occurs because the Triton JIT compiler fails to compile Mamba chunk_scan kernels on the first run. After the pod restarts, the model works correctly due to the warm compilation cache.

Qwen 3.5 and 3.6 models fail on CPU deployments.

Qwen 3.5 and 3.6 models fail to run correctly on CPU-only deployments due to an upstream vLLM issue with GDN attention operations.

Intel Gaudi deployments fail due to habana-torch-plugin version rejection.

When deploying models on Intel Gaudi accelerators, the vllm-gaudi configuration validation rejects the habana-torch-plugin version 1.23.0.post1 because it does not conform to PEP 440 versioning requirements.

Prefix caching with models requiring transformers 5.x crashes the inference scheduler.

Deploying models whose architecture was introduced in transformers 5.0 or later with prefix caching enabled (precise-prefix-cache-scorer) causes the inference scheduler to crash. The tokenizer sidecar (odh-llm-d-kv-cache-rhel9) ships transformers 4.57.6, which does not recognize architecture types added in transformers 5.x. Affected models include Gemma 4, GLM-5, Qwen3.5 MoE, and others added after the 4.x series.

To work around this issue, disable precise-prefix-cache-scorer in the EndpointPickerConfig when using these models.

This issue is fixed in 3.5 stable. The UDS tokenizer sidecar is replaced by a vLLM render sidecar that stays in sync with the serving image’s transformers version.

AMD ROCm deployments on MI300X encounter memory access faults during CUDA graph capture.

When running models such as meta-llama/Llama-3.2-1B-Instruct or ibm-granite/granite-3.2-2b-instruct on AMD MI300X accelerators, GPU memory access faults occur with the error "Memory access fault by GPU node-2". This issue is caused by an incompatibility between vLLM CUDA Graph FULL mode and MI300X VF memory behavior under ROCm.

To work around this issue, set cudagraph_mode=PIECEWISE in your vLLM configuration:

--compilation-config '{"cudagraph_mode":"PIECEWISE"}'

Phi-4 Vision model fails to load with SigLIP2 compatibility error.

The RedHatAI/Phi-4-reasoning-vision-15B model requires a filter_out_non_signature_kwargs() function that is missing from the transformers library bundled with vLLM. The vLLM API server crashes at startup and cannot serve the model.

Sarvam-105B model fails to load with RoPE validation error.

The RedHatAI/sarvam-105b-FP8-Dynamic model configuration calls validate_rope(ignore_keys=…​), but the ignore_keys parameter was removed in Transformers v5. The vLLM API server crashes during model initialization.

AMD Zen CPU deployments crash due to zentorch plugin incompatibility.

The zentorch vLLM plugin supports vLLM versions 0.15.0 through 0.18.0, but Red Hat AI Inference 3.5.0-ea.1 ships vLLM 0.19.1. The engine crashes during model loading with a segfault.

To work around this issue, unset the VLLM_PLUGINS environment variable:

unset VLLM_PLUGINS

Structured output generation fails on IBM Z when using the llguidance backend.

When using the llguidance structured decoding backend on IBM Z (s390x), JSON Schema-constrained generation may produce invalid output or become stuck generating whitespace indefinitely.

Gemma 4 12B model deployment might fail due to unsupported gemma4_unified architecture in the bundled Transformers version.

Gemma 4 12B models use the gemma4_unified architecture, which is not supported by the version of Transformers included in the Red Hat AI Inference container image. Deployment of the google/gemma-4-12B-it model fails during vLLM initialization with the error "The checkpoint you are trying to load has model type gemma4_unified but Transformers does not recognize this architecture." Other Gemma 4 variants deploy successfully because they use different model architectures that are already supported.