why vllm inference fails with Nvidia B300 gpu with cuda error?

Environment

• RHELAI 3.3.
• Nvidia B300 GPU.
• RHAIIS.

Issue

• why vllm inference fails with Nvidia B300 gpu with cuda error?

>>> print(torch.cuda.is_available())
/usr/local/lib/python3.12/dist-packages/torch/cuda/__init__.py:180: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 802: system not yet initialized (Triggered internally at /opt/pytorch/pytorch/c10/cuda/CUDAFunctions.cpp:119.)

Resolution

• As workaround, disable nvlink using following command.

# echo “options nvidia NVreg_NvLinkDisable=1” >> /etc/modprobe.d/nvidia.conf

Root Cause

• The error suggests that system having multiple SXM GPUs requires NVLink/NVSwitch fabric manager service to be running, currently RHELAI does not support/provide Infiniband/RoCE driver stack or OFED/MOFED drivers including InfiniBand-related utilities which are required for fabricmanager to work.

Diagnostic Steps

• To validate nvlink, check output for nvidia-smi -s nvlink. Following snippet shows before and after comparison of applying workaround.

Without workaround
------
    Platform Info
        Chassis Serial Number                          : 
        Slot Number                                    : N/A
        Tray Index                                     : N/A
        Host ID                                        : 1
        Peer Type                                      : Switch Connected    <----
        Module Id                                      : 4
        GPU Fabric GUID                                : 0xcb81c00003666d3c



Comparing output of 'nvidia-smi -q



With workaround
-----
    Platform Info
        Chassis Serial Number                          : 
        Slot Number                                    : 0
        Tray Index                                     : 0
        Host ID                                        : 1
        Peer Type                                      : Direct Connected    <----
        Module Id                                      : 1
        GPU Fabric GUID                                : 0x0000000000000000

• Another way to verify if cuda is working is to test following containers provided by nvidia repo.

# podman run --rm --device nvidia.com/gpu=all nvcr.io/nvidia/k8s/cuda-sample:devicequery
/cuda-samples/sample Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

cudaGetDeviceCount returned 3
-> initialization error
Result = FAIL

# podman run --rm --device nvidia.com/gpu=all nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
[Vector addition of 50000 elements]
Failed to allocate device vector A (error code initialization error)!

• But after applying workaround, the same container will show following outputs.

# podman run --rm --device nvidia.com/gpu=all nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0
[Vector addition of 50000 elements]
Copy input data from the host memory to the CUDA device
CUDA kernel launch with 196 blocks of 256 threads
Copy output data from the CUDA device to the host memory
Test PASSED
Done

# podman run --rm --device nvidia.com/gpu=all nvcr.io/nvidia/k8s/cuda-sample:devicequery
/cuda-samples/sample Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 8 CUDA Capable device(s)

Device 0: "NVIDIA B300 SXM6 AC"
  CUDA Driver Version / Runtime Version          13.0 / 12.5
  CUDA Capability Major/Minor version number:    10.3
  Total amount of global memory:                 274113 MBytes (287428640768 bytes)
MapSMtoCores for SM 10.3 is undefined.  Default to use 128 Cores/SM
MapSMtoCores for SM 10.3 is undefined.  Default to use 128 Cores/SM
  (148) Multiprocessors, (128) CUDA Cores/MP:    18944 CUDA Cores
  GPU Max Clock rate:                            2032 MHz (2.03 GHz)
  Memory Clock rate:                             3996 Mhz
  Memory Bus Width:                              7680-bit
  L2 Cache Size:                                 132644864 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total shared memory per multiprocessor:        233472 bytes
... 
> Peer access from NVIDIA B300 SXM6 AC (GPU7) -> NVIDIA B300 SXM6 AC (GPU6) : Yes

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 13.0, CUDA Runtime Version = 12.5, NumDevs = 8
Result = PASS

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.