Procedure

1. In a Jupyter notebook cell or other configured client environment, run a RAG query by using the file_search tool.

   response = client.responses.create(
       model=model_id,
       input=query,
       instructions=system_instructions,
       tools=[
           {
               "type": "file_search",
               "vector_store_ids": [vector_store_id],
           }
       ],
   )

   Copy code to clipboard

2. Inspect the full response object rather than only the output_text property.

   response.output

   Copy code to clipboard

3. Access the annotations array.

   annotations = response.output[0].content[0].annotations
   print(annotations)

   Copy code to clipboard

4. Review the file_citation annotation fields.

   Example output:

   [
     {
       "type": "file_citation",
       "file_id": "file-57610eaac6364459bfefae60377837b7",
       "filename": "redbankfinancial_about.pdf",
       "index": 139
     }
   ]

   Copy code to clipboard

1. In the OpenShift AI dashboard, navigate to the project details page and click the Deployments tab.
2. In the Model serving platform tile, click Select model.

3. Click the Deploy model button.

   The Deploy model dialog opens.

4. Configure the deployment properties for your model:

   1. In the Model deployment name field, enter a unique name for your deployment.
   2. In the Serving runtime field, select vLLM NVIDIA GPU serving runtime for KServe from the drop-down list.
   3. In the Deployment mode field, select KServe RawDeployment from the drop-down list.
   4. Set Number of model server replicas to deploy to 1.

   5. In the Model server size field, select Custom from the drop-down list.

      • Set CPUs requested to 1 core.
      • Set Memory requested to 10 GiB.
      • Set CPU limit to 2 core.
      • Set Memory limit to 14 GiB.
      • Set Accelerator to NVIDIA GPUs.
      • Set Accelerator count to 1.
   6. From the Connection type, select a relevant data connection from the drop-down list.

5. In the Additional serving runtime arguments field, specify the following recommended arguments:

   --dtype=half
   --max-model-len=20000
   --gpu-memory-utilization=0.95
   --enable-chunked-prefill
   --enable-auto-tool-choice
   --tool-call-parser=llama3_json
   --chat-template=/app/data/template/tool_chat_template_llama3.2_json.jinja

   Copy code to clipboard

   1. Click Deploy.

      Note

      Model deployment can take several minutes, especially for the first model that is deployed on the cluster. Initial deployment may take more than 10 minutes while the relevant images download.

Verification

1. Verify that the kserve-controller-manager and odh-model-controller pods are running:

   1. Open a new terminal window.
   2. Log in to your OpenShift cluster from the CLI:
   3. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   4. After you have logged in, click Display token.

   5. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

      $ oc login --token=<token> --server=<openshift_cluster_url>

      Copy code to clipboard

   6. Enter the following command to verify that the kserve-controller-manager and odh-model-controller pods are running:

      $ oc get pods -n redhat-ods-applications | grep -E 'kserve-controller-manager|odh-model-controller'

      Copy code to clipboard

   7. Confirm that you see output similar to the following example:

      kserve-controller-manager-7c865c9c9f-xyz12   1/1     Running   0          4m21s
      odh-model-controller-7b7d5fd9cc-wxy34        1/1     Running   0          3m55s

      Copy code to clipboard

   8. If you do not see either of the kserve-controller-manager and odh-model-controller pods, there could be a problem with your deployment. In addition, if the pods appear in the list, but their Status is not set to Running, check the pod logs for errors:

      $ oc logs <pod-name> -n redhat-ods-applications

      Copy code to clipboard

   9. Check the status of the inference service:

      $ oc get inferenceservice -n ogx
      $ oc get pods -n <project name> | grep llama

      Copy code to clipboard

      • The deployment automatically creates the following resources:

        • A ServingRuntime resource.
        • An InferenceService resource, a Deployment, a pod, and a service pointing to the pod.

      • Verify that the server is running. For example:

        $ oc logs llama-32-3b-instruct-predictor-77f6574f76-8nl4r  -n <project name>

        Copy code to clipboard

        Check for output similar to the following example log:

        INFO     2025-05-15 11:23:52,750 __main__:498 server: Listening on ['::', '0.0.0.0']:8321
        INFO:     Started server process [1]
        INFO:     Waiting for application startup.
        INFO     2025-05-15 11:23:52,765 __main__:151 server: Starting up
        INFO:     Application startup complete.
        INFO:     Uvicorn running on http://['::', '0.0.0.0']:8321 (Press CTRL+C to quit)

        Copy code to clipboard
      • The deployed model displays in the Deployments tab on the project details page for the project it was deployed under.

2. If you see a ConvertTritonGPUToLLVM error in the pod logs when querying the /v1/chat/completions API, and the vLLM server restarts or returns a 500 Internal Server error, apply the following workaround:

   Before deploying the model, remove the --enable-chunked-prefill argument from the Additional serving runtime arguments field in the deployment dialog.

   The error is displayed similar to the following:

   /opt/vllm/lib64/python3.12/site-packages/vllm/attention/ops/prefix_prefill.py:36:0: error: Failures have been detected while processing an MLIR pass pipeline
   /opt/vllm/lib64/python3.12/site-packages/vllm/attention/ops/prefix_prefill.py:36:0: note: Pipeline failed while executing [`ConvertTritonGPUToLLVM` on 'builtin.module' operation]: reproducer generated at `std::errs, please share the reproducer above with Triton project.`
   INFO:     10.129.2.8:0 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error

   Copy code to clipboard

Procedure

1. Open a new terminal window.

   1. Log in to your OpenShift cluster from the CLI:
   2. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   3. After you have logged in, click Display token.

   4. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

      $ oc login --token=<token> --server=<openshift_cluster_url>

      Copy code to clipboard

2. If you enabled Require token authentication during model deployment, retrieve your token:

   $ export MODEL_TOKEN=$(oc get secret default-name-llama-32-3b-instruct-sa -n <project name> --template={{ .data.token }} | base64 -d)

   Copy code to clipboard

3. Obtain your model endpoint URL:

   • If you enabled Make deployed models available through an external route during model deployment, click Endpoint details on the Deployments page in the OpenShift AI dashboard to obtain your model endpoint URL.

   • In addition, if you did not enable Require token authentication during model deployment, you can also enter the following command to retrieve the endpoint URL:

     $ export MODEL_ENDPOINT="https://$(oc get route llama-32-3b-instruct -n <project name> --template={{ .spec.host }})"

     Copy code to clipboard

4. Test the endpoint with a sample chat completion request:

   • If you did not enable Require token authentication during model deployment, enter a chat completion request. For example:

     $ curl -X POST $MODEL_ENDPOINT/v1/chat/completions \
      -H "Content-Type: application/json" \
      -d '{
      "model": "llama-32-3b-instruct",
      "messages": [
        {
          "role": "user",
          "content": "Hello"
        }
      ]
     }'

     Copy code to clipboard

   • If you enabled Require token authentication during model deployment, include a token in your request. For example:

     curl -s -k $MODEL_ENDPOINT/v1/chat/completions \
     --header "Authorization: Bearer $MODEL_TOKEN" \
     --header 'Content-Type: application/json' \
     -d '{
       "model": "llama-32-3b-instruct",
       "messages": [
         {
           "role": "user",
           "content": "can you tell me a funny joke?"
         }
       ]
     }' | jq .

     Copy code to clipboard
     Note

     The -k flag disables SSL verification and should only be used in test environments or with self-signed certificates.

Procedure

1. In the OpenShift console, click the Quick Create ( ) icon and then click the Import YAML option.
2. Verify that your project is the selected project.

3. In the Import YAML editor, paste the following manifest and click Create:

   apiVersion: v1
   kind: Secret
   metadata:
     name: milvus-secret
   type: Opaque
   stringData:
     root-password: "MyStr0ngP@ssw0rd"
   ---
   kind: PersistentVolumeClaim
   apiVersion: v1
   metadata:
     name: milvus-pvc
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 20Gi
     volumeMode: Filesystem
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: etcd-deployment
     labels:
       app: etcd
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: etcd
     strategy:
       type: Recreate
     template:
       metadata:
         labels:
           app: etcd
       spec:
         containers:
           - name: etcd
             image: quay.io/coreos/etcd:v3.5.5
             command:
               - etcd
               - --advertise-client-urls=http://127.0.0.1:2379
               - --listen-client-urls=http://0.0.0.0:2379
               - --data-dir=/etcd
             ports:
               - containerPort: 2379
             volumeMounts:
               - name: etcd-data
                 mountPath: /etcd
             env:
               - name: ETCD_AUTO_COMPACTION_MODE
                 value: revision
               - name: ETCD_AUTO_COMPACTION_RETENTION
                 value: "1000"
               - name: ETCD_QUOTA_BACKEND_BYTES
                 value: "4294967296"
               - name: ETCD_SNAPSHOT_COUNT
                 value: "50000"
         volumes:
           - name: etcd-data
             emptyDir: {}
         restartPolicy: Always
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: etcd-service
   spec:
     ports:
       - port: 2379
         targetPort: 2379
     selector:
       app: etcd
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     labels:
       app: milvus-standalone
     name: milvus-standalone
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: milvus-standalone
     strategy:
       type: Recreate
     template:
       metadata:
         labels:
           app: milvus-standalone
       spec:
         containers:
           - name: milvus-standalone
             image: milvusdb/milvus:v2.6.0
             args: ["milvus", "run", "standalone"]
             env:
               - name: DEPLOY_MODE
                 value: standalone
               - name: ETCD_ENDPOINTS
                 value: etcd-service:2379
               - name: COMMON_STORAGETYPE
                 value: local
               - name: MILVUS_ROOT_PASSWORD
                 valueFrom:
                   secretKeyRef:
                     name: milvus-secret
                     key: root-password
             livenessProbe:
               exec:
                 command: ["curl", "-f", "http://localhost:9091/healthz"]
               initialDelaySeconds: 90
               periodSeconds: 30
               timeoutSeconds: 20
               failureThreshold: 5
             ports:
               - containerPort: 19530
                 protocol: TCP
               - containerPort: 9091
                 protocol: TCP
             volumeMounts:
               - name: milvus-data
                 mountPath: /var/lib/milvus
         restartPolicy: Always
         volumes:
           - name: milvus-data
             persistentVolumeClaim:
               claimName: milvus-pvc
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: milvus-service
   spec:
     selector:
       app: milvus-standalone
     ports:
       - name: grpc
         port: 19530
         targetPort: 19530
       - name: http
         port: 9091
         targetPort: 9091

   Copy code to clipboard
   Note

   • Use the gRPC port (19530) for the MILVUS_ENDPOINT setting in OGX.
   • The HTTP port (9091) is reserved for health checks.
   • If you deploy Milvus in a different namespace, use the fully qualified service name in your OGX configuration. For example: http://milvus-service.<namespace>.svc.cluster.local:19530

Verification

1. In the OpenShift web console, click Workloads → Deployments.
2. Verify that both etcd-deployment and milvus-standalone show a status of 1 of 1 pods available.
3. Click Pods in the navigation panel and confirm that pods for both deployments are Running.
4. Click the milvus-standalone pod name, then select the Logs tab.

5. Verify that Milvus reports a healthy startup with output similar to:

   Milvus Standalone is ready to serve ...
   Listening on 0.0.0.0:19530 (gRPC)

   Copy code to clipboard
6. Click Networking → Services and confirm that the milvus-service and etcd-service resources exist and are exposed on ports 19530 and 2379, respectively.

7. (Optional) Click Pods → milvus-standalone → Terminal and run the following health check:

   curl http://localhost:9091/healthz

   Copy code to clipboard

   A response of {"status": "healthy"} confirms that Milvus is running correctly.

Procedure

1. Open a new terminal window and log in to your OpenShift cluster from the CLI:

   In the upper-right corner of the OpenShift web console, click your user name and select Copy login command. After you have logged in, click Display token. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

   $ oc login --token=<token> --server=<openshift_cluster_url>

   Copy code to clipboard

2. Create a secret that contains the inference model and the remote embeddings environment variables:

   # Remote LLM
   export INFERENCE_MODEL="llama-3-2-3b"
   export VLLM_URL="https://llama-32-3b-instruct-predictor:8443/v1"
   export VLLM_TLS_VERIFY="false"   # Use "true" in production
   export VLLM_API_TOKEN="<token identifier>"
   export VLLM_MAX_TOKENS=16384
   
   # Remote embedding configuration
   export EMBEDDING_MODEL="nomic-embed-text-v1-5"
   export EMBEDDING_PROVIDER_MODEL_ID="nomic-embed-text-v1-5"
   export VLLM_EMBEDDING_URL="<embedding-endpoint>/v1"
   export VLLM_EMBEDDING_API_TOKEN="<embedding-token>"
   export VLLM_EMBEDDING_MAX_TOKENS=8192
   export VLLM_EMBEDDING_TLS_VERIFY="true"
   
   oc create secret generic ogx-secret -n <project-name> \
     --from-literal=INFERENCE_MODEL="$INFERENCE_MODEL" \
     --from-literal=VLLM_URL="$VLLM_URL" \
     --from-literal=VLLM_TLS_VERIFY="$VLLM_TLS_VERIFY" \
     --from-literal=VLLM_API_TOKEN="$VLLM_API_TOKEN" \
     --from-literal=VLLM_MAX_TOKENS="$VLLM_MAX_TOKENS" \
     --from-literal=EMBEDDING_MODEL="$EMBEDDING_MODEL" \
     --from-literal=EMBEDDING_PROVIDER_MODEL_ID="$EMBEDDING_PROVIDER_MODEL_ID" \
     --from-literal=VLLM_EMBEDDING_URL="$VLLM_EMBEDDING_URL" \
     --from-literal=VLLM_EMBEDDING_TLS_VERIFY="$VLLM_EMBEDDING_TLS_VERIFY" \
     --from-literal=VLLM_EMBEDDING_API_TOKEN="$VLLM_EMBEDDING_API_TOKEN" \
     --from-literal=VLLM_EMBEDDING_MAX_TOKENS="$VLLM_EMBEDDING_MAX_TOKENS"

   Copy code to clipboard
3. Choose one of the following deployment examples:

Procedure

1. In a new notebook cell, install the client:

   %pip install ogx_client

   Copy code to clipboard

2. Install the requests library if it is not already available:

   %pip install requests

   Copy code to clipboard

3. Import OGXClient and create a client instance:

   from ogx_client import OGXClient
   client = OGXClient(base_url="<ogx-base-url>")

   Copy code to clipboard

4. List the available models:

   models = client.models.list()

   Copy code to clipboard

5. Verify that the list includes:

   • At least one LLM model.

   • At least one embedding model.

     [Model(identifier='llama-32-3b-instruct', model_type='llm', provider_id='vllm-inference'),
      Model(identifier='nomic-embed-text-v1-5', model_type='embedding', metadata={'embedding_dimension': 768})]

     Copy code to clipboard

6. Select one LLM and one embedding model:

   model_id = next(m.identifier for m in models if m.model_type == "llm")
   
   embedding_model = next(m for m in models if m.model_type == "embedding")
   embedding_model_id = embedding_model.identifier
   embedding_dimension = int(embedding_model.metadata["embedding_dimension"])

   Copy code to clipboard

7. (Optional) Create a vector store. Skip this step if you already have one.

   Note

   Provider IDs can differ between interfaces. In the Python SDK, you typically use the provider name directly (for example, provider_id: "pgvector"). In some CLI tools and examples, remote providers might use a prefixed identifier (for example, --vector-db-provider-id remote-pgvector). Use the provider ID format that matches the interface you are using.

1. If you already have a vector store, set its identifier:

   # vector_store_id = "<existing-vector-store-id>"

   Copy code to clipboard

2. Download a PDF, upload it to OGX, and add it to your vector store:

   import requests
   
   pdf_url = "https://www.federalreserve.gov/aboutthefed/files/quarterly-report-20250822.pdf"
   filename = "quarterly-report-20250822.pdf"
   
   response = requests.get(pdf_url)
   response.raise_for_status()
   
   with open(filename, "wb") as f:
       f.write(response.content)
   
   with open(filename, "rb") as f:
       file_info = client.files.create(
           file=(filename, f),
           purpose="assistants",
       )
   
   vector_store_file = client.vector_stores.files.create(
       vector_store_id=vector_store_id,
       file_id=file_info.id,
       chunking_strategy={
           "type": "static",
           "static": {
               "max_chunk_size_tokens": 800,
               "chunk_overlap_tokens": 400,
           },
       },
   )
   
   print(vector_store_file)

   Copy code to clipboard

Procedure

1. In a new notebook cell, install the client:

   %pip install -q ogx_client

   Copy code to clipboard

2. Import OgxClient:

   from ogx_client import OgxClient

   Copy code to clipboard

3. Create a client instance:

   # Use the OGX service or route URL that is reachable from the workbench.
   # Do not append /v1 when using ogx_client.
   client = OGXClient(base_url="<ogx-base-url>")

   Copy code to clipboard

4. List available models:

   models = client.models.list()

   Copy code to clipboard

5. Select an LLM. If you plan to register a new vector store, also capture an embedding model:

   model_id = next(m.identifier for m in models if m.model_type == "llm")
   
   embedding = next((m for m in models if m.model_type == "embedding"), None)
   if embedding:
       embedding_model_id = embedding.identifier
       embedding_dimension = int(embedding.metadata.get("embedding_dimension", 768))

   Copy code to clipboard
6. If you do not already have a vector store ID, register a vector store (choose one):

1. If you already have a vector store, set its identifier:

   # vector_store_id = "<existing-vector-store-id>"

   Copy code to clipboard

2. Query without using a vector store:

   system_instructions = """You are a precise and reliable AI assistant.
   Use retrieved context when it is available.
   If nothing relevant is found, say so clearly."""
   
   query = "How do you do great work?"
   
   response = client.responses.create(
       model=model_id,
       input=query,
       instructions=system_instructions,
   )
   
   print(response.output_text)

   Copy code to clipboard

3. Query by using the Responses API with file search:

   response = client.responses.create(
       model=model_id,
       input=query,
       instructions=system_instructions,
       tools=[
           {
               "type": "file_search",
               "vector_store_ids": [vector_store_id],
           }
       ],
   )
   
   print(response.output_text)

   Copy code to clipboard

Procedure

1. In a new notebook cell, install the client:

   %pip install -q ogx-client

   Copy code to clipboard

2. In a new notebook cell, import OgxClient:

   from ogx_client import OgxClient

   Copy code to clipboard

3. In a new notebook cell, assign your deployment endpoint to the base_url parameter to create a OgxClient instance:

   client = OgxClient(base_url="http://<ogx-service>:8321")

   Copy code to clipboard
   Note

   OgxClient requires the service root without the /v1 path suffix. For example, use http://ogx-service:8321.

   The /v1 suffix is required only when you use OpenAI-compatible SDKs or send raw HTTP requests to the OpenAI-compatible API surface.

4. List the available models:

   models = client.models.list()

   Copy code to clipboard

5. Select the first LLM and the first embedding model:

   model_id = next(m.identifier for m in models if m.model_type == "llm")
   embedding_model = next(m for m in models if m.model_type == "embedding")
   embedding_model_id = embedding_model.identifier
   embedding_dimension = int(embedding_model.metadata.get("embedding_dimension", 768))

   Copy code to clipboard
6. Register a vector store (choose one option). Skip this step if your pipeline registers the store automatically.

1. In the OpenShift web console, import the YAML file containing your Docling pipeline into your project, as described in Importing a pipeline.

2. Create a pipeline run to execute your Docling pipeline, as described in Executing a pipeline run. The pipeline run inserts your PDF documents into the vector store. If you run the Docling pipeline from the Content from github.com is not included.RAG demo samples repository, you can optionally customize the following parameters before starting the pipeline run:

   • base_url: The base URL to fetch PDF files from.
   • pdf_filenames: A comma-separated list of PDF filenames to download and convert.
   • num_workers: The number of parallel workers.
   • vector_store_id: The vector store identifier.
   • service_url: The Milvus service URL.
   • embed_model_id: The embedding model to use.
   • max_tokens: The maximum tokens for each chunk.
   • use_gpu: Enable or disable GPU acceleration.

Verification

1. In your Jupyter notebook, query the LLM with a question that relates to the ingested content:

   system_instructions = """You are a precise and reliable AI assistant.
   Use retrieved context when it is available.
   If nothing relevant is found in the available files, say so clearly."""
   
   prompt = "What can you tell me about the birth of word processing?"
   
   # Query using the Responses API with file search
   response = client.responses.create(
       model=model_id,
       input=prompt,
       instructions=system_instructions,
       tools=[
           {
               "type": "file_search",
               "vector_store_ids": [vector_store_id],
           }
       ],
   )
   
   print("Answer (with vector stores):")
   print(response.output_text)

   Copy code to clipboard

2. Query chunks from the vector store:

   query_result = client.vector_io.query(
       vector_store_id=vector_store_id,
       query="word processing",
   )
   print(query_result)

   Copy code to clipboard
   • The pipeline run completes successfully in your project.
   • Document embeddings are stored in the vector store and are available for retrieval.
   • No errors or warnings appear in the pipeline logs or your notebook output.

Procedure

1. Log in to the OpenShift web console.
2. From the Project list, select the project where you want to deploy Qdrant.
3. Click Import YAML.

4. Paste the following YAML:

   Important

   This example deploys a standalone Qdrant service for development or evaluation. For production deployments, review and adapt the configuration to meet your organization’s security, availability, backup, and lifecycle requirements.

   apiVersion: v1
   kind: Secret
   metadata:
     name: <qdrant_credentials_secret>
   type: Opaque
   stringData:
     QDRANT_API_KEY: "<api_key>"
   
   ---
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: <qdrant_pvc>
   spec:
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: <storage_size>
   
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: <qdrant_deployment>
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: <qdrant_app_label>
     template:
       metadata:
         labels:
           app: <qdrant_app_label>
       spec:
         containers:
         - name: qdrant
           image: qdrant/qdrant:v1.12.0
           ports:
           - name: http
             containerPort: 6333
           - name: grpc
             containerPort: 6334
           env:
           - name: QDRANT__SERVICE__API_KEY
             valueFrom:
               secretKeyRef:
                 name: <qdrant_credentials_secret>
                 key: QDRANT_API_KEY
           volumeMounts:
           - name: qdrant-storage
             mountPath: /qdrant/storage
           - name: qdrant-storage
             mountPath: /qdrant/snapshots
             subPath: snapshots
           readinessProbe:
             httpGet:
               path: /readyz
               port: 6333
             initialDelaySeconds: 5
             periodSeconds: 10
           livenessProbe:
             httpGet:
               path: /healthz
               port: 6333
             initialDelaySeconds: 10
             periodSeconds: 20
         volumes:
         - name: qdrant-storage
           persistentVolumeClaim:
             claimName: <qdrant_pvc>
   
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: <qdrant_service>
   spec:
     selector:
       app: <qdrant_app_label>
     ports:
     - name: http
       port: 6333
       targetPort: 6333
     - name: grpc
       port: 6334
       targetPort: 6334
     type: ClusterIP

   Copy code to clipboard
   Note

   If your Qdrant instance does not require authentication, remove the Secret and the QDRANT__SERVICE__API_KEY environment variable from the Deployment configuration.

5. Replace the placeholder values as follows:

   • <qdrant_credentials_secret>: A name for the Secret that stores the Qdrant API key, for example qdrant-credentials.
   • <api_key>: An API key for authenticating with Qdrant. If authentication is not required, remove the Secret and the QDRANT__SERVICE__API_KEY environment variable from the Deployment.
   • <qdrant_pvc>: A name for the PersistentVolumeClaim, for example qdrant-pvc.
   • <storage_size>: The storage capacity to request, for example 10Gi.
   • <qdrant_deployment>: A name for the Deployment, for example qdrant.
   • <qdrant_app_label>: A label for the application, for example qdrant.
   • <qdrant_service>: A name for the Service, for example qdrant-service.
6. Click Create.

Procedure

1. In the OpenShift web console, switch to the Administrator perspective.

2. Create a Secret that stores the Qdrant connection details used by OGX. This Secret must contain the URL of the Qdrant service and, if required, the API key.

   Note

   If you deployed Qdrant by using the procedure in Deploying a Qdrant vector database, create this Secret separately for the OGX configuration. The Secret created during the Qdrant deployment does not contain the QDRANT_URL value required by the OGX provider.

   1. From the Project list, select the project where the OGXServer resource is deployed.
   2. Click Workloads → Secrets.
   3. Click Create → From YAML.

   4. Paste the following YAML:

      apiVersion: v1
      kind: Secret
      metadata:
        name: qdrant-connection
      type: Opaque
      stringData:
        QDRANT_URL: "<qdrant_url>"
        QDRANT_API_KEY: "<api_key>"

      Copy code to clipboard

   5. Replace the placeholder values as follows:

      • <qdrant_url>: The full URL to the Qdrant service, for example http://qdrant-service:6333. For in-cluster deployments, use the Service name and port. For external deployments, use the external URL.
      • <api_key>: The API key for authenticating with Qdrant. If authentication is not enabled for your Qdrant instance, remove the QDRANT_API_KEY entry from both the Secret and the env section in the OGXServer configuration.
   6. Click Create.

3. Update your OGXServer custom resource to reference the Secret and supply the required environment variables.

   1. Click Operators → Installed Operators.
   2. Select the OGX Operator.
   3. Click the OGXServer tab.
   4. Select your OGXServer resource.
   5. Click YAML.

   6. Update the resource to include the following fields.

      Note

      The environment variable names and configuration fields used by the Qdrant provider can vary depending on the OGX version included with OpenShift AI. Before applying this configuration, verify that the variables and fields match the supported versions listed in Supported Configurations for 3.x.

      apiVersion: ogx.io/v1beta1
      kind: OGXServer
      metadata:
        name: ogx
      spec:
        server:
          containerSpec:
            env:
              - name: ENABLE_QDRANT
                value: "true"
              - name: QDRANT_URL
                valueFrom:
                  secretKeyRef:
                    name: qdrant-connection
                    key: QDRANT_URL
              - name: QDRANT_API_KEY
                valueFrom:
                  secretKeyRef:
                    name: qdrant-connection
                    key: QDRANT_API_KEY

      Copy code to clipboard
   7. Click Save.

Procedure

1. Determine how you will access the OGX API.

   You can access the API from within the cluster or from outside the cluster.

   • In-cluster access: Run the curl commands from a pod in the same project, or from a workstation that has network access to the OGX Service.

   • External access: Expose the OGX Service by creating a Route, and then use the Route URL from your local workstation.

     For this procedure, set OGX_URL to the service or route root URL without the /v1 suffix. The example commands append /v1 as part of the endpoint path.

     For more information about API compatibility and base URL requirements, see

     OpenAI compatibility for RAG APIs in OGX.

     Example base URL for in-cluster access

     OGX_URL="http://ogx-service:8321"

     Copy code to clipboard

     Example base URL for external access through a Route

     OGX_URL="https://ogx-route.example.com"

     Copy code to clipboard

2. Create a vector store and capture its ID.

   CREATE_RESPONSE=$(curl -s -X POST "${OGX_URL}/v1/vector_stores" \
     -H "Content-Type: application/json" \
     -d '{
       "name": "my-rag-store",
       "embedding_model": "vllm/ibm-granite/granite-embedding-125m-english",
       "embedding_dimension": 768,
       "provider_id": "qdrant-remote"
     }')
   
   VECTOR_STORE_ID=$(echo "$CREATE_RESPONSE" | jq -r '.id')
   echo "Vector store ID: ${VECTOR_STORE_ID}"

   Copy code to clipboard

   Ensure that the VECTOR_STORE_ID variable contains a valid value before continuing.

Procedure

1. Log in to your OpenShift cluster from the CLI:

   $ oc login --token=<token> --server=<openshift_cluster_url>

   Copy code to clipboard

2. Create a secret that contains the S3 credentials:

   $ oc create secret generic s3-files-credentials \
     --from-literal=AWS_ACCESS_KEY_ID=<access_key_id> \
     --from-literal=AWS_SECRET_ACCESS_KEY=<secret_access_key> \
     -n <project>

   Copy code to clipboard

   In the previous command, replace <access_key_id> with the access key ID for your S3-compatible storage backend. Replace <secret_access_key> with the secret access key for your S3-compatible storage backend. Replace <project> with the name of the project where the OGXServer resource is deployed.

Procedure

1. Log in to the OpenShift AI web console as a cluster administrator.
2. From the Project list, select the project that contains your OGXServer CR.

3. Update your OGXServer CR to enable the remote::s3 files provider and reference the secret that you created.

   1. Click Home → Search.
   2. From the Resources list, search for OGXServer and select it. The cluster also exposes a OGXOperator resource, which is an internal OpenShift AI resource that is managed by the Red Hat OpenShift AI Operator. Do not select OGXOperator.
   3. From the list of OGXServer instances, click the name of the instance that you want to update.
   4. Click the YAML tab.

   5. Update the resource to include the following fields, and then click Save:

      apiVersion: ogx.io/v1beta1
      kind: OGXServer
      metadata:
        name: my-ogx
        namespace: my-ogx-namespace
      spec:
        server:
          containerSpec:
            env:
              - name: ENABLE_S3 1
                value: "true"
              - name: S3_BUCKET_NAME 2
                value: "<bucket_name>"
              - name: AWS_DEFAULT_REGION 3
                value: "<region>"
              - name: S3_ENDPOINT_URL 4
                value: "<s3_endpoint_url>"
              - name: S3_AUTO_CREATE_BUCKET 5
                value: "false"
              - name: AWS_ACCESS_KEY_ID 6
                valueFrom:
                  secretKeyRef:
                    name: s3-files-credentials
                    key: AWS_ACCESS_KEY_ID
              - name: AWS_SECRET_ACCESS_KEY
                valueFrom:
                  secretKeyRef:
                    name: s3-files-credentials
                    key: AWS_SECRET_ACCESS_KEY
          name: ogx
          port: 8321
        distribution:
          name: rh-dev

      Copy code to clipboard

      1

      Enables the remote::s3 files provider.

      2

      Specifies the name of the S3 bucket where files are stored. S3 bucket names must be globally unique.

      3

      Specifies the region for the S3 bucket, for example, us-east-1. For non-AWS backends, set this value to match the region configuration of your backend, if it requires one. The default of us-east-1 is appropriate for many S3-compatible backends, including MinIO.

      4

      Optional: Specifies the S3 endpoint URL for S3-compatible backends other than AWS S3. Omit this field when using AWS S3. Set this field for MinIO, Ceph Object Gateway, OCI Object Storage, or other S3-compatible backends.

      5

      Optional: When set to "true", allows the provider to create the bucket if it does not exist. Requires the s3:CreateBucket IAM permission. Leave unset or set to "false" in production environments.

      6

      Specifies the S3 credentials, sourced from the secret that you created. Omit AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY if the OGX service account uses an IAM role.

Procedure

1. Open a terminal session in an environment that can reach the OGX API. The OGX service is reachable in the following ways:

   In-cluster access

   Run the commands from a pod in the same project as the OGX service, or from a workbench that has network access to the OGX service. The OGX Operator names the service <distribution-name>-service. Use that name with port 8321, for example:

   OGX_URL="http://ogx-service:8321"

   Copy code to clipboard

   External access

   Expose the OGX service by creating a route, and then run the commands from your local workstation. Use the route URL, for example:

   OGX_URL="https://ogx-route.example.com"

   Copy code to clipboard

   For more information about the correct base URL format and how to find the URL for your deployment, see OpenAI compatibility for RAG APIs in OGX.

2. Set the OGX_URL environment variable to the URL of your OGX service or route, without the /v1 suffix. The example commands in this procedure append /v1 as part of the endpoint path.

3. If your deployment requires authentication, set the OGX_TOKEN environment variable to your API token:

   $ export OGX_TOKEN="<api_token>"

   Copy code to clipboard

4. Upload a file. The purpose form field is required:

   $ curl -X POST \
     "${OGX_URL}/v1/files" \
     -H "Authorization: Bearer ${OGX_TOKEN}" \
     -F purpose="assistants" \
     -F file="@<path_to_file>"

   Copy code to clipboard

   In the previous command, replace <path_to_file> with the local path to the file that you want to upload.

   The output is similar to the following example:

   {
     "object": "file",
     "id": "file-53b3fd75ca2c421c9a292ac63ff924ce",
     "bytes": 16,
     "created_at": 1778608944,
     "expires_at": null,
     "filename": "test.txt",
     "purpose": "assistants"
   }

   Copy code to clipboard

   Note the value of the id field. You use this identifier in subsequent operations to refer to the file. The expires_at field is null for files that do not expire, which is the default behavior for files uploaded with purpose="assistants".

5. List files:

   $ curl -X GET \
     "${OGX_URL}/v1/files" \
     -H "Authorization: Bearer ${OGX_TOKEN}"

   Copy code to clipboard

   The output is similar to the following example:

   {
     "data": [
       {
         "object": "file",
         "id": "file-53b3fd75ca2c421c9a292ac63ff924ce",
         "bytes": 16,
         "created_at": 1778608944,
         "expires_at": null,
         "filename": "test.txt",
         "purpose": "assistants"
       }
     ],
     "has_more": false,
     "first_id": "file-53b3fd75ca2c421c9a292ac63ff924ce",
     "last_id": "file-53b3fd75ca2c421c9a292ac63ff924ce",
     "object": "list"
   }

   Copy code to clipboard

   When no files are present, the data array is empty and the first_id and last_id fields contain empty strings.

6. Retrieve file metadata:

   $ curl -X GET \
     "${OGX_URL}/v1/files/<file_id>" \
     -H "Authorization: Bearer ${OGX_TOKEN}"

   Copy code to clipboard

   In the previous command, replace <file_id> with the identifier of the file, which is returned by the upload or list operation.

   The output is similar to the following example:

   {
     "object": "file",
     "id": "file-53b3fd75ca2c421c9a292ac63ff924ce",
     "bytes": 16,
     "created_at": 1778608944,
     "expires_at": null,
     "filename": "test.txt",
     "purpose": "assistants"
   }

   Copy code to clipboard

7. Retrieve file content:

   $ curl -X GET \
     "${OGX_URL}/v1/files/<file_id>/content" \
     -H "Authorization: Bearer ${OGX_TOKEN}" \
     -o <output_path>

   Copy code to clipboard

   In the previous command, replace <output_path> with the local path where the downloaded file content is saved.

   The endpoint returns the raw bytes of the file. The command does not produce console output when you use the -o option. Confirm that the file was downloaded successfully by checking that the output file exists and has the expected size:

   $ ls -l <output_path>

   Copy code to clipboard

   The reported file size matches the bytes value returned by the metadata operation.

8. Delete a file:

   $ curl -X DELETE \
     "${OGX_URL}/v1/files/<file_id>" \
     -H "Authorization: Bearer ${OGX_TOKEN}"

   Copy code to clipboard

   The output is similar to the following example:

   {
     "id": "file-53b3fd75ca2c421c9a292ac63ff924ce",
     "object": "file",
     "deleted": true
   }

   Copy code to clipboard

   The deleted field is set to true, confirming that OGX has removed both the file metadata and the underlying S3 object.