1. The client submits a job through the REST API, SDK, or CLI.
2. The server validates the request, resolves benchmarks, and persists the job with a status of pending.

3. The runtime creates a Kubernetes Job for each benchmark. Each Job pod contains two containers:

   • The adapter container runs the evaluation framework. Adapters are provider-specific container images that implement a standard interface, translating the job specification into the evaluation framework-specific invocations and returning structured results.
   • The sidecar proxy container authenticates to the EvalHub server using a ServiceAccount token and forwards status events and results from the adapter. The sidecar also proxies authenticated requests to MLflow and OCI registries when configured. This design keeps credentials out of the adapter container, which can run custom user-provided code.
4. The adapter runs the evaluation and reports status events back to EvalHub through the sidecar.
5. The server aggregates and stores the results. If MLflow integration is enabled, the server also logs the results to MLflow.

Procedure

1. Create a Secret containing the PostgreSQL connection string. The Secret must contain a db-url key with a valid PostgreSQL connection URI:

   apiVersion: v1
   kind: Secret
   metadata:
     name: evalhub-db-credentials
   type: Opaque
   stringData:
     db-url: "postgres://evalhub:changeme@postgresql.evalhub.svc.cluster.local:5432/evalhub"

   Copy code to clipboard
   Note

   Replace the hostname, credentials including the changeme placeholder, and database name to match your PostgreSQL deployment.

2. Apply the created evalhub-db-credentials.yaml:

   $ oc apply -f evalhub-db-credentials.yaml -n <namespace>

   Copy code to clipboard

3. Create an EvalHub custom resource to deploy the service, such as evalhub_cr.yaml:

   apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: EvalHub
   metadata:
     name: evalhub
   spec:
     replicas: 1
     database:
       type: postgresql
       secret: evalhub-db-credentials
     providers:
       - lm-evaluation-harness
       - garak
       - guidellm
     collections:
       - safety-and-fairness-v1
     env:
       - name: MLFLOW_TRACKING_URI
         value: "http://mlflow.mlflow.svc.cluster.local:5000"

   Copy code to clipboard

   where:

   replicas defines the number of EvalHub pods to create.

   database.type defines the storage backend. Set to postgresql for PostgreSQL.

   database.secret defines the name of a Secret containing the PostgreSQL connection string.

   providers defines the list of evaluation provider configurations to load at startup.

   collections defines the list of benchmark collections to load at startup.

   otel defines the OpenTelemetry exporter configuration for traces and metrics (optional).

   env defines the environment variables to set in the EvalHub deployment containers.

4. Apply the custom resource to the cluster:

   $ oc apply -f evalhub_cr.yaml -n <namespace>

   Copy code to clipboard
   Note

   Use a dedicated namespace for EvalHub rather than redhat-ods-applications. The redhat-ods-applications namespace has NetworkPolicies that restrict cross-namespace traffic, which requires additional labeling on tenant namespaces. For more information, see Section 2.24, “Set up a tenant namespace”.

   The TrustyAI Operator automatically reconciles the EvalHub custom resource in your namespace.

Verification

1. Confirm that the EvalHub pod is running:

   $ oc get pods -l app=eval-hub -n <namespace>

   Copy code to clipboard

   NAME                       READY   STATUS    RESTARTS   AGE
   evalhub-7b9f4c6d88-x2k4p  1/1     Running   0          2m

   Copy code to clipboard

2. Query the health endpoint:

   $ export EVALHUB_URL=https://$(oc get routes evalhub -o jsonpath='{.spec.host}' -n <namespace>)
   $ curl $EVALHUB_URL/api/v1/health | jq .

   Copy code to clipboard

   {
     "status": "healthy",
     "timestamp": "2026-04-13T10:00:00Z",
     "version": "0.3.0",
     "uptime": 3600000000000,
   }

   Copy code to clipboard

Procedure

1. Install the EvalHub SDK with CLI support:

   $ pip install "eval-hub-sdk[cli]"

   Copy code to clipboard

   To install only the Python SDK without the CLI, run:

   $ pip install "eval-hub-sdk[client]"

   Copy code to clipboard

2. Configure the CLI to connect to your EvalHub server:

   $ evalhub config set base_url https://<evalhub_route>
   $ evalhub config set tenant <namespace>

   Copy code to clipboard

   where:

   • base_url defines the URL of your EvalHub server route.
   • tenant defines the namespace where your evaluation jobs will run.

3. Set your authentication token:

   $ export TOKEN=$(oc create token <serviceaccount> -n <namespace>)
   $ evalhub config set token $TOKEN

   Copy code to clipboard

   Replace <serviceaccount> with the name of a ServiceAccount that has EvalHub access. For more information about granting access, see Section 2.25, “Grant access to EvalHub”.

Procedure

1. List all registered providers:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" $EVALHUB_URL/api/v1/evaluations/providers | jq .

   Copy code to clipboard

   {
     "items": [
       {
         "resource": { "id": "lm_evaluation_harness", "owner": "system" },
         "name": "lm_evaluation_harness",
         "title": "LM Evaluation Harness",
         "benchmarks": [ ... ]
       },
       {
         "resource": { "id": "garak", "owner": "system" },
         "name": "garak",
         "title": "Garak",
         "benchmarks": [ ... ]
       }
     ]
   }

   Copy code to clipboard

2. Get a specific provider with its benchmarks:

   • To get a specific provider by using the REST API, run:

     $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" $EVALHUB_URL/api/v1/evaluations/providers/lm_evaluation_harness | jq .

     Copy code to clipboard

     {
       "resource": { "id": "lm_evaluation_harness", "owner": "system" },
       "name": "lm_evaluation_harness",
       "title": "LM Evaluation Harness",
       "benchmarks": [
         { "id": "mmlu", "name": "MMLU", "category": "reasoning" },
         { "id": "hellaswag", "name": "HellaSwag", "category": "reasoning" },
         { "id": "arc_challenge", "name": "ARC Challenge", "category": "reasoning" },
         ...
       ]
     }

     Copy code to clipboard

   • To get a specific provider by using the Python SDK, run:

     from evalhub.client import SyncEvalHubClient
     
     client = SyncEvalHubClient(
         base_url="https://evalhub.example.com",
         tenant="my-namespace"
     )
     
     for provider in client.providers.list():
         print(f"{provider.resource.id}: {provider.name}")
     
     benchmarks = client.benchmarks.list(provider_id="lm_evaluation_harness")
     for b in benchmarks:
         print(f"  {b.id}: {b.name}")

     Copy code to clipboard

     lm_evaluation_harness: LM Evaluation Harness
     garak: Garak
     guidellm: GuideLLM
       mmlu: Massive Multitask Language Understanding
       hellaswag: HellaSwag
       gsm8k: Grade School Math 8K
       ...

     Copy code to clipboard

   • To get a specific provider by using the CLI, run:

     $ evalhub providers list

     Copy code to clipboard

      ID                     NAME                   DESCRIPTION                              BENCHMARKS
      lm_evaluation_harness  LM Evaluation Harness  EleutherAI language model evaluation     167
      garak                  Garak                  LLM vulnerability and safety scanner     12
      guidellm              GuideLLM               Performance benchmarking                  4

     Copy code to clipboard

3. Optional: Get more details information about a specific provider. For example, for details about lm_evaluation_harness, run:

   $ evalhub providers describe lm_evaluation_harness

   Copy code to clipboard

   Provider: LM Evaluation Harness
   ID:       lm_evaluation_harness
   Description: EleutherAI language model evaluation framework
   
   Benchmarks (167):
    ID             NAME                             CATEGORY             METRICS
    mmlu           Massive Multitask Language Und…   knowledge            acc, acc_norm
    hellaswag      HellaSwag                         reasoning            acc, acc_norm
    gsm8k          Grade School Math 8K              math                 exact_match
    arc_easy       ARC Easy                          reasoning            acc, acc_norm
    ...

   Copy code to clipboard

Procedure

1. Submit a job by specifying the model endpoint and one or more benchmarks:

   • To use the REST API, run:

     $ curl -X POST $EVALHUB_URL/api/v1/evaluations/jobs \
       -H "Authorization: Bearer $TOKEN" \
       -H "Content-Type: application/json" \
       -H "X-Tenant: <namespace>" \
       -d '{
         "name": "my-eval",
         "model": {
           "url": "http://my-model.my-namespace.svc.cluster.local:8080/v1",
           "name": "my-model"
         },
         "benchmarks": [
           {
             "provider_id": "lm_evaluation_harness",
             "benchmark_id": "mmlu"
           },
           {
             "provider_id": "lm_evaluation_harness",
             "benchmark_id": "hellaswag"
           }
         ]
       }'

     Copy code to clipboard
     Note

     Most providers expect the model URL to point to an OpenAI-compatible inference endpoint. The required URL format might vary depending on the provider. Check the provider documentation for specific requirements.

     The server returns a 202 Accepted response with the job resource, including a job ID for tracking.

   • To use the Python SDK, enter the following command:

     from evalhub.client import SyncEvalHubClient
     from evalhub.models import JobSubmissionRequest, ModelConfig, BenchmarkConfig
     
     client = SyncEvalHubClient(
         base_url="https://evalhub.example.com",
         tenant="my-namespace"
     )
     
     job = client.jobs.create(JobSubmissionRequest(
         name="my-eval",
         model=ModelConfig(
             url="http://my-model.my-namespace.svc.cluster.local:8080/v1",
             name="my-model"
         ),
         benchmarks=[
             BenchmarkConfig(provider_id="lm_evaluation_harness", benchmark_id="mmlu"),
             BenchmarkConfig(provider_id="lm_evaluation_harness", benchmark_id="hellaswag"),
         ]
     ))
     
     print(f"Job ID: {job.resource.id}")

     Copy code to clipboard

   • To use the CLI, run the following command:

     $ evalhub eval run \
         --name my-eval \
         --model-url http://my-model.my-namespace.svc.cluster.local:8080/v1 \
         --model-name my-model \
         --provider lm_evaluation_harness \
         -b mmlu -b hellaswag

     Copy code to clipboard

   • To use a YAML config file, run:

     $ evalhub eval run --config evaljob.yaml

     Copy code to clipboard

Procedure

1. Check the status of a specific job:

   $ curl -s \
       -H "Authorization: Bearer $TOKEN" \
       -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq .

   Copy code to clipboard

   Example response for a completed job:

   {
     "resource": {
       "id": "<job_id>",
       "tenant": "<namespace>",
       "created_at": "2026-04-22T10:00:00Z"
     },
     "status": {
       "state": "completed",
       "benchmarks": [
         { "id": "mmlu", "provider_id": "lm_evaluation_harness", "status": "completed" },
         { "id": "hellaswag", "provider_id": "lm_evaluation_harness", "status": "completed" }
       ]
     },
     "results": {
       "benchmarks": [
         {
           "id": "mmlu",
           "provider_id": "lm_evaluation_harness",
           "metrics": { "acc": 0.65, "acc_norm": 0.68 }
         },
         {
           "id": "hellaswag",
           "provider_id": "lm_evaluation_harness",
           "metrics": { "acc": 0.72, "acc_norm": 0.75 }
         }
       ]
     },
     "name": "my-eval",
     "model": {
       "url": "http://my-model:8080/v1",
       "name": "my-model"
     },
     ...
   }

   Copy code to clipboard

2. After the job completes, retrieve the benchmark results:

   $ curl -s \
       -H "Authorization: Bearer $TOKEN" \
       -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq .results

   Copy code to clipboard

   The results object contains benchmark scores, metrics, and pass/fail outcomes. If pass criteria are configured, the results include a test field with the overall score, threshold, and pass/fail status.

3. List all jobs, optionally filtered by status:

   • To use the REST API, run:

     $ curl -s \
         -H "Authorization: Bearer $TOKEN" \
         -H "X-Tenant: <namespace>" \
         "$EVALHUB_URL/api/v1/evaluations/jobs?status=completed&limit=10" | jq .

     Copy code to clipboard

     Table 2.1. Job query parameters

     Parameter	Default	Description
     limit	50	Maximum number of results to return. The maximum allowed value is 100.
     offset	0	Number of results to skip for pagination.
     status	—	Filter by job state: pending, running, completed, failed, cancelled, partially_failed.
     name	—	Filter by job name. Uses exact, case-sensitive matching.
     tags	—	Filter by a single tag. Returns jobs that contain the specified tag in their tags list.
     owner	—	Filter by the authenticated username of the job owner, for example system:serviceaccount:<namespace>:<name> for a ServiceAccount or the OpenShift username.
     experiment_id	—	Filter by MLflow experiment ID.

   • To use the CLI and to watch a job’s status in real time, use the --watch flag. The CLI polls the job at regular intervals and displays benchmark progress until the job reaches a terminal state:

     $ evalhub eval status --watch <job_id>

     Copy code to clipboard

     To retrieve formatted results after a job completes:

     $ evalhub eval results <job_id> --format table

     Copy code to clipboard

      BENCHMARK   PROVIDER                METRIC     VALUE
      mmlu        lm_evaluation_harness   acc        0.65
      mmlu        lm_evaluation_harness   acc_norm   0.68
      hellaswag   lm_evaluation_harness   acc        0.72
      hellaswag   lm_evaluation_harness   acc_norm   0.75

     Copy code to clipboard

     The --format flag supports table, json, yaml, and csv.

   • To use the Python SDK and to check the status of a specific job, run:

     job = client.jobs.get(job_id)
     print(f"State: {job.state}")

     Copy code to clipboard

     To wait for a job to complete:

     result = client.jobs.wait_for_completion(job_id, timeout=3600, poll_interval=5.0)
     for b in result.results.benchmarks:
         print(f"{b.id}: {b.metrics}")

     Copy code to clipboard

     To list jobs filtered by status:

     from evalhub.models import JobStatus
     
     completed_jobs = client.jobs.list(status=JobStatus.COMPLETED, limit=10)
     for job in completed_jobs:
         print(f"{job.id}: {job.state}")

     Copy code to clipboard

Procedure

1. Create a collection:

   • By using the REST API:

     $ curl -X POST $EVALHUB_URL/api/v1/evaluations/collections \
       -H "Authorization: Bearer $TOKEN" \
       -H "Content-Type: application/json" \
       -H "X-Tenant: <namespace>" \
       -d '{
         "name": "my-safety-suite",
         "category": "safety",
         "benchmarks": [
           {"provider_id": "lm_evaluation_harness", "benchmark_id": "truthfulqa_mc2"},
           {"provider_id": "garak", "benchmark_id": "owasp_llm_top_10"}
         ]
       }'

     Copy code to clipboard

     Example response:

     {
       "resource": {
         "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
         "tenant": "<namespace>",
         "created_at": "2026-04-22T10:00:00Z",
         "owner": "<user_name>"
       },
       "name": "my-safety-suite",
       "category": "safety",
       "benchmarks": [
         {"provider_id": "lm_evaluation_harness", "id": "truthfulqa_mc2"},
         {"provider_id": "garak", "id": "owasp_llm_top_10"}
       ]
     }

     Copy code to clipboard

   • By using the CLI with a YAML spec file:

     my-safety-suite.yaml

     name: my-safety-suite
     category: safety
     benchmarks:
       - provider_id: lm_evaluation_harness
         benchmark_id: truthfulqa_mc2
       - provider_id: garak
         benchmark_id: owasp_llm_top_10

     Copy code to clipboard

     $ evalhub collections create --file my-safety-suite.yaml

     Copy code to clipboard

   • By using the Python SDK:

     collection = client.collections.create({
         "name": "my-safety-suite",
         "category": "safety",
         "benchmarks": [
             {"provider_id": "lm_evaluation_harness", "benchmark_id": "truthfulqa_mc2"},
             {"provider_id": "garak", "benchmark_id": "owasp_llm_top_10"}
         ]
     })

     Copy code to clipboard

2. Optional: After creating a collection, you can submit evaluation jobs that reference it. The following example shows a job submission by using the created collection:

   $ curl -X POST $EVALHUB_URL/api/v1/evaluations/jobs \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: <namespace>" \
     -d '{
       "name": "my-eval",
       "model": {
         "url": "http://my-model.my-namespace.svc.cluster.local:8080/v1",
         "name": "my-model"
       },
       "collection": {
         "id": "<collection_id>"
       }
     }'

   Copy code to clipboard

Procedure

1. Create a Secret containing your API key in the model-auth.yaml file:

   apiVersion: v1
   kind: Secret
   metadata:
     name: model-auth
   type: Opaque
   stringData:
     api-key: "<api_key>"

   Copy code to clipboard

2. Apply the Secret to the tenant namespace:

   $ oc apply -f model-auth.yaml -n <namespace>

   Copy code to clipboard

3. When you submit an evaluation job, include an auth field in the model object to reference the Secret:

   Example model configuration with API key authentication:

   "model": {
     "url": "http://my-model.my-namespace.svc.cluster.local:8080/v1",
     "name": "my-model",
     "auth": {
       "secret_ref": "model-auth"
     }
   }

   Copy code to clipboard

   where secret_ref specifies the name of the Secret that has the API key. For details, see Submit an evaluation job.

Procedure

1. Create a Secret containing your S3 credentials in the my-s3-credentials.yaml file:

   apiVersion: v1
   kind: Secret
   metadata:
     name: my-s3-credentials
     namespace: <namespace>
   type: Opaque
   stringData:
     AWS_ACCESS_KEY_ID: "<your_access_key>"
     AWS_SECRET_ACCESS_KEY: "<your_secret_key>"
     AWS_DEFAULT_REGION: "<your_region>"
     AWS_S3_ENDPOINT: "<your_s3_endpoint>"

   Copy code to clipboard

   where:

   • AWS_DEFAULT_REGION defines the region for your S3-compatible storage, for example us-east-1.
   • AWS_S3_ENDPOINT defines the endpoint URL for your S3-compatible storage, for example https://minio.example.com:9000 for MinIO. For Amazon S3, you can omit this field or use the default AWS endpoint.

2. Apply the Secret:

   $ oc apply -f my-s3-credentials.yaml

   Copy code to clipboard

3. When you submit an evaluation job, add a test_data_ref block to each benchmark that requires external data:

   Example S3 test data configuration in a job submission:

   "benchmarks": [
     {
       "provider_id": "lm_evaluation_harness",
       "benchmark_id": "mmlu",
       "test_data_ref": {
         "s3": {
           "bucket": "my-eval-data",
           "key": "datasets/mmlu",
           "secret_ref": "my-s3-credentials"
         }
       }
     }
   ]

   Copy code to clipboard

   where:

   • s3.bucket defines the S3 bucket name.
   • s3.key defines the S3 key prefix for the data set files.

   • s3.secret_ref defines the name of the Secret containing the S3 credentials.

     For the full job submission request, see Section 2.7, “Submit an evaluation job”.

     The init container downloads all objects under the specified S3 prefix to /test_data, preserving the relative directory structure. The secret_ref must reference a Secret in the tenant namespace.

     Note

     The expected file format and directory structure of the test data depend on the adapter and benchmark. See the adapter documentation for the required data layout.

     Alternatively, use the CLI:

     $ evalhub eval run \
         --name s3-data-eval \
         --model-url http://my-model.my-namespace.svc.cluster.local:8080/v1 \
         --model-name my-model \
         --provider lm_evaluation_harness \
         --benchmark mmlu \
         --test-data-s3-bucket my-eval-data \
         --test-data-s3-key datasets/mmlu \
         --test-data-s3-secret my-s3-credentials

     Copy code to clipboard

     Alternatively, use the Python SDK:

     from evalhub.models import (
         JobSubmissionRequest, ModelConfig, BenchmarkConfig,
         TestDataRef, S3TestDataRef
     )
     
     job = client.jobs.submit(JobSubmissionRequest(
         name="s3-data-eval",
         model=ModelConfig(
             url="http://my-model.my-namespace.svc.cluster.local:8080/v1",
             name="my-model"
         ),
         benchmarks=[
             BenchmarkConfig(
                 id="mmlu",
                 provider_id="lm_evaluation_harness",
                 test_data_ref=TestDataRef(
                     s3=S3TestDataRef(
                         bucket="my-eval-data",
                         key="datasets/mmlu",
                         secret_ref="my-s3-credentials",
                     )
                 ),
             )
         ],
     ))

     Copy code to clipboard

     Collections also support test_data_ref on individual benchmarks, allowing you to define custom data sources as part of a reusable evaluation suite.

Procedure

1. Create a kubernetes.io/dockerconfigjson Secret with your registry credentials:

   $ oc create secret docker-registry oci-registry-credentials \
       --docker-server=quay.io \
       --docker-username=<user_name> \
       --docker-password=<password> \
       -n <namespace>

   Copy code to clipboard

2. When you submit an evaluation job, include an exports block in the job submission body:

   Example OCI export configuration in a job submission:

   "benchmarks": [
     {
       "provider_id": "lm_evaluation_harness",
       "benchmark_id": "mmlu"
     }
   ],
   "exports": {
     "oci": {
       "coordinates": {
         "oci_host": "quay.io",
         "oci_repository": "my-org/eval-results"
       },
       "k8s": {
         "connection": "oci-registry-credentials"
       }
     }
   }

   Copy code to clipboard

   where:

   • oci.coordinates.oci_host defines the OCI registry hostname.
   • oci.coordinates.oci_repository defines the repository path within the registry.

   • oci.k8s.connection defines the name of the Secret containing the registry credentials.

     For the full job submission request, see Submit an evaluation job.

     Results artifact from the evaluation frameworks are stored as OCI artifacts with separate layers, allowing selective access to specific outputs.

Verification

1. After the job completes, retrieve the OCI artifact reference from the job results:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq '.results.benchmarks[0].artifacts'

   Copy code to clipboard

2. Verify the artifact exists in the registry by using skopeo:

   $ skopeo inspect --creds <user_name>:<password> docker://quay.io/my-org/eval-results:<tag>

   Copy code to clipboard

   The tag is in the format evalhub-<hash>, where the hash is derived from the job ID, provider, and benchmark. You can find the full OCI reference, including the tag, in the job results.

Procedure

1. Register the custom provider:

   $ curl -X POST $EVALHUB_URL/api/v1/evaluations/providers \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: <namespace>" \
     -d '{
       "name": "my-custom-provider",
       "title": "My Custom Provider",
       "description": "Custom evaluation framework for domain-specific benchmarks.",
       "benchmarks": [
         {
           "id": "domain_accuracy",
           "name": "Domain Accuracy",
           "category": "general",
           "metrics": ["accuracy", "f1"],
           "primary_score": {
             "metric": "accuracy",
             "lower_is_better": false
           },
           "pass_criteria": {
             "threshold": 0.8
           }
         }
       ],
       "runtime": {
         "k8s": {
           "image": "quay.io/my-org/my-adapter:latest",
           "cpu_request": "500m",
           "memory_request": "512Mi",
           "cpu_limit": "2000m",
           "memory_limit": "4Gi"
         }
       }
     }'

   Copy code to clipboard

   Example response:

   {
     "resource": {
       "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
       "tenant": "<namespace>",
       "created_at": "2026-04-22T10:00:00Z",
       "owner": "<user_name>"
     },
     "name": "my-custom-provider",
     "title": "My Custom Provider",
     "description": "Custom evaluation framework for domain-specific benchmarks.",
     "benchmarks": [
       {
         "id": "domain_accuracy",
         "name": "Domain Accuracy",
         "category": "general",
         "metrics": ["accuracy", "f1"],
         "primary_score": { "metric": "accuracy", "lower_is_better": false },
         "pass_criteria": { "threshold": 0.8 }
       }
     ],
     "runtime": {
       "k8s": {
         "image": "quay.io/my-org/my-adapter:latest",
         "cpu_request": "500m",
         "memory_request": "512Mi",
         "cpu_limit": "2000m",
         "memory_limit": "4Gi"
       }
     }
   }

   Copy code to clipboard

   The runtime.k8s section specifies the container image and resource requests for the adapter pod. Each benchmark must declare an id, name, and category. The optional primary_score and pass_criteria fields set default thresholds for the benchmark.

   User-created providers can be updated and deleted through the API. Built-in providers with owner: system are read-only.

   Note

   The Python SDK and CLI do not support creating providers. Use the REST API to register custom providers.

Procedure

1. Create a ConfigMap in the EvalHub custom resource namespace with the provider definition:

   evalhub-provider-my-custom-provider.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: evalhub-provider-my-custom-provider
     namespace: <evalhub_namespace>
     labels:
       trustyai.opendatahub.io/evalhub-provider-type: system
       trustyai.opendatahub.io/evalhub-provider-name: my-custom-provider
   data:
     my-custom-provider.yaml: |
       id: my-custom-provider
       name: My Custom Provider
       description: Custom evaluation framework for domain-specific benchmarks.
       runtime:
         k8s:
           image: quay.io/my-org/my-adapter:latest
           cpu_request: "500m"
           memory_request: "512Mi"
           cpu_limit: "2000m"
           memory_limit: "4Gi"
       benchmarks:
         - id: domain_accuracy
           name: Domain Accuracy
           category: general
           metrics:
             - accuracy
             - f1
           primary_score:
             metric: accuracy
             lower_is_better: false
           pass_criteria:
             threshold: 0.8

   Copy code to clipboard

2. Apply the created ConfigMap:

   $ oc apply -f evalhub-provider-my-custom-provider.yaml

   Copy code to clipboard

3. Reference the provider name in your EvalHub custom resource by adding it to the spec.providers list:

   Example spec.providers fragment:

   spec:
     providers:
       - lm-evaluation-harness
       - garak
       - my-custom-provider

   Copy code to clipboard

   For the full EvalHub custom resource structure, see Section 2.3, “Deploy EvalHub with the TrustyAI Operator”.

   The operator copies the ConfigMap to the instance namespace and mounts it as a projected volume at /etc/evalhub/config/providers. The EvalHub server loads all provider YAML files from this directory at startup.

Verification

1. Confirm that the ConfigMap was created:

   $ oc get configmap evalhub-provider-my-custom-provider -n <evalhub_namespace>

   Copy code to clipboard

2. Check that the EvalHub deployment has restarted and is ready:

   $ oc get pods -l app=eval-hub -n <evalhub_namespace>

   Copy code to clipboard

3. Confirm the custom provider is loaded:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/providers/my-custom-provider | jq .name

   Copy code to clipboard

   The output should return "My Custom Provider".

Procedure

1. Create a ConfigMap in the EvalHub custom resource namespace with the collection definition:

   evalhub-collection-my-eval-suite.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: evalhub-collection-my-eval-suite
     namespace: <evalhub_namespace>
     labels:
       trustyai.opendatahub.io/evalhub-collection-type: system
       trustyai.opendatahub.io/evalhub-collection-name: my-eval-suite
   data:
     my-eval-suite.yaml: |
       id: my-eval-suite
       name: My Evaluation Suite
       category: general
       description: Custom evaluation suite for internal model validation.
       pass_criteria:
         threshold: 0.7
       benchmarks:
         - id: mmlu
           provider_id: lm_evaluation_harness
           weight: 2
           primary_score:
             metric: acc_norm
             lower_is_better: false
           pass_criteria:
             threshold: 0.6
         - id: hellaswag
           provider_id: lm_evaluation_harness
           weight: 1
           primary_score:
             metric: acc_norm
             lower_is_better: false
           pass_criteria:
             threshold: 0.7

   Copy code to clipboard

2. Apply the evalhub-collection-my-eval-suite.yaml:

   $ oc apply -f evalhub-collection-my-eval-suite.yaml

   Copy code to clipboard

3. Reference the collection in your EvalHub custom resource by adding the collection name to the spec.collections list:

   Example spec.collections fragment:

   spec:
     collections:
       - leaderboard-v2
       - safety-and-fairness-v1
       - my-eval-suite

   Copy code to clipboard

   For the full EvalHub custom resource structure, see Section 2.3, “Deploy EvalHub with the TrustyAI Operator”.

   The operator mounts collection ConfigMap(s) at /etc/evalhub/config/collections.

Verification

1. Confirm that the ConfigMap was created:

   $ oc get configmap evalhub-collection-my-eval-suite -n <evalhub_namespace>

   Copy code to clipboard

2. Check that the EvalHub deployment has restarted and is ready:

   $ oc get pods -l app=eval-hub -n <evalhub_namespace>

   Copy code to clipboard

3. List collections and confirm the custom collection is present:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/collections/my-eval-suite | jq .name

   Copy code to clipboard

   The output should return "My Evaluation Suite".

Procedure

1. Install the EvalHub SDK with the adapter extra:

   $ pip install "eval-hub-sdk[adapter]"

   Copy code to clipboard

2. Create a class that extends FrameworkAdapter and implements run_benchmark_job:

   from evalhub.adapter import FrameworkAdapter
   from evalhub.models import JobSpec, JobCallbacks, JobResults, JobStatusUpdate, JobPhase
   
   class MyAdapter(FrameworkAdapter):
       def run_benchmark_job(self, config: JobSpec, callbacks: JobCallbacks) -> JobResults:
           callbacks.report_status(JobStatusUpdate(
               phase=JobPhase.RUNNING_EVALUATION,
               message="Running evaluation"
           ))
   
           # Replace with your framework's evaluation function
           scores = run_my_framework(
               model_url=config.model.url,
               benchmark=config.benchmark_id,
               parameters=config.parameters
           )
   
           return JobResults(
               id=config.id,
               benchmark_id=config.benchmark_id,
               benchmark_index=config.benchmark_index,
               model_name=config.model.name,
               results=scores,
               num_examples_evaluated=len(scores),
               duration_seconds=self._get_duration()  # Implement to return elapsed seconds
           )

   Copy code to clipboard

   The framework handles loading the job specification from the mounted ConfigMap, authenticating with the sidecar proxy container that communicates with the EvalHub server, and reporting results. Your adapter only needs to run the evaluation and return the results. For more information about the adapter and sidecar architecture, see Section 2.2, “EvalHub architecture overview”.

3. Package your adapter as a Red Hat Universal Base Image 9 (UBI9) container image:

   1. Create a Containerfile in your adapter directory:

      Containerfile

      FROM registry.access.redhat.com/ubi9/python-312
      
      WORKDIR /app
      
      COPY requirements.txt .
      RUN pip install --no-cache-dir -r requirements.txt
      
      COPY main.py /app/main.py
      
      ENTRYPOINT ["python", "main.py"]

      Copy code to clipboard

   2. Build the image:

      $ podman build -t quay.io/my-org/my-adapter:latest .

      Copy code to clipboard

   3. Push the image to a container registry:

      $ podman push quay.io/my-org/my-adapter:latest

      Copy code to clipboard

4. Reference the image in the provider’s runtime.k8s.image field when registering the provider. See Section 2.17, “Add a custom provider by using the API”.

   The following tables describe the JobSpec and JobCallbacks interfaces available to your adapter.

   Table 2.3. JobSpec fields

   Field	Description
   id	Unique job identifier.
   provider_id	Identifier of the provider that the benchmark belongs to.
   benchmark_id	Identifier of the benchmark to evaluate.
   benchmark_index	Index of this benchmark within the job.
   model	Model configuration, including url and name.
   parameters	Benchmark-specific parameters, for example num_fewshot or limit.
   num_examples	The number of examples to evaluate. When set to None, the adapter evaluates all examples.
   exports	Optional OCI artifact export specification.

   Table 2.4. JobCallbacks methods

   Method	Purpose
   report_status(update)	Sends progress updates including the phase, message, and completed/total steps.
   create_oci_artifact(spec)	Pushes evaluation artifacts to an OCI registry.
   report_results(results)	Reports the final results to the EvalHub server. This method is called automatically if you return JobResults.

Procedure

1. Add the tenant label to the namespace:

   $ oc label namespace <namespace> evalhub.trustyai.opendatahub.io/tenant=

   Copy code to clipboard

   The label value is intentionally empty. The TrustyAI Operator checks for the presence of the label, not its value.

   Note

   Use a dedicated namespace for EvalHub rather than redhat-ods-applications, as described in Section 2.3, “Deploy EvalHub with the TrustyAI Operator”. The redhat-ods-applications namespace has NetworkPolicy resources that restrict cross-namespace traffic, which requires additional labeling on tenant namespaces. If EvalHub is deployed in redhat-ods-applications, label each tenant namespace to allow the evaluation Job sidecar to communicate with the EvalHub server:

   $ oc label namespace <namespace> opendatahub.io/generated-namespace=true

   Copy code to clipboard

   Review the NetworkPolicy resources with oc get networkpolicy -n <evalhub-server-namespace> to determine any additional requirements.

   The TrustyAI Operator watches for this label and automatically provisions the following resources in the labeled namespace:

   • A job ServiceAccount used by evaluation Job pods as their identity.
   • A Role and RoleBinding granting the job ServiceAccount permission to create status-events for reporting job progress.
   • A RoleBinding granting the EvalHub API ServiceAccount permission to create and delete Job resources in the tenant namespace.
   • A RoleBinding granting the EvalHub API ServiceAccount permission to manage ConfigMap resources used to mount job specifications into Job pods.
   • A RoleBinding granting the job ServiceAccount access to MLflow resources when MLflow is configured.

   • A service CA ConfigMap with the cluster CA bundle injected by OpenShift, so that Job pods can make HTTPS requests to the EvalHub API.

     When the tenant label is removed from a namespace, the controller cleans up all provisioned resources automatically.

Verification

1. Confirm that the tenant label is set on the namespace:

   $ oc get namespace <namespace> --show-labels | grep evalhub

   Copy code to clipboard

2. Confirm that the operator provisioned the expected resources in the tenant namespace:

   $ oc get serviceaccount,rolebinding,configmap -n <namespace> | grep evalhub

   Copy code to clipboard

   The output should include a ServiceAccount, RoleBinding resources, and a service CA ConfigMap created by the operator.

Procedure

1. Select the type of principal that matches your use case from the following table:

   Table 2.15. Principal types

   Principal type	Token source	Use case
   ServiceAccount	Mounted pod token or long-lived token	Automation, CI/CD pipelines, agents using Model Context Protocol (MCP)
   OpenShift User	oc whoami -t	Interactive use
   OpenShift Group	User token with group membership	Team-based access

2. Create a Role in the tenant namespace that grants access to the required EvalHub virtual resources:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: evalhub-evaluator
     namespace: <namespace>
   rules:
     - apiGroups: ["trustyai.opendatahub.io"]
       resources: ["evaluations", "collections", "providers"]
       verbs: ["get", "list", "create", "update", "delete"]
     - apiGroups: ["mlflow.kubeflow.org"]
       resources: ["experiments"]
       verbs: ["create", "get"]

   Copy code to clipboard

   Apply the Role:

   $ oc apply -f evalhub-evaluator-role.yaml

   Copy code to clipboard

3. Create a RoleBinding to bind the principal to the Role depending on the selected type.

   • To grant access to a ServiceAccount:

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: my-sa-evalhub-access
       namespace: <namespace>
     subjects:
       - kind: ServiceAccount
         name: my-sa
         namespace: <namespace>
     roleRef:
       kind: Role
       name: evalhub-evaluator
       apiGroup: rbac.authorization.k8s.io

     Copy code to clipboard

     Apply the RoleBinding by the command:

     $ oc apply -f my-sa-evalhub-access.yaml

     Copy code to clipboard

     To obtain a bearer token for a ServiceAccount, run the following command:

     $ export TOKEN=$(oc create token my-sa -n <namespace> --duration=1h)

     Copy code to clipboard

   • To grant access to an OpenShift User:

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: user-evalhub-access
       namespace: <namespace>
     subjects:
       - kind: User
         name: <user_name>
     roleRef:
       kind: Role
       name: evalhub-evaluator
       apiGroup: rbac.authorization.k8s.io

     Copy code to clipboard

     Apply the user RoleBinding:

     $ oc apply -f user-evalhub-access.yaml

     Copy code to clipboard

     To obtain a bearer token for an OpenShift User, log in as the user and run the following command:

     $ export TOKEN=$(oc whoami -t)

     Copy code to clipboard

   • To grant access to an OpenShift Group:

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: team-evalhub-access
       namespace: <namespace>
     subjects:
       - kind: Group
         name: evalhub-users
     roleRef:
       kind: Role
       name: evalhub-evaluator
       apiGroup: rbac.authorization.k8s.io

     Copy code to clipboard

     Apply the group RoleBinding:

     $ oc apply -f team-evalhub-access.yaml

     Copy code to clipboard

     To obtain a bearer token for a Group member, log in as a user who belongs to the group and run the following command:

     $ export TOKEN=$(oc whoami -t)

     Copy code to clipboard

Procedure

1. In the dashboard, click Develop & train → Evaluations. The Evaluations page opens. It contains:

   1. A Start evaluation run button. If you have not run any previous evaluations, only this button is displayed.
   2. A list of evaluations you have previously run, if any exist.
   3. A Project dropdown option you can click to show the evaluations relating to one project instead of all projects.
   4. A filter to sort your evaluations by model or evaluation name.

   The following table outlines the elements and functions of the evaluations list:

2. From the Project dropdown menu, select the namespace of the project where you want to evaluate the model.
3. Click the Start evaluation run button. The Model evaluation form is displayed.

4. Fill in the details of the form. The model argument summary is displayed after you complete the form details:

   1. Model name: Select a model from all the deployed LLMs in your project.
   2. Evaluation name: Give your evaluation a unique name.
   3. Tasks: Choose one or more evaluation tasks against which to measure your LLM. The 100 most common evaluation tasks are supported.

   4. Model type: Choose the type of model based on the type of prompt-formatting you use:

      1. Local-completion: You assemble the entire prompt chain yourself. Use this when you want to evaluate models that take a plain text prompt and return a continuation.
      2. Local-chat-completion: The framework injects roles or templates automatically. Use this for models that simulate a conversation by taking a list of chat messages with roles like user and assistant and reply appropriately.

   5. Security settings:

      1. Available online: Choose enable to allow your model to access the internet to download datasets.

      2. Trust remote code: Choose enable to allow your model to trust code from outside of the project namespace.

         Note

         The Security settings section is grayed out if the security option in global settings is set to active.

5. Observe that a model argument summary is displayed as soon as you fill in the form details.

6. Complete the tokenizer settings:

   1. Tokenized requests: If set to true, the evaluation requests are broken down into tokens. If set to false, the evaluation dataset remains as raw text.
   2. Tokenizer: Type the model’s tokenizer URL that is required for the evaluations.

7. Click Evaluate. The screen returns to the model evaluation page of your project and your job is displayed in the evaluations list.

   Note

   • It can take time for your evaluation to complete, depending on factors including hardware support, model size, and the type of evaluation task(s). The status column reports the current status of the evaluation: completed, running, or failed.
   • If your evaluation fails, the evaluation pod logs in your cluster provide more information.

Procedure

1. Download the translation models:

   $ huggingface-cli download Helsinki-NLP/opus-mt-zh-en --cache-dir /tmp/hf-cache
   $ huggingface-cli download Helsinki-NLP/opus-mt-en-zh --cache-dir /tmp/hf-cache

   Copy code to clipboard

2. Upload the cache to S3:

   $ aws s3 sync /tmp/hf-cache s3://<bucket>/<prefix>/ --exclude ".locks/*"

   Copy code to clipboard

3. In your assessment request JSON, add the hf_cache_path parameter to the benchmarks[].parameters object, pointing to the S3 location where you uploaded the models:

   "parameters": {
       "hf_cache_path": "s3://<bucket>/<prefix>/",
       ...
   }

   Copy code to clipboard

Procedure

1. Create a JSON file called intents-scan.json with the following content:

   {
     "name": "intents-scan",
     "model": {
       "url": "https://<your-model-endpoint>/v1",
       "name": "<your-model-name>",
       "auth": {
         "secret_ref": "<your-secret-name>"
       }
     },
     "benchmarks": [
       {
         "id": "intents",
         "provider_id": "garak-kfp",
         "parameters": {
           "kfp_config": {
             "endpoint": "https://ds-pipeline-dspa.<namespace>.svc.cluster.local:8443",
             "namespace": "<namespace>",
             "s3_secret_name": "<s3-secret-name>",
             "s3_endpoint": "http://minio-dspa.<namespace>.svc.cluster.local:9000",
             "s3_bucket": "mlpipeline",
             "verify_ssl": false
           },
           "intents_models": {
             "judge": {
               "url": "https://<judge-model-endpoint>/v1",
               "name": "<judge-model-name>"
             },
             "sdg": {
               "url": "https://<sdg-model-endpoint>/v1",
               "name": "hosted_vllm/<sdg-model-name>"
             }
           },
           "hf_cache_path": "s3://<bucket>/<prefix>"
         }
       }
     ],
     "experiment": {
       "name": "intents"
     }
   }

   Copy code to clipboard

   where:

   model

   Specifies the target to test. This is either a bare model endpoint or a model combined with guardrails. Provide the OpenAI-compatible endpoint URL, the model name, and a reference to a Kubernetes secret containing the API key.

   benchmarks

   Configures the assessment. The "id": "intents" benchmark runs the intent-based risk assessment with the following parameters:

   • kfp_config: Connection details for the Kubeflow Pipelines backend that orchestrates the assessment. Please note that s3_endpoint and s3_bucket are optional when the referenced s3_secret_name contains these values in the standard AWS-style configuration.
   • intents_models.judge: The model used to classify whether the target model’s responses are compliant or refused. This should be a different model from the target.
   • intents_models.sdg: The model used to generate the adversarial prompts.
   • hf_cache_path: Optional. An S3 URI pointing to pre-downloaded HuggingFace translation models. Required on disconnected clusters where the translation strategy cannot download models at runtime. Omit this parameter if your cluster has internet access.

   experiment

   Specifies a grouping for related assessment runs. Results are recorded as MLflow experiments, so you can compare runs across different models, configurations, or time periods from the MLflow tracking UI.

2. Submit the risk assessment:

   curl -s -X POST "$EVALHUB_URL/api/v1/evaluations/jobs" \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: $NS" \
     -d @intents-scan.json

   Copy code to clipboard

3. The assessment runs as a pipeline with the following stages:

   • The prompt generation model creates adversarial test prompts across the harm categories, producing diverse prompts that vary by demographic, region, writing style, and other dimensions.
   • Each test prompt is sent unmodified to the target model in a baseline test. The judge model classifies whether the target complied or refused.
   • Prompts that the model refused in the baseline are progressively attacked with increasingly sophisticated techniques. Only prompts that remain refused continue to the next strategy.
   • Results are aggregated into a risk assessment report and optionally logged to MLflow as an experiment run.

Procedure

1. Create a Python script called intents-scan.py with the following content:

   from garak_pipeline import (
       PipelineRunner,
       KubeflowConfig,
       EvalConfig,
       ModelConfig,
       IntentsModelConfig,
   )
   
   runner = PipelineRunner(KubeflowConfig(
       pipelines_endpoint="https://<ds-pipeline-dspa-route>",
       namespace="<namespace>",
       s3_credentials_secret_name="<s3-secret-name>",
   ))
   
   job = runner.run_scan(EvalConfig(
       model=ModelConfig(
           model_endpoint="https://<your-model-endpoint>/v1",
           model_name="<your-model-name>",
       ),
       benchmark="intents",
       intents_models={
           "judge": IntentsModelConfig(
               url="https://<judge-model-endpoint>/v1",
               name="<judge-model-name>",
           ),
           "sdg": IntentsModelConfig(
               url="https://<sdg-model-endpoint>/v1",
               name="hosted_vllm/<sdg-model-name>",
           ),
       },
   ))
   
   completed = runner.wait_for_completion(job.job_id, verbose=True)
   runner.download_html_report(job.job_id)

   Copy code to clipboard

   where:

   pipelines_endpoint

   Specifies the Kubeflow Pipelines route that is accessible from where you run the script. You can get it with oc get routes ds-pipeline-dspa -o jsonpath='{.spec.host}'.

   namespace

   Specifies the Kubernetes namespace where the pipeline server is configured.

   s3_credentials_secret_name

   Specifies the Kubernetes secret containing S3 credentials for pipeline artifact storage.

   model_endpoint

   Specifies the OpenAI-compatible endpoint URL for the target model to test.

   model_name

   Specifies the name of the target model.

   intents_models

   Specifies the auxiliary models used by the assessment. The judge model classifies whether the target model’s responses are compliant or refused. The sdg model generates the adversarial prompts.

2. Run the script:

   $ python intents-scan.py

   Copy code to clipboard

1. Baseline: Sends each test prompt to the target model, unmodified. This establishes your model’s default behavior and provides the reference point for measuring how effective subsequent strategies are.
2. System Prompt Override (SPO): Replaces the system prompt with adversarial instructions designed to override your model’s safety controls. Multiple adversarial system prompts are paired with each user prompt to test different override approaches.
3. SPO variants: Combines adversarial system prompts with statistical manipulation of the user prompt, system prompt, or both.
4. Translation: Translates prompts into another language, by default Mandarin Chinese, and automatically translates responses back into English for classification. This reveals whether your model’s safety controls are language-dependent.
5. Tree of Attacks with Pruning (TAP): Uses a separate attacker LLM to dynamically generate new prompts based on the original prompt and your model’s previous responses. The attacker iteratively crafts prompts designed to bypass the specific safety controls it observes. TAP may generate prompts with completely different wording from the original, but results are tracked back to the original intent.

Procedure

1. Create a policy dataset as a JSON or CSV file with at least the following two columns:

   • policy_concept: The name of the harm category. For example Executive Compensation.
   • concept_definition: A natural-language description of what the category covers.

2. Write category descriptions.

   Write effective category descriptions by following these guidelines:

   • Start with "Prompts that…​" to clearly frame the scope.
   • Use the positive form. For example, write "Prompts that seek advice on…​" rather than "Do not include…​". Negative framing can prevent the generator from creating those prompts.
   • Include specific details, such as names of entities, products, people, and terminology relevant to your domain.
   • Describe the full scope, including variations, sub-types, and edge cases.

3. Upload the policy dataset to an S3 bucket accessible using the S3 secret provided via kfp_config.

   The assessment reads this dataset at the start of the prompt generation phase and uses it to generate adversarial test prompts specific to your domain.