Prometheus metrics reference
Automation orchestrator exposes the following Prometheus metrics in OpenMetrics format at /metrics endpoints on the API service and on each worker process. All metric names use the orchestrator_ prefix.
Important:
Prometheus metrics in automation orchestrator is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product capabilities, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.
Request metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
orchestrator_requests_total |
Counter | status,endpoint,interface |
Total number of requests processed. |
orchestrator_request_duration_seconds |
Histogram | endpoint,method,interface |
Request duration in seconds. |
orchestrator_errors_total |
Counter | error_type,interface |
Total number of errors by type. Theerror_type label classifies errors astimeout,rate_limit,validation, orinternal. |
orchestrator_auth_failures_total |
Counter | failure_type,interface |
Total number of authentication failures by type. |
Authentication failure types
The failure_type label on orchestrator_auth_failures_total uses the following values:
| Failure type | Description |
|---|---|
invalid_token |
Token signature or format is invalid. |
expired_token |
Token has passed its expiration time. |
missing_credentials |
Request did not include authentication credentials. |
globally_revoked |
All tokens were revoked by an administrator. |
refresh_revoked |
Refresh token was individually revoked. |
csrf_failed |
Cross-site request forgery validation failed. |
disabled_user |
User account is disabled. |
stale_token |
Token was issued before the most recent global revocation. |
disabled_sa |
Service account is disabled. |
revoked_sa_token |
Service account token was revoked. |
Workflow metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
orchestrator_workflows_total |
Counter | workflow_type |
Total number of workflow executions started. |
orchestrator_workflow_duration_seconds |
Histogram | (none) | Workflow execution duration in seconds. |
orchestrator_activity_duration_seconds |
Histogram | activity_name,status,workflow_type |
Activity execution duration in seconds. |
orchestrator_active_workflows |
Gauge | (none) | Number of currently active workflows. |
orchestrator_workflow_start_latency_seconds |
Histogram | component |
Workflow start latency in seconds. |
orchestrator_workflow_completion_rate |
Gauge | component |
Workflow completion rate. |
orchestrator_workflow_creation_success_rate |
Gauge | component |
Workflow creation success rate. |
orchestrator_workflow_serialization_duration_seconds |
Histogram | component |
Workflow serialization duration in seconds. |
orchestrator_workflow_validation_duration_seconds |
Histogram | component |
Workflow validation duration in seconds. |
orchestrator_context_duration_seconds |
Histogram | (none) | Context preparation overhead duration in seconds. |
orchestrator_temporal_execution_service_duration_seconds |
Histogram | component |
Temporal clientstart_workflow RPC duration including network and server startup. |
Large language model (LLM) metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
orchestrator_llm_calls_total |
Counter | model,status |
Total number of LLM API calls. |
orchestrator_llm_duration_seconds |
Histogram | model |
LLM API call duration in seconds. |
orchestrator_ttft_seconds |
Histogram | model |
Time to first token in seconds. |
orchestrator_llm_tokens_input_total |
Counter | model |
Total input tokens sent to LLM. |
orchestrator_llm_tokens_output_total |
Counter | model |
Total output tokens received from LLM. |
orchestrator_active_llm_requests |
Gauge | (none) | Number of in-flight LLM requests. |
Scheduled trigger metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
orchestrator_scheduled_trigger_fires_total |
Counter | status |
Total number of scheduled trigger executions. |
orchestrator_scheduled_trigger_latency_seconds |
Histogram | (none) | Latency between the scheduled fire time and the actual execution start, in seconds. |
Tool metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
orchestrator_tool_executions_total |
Counter | namespaced_name,status,error_code |
Total number of tool executions. |
orchestrator_tool_execution_duration_seconds |
Histogram | namespaced_name |
Tool execution duration in seconds. |
Authorization metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
orchestrator_authz_duration_seconds |
Histogram | resource_type,action |
Authorization check duration in seconds, including policy evaluation. |
orchestrator_opa_request_duration_seconds |
Histogram | resource_type,action |
Open Policy Agent (OPA) policy evaluation HTTP round-trip duration in seconds. |
Cache metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
orchestrator_cache_hits_total |
Counter | (none) | Total number of cache hits. |
orchestrator_cache_misses_total |
Counter | (none) | Total number of cache misses. |
orchestrator_cache_lookup_duration_seconds |
Histogram | (none) | Cache lookup duration in seconds. |
orchestrator_cache_utilization_ratio |
Gauge | (none) | Current cache utilization ratio, ranging from 0.0 to 1.0. |
Database metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
orchestrator_database_query_response_time_seconds |
Histogram | component,statement_type |
Database query response time in seconds. |
orchestrator_database_connection_pool_utilization |
Gauge | component |
Database connection pool utilization ratio. |
orchestrator_database_transaction_rate_tps |
Gauge | component |
Database transaction rate in transactions per second. |
API component metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
orchestrator_api_response_time_seconds |
Histogram | component,endpoint,method |
API response time in seconds, labeled by component. |
orchestrator_api_error_rate |
Gauge | component |
API error rate by component. |
orchestrator_api_throughput_rps |
Gauge | component |
API throughput in requests per second by component. |
System metrics
| Metric name | Type | Labels | Description |
|---|---|---|---|
orchestrator_system_uptime |
Gauge | component |
System uptime in seconds by component. |
orchestrator_system_e2e_latency_seconds |
Histogram | component |
System end-to-end latency in seconds. |
orchestrator_system_error_rate |
Gauge | component |
System-wide error rate by component. |
orchestrator_temporal_queue_depth |
Gauge | component,task_queue |
Temporal task queue depth. Thetask_queue label identifies which queue is measured (workflow or background). |
orchestrator_activity_execution_success_rate |
Gauge | component |
Activity execution success rate. |
Component labels
Metrics that include a component label use values from a predefined registry. The following component values are available:
| Component label | Description |
|---|---|
api_service |
API service layer |
workflow_engine |
Workflow engine |
temporal_worker |
Temporal worker |
execution_service |
Execution service |
invocation_service |
Invocation service |
routing_service |
Routing service |
tool_manager |
Tool manager |
database |
Database layer |
system_wide |
System-wide aggregate |
Interface labels
Request metrics that include an interface label distinguish between requests from the web interface and requests from external clients such as the API, CLI tools, or CI/CD pipelines.
| Interface label | Description |
|---|---|
ui |
Request originated from the automation orchestrator web interface. |
api |
Request originated from an external client (API call, CLI, CI/CD pipeline, or other integration). |
Histogram bucket boundaries
Histograms use tuned bucket boundaries based on expected latency profiles:
| Profile | Boundaries (seconds) | Used by |
|---|---|---|
| Fast | 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1.0 | TTFT, cache lookups, context preparation, authorization checks, OPA policy evaluation |
| Medium | 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 7.5, 10.0 | Request duration, LLM duration, API response time, database queries, tool execution, activity duration, workflow serialization duration, workflow validation duration, workflow start latency, Temporal execution service, scheduled trigger latency |
| Slow | 1.0, 5.0, 10.0, 30.0, 60.0, 120.0, 300.0 | Workflow execution duration, system end-to-end latency |
Endpoint and scrape configuration
| Setting | Value |
|---|---|
| API service endpoint path | /metrics (on the standard HTTP port) |
| Worker endpoint path | /metrics (on port9090 by default, configurable withAPP_METRICS_WORKER_PORT) |
| Format | OpenMetrics (Prometheus-compatible) |
| Excluded paths | /healthz/*,/api,/api/v1,/api_docs/v1/docs,/api_docs/v1/redoc,/api_docs/v1/openapi.json,/_internal/* |
Tip:
The operator creates ServiceMonitor resources with a 30-second scrape interval by default. If you configure an external Prometheus scrape target, a 30-second interval provides a good balance between data resolution and resource consumption.