Troubleshoot Prometheus metrics collection

Verify network connectivity to metrics endpoints

Confirm that your monitoring system can reach the API and worker services. Run the following command for each service, replacing service-name, namespace, and port with the values for your deployment:

$ oc run -i --tty --rm debug --image=curlimages/curl --restart=Never -- \
  curl -I "http://service-name.namespace.svc.cluster.local:port/metrics"

A successful response confirms the endpoint is reachable. If the connection fails, verify the service name and port with oc get svc -n namespace --show-labels.

Verify that the metrics endpoint returns data

List the services to find the backend service name:

$ oc get svc -n namespace -l app.kubernetes.io/component=orchestrator

Then query the metrics endpoint. Replace service-name with the backend service name from the previous command. If internal TLS is enabled on your deployment, use https and pass the -sk flags to skip certificate verification. If internal TLS is not enabled, use http instead:

$ oc run -i --tty --rm debug --image=registry.access.redhat.com/ubi9/ubi --restart=Never -- \
  curl -sk https://service-name.namespace.svc.cluster.local:8000/metrics | head -20

Verify that user workload monitoring is enabled

This check applies only to OpenShift User Workload Monitoring. Run the following command to check for running pods in the openshift-user-workload-monitoring namespace:

$ oc get pods -n openshift-user-workload-monitoring

If no pods are running in this namespace, user workload monitoring is not enabled. See Integrate automation orchestrator metrics with OpenShift monitoring for instructions to enable it.

Check the Prometheus Operator logs

This check applies only to OpenShift User Workload Monitoring. Review the Prometheus Operator logs for target discovery issues:

$ oc logs -l app.kubernetes.io/name=prometheus-operator -n openshift-user-workload-monitoring