How to get fired alerts on OpenShift using cli or script?

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 3.11
    • 4

Issue

  • While we can see any fired alerts on Alertmanager GUI of OpenShift, how to get fired alerts on OpenShift with cli for getting alerts in scripts?
  • Is it possible to get the fired alerts in OCP from the command line?

Resolution

It's possible to use curl to get the fired alerts from the Alertmanager API with the following commands:

### Gets alertmanager URL
$ oc get route alertmanager-main -n openshift-monitoring
NAME                HOST/PORT                                                     ...
alertmanager-main   alertmanager-main-openshift-monitoring.apps.url.openshift.com ...

$ ALERT_MANAGER=$(oc get route alertmanager-main -n openshift-monitoring -o jsonpath='{@.spec.host}')

### Gets fired alerts
$ curl -k -H "Authorization: Bearer $(oc create token prometheus-k8s -n openshift-monitoring)"  https://$ALERT_MANAGER/api/v1/alerts

### If you receive the error, "The Alertmanager v1 API was deprecated in version 0.16.0 and is removed as of version 0.27.0 - please use the equivalent route in the v2 API", replace v1 with v2 in the URL and execute the command again:

$ curl -k -H "Authorization: Bearer $(oc create token prometheus-k8s -n openshift-monitoring)"  https://$ALERT_MANAGER/api/v2/alerts

Its also possible to get the alerts from the Prometheus API 

oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -s   'http://localhost:9090/api/v1/alerts'

Also it is possible to query Prometheus for alerts historically - between two dates and times

oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -s   'http://localhost:9090/api/v1/query_range?query=ALERTS&start=2022-08-08T00:00:00.781Z&end=2022-08-09T00:00:00.781Z&step=1m'

You can also list the active alerts using "amtool" as follows.

oc rsh -n openshift-monitoring alertmanager-main-0 \
amtool alert query --alertmanager.url http://localhost:9093

### Additionally, you can filter a specific alert using "alertname" option as well.

oc rsh -n openshift-monitoring alertmanager-main-0 \
amtool alert query --alertmanager.url http://localhost:9093 alertname=XXXX
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.