Send test alerts to Alertmanager in OpenShift 4

Environment

• Red Hat OpenShift Container Platform 4

Issue

• A receiver has to be tested, for instance, with a critical alert. A real critical alert cannot be forced in production.
• Verify that your receiver is configured correctly by creating a test alert.

Resolution

1. Get a list of the configured routes:

$ oc exec alertmanager-main-0 -n openshift-monitoring -- amtool config routes show --alertmanager.url http://localhost:9093

Example output

Routing tree:
.
└── default-route  receiver: default
    ├── {alertname="Watchdog"}  receiver: Watchdog
    └── {service="example-app"}  receiver: default
        └── {severity="critical"}  receiver: team-frontend-page

2. Print the route to your chosen receiver. The following example shows the receiver used for alerts with service=example-app and severity=critical matchers.

$ oc exec alertmanager-main-0 -n openshift-monitoring -- amtool config routes test service=example-app severity=critical --alertmanager.url http://localhost:9093

Example output

team-frontend-page

3. Create a test alert and add it to the Alertmanager. The following example creates an alert with service=example-app and severity=critical to test the team-frontend-page receiver:

$ oc exec alertmanager-main-0 -n openshift-monitoring -- amtool alert add --alertmanager.url http://localhost:9093 alertname=myalarm  --start="2025-03-31T00:00:00-00:00" service=example-app severity=critical --annotation="summary=\"This is a test alert with a custom summary\""

4. Verify that the alert was generated:

$ oc exec alertmanager-main-0 -n openshift-monitoring -- amtool alert --alertmanager.url http://localhost:9093

Example output

Alertname  Starts At                Summary                                                                                  State   
myalarm    2025-03-31 00:00:00 UTC  This is a test alert with a custom summary                                               active  
Watchdog   2025-04-07 10:07:16 UTC  An alert that should always be firing to certify that Alertmanager is working properly.  active  

5. Verify that the receiver was notified with the myalarm alert.

IMPORTANT NOTE: The test alert does not appear in the console nor queries to Prometheus alerts API Content from prometheus-k8s.openshift-monitoring.svc is not included.Content from prometheus-k8s.openshift-monitoring.svc is not included.https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/alerts. The test alerts are not generated in Prometheus at all.

Root Cause

• Need to test receivers without generating real alerts.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.