Alertmanager fails to send SMTP notifications due to "starttls failed" errors regarding certificate hostname validity

Environment

• OpenShift Container Platform (OCP) 3.11
• OpenShift Container Platform (OCP) 4

Issue

• Alertmanager fails to send SMTP notifications for custom alerts.
• The following errors stream in the alertmanager container logs:

$ oc -n openshift-monitoring logs -f alertmanager-main-0 -c alertmanager
[...]
2019-06-11T09:51:00.326870516Z level=error ts=2019-06-11T09:51:00.326553215Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="starttls failed: x509: certificate is valid for smtp.example.com, not smtp-relay.example.com"
2019-06-11T09:51:00.326900117Z level=error ts=2019-06-11T09:51:00.326668215Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=1 err="starttls failed: x509: certificate is valid for smtp.example.com, not smtp-relay.example.com"

Resolution

• To amend this issue, the SMTP server should change to one with a valid TLS certificate.

See also:

• Alertmanager fails to send SMTP notifications due to "starttls failed" errors regarding certificate signed by unknown authority

Root Cause

• Golang does not support unencrypted connections to remote SMTP endpoints as is specified in the Prometheus documentation, detailed under the Content from prometheus.io is not included.global section

• Due to the common name mismatch in the TLS certificate, alertmanager fails to complete the SMTP authentication and send the email notifications.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.