Alertmanager fails to send SMTP notifications due to "starttls failed" errors regarding certificate signed by unknown authority

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform
    • 4

Issue

  • Alertmanager fails to send SMTP notifications for custom alerts.

  • The following errors stream in the alertmanager container logs:

    level=error ts=2019-09-05T20:41:09.059151352Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="starttls failed: x509: certificate signed by unknown authority"
level=error ts=2019-09-05T20:41:09.059462294Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="starttls failed: x509: certificate signed by unknown authority"

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Disable certificate verification in AlertManager by configuring insecure_skip_verify: True.

<email_config>
   <tls_config>
     [ insecure_skip_verify: <boolean> | default = false]

Reference:

See also:

Root Cause

The Certification Authority (CA) is unknown.

Diagnostic Steps

Check the alertmanager pod logs to find the error starttls failed: x509: certificate signed by unknown authority:

$ oc -n openshift-monitoring logs <alertmanager pod>
level=error ts=2019-09-05T20:41:09.059151352Z caller=notify.go:332 component=dispatcher msg="Error on notify" err="starttls failed: x509: certificate signed by unknown authority"
level=error ts=2019-09-05T20:41:09.059462294Z caller=dispatch.go:280 component=dispatcher msg="Notify for alerts failed" num_alerts=11 err="starttls failed: x509: certificate signed by unknown authority"
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.