Prometheus externalLabels `Cluster:` enforced from RHOCP 4.20

Environment

• Red Hat OpenShift Container Platform 4.20

Issue

• The protected label cluster can no longer be used as an externalLabel in Prometheus from Openshift 4.20

• Using the cluster key name can cause issues with developer dashboards

• In the events from the openshift-monitoring namespace could be present the error:

  error unmarshaling: reserved key "cluster" (one of [prometheus prometheus_replica cluster]) cannot be set in externalLabels
  

Resolution

If this label is required by an external system then it is possible to do a writeRelabelConfigs on the label within any remote_write configuration:

        writeRelabelConfigs:
        - sourceLabels: [cluster_id]
          targetLabel: cluster
        - regex: cluster_id
          action: labeldrop 

Root Cause

The following warning exists within our documentation:

Warning
Do not use prometheus or prometheus_replica as key names, because they are reserved and will be overwritten.
Do not use cluster as a key name. Using it can cause issues where you are unable to see data in the developer dashboards.

The protection of these labels is now enforced.

Diagnostic Steps

To check the currently set externalLabels run the following command:

oc get configmap -n openshift-monitoring cluster-monitoring-config -o json | jq 

The following error may also be seen in the event in the openshift-monitoring project:

error unmarshaling: reserved key "cluster" (one of [prometheus prometheus_replica cluster]) cannot be set in externalLabels

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.