Procedure

1. Add Platform metrics by creating a ConfigMap resource on the hub cluster in the open-cluster-management-observability namespace. Use observability-metrics-custom-allowlist as the name. See the following ConfigMap example that you can use to monitor Platform metrics:

Procedure

1. Set the configuration on the managed cluster in namespace where you want to capture the user workload metrics. The name must be observability-metrics-custom-allowlist.
2. The following example monitors the User workload metric sample_metrics from the namespace monitored_namespace. If you create the configuration in the open-cluster-management-addon-observability namespace instead, the metrics are collected from all the namespaces of the managed cluster. See the following example:

Procedure

1. Verify that mco observability is enabled by using the following command:

   oc get mco observability -o yaml

   Copy code to clipboard

2. Add the name of the default metric to the metrics_list.yaml parameter with a hyphen - at the start of the metric name. View the following metric example:

   -cluster_infrastructure_provider

   Copy code to clipboard

3. Create the observability-metrics-custom-allowlist config map in the open-cluster-management-observability namespace with the following command:

   oc apply -n open-cluster-management-observability -f observability-metrics-custom-allowlist.yaml

   Copy code to clipboard
4. Verify that the observability service is not collecting the specific metric from your managed clusters. When you query the metric from the Grafana dashboard, the metric is not displayed.

Procedure

1. To increase and decrease metrics collection on your clusters, edit the workers parameter in the multicluster-observability-operator resource.

2. If you want to use the same workers value for each of your managed clusters, set the value in your MultiClusterObservability custom resource definition. By default, the value is set to 1. For example, change the workers parameter value to 4. Your YAML file might resemble the following resource:

   apiVersion: observability.open-cluster-management.io/v1beta2
   kind: MultiClusterObservability
   metadata:
     name: observability
   spec:
     observabilityAddonSpec:
       enableMetrics: true
       workers: 4

   Copy code to clipboard

3. Optional: If you want to override the workers parameter for specific clusters, add the observability.open-cluster-management.io/addon-source: "override" annotation to the ObservabilityAddOn specification of your managed cluster.

   1. To revert the override, use the observability.open-cluster-management.io/addon-source: "mco" annotation.
4. To verify the value of the workers parameter, check the values in the MultiClusterObservability custom resource.

Procedure

1. Configure proxy settings by navigating to the cluster namespace on your hub cluster.

2. Create an AddOnDeploymentConfig resource with the proxy settings by adding a spec.proxyConfig parameter. View the following YAML example:

   apiVersion: addon.open-cluster-management.io/v1alpha1
   kind: AddOnDeploymentConfig
   metadata:
     name: <addon-deploy-config-name>
     namespace: <managed-cluster-name>
   spec:
     agentInstallNamespace: open-cluster-management-addon-observability
     proxyConfig:
       httpsProxy: "http://<username>:<password>@<ip>:<port>"
       noProxy: ".cluster.local,.svc,172.30.0.1"

   Copy code to clipboard
   • For the httpsProxy parameter, specify either a HTTP proxy or a HTTPS proxy.
   • For the noProxy parameter, include the IP address of the kube-apiserver.

3. To get the IP address, run following command on your managed cluster:

   oc -n default describe svc kubernetes | grep IP:

   Copy code to clipboard

4. Go to the ManagedClusterAddOn resource and update it by referencing the AddOnDeploymentConfig resource that you made. View the following YAML example:

   apiVersion: addon.open-cluster-management.io/v1alpha1
   kind: ManagedClusterAddOn
   metadata:
     name: observability-controller
     namespace: <managed-cluster-name>
   spec:
     installNamespace: open-cluster-management-addon-observability
     configs:
     - group: addon.open-cluster-management.io
       resource: addondeploymentconfigs
       name: <addon-deploy-config-name>
       namespace: <managed-cluster-name>

   Copy code to clipboard

   Note: The spec.installNamespace field is deprecated. See This content is not included.Product deprecations and removals to learn more.

5. Verify the proxy settings. If you successfully configured the proxy settings, the metric collector deployed by the observability add-on agent on the managed cluster sends the data to the hub cluster. Complete the following steps:

   1. Go to the hub cluster then the managed cluster on the Grafana dashboard.
   2. View the metrics for the proxy settings.

6. Disable proxy settings for observability add-ons if you need to make development changes.

   1. Go to the ManagedClusterAddOn resource.
   2. Remove the referenced AddOnDeploymentConfig resource.

Procedure

1. Customize certificates for accessing the object store. To validate the object store connection, create the Secret object in the file that contains the certificate authority by using the following command:

   oc create secret generic <tls_secret_name> --from-file=ca.crt=<path_to_file> -n open-cluster-management-observability

   Copy code to clipboard

2. Alternatively, you can apply the following YAML to create the secret:

   apiVersion: v1
   kind: Secret
   metadata:
     name: <tls_secret_name>
     namespace: open-cluster-management-observability
   type: Opaque
   data:
     ca.crt: <base64_encoded_ca_certificate>

   Copy code to clipboard

   Optional: If you want to enable mutual TLS, you need to add the public.crt and private.key keys in the previous secret.

3. Add the TLS secret details to the metricObjectStorage section by using the following command:

   oc edit mco observability -o yaml

   Copy code to clipboard

   Your file might resemble the following YAML:

   metricObjectStorage:
     key: thanos.yaml
     name: thanos-object-storage
     tlsSecretName: tls-certs-secret
     tlsSecretMountPath: /etc/<s3-directory>/certs

   Copy code to clipboard
   • The value for tlsSecretName is the name of the Secret object that you previously created.
   • The /etc/<s3-directory>/certs/ path defined for the tlsSecretMountPath parameter specifies where the certificates are mounted in the observability components. This path is required for the next step.

4. Update the thanos.yaml definition in the thanos-object-storage secret by adding the http_config.tls_config section with the certificate details. View the following example. Replace values where needed:

   thanos.yaml: |
      type: s3
      config:
        bucket: <bucket-name>
        endpoint: <s3.port>
        insecure: false
        access_key: <s3-access>
        secret_key: <s3-secret>
        http_config:
          tls_config:
            ca_file: /etc/<s3-directory>/certs/ca.crt
            insecure_skip_verify: false

   Copy code to clipboard
   • Set the insecure parameter to false to enable HTTPS.

   • The path for the ca_file parameter must match the tlsSecretMountPath from the MultiClusterObservability custom resource. The ca.crt must match the key in the <tls_secret_name> Secret resource.

     Optional: If you want to enable mutual TLS, you need to add the cert_file and key_file keys to the tls_config section. See the following example. Replace values where needed:

    thanos.yaml: |
       type: s3
       config:
         bucket: <bucket-name>
         endpoint: <s3.port>
         insecure: false
         access_key: <s3-access>
         secret_key: <s3-secret>
         http_config:
           tls_config:
             ca_file: /etc/<s3-directory>/certs/ca.crt
             cert_file: /etc/<s3-directory>/certs/public.crt
             key_file: /etc/<s3-directory>/certs/private.key
             insecure_skip_verify: false

   Copy code to clipboard
   • The path for ca_file, cert_file, and key_file must match the tlsSecretMountPath from the MultiClusterObservability custom resource. The ca.crt, public.crt, and private.crt must match the respective key in the tls_secret_name> Secret resource.

5. To verify that you can access the object store, check that the pods are deployed. Run the following command:

   oc -n open-cluster-management-observability get pods -l app.kubernetes.io/name=thanos-store

   Copy code to clipboard

Procedure

1. Create the following custom alert rule to get a notification for when your CPU usage passes your defined value:

   data:
     custom_rules.yaml: |
       groups:
         - name: cluster-health
           rules:
           - alert: ClusterCPUHealth-jb
             annotations:
               summary: Notify when CPU utilization on a cluster is greater than the defined utilization limit
               description: "The cluster has a high CPU usage: {{ $value }} core for {{ $labels.cluster }} {{ $labels.clusterID }}."
             expr: |
               max(cluster:cpu_usage_cores:sum) by (clusterID, cluster, prometheus) > 0
             for: 5s
             labels:
               cluster: "{{ $labels.cluster }}"
               prometheus: "{{ $labels.prometheus }}"
               severity: critical

   Copy code to clipboard

   Notes:

   • When you update your custom rules, observability-thanos-rule pods restart automatically.
   • You can create multiple rules in the configuration.
   • The default alert rules are in the observability-thanos-rule-default-rules config map of the open-cluster-management-observability namespace.

2. Create a custom recording rule to get the sum of the container memory cache of a pod. See the following example:

   data:
     custom_rules.yaml: |
       groups:
         - name: container-memory
           rules:
           - record: pod:container_memory_cache:sum
             expr: sum(container_memory_cache{pod!=""}) BY (pod, container)

   Copy code to clipboard

   Note: After you make changes to the config map, the configuration automatically reloads. The configuration reloads because of the config-reload within the observability-thanos-rule sidecar.

3. Verify that the alert rules are functioning correctly. Navigate to the Grafana dashboard by selecting the Explore page and query ALERTS. The alert is only available in Grafana if you created the alert.

Procedure

1. To reference secrets within your Alertmanager configuration, add the Secret resource content within the open-cluster-management-observability namespace and mount the content within the alertmanager pods.

2. To create a tls secret with TLS certificates, run the following command:

   oc create secret tls tls --cert=</path/to/cert.crt> --key=</path/to/cert.key> -n open-cluster-management-observability

   Copy code to clipboard

3. To mount the tls secret to your MultiClusterObservability resource, add it to the advanced section. Your resource might resemble the following content:

   ...
   advanced:
    alertmanager:
      secrets: ['tls']

   Copy code to clipboard

4. To add a reference of your tls secret within your Alertmanager configuration, add the path of your secret to the configuration. Your resource might resemble the following configuration:

   tls_config:
    cert_file: '/etc/alertmanager/secrets/tls/tls.crt'
    key_file: '/etc/alertmanager/secrets/tls/tls.key'

   Copy code to clipboard

5. To verify that the secrets are within your alertmanager pods, run the following command:

   oc -n open-cluster-management-observability get secret alertmanager-config --template='{{ index .data "alertmanager.yaml" }}' |base64 -d > alertmanager.yaml

   Copy code to clipboard

   Your YAML might resemble the following contents:

   "global":
     "http_config":
       "tls_config":
         "cert_file": "/etc/alertmanager/secrets/storyverify/tls.crt"
         "key_file": "/etc/alertmanager/secrets/storyverify/tls.key"

   Copy code to clipboard

6. To save the alertmanager.yaml configuration in the alertmanager-config secret, run the following command:

   oc -n open-cluster-management-observability create secret generic alertmanager-config --from-file=alertmanager.yaml --dry-run -o=yaml

   Copy code to clipboard

7. To replace the previous secret with your new secret, run the following command:

   oc -n open-cluster-management-observability replace secret --filename=-

   Copy code to clipboard

Procedure

1. Create the alertmanager-config YAML file to configure alerts with an external notification system.

   View the following example of the alertmanager-config YAML file:

1. If you want to configure a proxy for alert forwarding, add the following global entry to the alertmanager-config YAML file:

1. To forward user workload alerts, configure the alerts to be processed by the user workload Prometheus instance and not by Thanos ruler. See the following example of the PrometheusRule resource:

   apiVersion: monitoring.coreos.com/v1
   kind: PrometheusRule
   metadata:
     labels:
       openshift.io/prometheus-rule-evaluation-scope: leaf-prometheus

   Copy code to clipboard

Procedure

1. Run the following command to set the mco-disable-alerting parameter to "true":

   oc annotate MultiClusterObservability observability mco-disable-alerting=true

   Copy code to clipboard
2. Verify that the hub cluster alert manager is no longer propagating managed cluster alerts to third-party messaging tools.

Procedure

1. Navigate to the console from your hub cluster.
2. Locate the MultiClusterObservability custom resource, and update the replicas parameter value for the component where you want to change the replicas. Your updated YAML might resemble the following content:

Procedure

1. To increase the size of the persistent volume, update the MultiClusterObservability custom resource if the storage class support expanding volumes.

2. To decrease the size of the persistent volumes remove the pods using the persistent volumes, delete the persistent volume and re-create them. You might experience data loss in the persistent volume. Complete the following steps:

   1. Pause the MultiClusterObservability operator by adding the annotation mco-pause: "true" to the MultiClusterObservability custom resource.

   2. Look for the stateful sets or deployments of the desired component. Change their replica count to 0. This initiates a shutdown, which involves uploading local data when applicable to avoid data loss. For example, the Thanos Receive stateful set is named observability-thanos-receive-default and has three replicas by default. Therefore, you are looking for the following persistent volume claims:

      • data-observability-thanos-receive-default-0
      • data-observability-thanos-receive-default-1
      • data-observability-thanos-receive-default-2
   3. Delete the persistent volumes and persistent volume claims used by the desired component.
   4. In the MultiClusterObservability custom resource, edit the storage size in the configuration of the component to the desired amount in the storage size field. Prefix with the name of the component.
   5. Unpause the MultiClusterObservability operator by removing the previously added annotation.
   6. To initiate a reconcilation after having the operator paused, delete the multicluster-observability-operator and observatorium-operator pods. The pods are recreated and reconciled immediately.
3. Verify that persistent volume and volume claims are updated by checking the MultiClusterObservability custom resource.

Procedure

1. Add your URLs to the advanced section of the MultiClusterObservability spec. See the following example:

   spec:
     advanced:
       customObservabilityHubURL: <yourURL>
       customAlertmanagerHubURL: <yourURL>

   Copy code to clipboard

   Notes:

   • Only HTTPS URLs are supported. If you do not add https:// to your URL, the scheme is added automatically.
   • You can include the standard path for the Remote Write API, /api/metrics/v1/default/api/v1/receive in the customObservabilityHubURL spec. If you do not include the path, the observability service automatically adds the path at runtime.
   • Any intermediate component you use for the custom observability hub cluster URL cannot use TLS termination because the component relies on MTLS authentication. The custom Alertmanager hub cluster URL supports intermediate component TLS termination by using your own existing certificate instructions.

2. If you are using a customObservabilityHubURL, create a route object by using the following template. Replace <intermediate_component_url> with the intermediate component URL:

   apiVersion: route.openshift.io/v1
   kind: Route
   metadata:
     name: proxy-observatorium-api
     namespace: open-cluster-management-observability
   spec:
     host: <intermediate_component_url>
     port:
       targetPort: public
     tls:
       insecureEdgeTerminationPolicy: None
       termination: passthrough
     to:
       kind: Service
       name: observability-observatorium-api
       weight: 100
     wildcardPolicy: None

   Copy code to clipboard

3. If you are using a customAlertmanagerHubURL, create a route object by using the following template. Replace <intermediate_component_url> with the intermediate component URL:

   apiVersion: route.openshift.io/v1
   kind: Route
   metadata:
     name: alertmanager-proxy
     namespace: open-cluster-management-observability
   spec:
     host: <intermediate_component_url>
     path: /api/v2
     port:
       targetPort: oauth-proxy
     tls:
       insecureEdgeTerminationPolicy: Redirect
       termination: reencrypt
     to:
       kind: Service
       name: alertmanager
       weight: 100
     wildcardPolicy: None

   Copy code to clipboard

Procedure

1. To silence a Red Hat Advanced Cluster Management alert, you must have access to the alertmanager pods in the open-cluster-management-observability namespace. Run the following command in the observability-alertmanager-0 pod terminal to silence SampleAlert:

   amtool silence add --alertmanager.url="http://localhost:9093" --author="user" --comment="Silencing sample alert" alertname="SampleAlert"

   Copy code to clipboard

2. Silence an alert by using multiple match labels. The following command uses match-label-1 and match-label-2:

   amtool silence add --alertmanager.url="http://localhost:9093" --author="user" --comment="Silencing sample alert" <match-label-1>=<match-value-1> <match-label-2>=<match-value-2>

   Copy code to clipboard

3. If you want to silence an alert for a specific period of time, use the --duration flag. Run the following command to silence the SampleAlert for an hour:

   amtool silence add --alertmanager.url="http://localhost:9093" --author="user" --comment="Silencing sample alert" --duration="1h" alertname="SampleAlert"

   Copy code to clipboard

4. Define a start or end time for the silenced alert. Run the following command to silence the SampleAlert at a specific start time:

   amtool silence add --alertmanager.url="http://localhost:9093" --author="user" --comment="Silencing sample alert" --start="2023-04-14T15:04:05-07:00" alertname="SampleAlert"

   Copy code to clipboard

5. To view all silenced alerts that are created, run the following command:

   amtool silence --alertmanager.url="http://localhost:9093"

   Copy code to clipboard

6. If you no longer want to silence an alert, end the silencing by running the following command:

   amtool silence expire --alertmanager.url="http://localhost:9093" "d839aca9-ed46-40be-84c4-dca8773671da"

   Copy code to clipboard

7. To end the silencing of all alerts, run the following command:

   amtool silence expire --alertmanager.url="http://localhost:9093" $(amtool silence query --alertmanager.url="http://localhost:9093" -q)

   Copy code to clipboard

Procedure

1. Mute an alert when there is a set of parameter matches that match another set of existing matchers. In order for the rule to take effect, both the target and source alerts must have the same label values for the label names in the equal list. Your inhibit_rules might resemble the following:

Procedure

1. Define a ClusterRole resource with permissions to access metrics. Your resource might resemble the following YAML:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRole
   metadata:
    name: awesome-app-metrics-role
   rules:
    - apiGroups:
        - "cluster.open-cluster-management.io"
      resources:
        - managedclusters:
      resourceNames:
        - devcluster1
        - devcluster2
      verbs:
        - metrics/AwesomeAppNS

   Copy code to clipboard
   • managedclusters represents the parameter values for the managed clusters.
   • resourceNames represents the list of managed clusters.
   • verbs represents the namespace of the managed clusters.

2. Define a ClusterRoleBinding resource that binds the group, my-awesome-app-admins, with the ClusterRole resource for the awesome-app-metrics-role. Your resource might resemble the following YAML:

   kind: ClusterRoleBinding
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
    name: awesome-app-metrics-role-binding
   subjects:
    - kind: Group
      apiGroup: rbac.authorization.k8s.io
      name: my-awesome-app-admins
   roleRef:
    apiGroup: rbac.authorization.k8s.io
    kind: ClusterRole
    name: awesome-app-metrics-role

   Copy code to clipboard

   Note: When the users in the my-awesome-app-admins namespace log in to the Grafana console, they have the following restrictions:

   • Users see no data for dashboards that summarize fleet level data.
   • Users can only select managed clusters and namespaces specified in the ClusterRole resource.
3. To set up different types of user access, define separate ClusterRoles and ClusterRoleBindings resources to represent the different managed clusters in the namespaces.

Procedure

1. Create the MultiClusterObservability custom resource YAML file named multiclusterobservability_cr.yaml. View the following default YAML file for observability:

   apiVersion: observability.open-cluster-management.io/v1beta2
   kind: MultiClusterObservability
   metadata:
     name: observability
   spec:
     observabilityAddonSpec: {}
     storageConfig:
       metricObjectStorage:
         name: thanos-object-storage
         key: thanos.yaml

   Copy code to clipboard

   Note: You might want to modify the value for the retentionConfig parameter in the advanced section. Depending on the number of managed clusters, you might want to update the amount of storage for stateful sets. If your S3 bucket is configured to use STS tokens, annotate the service accounts to use STS with S3 role. View the following configuration:

   spec:
     advanced:
       compact:
          serviceAccountAnnotations:
              eks.amazonaws.com/role-arn: $S3_ROLE
       store:
          serviceAccountAnnotations:
             eks.amazonaws.com/role-arn: $S3_ROLE
       rule:
          serviceAccountAnnotations:
             eks.amazonaws.com/role-arn: $S3_ROLE
       receive:
          serviceAccountAnnotations:
             eks.amazonaws.com/role-arn: $S3_ROLE
       query:
          serviceAccountAnnotations:
             eks.amazonaws.com/role-arn: $S3_ROLE

   Copy code to clipboard

   See This content is not included.Observability API for more information.

2. To deploy on infrastructure machine sets, you must set a label for your set by updating the nodeSelector in the MultiClusterObservability YAML file. Your YAML file might resemble the following content:

     nodeSelector:
       node-role.kubernetes.io/infra: ""

   Copy code to clipboard

3. Apply the observability YAML to your cluster by running the following command:

   oc apply -f multiclusterobservability_cr.yaml

   Copy code to clipboard

   Note: By default, if you do not define the storageConfig.storageClass field in the MultiClusterObservability custom resource, platform default StorageClass fields are populated in the storageConfig section of the MultiClusterObservability resource. For example, AWS default storageClass is set to gp2.

4. Verify default storageClass by running the following command:

   oc get storageClass

   Copy code to clipboard

   See the following example output:

   NAME                PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
   gp2-csi             ebs.csi.aws.com   Delete          WaitForFirstConsumer   true                   151m
   gp3-csi (default)   ebs.csi.aws.com   Delete          WaitForFirstConsumer   true                   151m

   Copy code to clipboard
5. Validate that the observability service is enabled and the data is populated by launching the Grafana dashboards.
6. Click the Grafana link that is near the console header, from either the console Overview page or the Clusters page.

7. Access the multicluster-observability-operator deployment to verify that the multicluster-observability-operator pod is being deployed by the multiclusterhub-operator deployment. Run the following command:

   oc get deploy multicluster-observability-operator -n open-cluster-management --show-labels

   Copy code to clipboard

   You might receive the following results:

   NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE   LABELS
   multicluster-observability-operator   1/1     1            1           35m   installer.name=multiclusterhub,installer.namespace=open-cluster-management

   Copy code to clipboard

8. View the labels section of the multicluster-observability-operator deployment for labels that are associated with the resource. The labels section might contain the following details:

    labels:
       installer.name: multiclusterhub
       installer.namespace: open-cluster-management

   Copy code to clipboard
9. Optional: Add the observability: disabled cluster label to your clusters if you want to exclude specific managed clusters from collecting the observability data.

Procedure

1. Create an image pull-secret named, multiclusterhub-operator-pull-secret in the open-cluster-management-observability project.
2. Create your object storage secret named, thanos-object-storage in the open-cluster-management-observability project.
3. Enter the object storage secret details, then click Create. See step four of the Enabling observability section to view an example of a secret.
4. Create the MultiClusterObservability custom resource instance. When you receive the following message, the observability service is enabled successfully from OpenShift Container Platform: Observability components are deployed and running.
5. Verify the Thanos version from the command line interface (CLI). From your hub cluster, run the following command in the observability pods to receive the Thanos version:

Procedure

1. Set up the AWS environment. Run the following commands:

   export POLICY_VERSION=$(date +"%m-%d-%y")
   export TRUST_POLICY_VERSION=$(date +"%m-%d-%y")
   export CLUSTER_NAME=<my-cluster>
   export S3_BUCKET=$CLUSTER_NAME-acm-observability
   export REGION=us-east-2
   export NAMESPACE=open-cluster-management-observability
   export SA=tbd
   export SCRATCH_DIR=/tmp/scratch
   export OIDC_PROVIDER=$(oc get authentication.config.openshift.io cluster -o json | jq -r .spec.serviceAccountIssuer| sed -e "s/^https:\/\///")
   export AWS_ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
   export AWS_PAGER=""
   rm -rf $SCRATCH_DIR
   mkdir -p $SCRATCH_DIR

   Copy code to clipboard

2. Create an S3 bucket with the following command:

   aws s3 mb s3://$S3_BUCKET

   Copy code to clipboard

3. Create a s3-policy JSON file for access to your S3 bucket. Run the following command:

   {
       "Version": "$POLICY_VERSION",
       "Statement": [
           {
               "Sid": "Statement",
               "Effect": "Allow",
               "Action": [
                   "s3:ListBucket",
                   "s3:GetObject",
                   "s3:DeleteObject",
                   "s3:PutObject",
                   "s3:PutObjectAcl",
                   "s3:CreateBucket",
                   "s3:DeleteBucket"
               ],
               "Resource": [
                   "arn:aws:s3:::$S3_BUCKET/*",
                   "arn:aws:s3:::$S3_BUCKET"
               ]
           }
       ]
    }

   Copy code to clipboard

4. Apply the policy with the following command:

   S3_POLICY=$(aws iam create-policy --policy-name $CLUSTER_NAME-acm-obs \
   --policy-document file://$SCRATCH_DIR/s3-policy.json \
   --query 'Policy.Arn' --output text)
   echo $S3_POLICY

   Copy code to clipboard

5. Create a TrustPolicy JSON file. Run the following command:

   {
    "Version": "$TRUST_POLICY_VERSION",
    "Statement": [
      {
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "${OIDC_PROVIDER}:sub": [
              "system:serviceaccount:${NAMESPACE}:observability-thanos-query",
              "system:serviceaccount:${NAMESPACE}:observability-thanos-store-shard",
              "system:serviceaccount:${NAMESPACE}:observability-thanos-compact",
              "system:serviceaccount:${NAMESPACE}:observability-thanos-rule",
              "system:serviceaccount:${NAMESPACE}:observability-thanos-receive"
            ]
          }
        }
      }
    ]
   }

   Copy code to clipboard

6. Create a role for AWS Prometheus and CloudWatch with the following command:

   S3_ROLE=$(aws iam create-role \
     --role-name "$CLUSTER_NAME-acm-obs-s3" \
     --assume-role-policy-document file://$SCRATCH_DIR/TrustPolicy.json \
     --query "Role.Arn" --output text)
   echo $S3_ROLE

   Copy code to clipboard

7. Attach the policies to the role. Run the following command:

   aws iam attach-role-policy \
     --role-name "$CLUSTER_NAME-acm-obs-s3" \
     --policy-arn $S3_POLICY

   Copy code to clipboard

   Your secret might resemble the following file. The config section specifies signature_version2: false and does not specify access_key and secret_key:

   apiVersion: v1
   kind: Secret
   metadata:
     name: thanos-object-storage
     namespace: open-cluster-management-observability
   type: Opaque
   stringData:
     thanos.yaml: |
       type: s3
       config:
         bucket: $S3_BUCKET
         endpoint: s3.$REGION.amazonaws.com
         signature_version2: false

   Copy code to clipboard
8. Specify the service account annotations in the MultiClusterObservability custom resource as described in Creating the MultiClusterObservability custom resource section.

9. Retrieve the S3 access key and secret key for your cloud providers with the following commands. You must decode, edit, and encode your base64 string in the secret:

   1. To edit and decode the S3 access key for your cloud provider, run the following command:

      YOUR_CLOUD_PROVIDER_ACCESS_KEY=$(oc -n open-cluster-management-observability get secret <object-storage-secret> -o jsonpath="{.data.thanos\.yaml}" | base64 --decode | grep access_key | awk '{print $2}')

      Copy code to clipboard

   2. To view the access key for your cloud provider, run the following command:

      echo $YOUR_CLOUD_PROVIDER_ACCESS_KEY

      Copy code to clipboard

   3. To edit and decode the secret key for your cloud provider, run the following command:

      YOUR_CLOUD_PROVIDER_SECRET_KEY=$(oc -n open-cluster-management-observability get secret <object-storage-secret> -o jsonpath="{.data.thanos\.yaml}" | base64 --decode | grep secret_key | awk '{print $2}')

      Copy code to clipboard
   4. Run the following command to view the secret key for your cloud provider:

   echo $YOUR_CLOUD_PROVIDER_SECRET_KEY

   Copy code to clipboard

Procedure

1. Create an AWS Identity and Acess Management (IAM) policy that restricts access to your target Amazon S3 bucket.
2. Create an IAM role containing a trust policy that generate JWT tokens for OpenShift Container Platform service accounts.
3. Annotate the observability service accounts that require access to the Amazon S3 bucket.

Procedure

1. Clone the multicluster-observability-operator repository so that you are able to run the scripts that are in the tools folder:

   git clone https://github.com/open-cluster-management/multicluster-observability-operator.git
   cd multicluster-observability-operator/tools

   Copy code to clipboard

2. Run the setup-grafana-dev.sh script to set up your Grafana instance. When you run the script with the --deploy flag, the following resources are created: secret/grafana-dev-config, deployment.apps/grafana-dev, service/grafana-dev, ingress.extensions/grafana-dev, and persistentvolumeclaim/grafana-dev:

   ./setup-grafana-dev.sh --deploy

   Copy code to clipboard

   Your output might resemble the following content:

   secret/grafana-dev-config created
   deployment.apps/grafana-dev created
   service/grafana-dev created
   serviceaccount/grafana-dev created
   clusterrolebinding.rbac.authorization.k8s.io/open-cluster-management:grafana-crb-dev created
   route.route.openshift.io/grafana-dev created
   persistentvolumeclaim/grafana-dev created
   oauthclient.oauth.openshift.io/grafana-proxy-client-dev created
   deployment.apps/grafana-dev patched
   service/grafana-dev patched
   route.route.openshift.io/grafana-dev patched
   oauthclient.oauth.openshift.io/grafana-proxy-client-dev patched
   clusterrolebinding.rbac.authorization.k8s.io/open-cluster-management:grafana-crb-dev patched

   Copy code to clipboard

3. Switch your user role to Grafana administrator by using the switch-to-grafana-admin.sh script:

   1. Select the Grafana URL, https:grafana-dev-open-cluster-management-observability.{OPENSHIFT_INGRESS_DOMAIN}, then log in.

   2. From the CLI, run the following command to add the logged-in user as a Grafana administrator. For example, log in as the kubeadmin by running the following command:

      ./switch-to-grafana-admin.sh kube:admin

      Copy code to clipboard

      Your output might resemble the following content:

   User <kube:admin> switched to be grafana admin

   Copy code to clipboard

Procedure

1. Verify the Grafana version from the CLI:

   1. Log in to your hub cluster.
   2. Access the terminal for the observability-grafana pod.

2. To display the current Grafana version that is deployed within the cluster environment, run the following command:

   grafana-cli --version

   Copy code to clipboard

3. Verify the Grafana version from the Grafana console:

   1. Log in to the Grafana console dashboard.
   2. Navigate to the Manage tab.
   3. To view the version, scroll to the end of the page.

Procedure

1. From the Grafana console, create a dashboard by selecting the Create icon from the navigation panel.
2. Select Dashboard, and then click Add new panel.
3. From the New Dashboard/Edit Panel view, navigate to the Query tab.
4. Configure your query by selecting Observatorium from the data source selector.
5. Enter a valid Prometheus Query Language (PromQL) query.
6. From the Grafana dashboard header, click the Save icon that is in the dashboard header.
7. Add a descriptive name for the dashbard and click Save.

Procedure

1. If you have permissions to run the automation scripts, run the following command and replace the dashboard name:

   ./generate-dashboard-configmap-yaml.sh "Example dashboard name"

   Copy code to clipboard

   Your might receive the following output:

   Save dashboard <demo-dashboard> to ./demo-dashboard.yaml

   Copy code to clipboard

2. If you do not have permissions to run the previously mentioned script, complete the following steps:

   1. Open your custom dashboard from the Grafana console and click the Dashboard settings icon.
   2. Click the JSON Model icon from the side navigation panel.
   3. Copy the dashboard JSON data and paste it in the data section.
   4. Modify the name and replace $your-dashboard-name.
   5. Generate and enter a universally unique identifier (UUID) in the uid field in data.$your-dashboard-name.json.$$your_dashboard_json. You can use a program such as uuidegen to create a UUID.

   6. Create a YAML file Your ConfigMap might resemble the following file:

      kind: ConfigMap
      apiVersion: v1
      metadata:
        name: $your-dashboard-name
        namespace: open-cluster-management-observability
        labels:
          grafana-custom-dashboard: "true"
      data:
        $your-dashboard-name.json: |-
          $your_dashboard_json

      Copy code to clipboard

      Note: If your dashboard is not located in the General folder, use the observability.open-cluster-management.io/dashboard-folder annotation to specify the target folder name. See the following example:

   annotations:
     observability.open-cluster-management.io/dashboard-folder: Custom

   Copy code to clipboard

3. Apply the YAML file fromo the CLI or OpenShift Container Platform console to create the config map. Run the following command:

   oc apply -f demo-dashboard.yaml

   Copy code to clipboard
4. To apply the YAML from the console, navigate to the OpenShift Container Platform console.
5. Open the YAML editor and paste the config map definition.
6. Click Create to make the dashboard available in the folder.

Procedure

1. Import or modify a cluster into the managed cluster fleet. The observability-rbac-query-proxy pod watches for any changes in reference to the managed cluster labels and automatically updates the observability-managed-cluster-label-allowlist config map to reflect the changes. The config map contains only unique label names, which are either included in the ignore_labels or labels list. Your observability-managed-cluster-label-allowlist ConfigMap might resemble the following YAML file:

   data:
     managed_cluster.yaml: |
       ignore_labels:
         - clusterID
         - cluster.open-cluster-management.io/clusterset
         - feature.open-cluster-management.io/addon-application-manager
         - feature.open-cluster-management.io/addon-cert-policy-controller
         - feature.open-cluster-management.io/addon-cluster-proxy
         - feature.open-cluster-management.io/addon-config-policy-controller
         - feature.open-cluster-management.io/addon-governance-policy-framework
         - feature.open-cluster-management.io/addon-observability-controller
         - feature.open-cluster-management.io/addon-search-collector
         - feature.open-cluster-management.io/addon-work-manager
         - installer.name
         - installer.namespace
         - local-cluster
         - name
       labels:
         - cloud
         - vendor

   Copy code to clipboard
   • Any label that is listed in the ignore_labels keylist of the ConfigMap is removed from the drop-down filter on the ACM - Clusters Overview Grafana dashboard.
   • The labels that are enabled are displayed in the drop-down filter on the ACM - Clusters Overview Grafana dashboard. The values are from the acm_managed_cluster_labels metric, depending on the label key value that is selected.

2. Add a managed cluster label to the observability-managed-cluster-label-allowlist config map for the label to appear available as a filter option in Grafana. Be sure to add a unique label to the hub cluster, or managed cluster object that is associated with the managed cluster fleet. For example, if you add the label, department=finance to a managed cluster, the config map is updated and might resemble the following changes:

   data:
     managed_cluster.yaml: |
       ignore_labels:
         - clusterID
         - cluster.open-cluster-management.io/clusterset
         - feature.open-cluster-management.io/addon-application-manager
         - feature.open-cluster-management.io/addon-cert-policy-controller
         - feature.open-cluster-management.io/addon-cluster-proxy
         - feature.open-cluster-management.io/addon-config-policy-controller
         - feature.open-cluster-management.io/addon-governance-policy-framework
         - feature.open-cluster-management.io/addon-observability-controller
         - feature.open-cluster-management.io/addon-search-collector
         - feature.open-cluster-management.io/addon-work-manager
         - installer.name
         - installer.namespace
         - local-cluster
         - name
       labels:
         - cloud
         - department
         - vendor

   Copy code to clipboard

3. Enable a managed cluster label that is in the ignore_labels parameter section. From the observability-managed-cluster-label-allowlist config map, move the label to the labels parameter section.

   1. Enable the local-cluster and name labels. The config map resyncs after 30 seconds to ensure that the cluster labels are updated. Your observability-managed-cluster-label-allowlist config map might resemble the following content:

   data:
     managed_cluster.yaml: |
       ignore_labels:
         - clusterID
         - installer.name
         - installer.namespace
       labels:
         - cloud
         - vendor
         - local-cluster
         - name

   Copy code to clipboard

4. Check the observability-rbac-query-proxy pod logs in the open-cluster-management-observability namespace to verify where the label is listed. Run the following command:

   oc logs -n open-cluster-management-observability deployment/observability-rbac-query-proxy | grep -i "label"

   Copy code to clipboard

   The following information might be displayed in the pod log:

1. To remove the managed cluster labels from the Label drop-down filter in Grafana, add the label name to the ignore_labels list.

2. Check the observability-rbac-query-proxy pod logs in the open-cluster-management-observability namespace to verify that the label is disabled. Run the following command:

   oc logs -n open-cluster-management-observability deployment/observability-rbac-query-proxy | grep -i "label"

   Copy code to clipboard

   The following information might be displayed in the pod log:

Procedure

1. Explore the cluster_infrastructure_provider alert from a single-node OpenShift cluster, run the following query expression: cluster_infrastructure_provider{clusterType="SNO"}
2. From your hub cluster, select the Grafana link that is in the console header.
3. Edit your cluster dashboard by selecting Edit Panel.
4. From the Query front-end data source in Grafana, click the Query tab.
5. Select $datasource.
6. If you want to see more data, increase the value of the Step parameter section. If the Step parameter section is empty, it is automatically calculated.
7. Find the Custom query parameters field and select max_source_resolution=auto.
8. To verify that the data is displayed, refresh your Grafana page.
9. View the etcd table from the hub cluster dashboard in Grafana to learn the stability of the etcd as a data store. Select the Grafana link from your hub cluster to view the etcd table data, which is collected from your hub cluster. The Leader election changes across managed clusters are displayed.
10. View the Kubernetes API server dashboards to see the total number of clusters that are exceeding or meeting the targeted service-level objective (SLO) value for the past seven or 30-day period, offending and non-offending clusters, and API Server Request Duration.

11. View the cluster fleet Kubernetes API service-level overview from the hub cluster dashboard in Grafana:

    1. Navigate to the Grafana dashboard.
    2. Access the managed dashboard menu by selecting Kubernetes > Service-Level Overview > API Server. The Fleet Overview and Top Cluster details are displayed.

12. View the Kubernetes API service-level overview table from the hub cluster dashboard in Grafana to see the error budget for the past seven or 30-day period, the remaining downtime, and trend.

    1. Navigate to the Grafana dashboard from your hub cluster.
    2. Access the managed dashboard menu by selecting Kubernetes > Service-Level Overview > API Server. The Fleet Overview and Top Cluster details are displayed.
13. View the OpenShift Virtualization dashboard to see comprehensive insights for each cluster with the OpenShift Virtualization operator installed. The state of the operator is displayed, which is determined by active OpenShift Virtualization alerts and the conditions of the Hyperconverged Cluster Operator.
14. View the number of running virtual machines and the operator version for each cluster.
15. Filter the dashboard by cluster name, operator health alerts, health impact of alerts, and alert severity.

Procedure

1. Run the setup-grafana-dev.sh script with the --clean flag to uninstall the Grafana instance and clean up the environment. When you run the script with the --clean flag, the following resources are deleted: secret/grafana-dev-config, deployment.apps/grafana-dev, service/grafana-dev, ingress.extensions/grafana-dev, and persistentvolumeclaim/grafana-dev.

Procedure

1. Disable RightSizingRecommendation for namespaces.

2. Open the observability instance configuration. Run the following command:

   oc edit multiclusterobservability observability

   Copy code to clipboard

3. Add the following lines to the spec section of your MultiClusterObservability custom resource:

   apiVersion: observability.open-cluster-management.io/v1beta2
   kind: MultiClusterObservability
   metadata:
     name: observability
   spec:
     capabilities:
       platform:
         analytics:
           namespaceRightSizingRecommendation:
             enabled: false
   . . .

   Copy code to clipboard
4. To enable RightSizingRecommendation for namespaces, set the namespaceRightSizingRecommendation.enabled: to true .
5. View RightSizingRecommendation guides from your Grafana board by selecting RightSizing Recommendation > ACM Right-Sizing Namespace.
6. To view the aggregated data on Grafana, expand or collapse the dashboard by clicking the drop-down arrow. For example, expand the memory data by clicking the name of the dashboard.

Procedure

1. Configure your placement for the rs-namespace-config ConfigMap resource placement. See the following example where the policy only applies to clusters with the environment=prod label:

   placementConfiguration: |
    spec:
      . . .
      predicates:
      - requiredClusterSelector:
          labelSelector:
            matchLabels:
              environment: prod
      . . .

   Copy code to clipboard

2. Configure your PrometheusRule resource by completing changes to one of the following steps:

   1. Define the namespaces that you want to include or exclude in the namespaceFilterCriteria specification.
   2. Add a list of namespaces that you want to include in the inclusionCriteria specification.

   3. Add a list of namespaces that you want to exclude in the exclusionCriteria specification.

      Note: You cannot use both inclusionCriteria and exclusionCriteria at the same time.

   4. To adjust the CPU and memory percentages, define the percentage usage.

      See the following example where the inclusionCriteria specification includes prod and the exclusionCriteria excludes openshift:

   . . .
     prometheusRuleConfig: |
      namespaceFilterCriteria:
       inclusionCriteria:
       - prod.*
       exclusionCriteria:
       - openshift.*
     labelFilterCriteria:
     - labelName: label_kubernetes_io_metadata_name
       inclusionCriteria:
       - prod
       - staging
       exclusionCriteria:
       - kube.*
     recommendationPercentage: 120
     . . .

   Copy code to clipboard

3. Configure your namespace binding. By default, the RightSizingRecommendation feature uses the open-cluster-management-global-set namespace for binding policies to clusters.

   1. To bind generated policies to a different namespace and cluster set, add your namespace to the namespaceBinding specification. Important: Ensure that the target namespace is part of a valid ClusterSet before you apply the change. The policies that you generate might not apply to any clusters if there is no valid binding.

      See the following example where the <your-namespace-binding> namespace is the value for namespaceBinding:

   apiVersion: observability.open-cluster-management.io/v1beta2
   kind: MultiClusterObservability
   metadata:
     name: observability
   spec:
     . . .
     capabilities:
       platform:
         analytics:
           namespaceRightSizingRecommendation:
             enabled: true
             namespaceBinding: <your-namespace-binding>
     . . .

   Copy code to clipboard

Procedure

1. Open the observability instance configuration. Run the following command:

   oc edit multiclusterobservability observability

   Copy code to clipboard

2. Add the following lines to the spec section of your MultiClusterObservability custom resource:

   apiVersion: observability.open-cluster-management.io/v1beta2
   kind: MultiClusterObservability
   metadata:
     name: observability
   spec:
     capabilities:
       platform:
         analytics:
           virtualizationRightSizingRecommendation:
             enabled: false
   . . .

   Copy code to clipboard
3. To enable RightSizingRecommendation for your virtual machines, set the virtualizationRightSizingRecommendation.enabled: parameter to true.

4. View RightSizingRecommendation guides from your Grafana dashboard by selecting RightSizing Recommendation > ACM Right-Sizing OpenShift Virtualization.

   Note: The CPU and memory information at the cluster level is displayed.

5. To view CPU and memory details, and past resource usage for your virtual machines, select the name of your virtual machine.

Procedure

1. Configure placement for your virtual machines by editing the rs-virt-config ConfigMap resource and defining the placementConfiguration parameter. Customize your placement to where the policy only applies to clusters with the environment=prod label, as displayed in the following example:

   placementConfiguration: |
    spec:
      . . .
      predicates:
      - requiredClusterSelector:
          labelSelector:
            matchLabels:
              environment: prod
      . . .

   Copy code to clipboard

2. Configure PrometheusRule for your virtual machines by completing one of the following steps:

   1. Define the namespaces that you want to include or exclude in the namespaceFilterCriteria specification.
   2. Add a list of namespaces that you want to include in the inclusionCriteria specification.

   3. Add a list of namespaces that you want to exclude in the exclusionCriteria specification.

      Note: You cannot use both inclusionCriteria and exclusionCriteria at the same time.

   4. To adjust the CPU and memory percentages, define the percentage usage.

      See the following example where the inclusionCriteria specification includes prod and the exclusionCriteria excludes openshift:

   . . .
     prometheusRuleConfig: |
      namespaceFilterCriteria:
       inclusionCriteria:
       - prod.*
       exclusionCriteria:
       - openshift.*
     labelFilterCriteria:
     - labelName: label_kubernetes_io_metadata_name
       inclusionCriteria:
       - prod
       - staging
       exclusionCriteria:
       - kube.*
     recommendationPercentage: 120
     . . .

   Copy code to clipboard

3. Configure your namespace binding. By default, the RightSizingRecommendation feature uses the open-cluster-management-global-set namespace for binding policies to clusters.

   1. To bind generated policies to a different namespace and cluster set, use the namespaceBinding property. Important: Ensure that the target namespace is part of a valid ClusterSet before you apply the change. The policies that you generate might not apply to any clusters if there is no valid binding.

      See the following example where the <your-namespace-binding> namespace is the value for namespaceBinding:

   apiVersion: observability.open-cluster-management.io/v1beta2
   kind: MultiClusterObservability
   metadata:
     name: observability
   spec:
     . . .
     capabilities:
       platform:
         analytics:
           virtualizationRightSizingRecommendation:
             enabled: true
             namespaceBinding: <your-namespace-binding>
     . . .

   Copy code to clipboard

   Important: Ensure that the target namespace is part of a valid ClusterSet before you apply the change. The policies that you generate might not apply to any clusters if there is no valid binding.

Procedure

1. To enable platform monitoring and user workload monitoring, add the platform and userWorkloads specification to your MultiClusterObservability resource. Run the following command:

   oc patch mco observability -n open-cluster-management-observability --type=merge -p '{"spec":{"capabilities":{"platform":{"metrics":{"default":{"enabled": true}}},"userWorkloads":{"metrics":{"default":{"enabled": true}}}}}}'

   Copy code to clipboard

   Your MultiClusterObservability resource might resemble the following file example:

   apiVersion: observability.open-cluster-management.io/v1beta2
   kind: MultiClusterObservability
   metadata:
     name: observability
   spec:
     capabilities:
       platform:
         metrics:
           default:
             enabled: true
       userWorkloads:
         metrics:
           default:
             enabled: true

   Copy code to clipboard

2. To verify that the default configuration resources for the multicluster observability add-on are created, open your multicluster-observability-addon ClusterManagementAddon resource. Run the following command:

   oc get prometheusagents -n open-cluster-management-observability

   Copy code to clipboard

3. Verify that the default configurations are added to your placements. Run the following command:

   oc get cma multicluster-observability-addon -o yaml | yq '.spec.installStrategy.placements'

   Copy code to clipboard

Procedure

1. Optional Override the default scrape interval for your PrometheusAgent resource by changing the scrapeInterval parameter. The default value is 300s. You can also override the scrapeInterval of the scrapeConfig resource.

2. Configure the ScrapeConfig resource to define a set of metrics for independent federation from Prometheus of your managed clusters. Complete the following steps:

   1. Add the name of the job that you want to reference for the jobName parameter.
   2. To ensure that Prometheus federates metrics, add the /federate URL path for the metricsPath parameter.
   3. Add the metric name and labels that you want to collect. See the following YAML file example where the ScrapeConfig resource collects the up metric:

   apiVersion: monitoring.rhobs/v1alpha1
   kind: ScrapeConfig
   metadata:
     name: some-metrics-to-collect
     namespace: open-cluster-management-observability
     labels:
       - app.kubernetes.io/component: <platform-metrics-collector> or <user-workload-metrics-collector>
   spec:
     jobName: some-job-name
     metricsPath: /federate
     params:
       match[]:
       - '{__name__="up"}'

   Copy code to clipboard

3. Configure the PrometheusRule resource to limit the cardinality of your collected metrics on your hub cluster. Complete the following steps:

   1. Define alerting and recording rules for platform and user workload monitoring on your managed clusters.

   2. To target user workloads in your PrometheusRule resource, add the following annotation to define the namespace where you want to deploy the resource: observability.open-cluster-management.io/target-namespace.

      Notes: - When the observability.open-cluster-management.io/target-namespace is not set in your PrometheusRule resource, the PrometheusRule resource are deployed to the default installation namespace. If openshift.io/cluster-monitoring is set to true, the PrometheusRule resouces are not monitored by OpenShift Container Platform user workload monitoring stack. - Be sure to use the monitoring.coreos.com group for the PrometheusRule resource.

4. Configure the AddonDeploymentConfig resource to customize the deployment of the multicluster observability add-on on managed clusters. Note: The values in the AddonDeploymentConfig resource are override direct modifications of other resources. Complete the following steps:

   1. Define the namespace where you install the PrometheusAgent resource for the agentInstallNamespace parameter. The default namespace is open-cluster-management-agent-addon.
   2. Add the nodePlacement specification with the associated nodeSelector and tolerations parameters.
   3. Edit the proxyConfig specification by updating the httpProxy and noProxy configurations.

Procedure

1. Create a new ScrapeConfig resource in the open-cluster-management-observability namespace that includes values for the required parameters, jobName, metricsPath, and params.

2. Add the appropriate label for the app.kubernetes.io/component parameter to specify whether the metrics are for platform monitoring or user workload monitoring. Use one of the following label values, platform-metrics-collector or user-workload-metrics-collector.

   Note: When you use the platform-metrics-collector label, the multicluster observability add-on automatically sets the scrapeClass and targets parameters to enable federation from the platform Prometheus of your OpenShift Container Platform managed cluster. You can override the scrapeClass and targets parameters by adding the value that you need.

3. Optional: Manually set the scrapeClass and staticConfigs specifications for your user-workload-metrics-collector ScrapeConfig resource.

4. Add the ScrapeConfig resource reference to the placements of the ClusterManagementAddOn resource where you want the resource to be deployed.

   Note: Ensure that you reference the ScrapeConfig resource only after you create it. Otherwise, the add-on status updates to a Deploying status because the resource does not exist.

   Your ScrapeConfig resource might resemble the following YAML file:

   apiVersion: monitoring.rhobs/v1alpha1
   kind: ScrapeConfig
   metadata:
     name: add-custom-metrics
     namespace: open-cluster-management-observability
     labels:
       - app.kubernetes.io/component: platform-metrics-collector
   spec:
     jobName: some-job-name
     metricsPath: /federate
     params:
       match[]:
       - '{__name__="up"}'

   Copy code to clipboard

Procedure

1. Update your ScrapeConfig resource by adding the endpoint of the Cluster Observability Operator MonitoringStack resource. Your resource might resemble the following YAML file:

   apiVersion: monitoring.coreos.com/v1alpha1
   kind: ScrapeConfig
   spec:
     scrapeClass: “”
     scheme: HTTP
     staticConfigs:
     - targets:
       - my-monitoring-stack.my-monitoring-ns.svc:9090

   Copy code to clipboard
2. If you use a proxy with the Prometheus server, modify the ScrapeConfig resource to include your TLS configuration. Create a corresponding scrapeClass specification on the user workload PrometheusAgent resource. Then reference the PrometheusAgent resource within the ScrapeConfig resource.

Procedure

1. To remove the Watchdog alert metric, add the writeRelabelConfigs specification to the remoteWrite configuration of the PrometheusAgent resource. The metric removal is complete after it is federated. Your PrometheusAgent resource might resemble the following YAML file:

   apiVersion: monitoring.rhobs/v1alpha1
   kind: ScrapeConfig
   metadata:
     name: platform-metrics-alerts
     namespace: open-cluster-management-observability
   spec:
     jobName: alerts
     metricRelabelings:
       - action: labeldrop
         regex: managed_cluster|id
       - action: drop
         regex: ^Watchdog$
         sourceLabels:
           - alertname
     metricsPath: /federate
     params:
       'match[]':
         - '{__name__="ALERTS"}'
     scheme: HTTPS
     scrapeClass: not-configurable
     staticConfigs:
       - targets:
           - not-configurable

   Copy code to clipboard

Procedure

1. From the Fleet Management perspective in the console, then click Help > Command Line Tools.
2. Find the list of available downloads in the Advanced Cluster Management Observability Migration Tool - latest section.
3. Download the allowlist-migration binary file for your operating system and architecture.

4. Change the permissions. Run the following command:

   chmod +x /path/to/migration

   Copy code to clipboard

5. To generate your ScrapeConfig and PrometheusRule resources, run the following command:

   /path/to/migration allowlist.yaml output/dir

   Copy code to clipboard

6. Add the generated ScrapeConfig resources to your cluster by running the following command:

   oc apply -f <output/dir>/custom-scrapeconfig.yaml -n open-cluster-management-observability

   Copy code to clipboard

7. Add the generated PrometheusRule resources to your cluster by running the following command:

   oc apply -f <output/dir>/custom-prometheusrule.yaml -n open-cluster-management-observability

   Copy code to clipboard

8. Add the generated ScrapeConfig and PrometheusRule resources to the ClusterManagementAddOn resource. Run the following command:

   oc patch clustermanagementaddon multicluster-observability-addon --type=json -p='[
   {
      "op": "add",
      "path": "/spec/installStrategy/placements/0/configs/-",
      "value": {
         "group": "monitoring.rhobs",
         "resource": "scrapeconfigs",
         "name": "custom-scrapeconfig",
         "namespace": "open-cluster-management-observability"
      }
   },
   {
      "op": "add",
      "path": "/spec/installStrategy/placements/0/configs/-",
      "value": {
         "group": "monitoring.coreos.com",
         "resource": "prometheusrules",
         "name": "custom-prometheusrule",
         "namespace": "open-cluster-management-observability"
      }
   }
   ]'

   Copy code to clipboard

Verification

1. Verify that the generated ScrapeConfig resource exists on your hub cluster. Run the following command:

   oc get ScrapeConfig -n open-cluster-management-observability

   Copy code to clipboard

2. Ensure that the generated PrometheusRule resource exists on your hub cluster. Run the following command:

   oc get PrometheusRule -n open-cluster-management-observability

   Copy code to clipboard

3. Verify that the ScrapeConfig resource is propogated to the managed cluster. Run the following command:

   oc get ScrapeConfig -n open-cluster-management-agent-addon

   Copy code to clipboard

4. Check that the PrometheusRule resource is propogated to the managed cluster. Run the following command:

   oc get PrometheusRule -n open-cluster-management-agent-addon

   Copy code to clipboard
5. Verify custom metrics collection by entering a query in Grafana. For example, enter the following query to verify use of the ScrapeConfig resource: up{job="my-app"}.

Procedure

1. In the MultiClusterObservability, set the .spec.storageConfig.storageClass field to the new storage class.
2. To ensure the data of the earlier PersistentVolumes is retained even when you delete the PersistentVolumeClaim, go to all your existing PersistentVolumes.
3. Change the reclaimPolicy to "Retain": `oc patch pv <your-pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'.
4. Optional: To avoid losing data, see Migrate persistent data to another Storage Class in DG 8 Operator in OCP 4.

5. Delete both the StatefulSet and the PersistentVolumeClaim in the following StatefulSet cases:

   • alertmanager-db-observability-alertmanager-<REPLICA_NUMBER>
   • data-observability-thanos-<COMPONENT_NAME>
   • data-observability-thanos-receive-default
   • data-observability-thanos-store-shard
6. Optional: You might need to delete, then re-create the MultiClusterObservability operator pod so that you can create the new StatefulSet.
7. Re-create a new PersistentVolumeClaim with the same name but the correct StorageClass.
8. Create a new PersistentVolumeClaim referring to the old PersistentVolume.
9. Verify that the new StatefulSet and PersistentVolumes use the new StorageClass that you chose.

Procedure

1. Create the Kubernetes secret for an external endpoint with the access information of the external endpoint in the open-cluster-management-observability namespace. View the following example secret:

   apiVersion: v1
   kind: Secret
   metadata:
     name: victoriametrics
     namespace: open-cluster-management-observability
   type: Opaque
   stringData:
     ep.yaml: |
       url: http://victoriametrics:8428/api/v1/write
       http_client_config:
         basic_auth:
           username: test
           password: test
         tls_config:
           secret_name:
           ca_file_key:
           cert_file_key:
           key_file_key:
           insecure_skip_verify:

   Copy code to clipboard
   • The ep.yaml parameter is the key of the content and is used in the MultiClusterObservability custom resource in next step. Currently, observability supports exporting metrics to endpoints without any security checks, with Basic authentication or with tls enablement. View the following tables for a full list of supported parameters:
   • The url parameter is required and the URL for the external endpoint. Enter the value as a string.
   • The http_client_config parameter is optional and is the advanced configuration for the HTTP client.
   • The basic_auth parameter is optional and is the HTTP client configuration for Basic authentication.
   • The username parameter is optional and is the user name for basic authorization. Enter the value as a string.
   • The password is optional and is the password for basic authorization. Enter the value as a string.
   • The tls_config parameter is optional and is the HTTP client configuration for TLS.
   • The secret_name parameter is required and is the name of the secret that contains certificates. Enter the value as a string.
   • The ca_file_key parameter is required and the key of the CA certificate in the secret. This parameter is only optional if the insecure_skip_verify parameter is set to true. Enter the value as a string.
   • The cert_file_key parameter is required and is the key of the client certificate in the secret. Enter the value as a string.
   • The key_file_key parameter is required and is the key of the client key in the secret. Enter the value as a string.
   • The insecure_skip_verify parameter is optional and used to skip the verification for target certificate. Enter the value as a boolean value.

2. To add a list of external endpoints that you want to export, add the writeStorage parameter to the MultiClusterObservability custom resource. View the following example:

   spec:
     storageConfig:
       writeStorage:
       - key: ep.yaml
         name: victoriametrics

   Copy code to clipboard
   • Each item contains two attributes: name and key. Name is the name of the Kubernetes secret that contains endpoint access information, and key is the key of the content in the secret. If you add more than one item to the list, then the metrics are exported to multiple external endpoints.

3. View the status of metric export after the metrics export is enabled by checking the acm_remote_write_requests_total metric.

   1. From the OpenShift Container Platform console of your hub cluster, navigate to the Metrics page by clicking Metrics in the Observe section.
   2. Then query the acm_remote_write_requests_total metric. The value of that metric is the total number of requests with a specific response for one external endpoint, on one observatorium API instance. The name label is the name for the external endpoint. The code label is the return code of the HTTP request for the metrics export.

Procedure

1. Create the TLS Secret config maps within the open-cluster-management-observability namespace.
2. Add the secret names to the secrets specification of the PrometheusAgent resource.

3. Add the remoteWrite specification to the PrometheusAgent resource. Your PrometheusAgent resource might resemble the following YAML file, where the up metric is exported to a custom endpoint:

   apiVersion: monitoring.rhobs/v1alpha1
   kind: PrometheusAgent
   metadata:
     name: mcoa-default-platform-metrics-collector-global
     namespace: open-cluster-management-observability
   spec:
     secrets:
       - custom-endpoint-ca
       - custom-endpoint-cert
     remoteWrite:
       - name: custom-endpoint
         tlsConfig:
           caFile: /etc/prometheus/secrets/custom-endpoint-ca/ca.crt
           certFile: /etc/prometheus/secrets/custom-endpoint-cert/tls.crt
           keyFile: /etc/prometheus/secrets/custom-endpoint-cert/tls.key
         url: 'https://my-custom-remote-write-endpoint.io/api/v1/receive'
         writeRelabelConfigs:
           - action: keep
             regex: ^up$
             sourceLabels:
               - __name__
       - name: acm-observability
         ...

   Copy code to clipboard

Procedure

1. To configure alert forwarding for platform metrics, create a YAML file named mcoa-alert-forward-platform.yaml.

   1. Add the base domain of your hub cluster for the $hubBaseDomain parameter. If you make custom changes to the additionalAlertmanagerConfigs specification of the config map, be sure to add the changes directly into the policy, before applying the policy.

      See the following example:

      apiVersion: policy.open-cluster-management.io/v1
      kind: Policy
      metadata:
        name: mcoa-alert-forward-platform
        namespace: open-cluster-management-global-set
      spec:
        disabled: false
        policy-templates:
        - objectDefinition:
            apiVersion: policy.open-cluster-management.io/v1
            kind: ConfigurationPolicy
            metadata:
              name: mcoa-alert-forward-platform
            spec:
              namespaceSelector:
                exclude:
                - kube-*
                include:
                - default
              object-templates-raw: |
                {{ $hubBaseDomain:= "INPUT_BASE_DOMAIN" }}
                {{ $hubName := (split "." $hubBaseDomain)._0 }}
                {{- $cmo := (lookup "v1" "ConfigMap" "openshift-monitoring" "cluster-monitoring-config") }}
                {{- $cy := dict }}
                {{- if and $cmo $cmo.data }}
                  {{- $cy = (index $cmo "data" "config.yaml") | fromYaml }}
                {{- end }}
      
                {{- $mangedConfig := dict }}
      
                {{- $pm := `
                  prometheusK8s:
                    additionalAlertmanagerConfigs:
                    - apiVersion: v2
                      bearerToken:
                        key: token
                        name: observability-alertmanager-accessor-%[1]s
                      scheme: https
                      staticConfigs:
                      - alertmanager-open-cluster-management-observability.apps.%[2]s
                      tlsConfig:
                        ca:
                          key: service-ca.crt
                          name: hub-alertmanager-router-ca-%[1]s
                        insecureSkipVerify: false
                    externalLabels:
                      managed_cluster: %[3]s
                ` }}
      
                  {{- $mangedConfig = merge $mangedConfig
                                    ((printf $pm $hubName $hubBaseDomain (fromClusterClaim "id.openshift.io"))| fromYaml)
                }}
                - complianceType: mustonlyhave
                  objectDefinition:
                    apiVersion: v1
                    data:
                      config.yaml: |
                        {{ (merge $cy $mangedConfig)| toYaml |autoindent }}
                    kind: ConfigMap
                    metadata:
                      name: cluster-monitoring-config
                      namespace: openshift-monitoring
                  recordDiff: InStatus
        remediationAction: inform

      Copy code to clipboard

   2. Create a PlacementBinding resource to bind the policy to the global placement.See the following sample:

      apiVersion: policy.open-cluster-management.io/v1
      kind: PlacementBinding
      metadata:
        name: mcoa-alert-forward-platform-placement
        namespace: open-cluster-management-global-set
      placementRef:
        apiGroup: cluster.open-cluster-management.io
        kind: Placement
        name: global
      subjects:
      - apiGroup: policy.open-cluster-management.io
        kind: Policy
        name: mcoa-alert-forward-platform

      Copy code to clipboard

   3. To apply the policy changes across all managed clusters and enable alert forwarding from your managed clusters to the hub cluster, change the remediationAction parameter value to enforce.

      Note: The remediationAction of the previous policy is set to inform mode. From the console, validate your changes by viewing the policy from the Governance page. Your policy appears as non-compliant.

   4. Apply your YAML file. Run the following command:

      oc apply -f ./forward-platform-alerts.yaml

      Copy code to clipboard

   Note: If you have managed clusters other than a Red Hat OpenShift Container Platform cluster, you must create the appropriate PlacementBinding resource targeting the managed clusters.

2. To configure alert forwarding for user workload metrics, create a YAML file named mcoa-alert-forward-uwl.

   1. Add the base domain for your hub cluster in the $hubBaseDomain parameter. Your YAML file might resemble the following example:

      apiVersion: policy.open-cluster-management.io/v1
      kind: Policy
      metadata:
        name: mcoa-alert-forward-uwl
        namespace: open-cluster-management-global-set
      spec:
        disabled: false
        policy-templates:
        - objectDefinition:
            apiVersion: policy.open-cluster-management.io/v1
            kind: ConfigurationPolicy
            metadata:
              name: mcoa-alert-forward-uwl
            spec:
              namespaceSelector:
                exclude:
                - kube-*
                include:
                - default
              object-templates-raw: |
                {{ $hubBaseDomain:= "INPUT_BASE_DOMAIN" }}
                {{ $hubName := (split "." $hubBaseDomain)._0 }}
      
                {{- $cmo := (lookup "v1" "ConfigMap" "openshift-user-workload-monitoring" "user-workload-monitoring-config") }}
                {{- $cy := dict }}
                {{- if and $cmo $cmo.data }}
                  {{- $cy = (index $cmo "data" "config.yaml") | fromYaml }}
                {{- end }}
      
                {{- $mangedConfig := dict }}
      
                {{- $pm := `
                  prometheus:
                    additionalAlertmanagerConfigs:
                    - apiVersion: v2
                      bearerToken:
                        key: token
                        name: observability-alertmanager-accessor-%[1]s
                      scheme: https
                      staticConfigs:
                      - alertmanager-open-cluster-management-observability.apps.%[2]s
                      tlsConfig:
                        ca:
                          key: service-ca.crt
                          name: hub-alertmanager-router-ca-%[1]s
                        insecureSkipVerify: false
                    externalLabels:
                      managed_cluster: %[3]s
                ` }}
      
                  {{- $mangedConfig = merge $mangedConfig
                                    ((printf $pm $hubName $hubBaseDomain (fromClusterClaim "id.openshift.io"))| fromYaml)
                }}
                - complianceType: mustonlyhave
                  objectDefinition:
                    apiVersion: v1
                    data:
                      config.yaml: |
                        {{ (merge $cy $mangedConfig )| toYaml |autoindent }}
                    kind: ConfigMap
                    metadata:
                      name: user-workload-monitoring-config
                      namespace: openshift-user-workload-monitoring
                  recordDiff: InStatus
        remediationAction: inform

      Copy code to clipboard

   2. Create a PlacementBinding resource to bind the policy to the global placement. Your YAML file might resemble the following sample:

      apiVersion: policy.open-cluster-management.io/v1
      kind: PlacementBinding
      metadata:
        name: mcoa-alert-forward-uwl-placement
        namespace: open-cluster-management-global-set
      placementRef:
        apiGroup: cluster.open-cluster-management.io
        kind: Placement
        name: global
      subjects:
      - apiGroup: policy.open-cluster-management.io
        kind: Policy
        name: mcoa-alert-forward-uwl

      Copy code to clipboard
   3. Apply your YAML file. Run the following command:

   oc apply -f ./forward-uwl-alerts.yaml

   Copy code to clipboard