Procedure

1. Create a service account for the collector. If you want to write logs to a storage system that requires a token for authentication, you must include a token in the service account.

2. Bind the appropriate cluster roles to the service account:

   Example binding command

   $ oc adm policy add-cluster-role-to-user <cluster_role_name> system:serviceaccount:<namespace_name>:<service_account_name>

   Copy code to clipboard

Procedure

1. Extract the existing ClusterLogForwarder configuration and save it as a local file.

   $ oc get clusterlogforwarder <name> -n <namespace> -o yaml > <filename>.yaml

   Copy code to clipboard

   Where:

   • <name> is the name of the ClusterLogForwarder instance you want to configure.
   • <namespace> is the namespace where you created the ClusterLogForwarder instance, for example openshift-logging.
   • <filename> is the name of the local file where you save the configuration.

2. Add a configuration to drop unwanted log records to the filters spec in the ClusterLogForwarder CR.

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     # ...
     filters:
     - name: drop-filter
       type: drop 1
       drop: 2
       - test: 3
         - field: .kubernetes.labels."app.version-1.2/beta" 4
           matches: .+ 5
         - field: .kubernetes.pod_name
           notMatches: "my-pod" 6
     pipelines:
     - name: my-pipeline 7
       filterRefs:
       - drop-filter
     # ...

   Copy code to clipboard

   1

   Specify the type of filter. The drop filter drops log records that match the filter configuration.

   2

   Specify configuration options for the drop filter.

   3

   Specify conditions for tests to evaluate whether the filter drops a log record.

   4

   Specify dot-delimited paths to fields in log records.

   • Each path segment can contain alphanumeric characters and underscores, a-z, A-Z, 0-9, _, for example, .kubernetes.namespace_name.
   • If segments contain different characters, the segment must be in quotes, for example, .kubernetes.labels."app.version-1.2/beta".
   • You can include several field paths in a single test configuration, but they must all evaluate to true for the test to pass and the drop filter to apply.

   5

   Specify a regular expression. If log records match this regular expression, they are dropped.

   6

   Specify a regular expression. If log records do not match this regular expression, they are dropped.

   7

   Specify the pipeline that uses the drop filter.

   Note

   You can set either the matches or notMatches condition for a single field path, but not both.

   Example configuration that keeps only high-priority log records

   # ...
   filters:
   - name: important
     type: drop
     drop:
     - test:
       - field: .message
         notMatches: "(?i)critical|error"
       - field: .level
         matches: "info|warning"
   # ...

   Copy code to clipboard

   Example configuration with several tests

   # ...
   filters:
   - name: important
     type: drop
     drop:
     - test: 1
       - field: .kubernetes.namespace_name
         matches: "openshift.*"
     - test: 2
       - field: .log_type
         matches: "application"
       - field: .kubernetes.pod_name
         notMatches: "my-pod"
   # ...

   Copy code to clipboard

   1

   The filter drops logs that contain a namespace that starts with openshift.

   2

   The filter drops application logs that do not have my-pod in the pod name.

3. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy code to clipboard

Procedure

1. Add a configuration for a filter to the input spec in the ClusterLogForwarder CR.

   The following example shows how to configure the ClusterLogForwarder CR to include logs based on label expressions or matched label key/values:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     inputs:
       - name: mylogs
         application:
           selector:
             matchExpressions:
             - key: env 1
               operator: In 2
               values: ["prod", "qa"] 3
             - key: zone
               operator: NotIn
               values: ["east", "west"]
             matchLabels: 4
               app: one
               name: app1
         type: application
   # ...

   Copy code to clipboard

   1

   Specifies the label key to match.

   2

   Specifies the operator. Valid values include: In, NotIn, Exists, and DoesNotExist.

   3

   Specifies an array of string values. If the operator value is either Exists or DoesNotExist, the value array must be empty.

   4

   Specifies an exact key or value mapping.

2. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy code to clipboard

Procedure

1. Extract the existing ClusterLogForwarder configuration and save it as a local file.

   $ oc get clusterlogforwarder <name> -n <namespace> -o yaml > <filename>.yaml

   Copy code to clipboard

   Where:

   • <name> is the name of the ClusterLogForwarder instance you want to configure.
   • <namespace> is the namespace where you created the ClusterLogForwarder instance, for example openshift-logging.
   • <filename> is the name of the local file where you save the configuration.

2. Add a configuration to prune log records to the filters spec in the ClusterLogForwarder CR.

   Important

   If you specify both in and notIn parameters, the notIn array takes precedence over in during pruning. After records are pruned by using the notIn array, they are then pruned by using the in array.

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     serviceAccount:
       name: my-account
     filters:
     - name: prune-filter
       type: prune 1
       prune: 2
         in: [.kubernetes.annotations, .kubernetes.namespace_id] 3
         notIn: [.kubernetes,.log_type,.message,."@timestamp",.log_source] 4
     pipelines:
     - name: my-pipeline 5
       filterRefs: ["prune-filter"]
     # ...

   Copy code to clipboard

   1

   Specify the type of filter. The prune filter prunes log records by configured fields.

   2

   Specify configuration options for the prune filter.

   • The in and notIn fields are arrays of dot-delimited paths to fields in log records.
   • Each path segment can contain alpha-numeric characters and underscores, a-z, A-Z, 0-9, _, for example, .kubernetes.namespace_name.
   • If segments contain different characters, the segment must be in quotes, for example, .kubernetes.labels."app.version-1.2/beta".

   3

   Optional: Specify fields to remove from the log record. The log collector keeps all other fields.

   4

   Optional: Specify fields to keep in the log record. The log collector removes all other fields.

   5

   Specify the pipeline that the prune filter is applied to.

   Important

   • The filters cannot remove the .log_type, .log_source, .message fields from the log records. You must include them in the notIn field.
   • If you use the googleCloudLogging output, you must include .hostname in the notIn field.

3. Apply the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy code to clipboard

Procedure

1. Modify the ClusterLogForwarder CR to add configuration for the http receiver input:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <clusterlogforwarder_name> 1
     namespace: <namespace>
   # ...
   spec:
     serviceAccount:
       name: <service_account_name>
     inputs:
     - name: http-receiver 2
       type: receiver
       receiver:
         type: http 3
         port: 8443 4
         http:
           format: kubeAPIAudit 5
     outputs:
     - name: <output_name>
       type: http
       http:
         url: <url>
     pipelines: 6
       - name: http-pipeline
         inputRefs:
           - http-receiver
         outputRefs:
           - <output_name>
   # ...

   Copy code to clipboard

   1

   Specify a name for the ClusterLogForwarder CR.

   2

   Specify a name for your input receiver.

   3

   Specify the input receiver type as http.

   4

   Optional: Specify the port that the input receiver listens on. This must be a value between 1024 and 65535. The default value is 8443 if this is not specified.

   5

   Currently, only the kube-apiserver webhook format is supported for http input receivers.

   6

   Configure a pipeline for your input receiver.

2. Apply the changes to the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy code to clipboard

3. Verify that the collector is listening on the service that has a name in the <clusterlogforwarder_resource_name>-<input_name> format by running the following command:

   $ oc get svc

   Copy code to clipboard

   Example output

   NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)            AGE
   collector                 ClusterIP   172.30.85.239    <none>        24231/TCP          3m6s
   collector-http-receiver   ClusterIP   172.30.205.160   <none>        8443/TCP           3m6s

   Copy code to clipboard

   In the example, the service name is collector-http-receiver.

Verification

1. Extract the certificate authority (CA) certificate file by running the following command:

   $ oc extract cm/openshift-service-ca.crt -n <namespace>

   Copy code to clipboard
   Note

   If the CA in the cluster where the collectors are running changes, you must extract the CA certificate file again.

2. As an example, use the curl command to send logs by running the following command:

   $ curl --cacert <openshift_service_ca.crt> https://collector-http-receiver.<namespace>.svc:8443 -XPOST -d '{"<prefix>":"<message>"}'

   Copy code to clipboard

   Replace <openshift_service_ca.crt> with the extracted CA certificate file.

Procedure

1. Grant the collect-infrastructure-logs cluster role to the service account by running the following command:

   Example binding command

   $ oc adm policy add-cluster-role-to-user collect-infrastructure-logs -z logcollector

   Copy code to clipboard

2. Modify the ClusterLogForwarder CR to add configuration for the syslog receiver input:

   Example ClusterLogForwarder CR

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: <clusterlogforwarder_name> 1
     namespace: <namespace>
   # ...
   spec:
     serviceAccount:
       name: <service_account_name> 2
     inputs:
       - name: syslog-receiver 3
         type: receiver
         receiver:
           type: syslog 4
           port: 10514 5
     outputs:
     - name: <output_name>
       lokiStack:
         authentication:
           token:
             from: serviceAccount
         target:
           name: logging-loki
           namespace: openshift-logging
       tls: 6
         ca:
           key: service-ca.crt
           configMapName: openshift-service-ca.crt
       type: lokiStack
   # ...
     pipelines: 7
       - name: syslog-pipeline
         inputRefs:
           - syslog-receiver
         outputRefs:
           - <output_name>
   # ...

   Copy code to clipboard

   1 2

   Use the service account that you granted the collect-infrastructure-logs permission in the previous step.

   3

   Specify a name for your input receiver.

   4

   Specify the input receiver type as syslog.

   5

   Optional: Specify the port that the input receiver listens on. This must be a value between 1024 and 65535.

   6

   If TLS configuration is not set, the default certificates will be used. For more information, run the command oc explain clusterlogforwarders.spec.inputs.receiver.tls.

   7

   Configure a pipeline for your input receiver.

3. Apply the changes to the ClusterLogForwarder CR by running the following command:

   $ oc apply -f <filename>.yaml

   Copy code to clipboard

4. Verify that the collector is listening on the service that has a name in the <clusterlogforwarder_resource_name>-<input_name> format by running the following command:

   $ oc get svc

   Copy code to clipboard

   Example output

   NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)            AGE
   collector                   ClusterIP   172.30.85.239    <none>        24231/TCP          33m
   collector-syslog-receiver   ClusterIP   172.30.216.142   <none>        10514/TCP          2m20s

   Copy code to clipboard

   In this example output, the service name is collector-syslog-receiver.

Verification

1. Extract the certificate authority (CA) certificate file by running the following command:

   $ oc extract cm/openshift-service-ca.crt -n <namespace>

   Copy code to clipboard
   Note

   If the CA in the cluster where the collectors are running changes, you must extract the CA certificate file again.

2. As an example, use the curl command to send logs by running the following command:

   $ curl --cacert <openshift_service_ca.crt> collector-syslog-receiver.<namespace>.svc:10514  “test message”

   Copy code to clipboard

   Replace <openshift_service_ca.crt> with the extracted CA certificate file.

Procedure

1. Copy the service account credentials received from GCP into a file called key.json.

2. Create an object storage secret with the name logging-loki-gcs by running the following command:

   $ oc create secret generic logging-loki-gcs \
     --from-literal=bucketname="<bucket_name>" \
     --from-file=key.json="<path/to/key.json>"

   Copy code to clipboard

Procedure

1. Create an ObjectBucketClaim custom resource in the openshift-logging namespace:

   apiVersion: objectbucket.io/v1alpha1
   kind: ObjectBucketClaim
   metadata:
     name: loki-bucket-odf
     namespace: openshift-logging
   spec:
     generateBucketName: loki-bucket-odf
     storageClassName: openshift-storage.noobaa.io

   Copy code to clipboard

2. Get bucket properties from the associated ConfigMap object by running the following command:

   BUCKET_HOST=$(oc get -n openshift-logging configmap loki-bucket-odf -o jsonpath='{.data.BUCKET_HOST}')
   BUCKET_NAME=$(oc get -n openshift-logging configmap loki-bucket-odf -o jsonpath='{.data.BUCKET_NAME}')
   BUCKET_PORT=$(oc get -n openshift-logging configmap loki-bucket-odf -o jsonpath='{.data.BUCKET_PORT}')

   Copy code to clipboard

3. Get bucket access key from the associated secret by running the following command:

   ACCESS_KEY_ID=$(oc get -n openshift-logging secret loki-bucket-odf -o jsonpath='{.data.AWS_ACCESS_KEY_ID}' | base64 -d)
   SECRET_ACCESS_KEY=$(oc get -n openshift-logging secret loki-bucket-odf -o jsonpath='{.data.AWS_SECRET_ACCESS_KEY}' | base64 -d)

   Copy code to clipboard

4. Create an object storage secret with the name logging-loki-odf by running the following command:

   $ oc create -n openshift-logging secret generic logging-loki-odf \
   --from-literal=access_key_id="<access_key_id>" \
   --from-literal=access_key_secret="<secret_access_key>" \
   --from-literal=bucketnames="<bucket_name>" \
   --from-literal=endpoint="https://<bucket_host>:<bucket_port>"

   Copy code to clipboard

Procedure

1. Enter the following command to create a new group:

   $ oc adm groups new cluster-admin

   Copy code to clipboard

2. Enter the following command to add the desired user to the cluster-admin group:

   $ oc adm groups add-users cluster-admin <username>

   Copy code to clipboard

3. Enter the following command to add the cluster-admin user role to the group:

   $ oc adm policy add-cluster-role-to-group cluster-admin cluster-admin

   Copy code to clipboard

Procedure

1. Create a LokiStack CR:

   • Enable stream-based retention globally as shown in the following example:

     Example global stream-based retention for AWS

     apiVersion: loki.grafana.com/v1
     kind: LokiStack
     metadata:
       name: logging-loki
       namespace: openshift-logging
     spec:
       limits:
        global: 1
           retention: 2
             days: 20
             streams:
             - days: 4
               priority: 1
               selector: '{kubernetes_namespace_name=~"test.+"}' 3
             - days: 1
               priority: 1
               selector: '{log_type="infrastructure"}'
       managementState: Managed
       replicationFactor: 1
       size: 1x.small
       storage:
         schemas:
         - effectiveDate: "2020-10-11"
           version: v13
         secret:
           name: logging-loki-s3
           type: aws
       storageClassName: gp3-csi
       tenants:
         mode: openshift-logging

     Copy code to clipboard

     1

     Sets retention policy for all log streams. Note: This field does not impact the retention period for stored logs in object storage.

     2

     Retention is enabled in the cluster when this block is added to the CR.

     3

     Contains the Content from grafana.com is not included.LogQL query used to define the log stream.spec: limits:

   • Enable stream-based retention per-tenant basis as shown in the following example:

     Example per-tenant stream-based retention for AWS

     apiVersion: loki.grafana.com/v1
     kind: LokiStack
     metadata:
       name: logging-loki
       namespace: openshift-logging
     spec:
       limits:
         global:
           retention:
             days: 20
         tenants: 1
           application:
             retention:
               days: 1
               streams:
                 - days: 4
                   selector: '{kubernetes_namespace_name=~"test.+"}' 2
           infrastructure:
             retention:
               days: 5
               streams:
                 - days: 1
                   selector: '{kubernetes_namespace_name=~"openshift-cluster.+"}'
       managementState: Managed
       replicationFactor: 1
       size: 1x.small
       storage:
         schemas:
         - effectiveDate: "2020-10-11"
           version: v13
         secret:
           name: logging-loki-s3
           type: aws
       storageClassName: gp3-csi
       tenants:
         mode: openshift-logging

     Copy code to clipboard

     1

     Sets retention policy by tenant. Valid tenant types are application, audit, and infrastructure.

     2

     Contains the Content from grafana.com is not included.LogQL query used to define the log stream.

2. Apply the LokiStack CR:

   $ oc apply -f <filename>.yaml

   Copy code to clipboard

Procedure

1. List the pods in Pending status by running the following command:

   $ oc get pods --field-selector status.phase==Pending -n openshift-logging

   Copy code to clipboard

   Example oc get pods output

   NAME                           READY   STATUS    RESTARTS   AGE 1
   logging-loki-index-gateway-1   0/1     Pending   0          17m
   logging-loki-ingester-1        0/1     Pending   0          16m
   logging-loki-ruler-1           0/1     Pending   0          16m

   Copy code to clipboard

   1

   These pods are in Pending status because their corresponding PVCs are in the failed zone.

2. List the PVCs in Pending status by running the following command:

   $ oc get pvc -o=json -n openshift-logging | jq '.items[] | select(.status.phase == "Pending") | .metadata.name' -r

   Copy code to clipboard

   Example oc get pvc output

   storage-logging-loki-index-gateway-1
   storage-logging-loki-ingester-1
   wal-logging-loki-ingester-1
   storage-logging-loki-ruler-1
   wal-logging-loki-ruler-1

   Copy code to clipboard

3. Delete the PVC(s) for a pod by running the following command:

   $ oc delete pvc <pvc_name> -n openshift-logging

   Copy code to clipboard

4. Delete the pod(s) by running the following command:

   $ oc delete pod <pod_name> -n openshift-logging

   Copy code to clipboard

   Once these objects have been successfully deleted, they should automatically be rescheduled in an available zone.

Procedure

1. Create an AlertingRule custom resource (CR):

   Example infrastructure AlertingRule CR

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: loki-operator-alerts
       namespace: openshift-operators-redhat 1
       labels: 2
         openshift.io/<label_name>: "true"
     spec:
       tenantID: "infrastructure" 3
       groups:
         - name: LokiOperatorHighReconciliationError
           rules:
             - alert: HighPercentageError
               expr: | 4
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"} |= "error" [1m])) by (job)
                   /
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"}[1m])) by (job)
                   > 0.01
               for: 10s
               labels:
                 severity: critical 5
               annotations:
                 summary: High Loki Operator Reconciliation Errors 6
                 description: High Loki Operator Reconciliation Errors 7

   Copy code to clipboard

   1

   The namespace where this AlertingRule CR is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   2

   The labels block must match the LokiStack spec.rules.selector definition.

   3

   AlertingRule CRs for infrastructure tenants are only supported in the openshift-*, kube-\*, or default namespaces.

   4

   The value for kubernetes_namespace_name: must match the value for metadata.namespace.

   5

   The value of this mandatory field must be critical, warning, or info.

   6

   This field is mandatory.

   7

   This field is mandatory.

   Example application AlertingRule CR

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: app-user-workload
       namespace: app-ns 1
       labels: 2
         openshift.io/<label_name>: "true"
     spec:
       tenantID: "application"
       groups:
         - name: AppUserWorkloadHighError
           rules:
             - alert:
               expr: | 3
                 sum(rate({kubernetes_namespace_name="app-ns", kubernetes_pod_name=~"podName.*"} |= "error" [1m])) by (job)
               for: 10s
               labels:
                 severity: critical 4
               annotations:
                 summary:  5
                 description:  6

   Copy code to clipboard

   1

   The namespace where this AlertingRule CR is created must have a label matching the LokiStack spec.rules.namespaceSelector definition.

   2

   The labels block must match the LokiStack spec.rules.selector definition.

   3

   Value for kubernetes_namespace_name: must match the value for metadata.namespace.

   4

   The value of this mandatory field must be critical, warning, or info.

   5

   The value of this mandatory field is a summary of the rule.

   6

   The value of this mandatory field is a detailed description of the rule.

2. Apply the AlertingRule CR:

   $ oc apply -f <filename>.yaml

   Copy code to clipboard