Procedure

1. Check the status of collector pods:

   $ oc get pods -n openshift-logging -l app.kubernetes.io/component=collector

   Copy code to clipboard

2. Examine the details of any problematic pods:

   $ oc describe pod _<pod_name>_ -n openshift-logging

   Copy code to clipboard

   Replace <pod_name> with the name of the affected collector pod.

3. Review the collector pod logs:

   $ oc logs _<pod_name>_ -n openshift-logging

   Copy code to clipboard

4. Test the metrics endpoint directly:

   $ oc exec _<pod_name>_ -n openshift-logging -- curl -s http://localhost:24231/metrics

   Copy code to clipboard

   The default metrics port for the collector is 24231.

5. Check for network policies that might block metrics scraping:

   $ oc get networkpolicies -n openshift-logging

   Copy code to clipboard

6. Verify that the ServiceMonitor resource exists:

   $ oc get servicemonitor -n openshift-logging

   Copy code to clipboard

7. Check the service and endpoints configuration:

   $ oc get service -n openshift-logging
   $ oc get endpoints -n openshift-logging

   Copy code to clipboard

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • app_kubernetes_io_instance: The collector name
   • namespace: The collector namespace
   • component_id: The specific failing Vector sink, such as output_lokistack_otlp_application or output_my_splunk

2. Review the ClusterLogForwarder configuration to identify the output definition that matches the component_id from the alert:

   $ oc get clusterlogforwarder _<clusterlogforwarder_name>_ -n _<namespace>_ -o yaml

   Copy code to clipboard

3. Locate the Kubernetes secret referenced by the output definition and verify the secret exists:

   $ oc get secret _<secret_name>_ -n _<namespace>_

   Copy code to clipboard

4. Verify that the secret contains the expected keys with valid, non-empty values:

   $ oc get secret _<secret_name>_ -n _<namespace>_ -o yaml

   Copy code to clipboard
5. Confirm that the credentials in the secret have the necessary permissions for write access to the target log store.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • app_kubernetes_io_instance: The collector name
   • namespace: The collector namespace
   • component_id: The failing sink ID, such as output_lokistack_otlp_application or output_my_splunk
   • error_kind: The general error category

2. Check the collector pod logs to identify the specific error details:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/instance=_<collector_name>_ | grep -i error | tail -50

   Copy code to clipboard

   The logs contain detailed error messages such as:

   • error trying to connect: dns error: failed to lookup address information: Name or service not known - DNS resolution failure
   • connection refused - Service not listening on the configured port
   • certificate verify failed - TLS certificate validation error
   • 401 Unauthorized or 403 Forbidden - Authentication or authorization failure

3. Review the ClusterLogForwarder configuration to find the output definition that matches the component_id from the alert:

   $ oc get clusterlogforwarder _<clusterlogforwarder_name>_ -n _<namespace>_ -o yaml

   Copy code to clipboard
4. Verify that the log receiver service is available and reachable from the cluster.
5. Validate the authorization credentials for the log receiver.

6. In the ClusterLogForwarder configuration, verify the following:

   • The output URL is correct
   • The TLS configuration in secrets and config maps is valid
   • The authorization credentials in secrets are correct

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The collector namespace
   • app_kubernetes_io_instance: The ClusterLogForwarder instance name
   • component_id: The source component that is discarding logs

2. Check the collector pod logs for details about discarded log lines:

   $ oc logs _<collector_pod_name>_ -n openshift-logging | grep -i discard

   Copy code to clipboard

3. Review the ClusterLogForwarder configuration to check the maxMessageSize setting:

   $ oc get clusterlogforwarder _<clusterlogforwarder_name>_ -n _<namespace>_ -o yaml

   Copy code to clipboard

   The default maxMessageSize is 3 MB (3145728 bytes). Log lines exceeding this size are discarded.

4. Identify which pods or containers are producing oversized log lines by querying the collector metrics:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'rate(log_source_internal_metrics_events_discarded_total{reason="message_too_large"}[5m]) > 0'

   Copy code to clipboard

   The output shows which collector sources (identified by component_id label) are discarding logs.

5. Identify the specific pods or namespaces affected:

   $ oc logs _<collector_pod_name>_ -n openshift-logging | grep -i "discarded\|too large" | head -20

   Copy code to clipboard

   Look for log entries indicating which application pod or namespace is generating oversized messages.

6. Investigate the source of oversized log lines:

   • Check application logs for unusually large log entries such as stack traces, debug output, or data dumps
   • Review container configurations for verbose logging settings
   • Identify if specific error conditions trigger large log messages

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The namespace containing the LokiStack
   • pod: The specific ingester pod experiencing flush failures

2. Check the status of the affected ingester pod:

   $ oc get pod _<pod_name>_ -n _<namespace>_

   Copy code to clipboard

3. Review the ingester pod logs for flush-related errors:

   $ oc logs _<pod_name>_ -n _<namespace>_ | grep -i flush

   Copy code to clipboard

   Look for error messages related to storage connectivity, authentication, or capacity issues.

4. Verify that the object storage bucket is accessible and has not been deleted or modified:

   $ oc get lokistack _<lokistack_name>_ -n _<namespace>_ -o yaml

   Copy code to clipboard

   Check the spec.storage section for the object storage configuration.

5. Check the secret containing the object storage credentials:

   $ oc get secret _<storage_secret_name>_ -n _<namespace>_ -o yaml

   Copy code to clipboard

   Verify that the secret exists and contains valid credentials.

6. Verify network connectivity from the ingester pod to the object storage endpoint:

   $ oc exec _<pod_name>_ -n _<namespace>_ -- curl -v _<storage_endpoint_url>_

   Copy code to clipboard
7. Check for storage capacity issues in the object storage backend by reviewing bucket usage and quotas through your cloud provider’s console or CLI tools.

1. Check that the flush failure rate decreases:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'rate(loki_ingester_chunks_flush_failures_total[5m]) / rate(loki_ingester_chunks_flush_requests_total[5m])'

   Copy code to clipboard
2. Verify that the alert clears in the Red Hat OpenShift Logging web console after approximately 15 minutes.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The namespace containing the LokiStack
   • job: The LokiStack gateway job experiencing write errors

2. Check the status of the LokiStack gateway pods:

   $ oc get pods -n _<namespace>_ -l app.kubernetes.io/component=lokistack-gateway

   Copy code to clipboard

3. Review the LokiStack gateway logs for write request errors:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=lokistack-gateway --tail=100 | grep -i error

   Copy code to clipboard

4. Check the status of the LokiStack distributor and ingester components:

   $ oc get pods -n _<namespace>_ -l app.kubernetes.io/component=distributor
   $ oc get pods -n _<namespace>_ -l app.kubernetes.io/component=ingester

   Copy code to clipboard

5. Review the LokiStack custom resource status for component health:

   $ oc get lokistack _<lokistack_name>_ -n _<namespace>_ -o jsonpath='{.status}'

   Copy code to clipboard

6. Check for resource constraints on the ingester pods:

   $ oc describe pod -n _<namespace>_ -l app.kubernetes.io/component=ingester

   Copy code to clipboard

   Look for memory or CPU throttling, OOMKilled events, or resource limit warnings.

7. Verify that the object storage backend is accessible and functioning:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=ingester | grep -i storage

   Copy code to clipboard

1. Check the error rate for write requests:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'sum(rate(lokistack_gateway_http_requests_total{code=~"5..", handler="push"}[5m])) / sum(rate(lokistack_gateway_http_requests_total{handler="push"}[5m]))'

   Copy code to clipboard
2. Verify that the alert clears in the Red Hat OpenShift Logging web console after approximately 15 minutes.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The namespace containing the LokiStack
   • job: The LokiStack gateway job experiencing read errors

2. Check the status of the LokiStack gateway pods:

   $ oc get pods -n _<namespace>_ -l app.kubernetes.io/component=lokistack-gateway

   Copy code to clipboard

3. Review the LokiStack gateway logs for query request errors:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=lokistack-gateway --tail=100 | grep -i error

   Copy code to clipboard

4. Check the status of the LokiStack querier and query front end components:

   $ oc get pods -n _<namespace>_ -l app.kubernetes.io/component=querier
   $ oc get pods -n _<namespace>_ -l app.kubernetes.io/component=query front end

   Copy code to clipboard

5. Review the querier logs for specific query errors:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=querier --tail=100 | grep -E 'error|failed|timeout'

   Copy code to clipboard

6. Check for resource constraints on the querier pods:

   $ oc describe pod -n _<namespace>_ -l app.kubernetes.io/component=querier

   Copy code to clipboard

   Look for memory or CPU throttling, OOMKilled events, or resource limit warnings.

7. Verify that the object storage backend is accessible:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=querier | grep -i storage

   Copy code to clipboard

8. Check if query timeouts are occurring due to expensive queries:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=querier | grep -i timeout

   Copy code to clipboard

1. Check the error rate for query requests:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'sum(rate(lokistack_gateway_http_requests_total{code=~"5..", handler=~"query|query_range"}[5m])) / sum(rate(lokistack_gateway_http_requests_total{handler=~"query|query_range"}[5m]))'

   Copy code to clipboard
2. Verify that the alert clears in the Red Hat OpenShift Logging web console after approximately 15 minutes.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The namespace containing the LokiStack
   • job: The specific Loki component job experiencing errors, such as lokistack-distributor, lokistack-ingester, or lokistack-querier
   • route: The API route experiencing errors

2. Identify the affected Loki component from the job label and check the pod status:

   $ oc get pods -n _<namespace>_ | grep _<component_name>_

   Copy code to clipboard

   For example, if job=lokistack-ingester, check ingester pods.

3. Review the component logs for error details:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=_<component_name>_ --tail=100 | grep -E 'error|ERROR|5[0-9]{2}'

   Copy code to clipboard

4. Check the LokiStack custom resource status:

   $ oc get lokistack _<lokistack_name>_ -n _<namespace>_ -o jsonpath='{.status.conditions}'

   Copy code to clipboard

5. Check for resource constraints on the affected component:

   $ oc describe pod -n _<namespace>_ -l app.kubernetes.io/component=_<component_name>_

   Copy code to clipboard

   Look for memory or CPU throttling, OOMKilled events, or resource limit warnings.

6. Review recent configuration changes to the LokiStack:

   $ oc get lokistack _<lokistack_name>_ -n _<namespace>_ -o yaml

   Copy code to clipboard

7. Check for related alerts that might provide more context:

   $ oc get prometheusrule -n _<namespace>_ -o yaml | grep -A 10 "LokiStack\|Loki"

   Copy code to clipboard

1. Check the error rate for the affected component:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'sum(rate(loki_request_duration_seconds_count{status_code=~"5.."}[5m])) by (job, route) / sum(rate(loki_request_duration_seconds_count[5m])) by (job, route)'

   Copy code to clipboard
2. Verify that the alert clears in the Red Hat OpenShift Logging web console after approximately 15 minutes.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The namespace containing the ClusterLogForwarder
   • app_kubernetes_io_instance: The ClusterLogForwarder instance name

2. Review the ClusterLogForwarder configuration:

   $ oc get clusterlogforwarder _<clusterlogforwarder_name>_ -n _<namespace>_ -o yaml

   Copy code to clipboard

3. Check the status of the ClusterLogForwarder for configuration errors:

   $ oc get clusterlogforwarder _<clusterlogforwarder_name>_ -n _<namespace>_ -o jsonpath='{.status.conditions}'

   Copy code to clipboard

4. Review the collector pod logs for unmatched log entry warnings:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/instance=_<clusterlogforwarder_name>_ | grep -i unmatched

   Copy code to clipboard

5. Examine the generated collector configuration:

   $ oc get configmap _<collector_configmap_name>_ -n _<namespace>_ -o yaml

   Copy code to clipboard

   The collector configuration is generated from the ClusterLogForwarder specification.

1. Apply the corrected ClusterLogForwarder configuration:

   $ oc apply -f clusterlogforwarder.yaml

   Copy code to clipboard

2. Monitor the collector pod logs to confirm that unmatched log warnings no longer appear:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/instance=_<clusterlogforwarder_name>_ --follow | grep -i unmatched

   Copy code to clipboard
3. Verify that the alert clears in the Red Hat OpenShift Logging web console after approximately 5 minutes.

Procedure

1. Identify the ClusterLogForwarder instances using the deprecated API:

   $ oc get clusterlogforwarder --all-namespaces -o json | jq -r '.items[] | select(.spec.outputs[]?.type == "azureMonitor") | "\(.metadata.namespace)/\(.metadata.name)"'

   Copy code to clipboard

2. For each affected ClusterLogForwarder, review the current configuration:

   $ oc get clusterlogforwarder _<clusterlogforwarder_name>_ -n _<namespace>_ -o yaml

   Copy code to clipboard

3. In the Azure portal, configure the Azure Monitor Logs Ingestion API:

   1. Create a Data Collection Endpoint (DCE).
   2. Create a Data Collection Rule (DCR) and associate it with the DCE.
   3. Create a custom table in your Log Analytics workspace.
   4. Assign the "Monitoring Metrics Publisher" role to the service principal or managed identity.
   5. Note the DCE endpoint URL, DCR immutable ID, and stream name for the next step.

4. Update the ClusterLogForwarder configuration to use the Azure Monitor Logs Ingestion output type:

   apiVersion: observability.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging
   spec:
     outputs:
     - name: azure-monitor-ingestion
       type: azureLogsIngestion  # New output type
       azureMonitor:
         authentication:
           type: servicePrincipal
           servicePrincipal:
             clientId: _<client_id>_
             tenantId: _<tenant_id>_
             clientSecretName: azure-secret
         dceEndpoint: _<dce_endpoint_url>_
         dcrImmutableId: _<dcr_immutable_id>_
         streamName: _<stream_name>_
     pipelines:
     - name: application-logs
       inputRefs:
       - application
       outputRefs:
       - azure-monitor-ingestion

   Copy code to clipboard

5. Create or update the secret containing the Azure service principal credentials:

   $ oc create secret generic azure-secret \
     --from-literal=clientSecret=_<client_secret>_ \
     -n _<namespace>_

   Copy code to clipboard

6. Apply the updated ClusterLogForwarder configuration:

   $ oc apply -f clusterlogforwarder.yaml

   Copy code to clipboard

1. Check the collector pod logs for successful Azure Monitor connections:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/instance=_<clusterlogforwarder_name>_ | grep -i azure

   Copy code to clipboard
2. In the Azure portal, verify that logs are appearing in the custom table.
3. Verify that the alert clears in the Red Hat OpenShift Logging web console after approximately 5 minutes.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the hostname label to identify the affected node.

2. Check the disk usage on the affected node:

   $ oc debug node/_<node_name>_ -- df -h /var

   Copy code to clipboard

3. Review the Vector disk buffer configuration and current usage:

   $ oc exec -n openshift-logging _<collector_pod_name>_ -- ls -lh /var/lib/vector

   Copy code to clipboard

4. Check the collector pod logs for slow output or connection errors:

   $ oc logs -n openshift-logging _<collector_pod_name>_ --tail=100 | grep -E 'slow|timeout|error|buffer'

   Copy code to clipboard

5. Review the ClusterLogForwarder configuration to identify output destinations:

   $ oc get clusterlogforwarder -n openshift-logging -o yaml

   Copy code to clipboard

6. Check if the output destinations are reachable and responding:

   $ oc exec -n openshift-logging _<collector_pod_name>_ -- curl -v _<output_destination_url>_

   Copy code to clipboard

7. Review the rate of log generation on the node to determine if it exceeds normal levels:

   $ oc exec -n openshift-logging _<collector_pod_name>_ -- vector top

   Copy code to clipboard

1. Check that the buffer usage is decreasing:

   $ oc exec -n openshift-logging _<collector_pod_name>_ -- du -sh /var/lib/vector

   Copy code to clipboard
2. Verify that the alert clears in the Red Hat OpenShift Logging web console after buffer usage drops below 15%.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The namespace containing the LokiStack
   • stack_name: The LokiStack instance name

2. Check the LokiStack status for component readiness:

   $ oc get lokistack _<lokistack_name>_ -n _<namespace>_ -o jsonpath='{.status.conditions}' | jq

   Copy code to clipboard

3. List all pods associated with the LokiStack and check their status:

   $ oc get pods -n _<namespace>_ -l app.kubernetes.io/instance=_<lokistack_name>_

   Copy code to clipboard

4. Identify pods that are not in Running or Ready state and examine their details:

   $ oc describe pod _<pod_name>_ -n _<namespace>_

   Copy code to clipboard

5. Review the logs of unhealthy pods:

   $ oc logs _<pod_name>_ -n _<namespace>_

   Copy code to clipboard

6. Check for resource constraints, image pull errors, or configuration issues in the pod events:

   $ oc get events -n _<namespace>_ --field-selector involvedObject.name=_<pod_name>_ --sort-by='.lastTimestamp'

   Copy code to clipboard

1. Check that all pods are running and ready:

   $ oc get pods -n _<namespace>_ -l app.kubernetes.io/instance=_<lokistack_name>_

   Copy code to clipboard

2. Verify the LokiStack status shows all components ready:

   $ oc get lokistack _<lokistack_name>_ -n _<namespace>_ -o jsonpath='{.status.conditions}' | jq '.[] | select(.type=="Ready")'

   Copy code to clipboard
3. Verify that the alert clears in the Red Hat OpenShift Logging web console after all components reach Ready state.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • stack_namespace: The namespace containing the LokiStack
   • stack_name: The LokiStack instance name

2. Check the current LokiStack storage schema configuration:

   $ oc get lokistack _<lokistack_name>_ -n _<namespace>_ -o jsonpath='{.spec.storage.schemas}' | jq

   Copy code to clipboard

3. Review the LokiStack status for schema update information:

   $ oc get lokistack _<lokistack_name>_ -n _<namespace>_ -o jsonpath='{.status.conditions}' | jq '.[] | select(.reason=="StorageNeedsSchemaUpdate")'

   Copy code to clipboard

4. Check the Loki Operator logs for recommended schema version:

   $ oc logs -n openshift-logging-operator deployment/loki-operator-controller-manager | grep -i schema

   Copy code to clipboard

1. Check the LokiStack status to confirm the schema update is acknowledged:

   $ oc get lokistack _<lokistack_name>_ -n _<namespace>_ -o jsonpath='{.status.conditions}' | jq

   Copy code to clipboard
2. Verify that the alert clears in the Red Hat OpenShift Logging web console after approximately 5 minutes.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The namespace containing the LokiStack
   • tenant: The tenant experiencing discarded samples
   • reason: The specific reason for sample discard, such as out_of_order, too_far_in_past, too_far_in_future, or line_too_long

2. Review the Loki ingester logs for discard details:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=ingester | grep -i discard

   Copy code to clipboard

3. Check the rate of discarded samples:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'sum by(namespace, tenant, reason) (rate(loki_discarded_samples_total[5m]))'

   Copy code to clipboard

4. Review the LokiStack limits configuration:

   $ oc get lokistack _<lokistack_name>_ -n _<namespace>_ -o jsonpath='{.spec.limits}' | jq

   Copy code to clipboard

1. Monitor the discard rate:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'sum by(reason) (rate(loki_discarded_samples_total[5m]))'

   Copy code to clipboard
2. Verify that the alert clears in the Red Hat OpenShift Logging web console after the discard rate drops to zero.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The namespace containing the LokiStack
   • job: The specific Loki component experiencing panics

2. Review logs from the affected component to identify the panic details:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=_<component_name>_ --tail=200 | grep -A 20 -i panic

   Copy code to clipboard

3. Check if the component pods are in a crash loop:

   $ oc get pods -n _<namespace>_ -l app.kubernetes.io/component=_<component_name>_

   Copy code to clipboard

4. Examine the pod events for restart information:

   $ oc describe pod _<pod_name>_ -n _<namespace>_

   Copy code to clipboard
5. Check if the panics are occurring during specific operations by reviewing the panic stack traces in the logs.

1. Check that the component pods are running without restart loops:

   $ oc get pods -n _<namespace>_ -l app.kubernetes.io/component=_<component_name>_ -w

   Copy code to clipboard

2. Monitor the panic rate:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'sum(increase(loki_panic_total[10m])) by (job)'

   Copy code to clipboard
3. Verify that the alert clears in the Red Hat OpenShift Logging web console after panics stop occurring.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The namespace containing the LokiStack
   • job: The Loki component experiencing high latency
   • route: The API route with slow responses

2. Check the current latency metrics for the affected route:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[5m])) by (job, route, le))'

   Copy code to clipboard

3. Identify whether latency is affecting reads (queries) or writes (ingestion):

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=_<component_name>_ | grep -E 'slow|latency|timeout'

   Copy code to clipboard

4. Check for resource constraints on the affected component:

   $ oc describe pod -n _<namespace>_ -l app.kubernetes.io/component=_<component_name>_ | grep -A 5 "Limits\|Requests"

   Copy code to clipboard
5. Review object storage performance metrics if available through your cloud provider.

1. Check that the 99th percentile latency decreases:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'histogram_quantile(0.99, sum(rate(loki_request_duration_seconds_bucket[5m])) by (job, route, le))'

   Copy code to clipboard
2. Verify that the alert clears in the Red Hat OpenShift Logging web console after latency drops below 5 seconds.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The namespace containing the LokiStack
   • job: The Loki component experiencing slow storage writes

2. Check the current storage write latency:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'histogram_quantile(0.99, sum(rate(loki_boltdb_shipper_request_duration_seconds_bucket{operation="WRITE"}[5m])) by (job, le))'

   Copy code to clipboard

3. Review ingester logs for storage-related errors or slow flush operations:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=ingester | grep -E 'flush|storage|slow'

   Copy code to clipboard

4. Verify object storage credentials and connectivity:

   $ oc get secret _<storage_secret_name>_ -n _<namespace>_ -o yaml

   Copy code to clipboard
5. Check cloud provider storage service status and performance metrics.

1. Monitor storage write latency:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'histogram_quantile(0.99, sum(rate(loki_boltdb_shipper_request_duration_seconds_bucket{operation="WRITE"}[5m])) by (job, le))'

   Copy code to clipboard
2. Verify that the alert clears in the Red Hat OpenShift Logging web console after write latency drops below 1 second.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The namespace containing the LokiStack
   • job: The Loki component experiencing slow storage reads

2. Check the current storage read latency:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'histogram_quantile(0.99, sum(rate(loki_boltdb_shipper_request_duration_seconds_bucket{operation="Shipper.Query"}[5m])) by (job, le))'

   Copy code to clipboard

3. Review querier logs for storage-related errors or slow read operations:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=querier | grep -E 'storage|slow|timeout'

   Copy code to clipboard
4. Check cloud provider storage service status and performance metrics.

1. Monitor storage read latency:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'histogram_quantile(0.99, sum(rate(loki_boltdb_shipper_request_duration_seconds_bucket{operation="Shipper.Query"}[5m])) by (job, le))'

   Copy code to clipboard
2. Verify that the alert clears in the Red Hat OpenShift Logging web console after read latency drops below 5 seconds.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The namespace containing the LokiStack
   • job: The Loki ingester job experiencing backpressure

2. Check the status of ingester pods and their resource usage:

   $ oc get pods -n _<namespace>_ -l app.kubernetes.io/component=ingester
   $ oc top pods -n _<namespace>_ -l app.kubernetes.io/component=ingester

   Copy code to clipboard

3. Review ingester logs for flush-related warnings or errors:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=ingester | grep -E 'flush|backpressure|WAL'

   Copy code to clipboard

4. Check ingester memory usage and limits:

   $ oc describe pod -n _<namespace>_ -l app.kubernetes.io/component=ingester | grep -A 5 "Limits\|Requests"

   Copy code to clipboard
5. Verify object storage write performance. See the LokiStorageSlowWrite troubleshooting procedure for storage diagnostics.

1. Check that WAL replay flushing has stopped:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'sum(loki_ingester_wal_replay_flushing) by (job)'

   Copy code to clipboard

2. Monitor ingester memory usage to ensure it stabilizes:

   $ oc top pods -n _<namespace>_ -l app.kubernetes.io/component=ingester -w

   Copy code to clipboard
3. Verify that the alert clears in the Red Hat OpenShift Logging web console after backpressure resolves.

Procedure

1. In the Red Hat OpenShift Logging web console, navigate to the alert and examine the following labels:

   • namespace: The namespace containing the LokiStack
   • job: The Loki component experiencing high query load

2. Check the current query latency:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'histogram_quantile(0.99, sum(rate(loki_logql_querystats_latency_seconds_bucket[5m])) by (job, le))'

   Copy code to clipboard

3. Review the number of concurrent queries:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'sum(loki_querier_queries_running) by (job)'

   Copy code to clipboard

4. Check querier component resource usage:

   $ oc get pods -n _<namespace>_ -l app.kubernetes.io/component=querier
   $ oc top pods -n _<namespace>_ -l app.kubernetes.io/component=querier

   Copy code to clipboard

5. Review querier logs for slow queries:

   $ oc logs -n _<namespace>_ -l app.kubernetes.io/component=querier | grep -E 'slow|timeout'

   Copy code to clipboard

1. Check that query latency decreases:

   $ oc exec -n openshift-monitoring prometheus-k8s-0 -- promtool query instant http://localhost:9090 'histogram_quantile(0.99, sum(rate(loki_logql_querystats_latency_seconds_bucket[5m])) by (job, le))'

   Copy code to clipboard

2. Monitor querier resource usage to ensure it stabilizes:

   $ oc top pods -n _<namespace>_ -l app.kubernetes.io/component=querier -w

   Copy code to clipboard
3. Verify that the alert clears in the Red Hat OpenShift Logging web console after query latency drops below 30 seconds.

Procedure

1. Create an AlertingRule custom resource (CR):

   The following example displays infrastructure AlertingRule CR:

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: loki-operator-alerts
       namespace: openshift-operators-redhat
       labels:
         openshift.io/cluster-monitoring: "true"
     spec:
       tenantID: infrastructure
       groups:
         - name: LokiOperatorHighReconciliationError
           rules:
             - alert: HighPercentageError
               expr: |
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"} |= "error" [1m])) by (job)
                   /
                 sum(rate({kubernetes_namespace_name="openshift-operators-redhat", kubernetes_pod_name=~"loki-operator-controller-manager.*"}[1m])) by (job)
                   > 0.01
               for: 10s
               labels:
                 severity: critical
               annotations:
                 summary: High Loki Operator Reconciliation Errors
                 description: High Loki Operator Reconciliation Errors

   Copy code to clipboard

   metadata.namespace

   The namespace must have a label that matches the LokiStack spec.rules.namespaceSelector configuration.

   metadata.labels

   The labels must match the LokiStack spec.rules.selector configuration.

   spec.tenantID

   For infrastructure tenants, create the AlertingRule only in openshift-*, kube-*, or default namespaces.

   expr

   The value of kubernetes_namespace_name must match the metadata.namespace value.

   labels.severity

   Set this mandatory field to critical, warning, or info.

   annotations.summary

   This field is mandatory.

   annotations.description

   This field is mandatory.

   The following example displays application AlertingRule CR:

     apiVersion: loki.grafana.com/v1
     kind: AlertingRule
     metadata:
       name: app-user-workload
       namespace: app-ns
       labels:
         openshift.io/cluster-monitoring: "true"
     spec:
       tenantID: application
       groups:
         - name: AppUserWorkloadHighError
           rules:
             - alert:
               expr: |
                 sum(rate({kubernetes_namespace_name="app-ns", kubernetes_pod_name=~"podName.*"} |= "error" [1m])) by (job)
               for: 10s
               labels:
                 severity: critical
               annotations:
                 summary: This is an example summary.
                 description: This is an example description.

   Copy code to clipboard

   metadata.namespace

   The namespace must have a label that matches the LokiStack spec.rules.namespaceSelector configuration.

   metadata.labels

   The labels must match the LokiStack spec.rules.selector configuration.

   expr

   The value of kubernetes_namespace_name must match the metadata.namespace value.

   labels.severity

   Set this mandatory field to critical, warning, or info.

   annotations.summary

   Provide a summary of the rule. This field is mandatory.

   annotations.description

   Provide a detailed description of the rule. This field is mandatory.

2. Apply the AlertingRule CR:

   $ oc apply -f <filename>.yaml

   Copy code to clipboard