Loki Ingester pod in 0/1 state with probe giving 503 errors after restarting in RHOCP 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
• Red Hat OpenShift Logging (RHOL)
  • 6.2.11
• Lokistack
• Loki

Issue

• Loki Ingester pod in Status: 0/1 with probe giving 503 errors after restarting
• LivenessProbe fails on Loki Ingester pod, but logs just show "uploading tables" section of startup logs
• The WAL directory (/tmp/wal) in the Loki Ingester pod has a big size

Resolution

This is being tracked in the bug report Content from redhat.atlassian.net is not included.LOG-9583. For more information, please open a This content is not included.new support case on the Red Hat Customer Portal referring to this solution.

Workaround

As a workaround, deleting the WAL directory from the Loki Ingester in status 0/1 to initiate properly.

Access the problematic Loki Ingester pod either via rsh or debug:

//RSH
$ oc -n openshift-logging rsh <ingester pod>
//DEBUG
$ oc -n openshift-logging debug pod/<ingester pod>

Delete the WAL segment shown in the Loki Ingester pod logs and exit the debug pod.

# rm /tmp/wal/*
# exit

Delete manually the Loki Ingester failing pod.

$ oc delete pod <ingester pod> -n openshift-logging

Root Cause

The root cause is being analyzed in the bug Content from redhat.atlassian.net is not included.LOG-9583.

Diagnostic Steps

1. Verify a Loki Ingester pod is in status 0/1:

   $ oc get pods -l app.kubernetes.io/component=ingester -n openshift-logging
   NAME                              READY   STATUS    RESTARTS   AGE
   lokistack-example-ingester-0      0/1     Running   0          7d
   lokistack-example-ingester-1      1/1     Running   0          26d
   

2. Verify that no errors present in Loki Ingester pod in status 0/1:

   $ oc logs lokistack-example-ingester-0|grep -ic error
   0
   

3. Verify that the Loki Ingester is always cleaning up unwanted indexes from the same table, in this example: index_20622:

   $ oc logs lokistack-example-ingester-0|grep "cleaning up unwanted indexes"
   

4. Verify that the Loki Wal (/tmp/wal) in the Loki Ingester in status 0/1 has a big size:

   $ oc rsh lokistack-example-ingester-0
   sh-5.1$ df -h 
   Filesystem Size Used Avail Use% Mounted on 
   ...
   /dev/sdb 9.8G 1.2M 9.8G 1% /tmp/loki 
   /dev/sdc 147G 92G 56G 63% /tmp/wal
   

5. Check if the directories inside the /tmp/loki contain 0 files

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.