How to add fluentd liveness probe in OCP 3

Environment

• Red Hat OpenShift Container Platform (OCP) 3.X

Issue

• Currently it looks like the Fluentd daemonset that comes out of the box isn't configured with a liveness probe and we have seen instances where the pods have become stuck and stop working. This is usually resolved with a cycling of the pods but it would be better to have a liveness probe to automatically do this rather than a manual check.
• Is there a set of recommended/supported liveness probes to implement across the clusters?

Resolution

There a few possible ways to implement a Liveness probe.

• /var/log/es-containers.log.pos- - Should have files under /var/log/containers/*.log listed in the file, and you should see the file positions change as fluentd is reading from those files - note that the file may not be updated immediately upon read, but over time it should have changes.

• /var/log/journal.pos - this should change as Fluentd reads from systemd journald - the contents are the journald cursor (journalctl --show-cursor)

• /var/lib/fluentd - This is where lives the buffer directories, buffer files are written to sub-directories here.

• At Fluentd documentation has a proposal to enable the Content from docs.fluentd.org is not included.monitoring REST API.

• This expression checks the last timestamp for the modification of directory /var/lib/fluentd .

livenessProbe:
            exec:
              command:
                - /bin/sh
                - '-c'
                - >-
                  LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300};
                  if [ ! -e /var/lib/fluentd ]; then exit 1; fi;
                  LAST_MODIFIED_DATE=$(stat /var/lib/fluentd  | grep Modify |
                  sed -r "s/Modify\:"" (.*)/\1/");
                  LAST_MODIFIED_TIMESTAMP=$(date -d "$LAST_MODIFIED_DATE" +%s);
                  if [ $(date +%s) -gt $(expr $LAST_MODIFIED_TIMESTAMP +
                  $LIVENESS_THRESHOLD_SECONDS) ]; then exit 1; fi;
            failureThreshold: 3
            initialDelaySeconds: 60
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.