Why does the Red Hat Logging stack not guarantee log sink?

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Red Hat OpenShift Logging (RHOL)
  • 5
  • 6

Issue

• What's the reason for the Red Hat Logging documentation indicating that it does not guarantee log sink?
• Need OpenShift Logging to send complete logs for critical audit reports

Resolution

The Red Hat Logging stack, or any other Logging solution, for the architecture of Kubernetes and OpenShift can not guarantee the log sink.

Needed to guarantee the log sink

A different technology solution should be used that integrates with the application as it's a messaging solution. The Red Hat product for this purpose is Red Hat AMQ.

Not needed to guarantee the log sink

If the reason is because from time to time an application generates a big spike of logs causing the logs being rotated so fast by CRI-O before giving time to the collector to ingest them, then, it could be considered to modify the log rotation managed by CRI-O following the Red Hat Knowledge Article "Modification of log rotation of CRI-O in Openshift 4" increasing the containerLogMaxSize.

Increasing the containerLogMaxSize will not work if the application is constantly generating high log volume, only when periodic spikes occur.

Other considerations

• Feature request "Reducing log loss during node scaledown by ensuring Vector terminates last"
• "Tuning log payloads and delivery in RHOCP 4"

Root Cause

Let's understand how the logs are written and later read by the collectors:

1. The containers write to the standard output and this standard output is captured by CRIO and written to the node filesystem
2. The collectors read from the filesystem to forward the logs
3. The collector could have problems on delivering the logs read to the destinations
4. Original log files are rotated by CRI-O when reached the limits as described in the Red Hat Knowledge Article "Modification of log rotation of CRI-O in Openshift 4".

If the log file is rotated by CRI-O before the collector has time to read and log forward it, it could be missed to get these logs read, log forwarded and delivered to the destination.

Another situation is when a node is being restarted. The collector pod is not guaranteed to be the last to be stopped. Thus, logs from the pods stopping after the collector during node restart could not be collected.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.