Reducing log loss during node scaledown by ensuring Vector terminates last.

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Red Hat OpenShift Logging(RHOL)
  • 5
  • 6
• Vector

Issue

• Reducing log loss during node scaledown by ensuring Vector terminates last.
• During node shutdown/scaledown, the Vector pod terminates before other workload pods, leading to log loss. Ideally, the Vector pod should be the last to terminate, ensuring that workload pod logs are captured and forwarded before shutdown.

Resolution

• Red Hat Logging stack is not a guaranteed log sink, feature request This content is not included.OBSDA-1206 is raised to review if log loss caused by the Vector's early during node scaledown/shutdown event can be reduced.
• For more information please open a new support case with Red Hat Support.

Diagnostic Steps

Steps to Validate:

1> Deployed a log-generator application that continuously writes logs to both Persistent Volume and STDOUT.

2> Initiated a shutdown of the node hosting this log-generator pod.

3> Compare the logs:

Example:
PV recorded up to line 95999
Loki recorded only up to line 93627
This confirms a loss of 2372 log lines during shutdown.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.