Vector splits the logs when log forwarding to syslog in RHOCP 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift Logging (RHOL)
    • 5
    • 6
  • Vector
  • syslog
  • rsyslog server

Issue

  • Vector splits a single log line in several log lines
  • Application log lines are arriving as partial to the syslog server
  • Vector DaemonSet in OpenShift collects incomplete or truncated JSON log entries.
  • JSON log lines appear intact in application pods but get truncated during collection.
  • Critical fields and information at the end of log entries are missing.
  • Impact: Audit/compliance gaps due to missing critical fields.

Resolution

  1. Configure the ClusterLogForwarder for the RFC 5424 and TCP protocol
  2. Increase the message size in the syslog server. If it's used rsyslog, review the Red Hat Knowledge Article " How to set the message size of rsyslog and what is the default and maximum value? " for modifying the variable $MaxMessageSize to a value that can allow to receive the log message sent by the collector included the metadata.

Note: For the audit logs, take a look to the Red Hat Knowledge Article "ClusterLogForwarder is not sending full audit logs to external rsyslog

Root Cause

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Several reasons could help to split or truncate the log lines when log forwarding to a syslog server:

* When the UDP Protocol, the UDP Protocol has length max size for the network package by definition
* When using the RFC 3164, the message length must not excee 1024 bytes. See RFC 3164 Section [Package Parameters](https://datatracker.ietf.org/doc/html/rfc3164#section-6.1) for more details
* When using the RFC 524, the receiver must be able to accept messages larger than 2048 octects in length. See RFC 5424 Section "[Message Length](https://www.rfc-editor.org/rfc/rfc5424.html#page-9)"
* The syslog server can have a max message size. See the Red Hat Knowledge Article "[How to set the message size of rsyslog and what is the default and maximum value?](https://access.redhat.com/solutions/2075423)"

For the current Red Hat Knowledge Article, the root cause was that the Rsyslog server had $MaxMessageSize to the default causing that the logs sent by the collector to the rsyslog server exceeding this default size were received in the rsyslog server in several log lines.

Diagnostic Steps

  1. Verify the ClusterLogForwarder that has configured an output using the RFC 5424 and using the TCP protocol:

        // In Logging v5
    $ oc get clusterLogForwarder -o yaml -n openshift-logging

    // In Logging v6
    $ oc get obsclf -o yaml -n openshift-logging

  2. Verify in the rsyslog server that a single log line produced by the application is split in several log lines

  3. Verify what's the size of the application log sent after including the metadata. For doing it, review the Red Hat Knowledge Article "How to use vector tap and vector top for troubleshooting in RHOCP 4"

  4. Verify in the rsyslog server what's the value for the variable $MaxMessageSize. If it's not set, the default value is used

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.