MultilineError logs in OpenShift 4 with ClusterLogForwarder API

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4.8 and newer versions
• Red Hat OpenShift Logging (RHOL)
  • 5.3 and newer versions

Issue

• How to forward multiline Error logs?
• What and where do we have to configure in fluentd in OpenShift Platform to forward multiline Error logs via official ClusterLogForwarder (4.8+)?
• How can Java stack traces be sent to ElasticSearch as a single log line?
• Java stack traces are shown on multiple lines in Kibana, how can they be shown as a single line?

Resolution

For Logging 5.3

OpenShift Logging 5.3 (starting from OpenShift Container Platform 4.8) allows to specify detectMultilineErrors: true in their ClusterLogForwarder as part of their pipeline definition that will summarise multi-line log entries:

  apiVersion: logging.openshift.io/v1
  kind: ClusterLogForwarder
  metadata:
      name: instance 
      namespace: openshift-logging 
  spec:
  ...
      pipelines:
       - name: forward-to-fluentd-secure 
         detectMultilineErrors: true
         inputRefs:  
         ...

For OpenShift 3

See this related solution

NOTE: In cases where the option is enabled and is observed that detectMultilineErrors does not work as expected, check the article: How detectMultilineErrors works in Logging stack with RHOCP 4.

Root Cause

Each error log line generated by the application is treated as a single line of log, then, it's processed like that by the collector. When a multi-line stack trace is generated by one application, each line of the stack trace is seen as an independent log entry, being not able to see the complete stack trace in one log entry.

Diagnostic Steps

• The ClusterLogForwarder CRD provides a field to detect multiline errors:

  $ oc explain clusterlogforwarders.spec.pipelines.detectMultilineErrors
  KIND: ClusterLogForwarder
  VERSION: logging.openshift.io/v1
  
  FIELD: detectMultilineErrors <boolean>
  
  DESCRIPTION:
  DetectMultilineErrors enables multiline error detection of container logs
  

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.