Multi-line logging in Openshift v3

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform
    • 3.X
  • JBOSS EAP Openshift image

Issue

  • We have tested the 3.1 centralized logging feature and saw that (like in v2) you do not treat multiline logging messages. We would like to have some way of handling multiline messages, and out-of-the-box support for this in the JBOSS image.
  • In v2, we used rsyslog to store the data in a file. Now you provide an elasticsearch/kibana access, but this does not work properly if you do not support multiline messages. Main use case are java exceptions : if you cannot keep the stacktrace as a whole, your metrics in kibana are useless and when you search on a given exception, you do not have the contaxt
  • Instead of using docker-like logging to stdout and treating that line based, you should be able to cope with logging in a multiline capable format (like JSON). The transport/storage/acces is ok, but then the log message is preserved, and not broken up into multiple independent lines.
  • We would like to achieve this by provoking a stacktrace in a JBOSS image and make sure it is logged as a single event. The result in kibana should show the complete sctacktrace, not a single line of it.

Resolution

A This content is not included.Request for feature enhancement was filed for the same. It has been closed.

The suggested workflow for this is to configure the application to not use the newline character \n in the middle of the log message. For JBOSS EAP, this is resolved in EAP7 with the merge of a Content from github.com is not included.pull request which sets ENABLE_JSON_LOGGING=true

For multiline logging in OCP 4.x, see this related solution

Root Cause

Fluentd, the process which gathers application logs from pods running in OpenShift, interprets logs on a per-line basis. Each line is divided by a newline character, \n. As such, any instance of the newline character will cause fluentd to treat it as a new line, which will be stored in Elasticsearch as a separate message.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.