Logs with detectMultilineException option enabled not properly parsed in RHOCP 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Red Hat OpenShift Logging (RHOL)
  • 6
• Vector
• detectMultilineException

Issue

• Logs are being printed with a multiline format for certain services.
• Each line in the log is treated as a separate line in Loki and it causes difficulties in searching and identifying logs.
• When log forwarder sends multiline exception, log is forwarded in multiple single lines
• Remap support for the reduce transform (multi-line logs)
• Certain Java stack traces appear split line-by-line in Loki instead of a single aggregated event.

Resolution

Red Hat is aware of a request for being able to create custom remap in the collector for being able to parse application multi-line logs as a single log line, currently under review in RFE This content is not included.OBSDA-762.

If this feature is required, open a support case on the Red Hat Customer Portal referring to this solution.

Root Cause

The feature multi-line exception assembles multi-lines exception stack trace generated by the languages described in the Red Hat OpenShift Logging Documentation, not multi line logs generated by the application.

For example in case of Java language:

• Multiline aggregation is based on strict pattern matching for Java exceptions.
• Stack traces are aggregated only while the log lines match a recognized Java exception pattern
• If a non-Java line appears within the stack trace (for example, URLs such as
  Content from docs.oracle.com is not included.https://docs.oracle.com/error-help/db/ora-12541/),%20the%20filter%20exits%20the%20Java%20exception%20state
• When the filter exits the Java exception state, subsequent at ... lines and chained exceptions are ingested as separate log events in Loki

Read the Red Hat Knowledge Article "How detectMultilineErrors or detectMultilineException works in Logging stack with RHOCP 4" for understanding better how the detectMultilineException feature works.

Diagnostic Steps

1. Verify that the collector has enabled detectMultilineException for that output
2. Identify an application that prints a log being multi line
3. View the logs in the Red Hat OpenShift console UI or in the destination and observe that it's not assembled in a single log line being in multiple log lines as generated by the application.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.