How detectMultilineErrors or detectMultilineException works in Logging stack with RHOCP 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift Logging (RHOL)
    • 5
    • 6
  • Log Collector
  • Fluentd
  • Vector

Issue

  • The option detectMultilineErrors is configured in Logging stack v5 (the same feature is called detectMultilineException in the Logging stack v6), but the detection of multi lines is not working
  • Why the applications logs are visible in different lines in the Log Store when the clusterLogForwarder is configured for detecting multi line Errors?
  • How does detectMultilineErrors in the Logging Stack v5 or detectMultilineException in the Logging Stack v6 works?

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Notes:

When detectMultilineErrors: true is configured in the clusterlogforwarder for RHOL v5 as documented in the article Multiline logs in OpenShift 4 with ClusterLogForwarder API then the plugin Content from github.com is not included.fluent-plugin-detect-exceptions it's enabled in the Fluentd Log Collector.

For the Vector collector (and for RHOL v6), the rules applied are a translation of the fluent-plugin-detect-exceptions plugin, that can be evolved in time. These rules are avaible in the "Content from github.com is not included.ViaQ/Vector" repository.

How does the Fluentd plugin work

The Content from github.com is not included.fluent-plugin-detect-exceptions plugin uses regular expressions for Ruby when, after detecting the language code used in the log, tries to find if the next lines are part of the same error.

It is important to observe that the rules try to match always regular expressions using the English language, and no other languages or translation.

For example, for Java, the rule (regular expression in Ruby) that evaluates if a log line is the first line of the error is:

    JAVA_RULES = [
      rule(%i[start_state java_start_exception],
           /(?:Exception|Error|Throwable|V8 errors stack trace)[:\r\n]/,

If the first line is detected, the parsing continues with the next lines in java_after_exception to verify if they are part of the initial error:

...
      rule(:java_after_exception, /^[\t ]*nested exception is:[\t ]*/,
           :java_start_exception),
      rule(:java_after_exception, /^[\r\n]*$/, :java_after_exception),
      rule(%i[java_after_exception java], /^[\t ]+(?:eval )?at /, :java),
...

Example 1. Java example detected as MultiLine

Java MultiLine log example where detectMultilineErrors is detecting as MultiLine:

Caused by: java.io.FileNotFoundException: The requested resource [/ztl/jsp/login/favicon.ico] is not available
    at org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:868)
    at org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:519)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)

Where the first line is matching the regular expression for the java_start_exception. Exactly this line:

Caused by: java.io.FileNotFoundException: The requested resource [/ztl/jsp/login/favicon.ico] is not available

The next lines are part of the multiline exception. It should be checked if they are matching any of the regular expressions from java_after_exception and it does matching java_after_exception ^[\t ]+(?:eval )?at`:

    at org.apache.catalina.servlets.DefaultServlet.serveResource(DefaultServlet.java:868)
    at org.apache.catalina.servlets.DefaultServlet.doGet(DefaultServlet.java:519)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:655)

Example 2 Java example not detected as MultiLine

Java MultiLine log example where detectMultilineErrors is not detecting as MultiLine:

22-12-29 15:59:07 [ERROR] ServletException; request:/ztl/jsp/home.jsp
Attributi:
org.apache.catalina.AccessLog.RemoteAddr = x.x.x.x 
com.dynatrace.oneagent.sensors.uem = com.dynatrace.agent.introspection.uem.UemAttribute@5a0c3df3 
org.apache.catalina.AccessLog.Protocol = HTTP/1.1 
org.apache.tomcat.remoteAddr = x.x.x.x 
org.apache.catalina.AccessLog.ServerName = server.example.com 
org.apache.catalina.AccessLog.ServerPort = 443 
org.apache.tomcat.request.forwarded = true 
org.apache.catalina.AccessLog.RemoteHost = x.x.x.x 
Parametri:
modo = login/favicon.ico, [1] 
Segue l'eccezione di origine:

The first line below is not detected as the first line of a Java MultiLine error because it's not matching the java_start_exception regular expression.

22-12-29 15:59:07 [ERROR] ServletException; request:/ztl/jsp/home.jsp

The next lines below are not detected as part of a continuation of a Java MultiLine log since they are not indented to the right as a normal Java stack trace. Then, they are not matching any of the regular expressions of java_after_exception:

Attributi:
org.apache.catalina.AccessLog.RemoteAddr = x.x.x.x 
com.dynatrace.oneagent.sensors.uem = com.dynatrace.agent.introspection.uem.UemAttribute@5a0c3df3 
org.apache.catalina.AccessLog.Protocol = HTTP/1.1 
org.apache.tomcat.remoteAddr = x.x.x.x 
org.apache.catalina.AccessLog.ServerName = server.example.com 
org.apache.catalina.AccessLog.ServerPort = 443 
org.apache.tomcat.request.forwarded = true 
org.apache.catalina.AccessLog.RemoteHost = x.x.x.x 
Parametri:
modo = login/favicon.ico, [1] 
Segue l'eccezione di origine:

Root Cause

The stack traces log lines must match the rules:

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.