High CPU due to threads looping in OutputBuffer.realWriteChars

Environment

• Red Hat JBoss Enterprise Application Platform (EAP) 5.x
• Red Hat JBoss Enterprise Portal Platform (EPP) 5.x
• Red Hat JBoss Enterprise SOA Platform (SOA-P) 5.x

Issue

• We're seeing JBoss consumes high CPU and the thread dumps and CPU data indicates that the high consuming threads are always in some point of the OutputBuffer.realWriteChars method and its underlying char to byte conversions and buffer flushes, for example:

    "ajp-127.0.0.1-8009-1" daemon prio=10 tid=0x00002aabd4fb2000 nid=0x7a6 runnable [0x00002b24a1517000]
       java.lang.Thread.State: RUNNABLE
            at sun.nio.cs.ISO_8859_1$Encoder.encodeArrayLoop(ISO_8859_1.java:148)
            at sun.nio.cs.ISO_8859_1$Encoder.encodeLoop(ISO_8859_1.java:187)
            at java.nio.charset.CharsetEncoder.encode(CharsetEncoder.java:544)
            at org.apache.tomcat.util.buf.C2BConverter.convert(C2BConverter.java:91)
            at org.apache.catalina.connector.OutputBuffer.realWriteChars(OutputBuffer.java:465)
            at org.apache.tomcat.util.buf.CharChunk.flushBuffer(CharChunk.java:440)
            at org.apache.tomcat.util.buf.CharChunk.append(CharChunk.java:295)
            at org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:499)
            at org.apache.catalina.connector.CoyoteWriter.write(CoyoteWriter.java:143)
            at org.apache.jasper.runtime.JspWriterImpl.flushBuffer(JspWriterImpl.java:119)
            at org.apache.jasper.runtime.JspWriterImpl.write(JspWriterImpl.java:326)
  

Resolution

• Upgrade to
  • EAP 5.2.0 or later
  • EPP 5.2.1 or later
  • SOA-P 5.3.0 or later

Or

• Apply the available relevant fix:
  • This content is not included.EAP 5.1.0
  • This content is not included.EAP 5.1.1
  • This content is not included.EAP 5.1.2
  • This content is not included.EPP 5.2.0
  • This content is not included.SOA-P 5.2.0

Root Cause

• When a surrogate pair character is placed at the boundary of a buffer, the thread handling it will loop infinitely.
• This is an identified bug/vulnerability, CVE-2011-4610.

Diagnostic Steps

• Troubleshoot with thread dumps and CPU utilization data as described in Java application high CPU to identify high CPU consumers.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.