High CPU Utilization due to GZIP compression issues at the CXF level

Environment

• Red Hat JBoss Enterprise Application Platform [EAP].
  • 5.1.x & above
  • 6.x.x

Issue

• Getting some thread dumps will yield multiple threads with the following description:

    "http-0.0.0.0-8580-7" daemon prio=10 tid=0x000000004a6cd800 nid=0x38e5 runnable [0x0000000049400000]
       java.lang.Thread.State: RUNNABLE
        at java.util.zip.Deflater.deflateBytes(Native Method)
        at java.util.zip.Deflater.deflate(Deflater.java:306)
        - locked <0x00002aab473baab0> (a java.util.zip.ZStreamRef)
        at java.util.zip.DeflaterOutputStream.deflate(DeflaterOutputStream.java:159)
        at java.util.zip.DeflaterOutputStream.write(DeflaterOutputStream.java:118)
        at java.util.zip.GZIPOutputStream.write(GZIPOutputStream.java:72)
        - locked <0x00002aab473baa78> (a java.util.zip.GZIPOutputStream)
    ....
  

Resolution

• This component is NOT supported in EAP 5.0.1 (it is only a tech preview). We only possibly support CXF in EAP 5.1 and beyond.
• It sounds like you could avoid this by disabling GZIP compression at the CXF level from reports of others. If you need to use GZIP compression after disabling GZIP compression at the CXF level, you should be able to add an attribute to the HTTP connector in the server.xml and enable compression through the connector. This can be done by adding compression="on" to your HTTP connector.

For EAP 6.x.x:-

• Use System Property org.apache.coyote.http11.Http11Protocol.COMPRESSION in standalone.xml after extensions tag and within system property tag. For Example :-

<extensions>
</extensions>

<system-properties>
     <property name="org.apache.coyote.http11.Http11Protocol.COMPRESSION" value="on"/>
</system-properties>

Above change requires restart of instance. Refer Content from docs.jboss.org is not included.EAP 6.x system properties Content from issues.jboss.org is not included.GZIP compression support EAP 6.x AS7-2991

• Otherwise, engaging the JDK vendor might be the next best bet to troubleshoot any high CPU/concurrency issues in GZIPOutputStream and Deflater

Root Cause

• Apparently Deflater.deflateBytes is called through GZIPOutputStream which may not be thread safe and is causing this infinite loop and CPU consumption.

Diagnostic Steps

• Troubleshoot with thread dumps and CPU utilization data as described in Java application high CPU to identify high CPU consumers.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.