ajp-apr-<port>-Poller thread consumes high cpu in Tomcat 8

Environment

• Red Hat JBoss Web Server (JWS) 3.0.x
• Tomcat 8

Issue

• I observed JWS 3.0.x Tomcat 8 Java process consumed high cpu. And, when the customer checked the java process with Java Mission Control, it indicates ajp-apr-<port>-Poller thread consumes a lot of cpu time.
• We restarted our httpd AJP proxy and then saw increased CPU usage in the Tomcat 8 instances behind it.

Resolution

• This is fixed in Tomcat 8.0.25+. The fix is available in JWS 3.1.0+.
• Prior to JWS 3.1.0, you can workaround this issue by disabling APR. You can disable APR by either of the following two methods:

Option 1. Comment out org.apache.catalina.core.AprLifecycleListener in $CATALINE_HOME/conf/server.xml .

Modify $CATALINE_HOME/conf/server.xml and change from:

  <!--APR library loader. Documentation at /docs/apr.html -->
  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" />

to:

  <!--APR library loader. Documentation at /docs/apr.html -->
  <!-- <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on" /> -->

Option 2. Change the protocol attribute of AJP connector to Java connector

Modify $CATALINE_HOME/conf/server.xml and change from:

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

to either:

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="org.apache.coyote.ajp.AjpNioProtocol" redirectPort="8443" />

or

    <!-- Define an AJP 1.3 Connector on port 8009 -->
    <Connector port="8009" protocol="org.apache.coyote.ajp.AjpNio2Protocol" redirectPort="8443" />

Root Cause

• Content from bz.apache.org is not included.Apache BZ#58151 – High CPU Usage randomly on ajp-apr-8009-Poller
• Content from issues.jboss.org is not included.JWS-430
• Thread dumps and thread-level cpu usage of java process indicates that "ajp-apr-8009-Poller" repeats calling org.apache.tomcat.jni.Poll.poll in the loop with the following same stacks and it consumes high cpu. Looping poll itself is generally ok. However, the problem was that a return value of zero was treated as a non-blocking read that returned no data rather than as EOF. That meant that the socket was put straight back into the Poller only for the Poller to trigger a read event immediately for EOF and so the loop continued.

"ajp-apr-8009-Poller" #14 daemon prio=5 os_prio=0 tid=0x00007fc0e41ab000 nid=0x5840 runnable [0x00007fc0aeab0000]
   java.lang.Thread.State: RUNNABLE
	at org.apache.tomcat.jni.Poll.poll(Native Method)
	at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1757)
	at java.lang.Thread.run(Thread.java:745)

"ajp-apr-8009-Poller" #14 daemon prio=5 os_prio=0 tid=0x00007fc0e41ab000 nid=0x5840 runnable [0x00007fc0aeab0000]
   java.lang.Thread.State: RUNNABLE
	at org.apache.tomcat.jni.Poll.poll(Native Method)
	at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1757)
	at java.lang.Thread.run(Thread.java:745)

Diagnostic Steps

• Ask for thread dumps and thread-level cpu usage by using the script attached in this article at the time of the issue.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.