What happens if requests exceeds specified maxThreads on JBoss Web HTTP Connector

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 4
    • 5
  • Linux
  • Solaris

Issue

  • What happens if 200 requests go to the JBoss Web HTTP Connector on port 8080 at the same time? The maxThreads is 100 and the acceptCount is 30.

  • The acceptCount does not seem to be used within JBoss Enterprise Application Platform (EAP). The actual value when the requests are rejected are much higher than the set value.

  • When the requests are rejected, there is no error returned. The client will actually timeout. I would expect some error returned when the server starts rejecting messages.

  • JBoss 5 EAP doesn't use acceptCount (in server.xml) as expected. When the threads get all busy, the queue seems not to exist. All subsequent requests are rejected. This is not the expected behaviour. This problem only occurs in JBoss EAP 5.1, not in JBOSS 5.1 AS - based on my tests.

  • What caused the following message:

      INFO  [JIoEndpoint] Maximum number of threads (###) created for connector with address /127.0.0.1 and port 8080

Resolution

EAP 5.1

The acceptCount parameter is ignored. Once the maxThreads limit is reached no further connections are accepted. In the console log, the following message appears

INFO  [JIoEndpoint] Maximum number of threads (###) created for connector with address /127.0.0.1 and port 8080

This was done to prevent a Denial Of Service (DOS) attack. The general idea is to serve the keepAlive connections and reject new ones once the limit is reached.

To retain the JBossWeb 2.0.x behaviour from EAP 4.x, add the parameter -Dorg.apache.tomcat.util.net.WAITFORWORKER=true to the startup options.

JIRA Content from issues.jboss.org is not included.Content from issues.jboss.org is not included.https://issues.jboss.org/browse/JBPAPP-5016 has been raised to get the existing EAP 5.1.x documentation updated.

These threads are created on demand only when needed. If you reach your max threads and see the above warning, this means all threads were in use at the moment. Slow request processing can often lead to that behavior as it will keep threads occupied longer, thus requiring more threads to keep servicing new incoming requests. To check for such slowness, capture:

  • JBoss access logs with response time and thread info, which you can enable in your server.xml with a pattern on the AccessLogValve like below:

      <Valve className="org.apache.catalina.valves.AccessLogValve"
         prefix="localhost_access_log." suffix=".log"
         pattern="%T %h %l %u %t %r %s %b %S %I" directory="${jboss.server.home.dir}/log"
         resolveHosts="false" />

  • GC logs and thread dumps and other performance diagnostic steps are mentioned in the KB article "Java application unresponsive".

EAP 4.x

There is also an open bugzilla on this same subject: Content from issues.apache.org is not included.Content from issues.apache.org is not included.https://issues.apache.org/bugzilla/show_bug.cgi?id=48488

Diagnostic Steps

To understand how it works, you need to set up a test case on Linux as follows:

  • Test case comprised of a http request to a JSP page which had an extended delay of 20s (Threed.sleep(20000))
  • Use JMeter as the client (each test case repeated multiple times varying the number of concurrent requests i.e 40, 100, 200)

EAP 5.1.x

  • Using netsat -lantp, grep for jboss-eap pid and the same for the jmeter pid. The number of ESTABLISHED connection (client) = ESTABLISHED connection (server) = maxThreads

EAP 4.x / EAP 5.1.x (with parameter -Dorg.apache.tomcat.util.net.WAITFORWORKER=true)

  • Using netsat -lantp, grep for jboss-eap pid and the same for the JMeter pid. The number of ESTABLISHED connection on the server side appear to be lower than those on the client side.

  • Comparing the surplus ESTABLISHED connection on the client side, you'll observe that these same connections appear on the server side in either ESTABLISHED or SYN_REC state, but do not have a pid associated with them. The reason is the kernel had already done the 3 way handshake, but it had not been accepted by the java (hence no pid).

  • The following test was run on Linux using a JSP page with Thread.sleep(10000) and JMeter

Test Case 1: [connections=38, maxThreads=10 acceptCount=20]
Server:
11 connections ESTABLISHED state pid: jboss-pid
21 connections ESTABLISHED state pid: -
6 connections SYN_REC state pid: -

Client:
38 connections ESTABLISHED pid: jmeter-pid
rest in state SYN_SENT

  • of the 38 connections in ESTABLISHED state, the 6 which correspond to SYN_REC on server side have Send-Q=176

Test Case 2: [connections=29, maxThreads=10, acceptCount=10]
Server:
11 connections ESTABLISHED state pid: jboss-pid
11 connections ESTABLISHED state pid: -
7 connections SYN_REC state pid: -

Client:
29 connections ESTABLISHED pid: jmeter-pid
rest in state SYN_SENT

  • of the 29 connections in ESTABLISHED state, the 7 which correspond to SYN_REC on server side have Send-Q=176

Test Case 3: [connections=23, maxThreads=10, acceptCount=5]
Server:
11 connections ESTABLISHED state pid: jboss-pid
6 connections ESTABLISHED state pid: -
6 connections SYN_REC state pid: -

Client:
23 connections ESTABLISHED pid: jmeter-pid
rest in state SYN_SENT

  • of the 23 connections in ESTABLISHED state, the 6 which correspond to SYN_REC on server side have Send-Q=176

The tests do confirm that acceptCount is taken into account. The sockets in states ESTABLISHED and SYN_REC, on the server side without any pid, are those in the backlog.

The following will also be useful in a support ticket:

Linux

  • tcpdump from the server side while running the tests
  • netstat output on both client and server side at 10 second interval (can make this lower depending on the extended delay on the test page).
    • include a timestamp each time netstat output is captured
    • netstat command to run netstat -lantp

Solaris

  • snoop output from server side when running the tests
  • lsof output on both client and server side at 10 second interval (can make this lower depending on the extended delay on the test page).
    • include a timestamp each time lsof output is captured
    • lsof command to run netstat lsof -iTCP -n

** N.B. Solaris netstat output doesn't seem to print pid that's why lsof is being used here. You can also use lsof on Linux instead of netstat

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.