Tuning the Undertow Thread Pool in JBoss EAP 7.x and 8.x

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 7.x
    • 8.x

Issue

  • How to tune Undertow thread pool in JBoss EAP 7
  • I'm curious how many connector threads we have for https by default?
  • How do I configure the max servlet threads
  • How does request queuing work in JBoss EAP 7 when under heavy load

Resolution

  • To set the value for task-max-threads in JBoss CLI, for example 128

    Standalone Mode

      [standalone@localhost:9990 /] /subsystem=io/worker=default/:write-attribute(name=task-max-threads,value=128)
  {
      "outcome" => "success",
      "response-headers" => {
          "operation-requires-reload" => true,
          "process-state" => "reload-required"
      }
  }
      
  [standalone@localhost:9990 /] :reload
  {
      "outcome" => "success",
      "result" => undefined
  }

    Domain Mode

    you need to specify /profile=<your_profile_name>/ to suite your environment. The following is for example with default profile.

      [domain@localhost:9990 /] /profile=default/subsystem=io/worker=default/:write-attribute(name=task-max-threads,value=128)
  {
      "outcome" => "success",
      "result" => undefined,
      "server-groups" => undefined
  }

    JBoss EAP for OpenShift

    It can be configured by mounting a custom script in ConfigMap. The custom script function is available in jboss-eap-7/eap72-openshift:1.2 or later.

      $ cat extensions/postconfigure.sh
  #!/bin/sh
  echo "Executing postconfigure.sh"
  $JBOSS_HOME/bin/jboss-cli.sh --file=$JBOSS_HOME/extensions/extensions.cli
   
  $ cat extensions/extensions.cli
  embed-server --std-out=echo --server-config=standalone-openshift.xml
  batch
  /subsystem=io/worker=default:write-attribute(name=task-max-threads,value=10)
  run-batch
  quit
   
  $ oc create configmap jboss-cli --from-file=postconfigure.sh=extensions/postconfigure.sh --from-file=extensions.cli=extensions/extensions.cli
  $ oc set volume dc/eap-app --add --name=jboss-cli -m /opt/eap/extensions -t configmap --configmap-name=jboss-cli --default-mode='0755'

  • To set it directly in standalone.xml or domain xml, add the task-max-threads parameter to the io subsystem's default worker, for example:

      <subsystem xmlns="urn:jboss:domain:io:2.0">
      <worker name="default" task-max-threads="128"/>
      <buffer-pool name="default"/>
  </subsystem>

  • To avoid performance issues related to thread pool exhaustion task-max-threads should always be greater than the expected maximum concurrent request count. Note that certain application designs (like using asynchronous servlets or invoking EJBs residing on the same container) may lead to consuming more than one task thread per request so in these cases task-max-threads need to be adjusted accordingly.

Note The task thread pool uses an unbounded queue which cannot be adjusted. If limiting request queuing is required, the request limit filter can be configured with a desired queue size as per How to configure max concurrent requests limit for specific application in JBoss EAP 7.

Note: There is a known issue per I/O threads experience increased CPU usage for https-listener when task worker thread pool is exhausted in JBoss EAP 7 allowing for increased CPU if all task threads are in use. As written in the knowledge, you can try either of the following as a workaround:

Root Cause

  • JBoss EAP 7 uses Undertow as the default web container, In Undertow by default all listener will use the default worker which is provided by the IO subsystem. This worker instance manages the listeners (AJP/HTTP/HTTPS) IO threads.

  • The IO Threads are responsible to handle incoming requests. The IO subsystem worker will provide the following options to tune it further.

      /subsystem=io/worker=default/:read-resource(recursive=false)
  {
      "outcome" => "success",
      "result" => {
          "io-threads" => undefined,
          "stack-size" => 0L,
          "task-keepalive" => 60,
          "task-max-threads" => undefined
      }
  }

    Where,

    • io-threads : Specify the number of I/O threads to create for the worker. If not specified, a default will be chosen, which is calculated by cpuCount * 2

    • stack-size : The stack size (in bytes) to attempt to use for worker threads.

    • task-keepalive : Specify the number of milliseconds to keep non-core task threads alive. However, this setting does not become effective because there's no setting for core size of task threads and core size is always same to max size.

    • task-max-threads : Specify the maximum number of threads for the worker task thread pool. If not specified, default value used which is calculated by formula cpuCount * 16

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.