How can we monitor the number of active busy threads of task thread pool and and active connection count for XNIO/Undertow in JBoss EAP 7.x and 8.x?

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 7.x
    • 8.x

Issue

  • How can we monitor the number of active busy threads in XNIO/Undertow task thread pool in JBoss EAP 7?
  • How can we monitor the number of active busy threads of XNIO/Undertow web threads in JBoss EAP 7?
  • How can we monitor the number of active connection count handled by XNIO/Undertow in EAP 7?
  • I need to know the number of current request count in ajp/http listener in jboss EAP 7. Is there a way to know the actual ajp (or http) busy thread?

Resolution

  • It's available in EAP 7.0.9 and 7.1+. The runtime statistics can be obtained via JMX MBeans org.xnio:type=Xnio,provider="nio",worker="your-workername" ("your-workername" defaults to default which is defined in standalone-*.xml).

  • Some useful metrics in org.xnio MBean:

    • BusyWorkerThreadCount: This shows the number of busy threads in the task worker thread pool. This is available since EAP 7.1.0 and EAP 7.0.9 or later.
    • WorkerQueueSize: This shows how many counts await for free task worker thread in workQueue. This can be used for monitoring to know if the worker thread pool is full or not. When this is greater than 0, it means the task worker thread pool is full. For example, if it's 1, all task worker threads are busy, and 1 request is awaiting for available task thread.
    • ConnectionCount: This shows the number of active established TCP connections. This has been available since EAP 7.0.5 or later. As noted in the following root cause section, there's a known issue in this metric, which was reported as This content is not included.XNIO-297 and fixed in EAP 7.0.7 or later.

  • Since JBoss EAP 7.2.0+, the same metrics that can be checked via JBoss-CLI:

    • busy-task-thread-count: he same metrics as BusyWorkerThreadCount in org.xnio MBean

    • queue-size: the same metrics as WorkerQueueSize in org.xnio MBean

    • connection-count: the same metrics as ConnectionCount in org.xnio MBean

    • For example:

      • standalone mode:
              [standalone@localhost:9990 /] /subsystem=io/worker=default:read-resource(include-runtime=true)
        {
            "outcome" => "success",
            "result" => {
                "busy-task-thread-count" => 0,
                "core-pool-size" => 2,
                "io-thread-count" => 24,
                "io-threads" => "24",
                "max-pool-size" => 192,
                "queue-size" => 0,
                "shutdown-requested" => false,
                "stack-size" => "0",
                "task-core-threads" => "2",
                "task-keepalive" => "60000",
                "task-max-threads" => "192",
                "outbound-bind-address" => undefined,
                "server" => {
                    "/127.0.0.1:8080" => undefined,
                    "/127.0.0.1:8443" => undefined
                }
            }
        }

        [standalone@localhost:9990 /] /subsystem=io/worker=default/server=\/127.0.0.1\:8080:read-resource(include-runtime=true)
        {
            "outcome" => "success",
            "result" => {
                "connection-count" => 0,
                "connection-limit-high-water-mark" => 2147483647,
                "connection-limit-low-water-mark" => 2147483647
            }
        }
      • Domain mode:
        For Domain mode worker threads need to be monitored for each server instance separately. The below CLI command is executed for server-one in master host:
              [domain@localhost:9990 /] /host=master/server=server-one/subsystem=io/worker=default:read-resource(include-runtime=true)
        {
            "outcome" => "success",
            "result" => {
                "busy-task-thread-count" => 0,
                "core-pool-size" => 2,
                "io-thread-count" => 16,
                "io-threads" => "16",
                "max-pool-size" => 128,
                "queue-size" => 0,
                "shutdown-requested" => false,
                "stack-size" => "0",
                "task-core-threads" => "2",
                "task-keepalive" => "60000",
                "task-max-threads" => "128",
                "outbound-bind-address" => undefined,
                "server" => {"/127.0.0.1:8080" => undefined}
            }
        }

        [domain@localhost:9990 /] /host=master/server=server-one/subsystem=io/worker=default/server=\/127.0.0.1\:8080:read-resource(include-runtime=true)
        {
            "outcome" => "success",
            "result" => {
                "connection-count" => 0,
                "connection-limit-high-water-mark" => 2147483647,
                "connection-limit-low-water-mark" => 2147483647
            }
        }

  • Since JBoss EAP 7.3.0+, the same metrics that can be checked through /metrics for standalone mode. (Note: this method is not available for server running in domain mode) The metrics name starts with jboss_io_ like the following:

    $ curl -s http://127.0.0.1:9990/metrics | grep -e "^jboss_io_"
jboss_io_busy_task_thread_count{worker="default"} 0.0
jboss_io_connection_count{worker="default",server="/127.0.0.1:8080"} 0.0
jboss_io_connection_count{worker="default",server="/127.0.0.1:8443"} 0.0
jboss_io_core_pool_size{worker="default"} 2.0
jboss_io_io_thread_count{worker="default"} 48.0
jboss_io_max_pool_size{worker="default"} 384.0
jboss_io_queue_size{worker="default"} 0.0

Caveats: Note that the number of established TCP connections (ConnectionCount) and busy worker threads (BusyWorkerThreadCount) are not directly linked as Undertow/XNIO uses non-blocking I/O.

Root Cause

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.