How do I configure the StuckThreadDetectionHandler in EAP 7/8?

Solution Verified - Updated

Environment

Red Hat JBoss Enterprise Application Platform

  • 7.x
  • 8.x

Issue

  • How do I configure the StuckThreadDetectionHandler in EAP 7?
  • How do I configure the StuckThreadDetectionHandler in EAP 8?
  • Is there any alert or notification indicating when request processing exceeds a certain amount of time?
  • Is there an MBean for the overall JVM health? E.g. if there are stuck threads the health of the JVM will be in WARNING state.

Resolution

  • The StuckThreadDetectionHandler can be used for logging notifications if request processing exceeds a specified amount of time. This does not expose any status or information via JMX MBean, but it will produce log messages with stacktraces indicating any stuck thread concerns.
    If the configured threshold is exceeded, the StuckThreadDetectionHandler logs a message like the following:

    Thread <thread-name> has been active for <duration> milliseconds (since <time>) to serve the same request for <context-path> and may be stuck (configured threshold for this StuckThreadDetectionValve is <configured-number-of-seconds> seconds). There is/are <total-umber-of-detected-stuck-threads> thread(s) in total that are monitored by this Valve and may be stuck.: java.lang.Throwable
        ...(stacktrace)...
    

    For example, if you have configured a 10 second threshold for StuckThreadDetectionHandler and you have a slow application (calling Thread.sleep() in JSP for this example):

    WARN  [io.undertow.request] (default I/O-20) UT005072: Thread default task-3 (id=250) has been active for 10013 milliseconds (since Thu Mar 16 14:14:35 JST 2023) to serve the same request for /test/sleep.jspsleep=15000 and may be stuck (configured threshold for this StuckThreadDetectionValve is 10 seconds). There is/are 1 thread(s) in total that are monitored by this Valve and may be stuck.: java.lang.Throwable
          at java.lang.Thread.sleep(Native Method)
          at org.apache.jsp.sleep_jsp._jspService(sleep_jsp.java:103)
          at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
          at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433)
          at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:403)
          at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:347)
          at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
          at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
          at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
          at io.undertow.jsp.JspFileHandler.handleRequest(JspFileHandler.java:32)
          at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
          at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
          at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
          at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
          at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
          at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
          at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
          at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
          at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50)
          at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68)
          at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
          at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
          at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269)
          ...(snip)..
    
  • To enable StuckThreadDetectionHandler globally, you can add <expression-filter name="stuck-detector" expression="blocking; stuck-thread-detector(#ofSeconds)"/> with <filter-ref name="stuck-detector" /> in your undertow subsystem. The #ofSeconds specifies the threshold of the StuckThreadDetectionHandler (stuck-thread-detector) will wait for a thread to complete before issuing a warning. Note that the time unit that can be specified for stuck-thread-detector is seconds.
    For example, if you want to configure StuckThreadDetectionHandler (stuck-thread-detector) globally with a 10 second threshold, you can use the following JBoss-CLI command:

    /subsystem=undertow/configuration=filter/expression-filter=stuck:add(expression="blocking; stuck-thread-detector(10)")
    /subsystem=undertow/server=default-server/host=default-host/filter-ref=stuck:add
    

    then, the above CLI will add the configuration to your undertow subsystem like the following:

                    <host name="default-host" alias="localhost">
                        <location name="/" handler="welcome-content"/>
                        ...(snip)...
                        <filter-ref name="stuck-detector" /> <!-- added -->
                    </host>
    
                ...(snip)...
    
                <filters>
                    ...(snip)...
                    <expression-filter name="stuck-detector" expression="blocking; stuck-thread-detector(10)"/> <!-- added -->
                </filters>
    

    Note: the blocking handler is needed in the expression-filter as well so that the StuckThreadDetectionHandler is invoked in the task thread that processes the request. Without the blocking handler, the StuckThreadDetectionHandler is invoked only in the I/O thread and is not possible to track any stuck threads in application request processing that run on task threads

  • To enable StuckThreadDetectionHandler (stuck-thread-detector) for the specific application, you can enable it with WEB-INF/undertow-handlers.conf in your application.

    • On EAP 7.0.x, if you want to configure StuckThreadDetectionHandler (stuck-thread-detector) with a 10 second threshold, the content of undertow-handlers.conf would be like the following. Again, the blocking handler is required to invoke StuckThreadDetectionHandler in the task thread during application request processing:

      blocking
      stuck-thread-detector(10)
      
    • With EAP 7.1+ that contains This content is not included.WFLY-6667/This content is not included.JBEAP-4846, you can enable stuck-thread-detector without having the blocking handler and still have the StuckThreadDetectionHandler invoked in the task thread. So, if you want to configure StuckThreadDetectionHandler with a 10 second threshold, the content of undertow-handlers.conf would be like the following:

      stuck-thread-detector(10)
      
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.