How do I configure the StuckThreadDetectionHandler in EAP 7/8?
Environment
Red Hat JBoss Enterprise Application Platform
- 7.x
- 8.x
Issue
- How do I configure the StuckThreadDetectionHandler in EAP 7?
- How do I configure the StuckThreadDetectionHandler in EAP 8?
- Is there any alert or notification indicating when request processing exceeds a certain amount of time?
- Is there an
MBeanfor the overall JVM health? E.g. if there are stuck threads the health of theJVMwill be in WARNING state.
Resolution
-
The
StuckThreadDetectionHandlercan be used for logging notifications if request processing exceeds a specified amount of time. This does not expose any status or information via JMX MBean, but it will produce log messages with stacktraces indicating any stuck thread concerns.
If the configured threshold is exceeded, theStuckThreadDetectionHandlerlogs a message like the following:Thread <thread-name> has been active for <duration> milliseconds (since <time>) to serve the same request for <context-path> and may be stuck (configured threshold for this StuckThreadDetectionValve is <configured-number-of-seconds> seconds). There is/are <total-umber-of-detected-stuck-threads> thread(s) in total that are monitored by this Valve and may be stuck.: java.lang.Throwable ...(stacktrace)...For example, if you have configured a 10 second threshold for
StuckThreadDetectionHandlerand you have a slow application (calling Thread.sleep() in JSP for this example):WARN [io.undertow.request] (default I/O-20) UT005072: Thread default task-3 (id=250) has been active for 10013 milliseconds (since Thu Mar 16 14:14:35 JST 2023) to serve the same request for /test/sleep.jspsleep=15000 and may be stuck (configured threshold for this StuckThreadDetectionValve is 10 seconds). There is/are 1 thread(s) in total that are monitored by this Valve and may be stuck.: java.lang.Throwable at java.lang.Thread.sleep(Native Method) at org.apache.jsp.sleep_jsp._jspService(sleep_jsp.java:103) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:433) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:403) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:347) at javax.servlet.http.HttpServlet.service(HttpServlet.java:590) at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) at io.undertow.jsp.JspFileHandler.handleRequest(JspFileHandler.java:32) at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68) at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) at org.wildfly.extension.undertow.security.SecurityContextAssociationHandler.handleRequest(SecurityContextAssociationHandler.java:78) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68) at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117) at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) at io.undertow.security.handlers.NotificationReceiverHandler.handleRequest(NotificationReceiverHandler.java:50) at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.security.jacc.JACCContextIdHandler.handleRequest(JACCContextIdHandler.java:61) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at org.wildfly.extension.undertow.deployment.GlobalRequestControllerHandler.handleRequest(GlobalRequestControllerHandler.java:68) at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52) at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:269) ...(snip).. -
To enable
StuckThreadDetectionHandlerglobally, you can add<expression-filter name="stuck-detector" expression="blocking; stuck-thread-detector(#ofSeconds)"/>with<filter-ref name="stuck-detector" />in your undertow subsystem. The#ofSecondsspecifies the threshold of theStuckThreadDetectionHandler(stuck-thread-detector) will wait for a thread to complete before issuing a warning. Note that the time unit that can be specified forstuck-thread-detectoris seconds.
For example, if you want to configureStuckThreadDetectionHandler(stuck-thread-detector) globally with a 10 second threshold, you can use the following JBoss-CLI command:/subsystem=undertow/configuration=filter/expression-filter=stuck:add(expression="blocking; stuck-thread-detector(10)") /subsystem=undertow/server=default-server/host=default-host/filter-ref=stuck:addthen, the above CLI will add the configuration to your undertow subsystem like the following:
<host name="default-host" alias="localhost"> <location name="/" handler="welcome-content"/> ...(snip)... <filter-ref name="stuck-detector" /> <!-- added --> </host> ...(snip)... <filters> ...(snip)... <expression-filter name="stuck-detector" expression="blocking; stuck-thread-detector(10)"/> <!-- added --> </filters>Note: the
blockinghandler is needed in the expression-filter as well so that theStuckThreadDetectionHandleris invoked in the task thread that processes the request. Without theblockinghandler, theStuckThreadDetectionHandleris invoked only in the I/O thread and is not possible to track any stuck threads in application request processing that run on task threads -
To enable
StuckThreadDetectionHandler(stuck-thread-detector) for the specific application, you can enable it withWEB-INF/undertow-handlers.confin your application.-
On EAP 7.0.x, if you want to configure
StuckThreadDetectionHandler(stuck-thread-detector) with a 10 second threshold, the content ofundertow-handlers.confwould be like the following. Again, theblockinghandler is required to invokeStuckThreadDetectionHandlerin the task thread during application request processing:blocking stuck-thread-detector(10) -
With EAP 7.1+ that contains This content is not included.WFLY-6667/This content is not included.JBEAP-4846, you can enable
stuck-thread-detectorwithout having theblockinghandler and still have theStuckThreadDetectionHandlerinvoked in the task thread. So, if you want to configureStuckThreadDetectionHandlerwith a 10 second threshold, the content ofundertow-handlers.confwould be like the following:stuck-thread-detector(10)
-
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.