EAP 7 shutdown stalls with in process requests

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP) 7.x

Issue

  • JBoss shutdown is not completing and the only activity shown in thread dumps is pending request processing like the following:
"default task-23" #174 prio=5 os_prio=0 tid=0x00002b39c0006800 nid=0x5914 runnable [0x00002b394aa7b000]
   java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000006d240bfe0> (a sun.nio.ch.Util$3)
	- locked <0x00000006d240bfd0> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000006d240bf88> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
	at org.xnio.nio.SelectorUtils.await(SelectorUtils.java:51)
	at org.xnio.nio.NioSocketConduit.awaitWritable(NioSocketConduit.java:233)
	at io.undertow.protocols.ssl.SslConduit.awaitWritable(SslConduit.java:459)
	at org.xnio.conduits.AbstractSinkConduit.awaitWritable(AbstractSinkConduit.java:66)
	at org.xnio.conduits.AbstractSinkConduit.awaitWritable(AbstractSinkConduit.java:66)
	at io.undertow.conduits.ChunkedStreamSinkConduit.awaitWritable(ChunkedStreamSinkConduit.java:361)
	at org.xnio.conduits.ConduitStreamSinkChannel.awaitWritable(ConduitStreamSinkChannel.java:134)
	at io.undertow.channels.DetachableStreamSinkChannel.awaitWritable(DetachableStreamSinkChannel.java:87)
	at io.undertow.server.HttpServerExchange$WriteDispatchChannel.awaitWritable(HttpServerExchange.java:1973)
	at org.xnio.channels.Channels.flushBlocking(Channels.java:64)
	at io.undertow.servlet.spec.ServletOutputStreamImpl.close(ServletOutputStreamImpl.java:609)
	at io.undertow.servlet.spec.HttpServletResponseImpl.closeStreamAndWriter(HttpServletResponseImpl.java:478)
	at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:561)

Resolution

  • If seeing request processing persist in request reads or response timeouts, then set appropriate timeouts (in ms) on your undertow listeners. These timeouts are infinite by default, so capping them can limit any shutdown delay due to interactions with a slow client:
                <http-listener name="https" socket-binding="https" read-timeout="60000" write-timeout="60000"/>
  • If seeing request processing stalled elsewhere, then investigate any stall concerns in other indicated code calls
  • Upgrade to EAP 7.3+. In EAP 7.3+, a normal shutdown such as SIGTERM or Cntl+C will complete without the completion of an in-flight request. This fix is due to This content is not included.WFCORE-1632.

Root Cause

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.