High amounts of I/O dispatcher threads accumulated in java application

Environment

• Java
• Apache HTTP Client
  • nio HttpAsyncClient

Issue

• We see a large amount of accumulating I/O dispatcher threads like below until our JVM crashes:

"I/O dispatcher 9207"
  at sun.nio.ch.EPollArrayWrapper.epollWait(JIJI)I (Native Method)
  at sun.nio.ch.EPollArrayWrapper.poll(J)I (EPollArrayWrapper.java:269)
  at sun.nio.ch.EPollSelectorImpl.doSelect(J)I (EPollSelectorImpl.java:93)
  at sun.nio.ch.SelectorImpl.lockAndDoSelect(J)I (SelectorImpl.java:86)
  at sun.nio.ch.SelectorImpl.select(J)I (SelectorImpl.java:97)
  at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute()V (AbstractIOReactor.java:257)
  at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(Lorg/apache/http/nio/reactor/IOEventDispatch;)V (BaseIOReactor.java:106)
  at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run()V (AbstractMultiworkerIOReactor.java:590)
  at java.lang.Thread.run()V (Thread.java:748)

Resolution

• Ensure an application shuts down any HttpAsyncClient when done with. Cache and reuse any client objects as possible instead of perpetually recreating them.

Root Cause

• Content from issues.apache.org is not included.HTTPASYNC-29
• Application is not shutting down HttpAsyncClients when done with them, leaving their lingering threads.

Diagnostic Steps

• Capture a heap dump from a high point of threads and use the following OQL to check the number of HttpAsyncClients present in the heap and still active:

SELECT status.value.name.value.toString() FROM org.apache.http.impl.nio.client.InternalHttpAsyncClient 

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.