NIO EventPoller thread dies from uncaught NPE, leading to unresponsiveness

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP) 6

Issue

  • We use an NIO connector and it became unresponsive. The acceptor thread was refusing connections with errors like the following:
INFO [org.apache.tomcat.util] (http-/0:0:0:0:0:0:0:0:38092-Acceptor) JBWEB003022: Channel processing failed
  • There were limited sockets established, but still in a heap dump, it looks like the NioEndpoint has exhausted connections:
    NioEndpoint.maxConnection = 32768
    NioEndpoint.counter = 32768
    NioEndpoint.connections hold 32768 instances of NioChannel
  • A thread dump also shows the EventPoller thread for this connector is missing

Resolution

  • Upgrade to EAP 6.2.4
  • Upgrade to EAP 6.3.0+

Root Cause

  • The EventPoller thread died and so it isn't properly cleaning up old nio channels, leading to the connector's exhaustion and unresponsiveness.
    • The EventPoller thread has been seen to die from an uncaught NPE. This results from a request thread finishing processing an event and recylcing a ChannelInfo that the EventPoller then tries use shortly after in an iteration of its maintain loop.

Diagnostic Steps

  • Capture a thread dump and check EventPoller thread states
  • Capture a heap dump and check NioEndpoint connection counts
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.