EAP 7 has growing amounts of pipe and eventpoll file handles between full GC cycles

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 7.2.x
  • 7.3.x (up to 7.3.2)

Issue

• We see a growing number of open file handles from pipes and eventpolls on EAP 7:

java    16990 jbossadmin  901u  a_inode               0,10         0        6123 [eventpoll]
java    16990 jbossadmin  879w     FIFO                0,9       0t0   392721389 pipe
java    16990 jbossadmin  489r     FIFO                0,9       0t0   392721389 pipe

• It looks like these decrease after a full GC cycle.

Resolution

This has been reported as CVE-2020-14340. pipe and eventpoll file handles that are linked to Java NIO Selectors will be closed immediately at the task thread exit in JBoss EAP 7.3.3 or later. Update to JBoss EAP 7.3.3 or later.

If you can not update immediately, either of the following can be used as mitigations:

• Force a GC if needed to release pollers awaiting finalization and their associated file descriptors

• Increase file limit as needed to sustain file descriptor churn between typical GC cycles

• To decrease file churn from temporary selectors, set the IO task thread to have task-core-threads equal to task-max-threads so threads and their associated thread local selectors are more effectively pooled and reused:

      <subsystem xmlns="urn:jboss:domain:io:3.0">
          <worker name="default" task-core-threads="64" task-max-threads="64" />
  

Root Cause

• Temporary selectors within XNIO are created and discarded but they are not fully released until GC has run to mark them for finalization.
• These selectors are cached as thread locals within task threads. By default, the core thread count is a fraction of the max thread count so any threads above the core thread count can be discarded if idle. When those threads are discarded their selectors are discarded along with them, but not fully released until GC and finalization. So with enough task threads being released and recreated in the pool, the discarded selectors can grow in count as well.

Diagnostic Steps

• Note that on Windows this may become evident with many localhost ports per maximum connections reached on Windows running several EAP 7 instances together
• Capture a heap dump (without triggering GC, for example with the -all=true option for jcmd GC.heap_dump) and output of lsof -p $JBOSS_PID from an issue time
• In Eclipse MAT, use the following OQL with the heap dump to compare nio poller file descriptors to lsof output (each poller would account for 3 file descriptors):

SELECT epfd, outgoingInterruptFD, incomingInterruptFD FROM sun.nio.ch.EPollArrayWrapper 

 epfd | outgoingInterruptFD | incomingInterruptFD
--------------------------------------------------
  901 |                 879 |                 489
--------------------------------------------------

java    16990 jbossadmin  901u  a_inode               0,10         0        6123 [eventpoll]
java    16990 jbossadmin  879w     FIFO                0,9       0t0   392721389 pipe
java    16990 jbossadmin  489r     FIFO                0,9       0t0   392721389 pipe

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.