JVM hangs at the JCA PoolFiller Thread

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 6.x.x
• Oracle Driver

Issue

• JBoss instance is not stopping, hanged in JCA Pool filler thread.
• Periodic Recovery thread is not able to make new connection, trying to acquire lock held by JCA Pool filler thread.

Stack trace :

JCA PoolFiller" daemon prio=3 tid=0x014fd400 nid=0x81 runnable [0x28ebe000..0x28ebf970]
      java.lang.Thread.State: RUNNABLE      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
      - locked <0x3d87df68> (a java.net.SocksSocketImpl)
      at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
      at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
      at java.net.Socket.connect(Socket.java:519)
      at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:150)
      at oracle.net.nt.ConnOption.connect(ConnOption.java:133)
      at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:370)
      at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:434)
      at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:687) 
      at oracle.net.ns.NSProtocol.connect(NSProtocol.java:247)
      at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1102)
      at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:320)
      at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:546)
      at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:236)
      at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:32)
      at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:521)
      at oracle.jdbc.pool.OracleDataSource.getPhysicalConnection(OracleDataSource.java:280)
      at oracle.jdbc.xa.client.OracleXADataSource.getPooledConnection(OracleXADataSource.java:469)
      at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:156)
      at oracle.jdbc.xa.client.OracleXADataSource.getXAConnection(OracleXADataSource.java:130)
      at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.getXAManagedConnection(XAManagedConnectionFactory.java:443)
      at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java:397)
      - locked <0x3d401ca8> (a org.jboss.as.connector.subsystems.datasources.AbstractDataSourceService$MyXaMCF)
      at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.createConnectionEventListener(SemaphoreArrayListManagedConnectionPool.java:781)
      at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.fillToMin(SemaphoreArrayListManagedConnectionPool.java:725)           at org.jboss.jca.core.connectionmanager.pool.mcp.PoolFiller.run(PoolFiller.java:97)
      at java.lang.Thread.run(Thread.java:619)

Resolution

• Check the connectivity to the DB IP and port, try telnet to check connectivity.
• Number of retries set in DS URL, should not be set too high, that let the thread go in long loop.

Root Cause

• JCA Pool Filler thread, when initiated after the JBoss EAP started, is trying to fill the pool with minimum connections. So if there are threads blocked at JCA Pool filler, make sure you are able to connect to the DB IP and Port. Or also check the number of retries in DS URL are not to set too high.
• Threads stalled in T2CConnection.t2cCreateState

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.