Connection pool running out of connections in JBoss EAP

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 4
    • 5
    • 6
    • 7

Issue

  • Unable to get a new connection from a JBoss EAP managed connection pool

  • In JBoss EAP 6 and later IJ000453 and IJ000655 messages such as the following appear together in the server.log

    ... javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:jboss/datasources/MyDS
...
... javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (30000 [ms])

  • In JBoss EAP 5 and earlier, messages such as the following appear in the server.log

    ... No ManagedConnections available within configured blocking timeout ( 30000 [ms] ]

  • Server hangs with many threads blocking with stack traces like the following:

    "ajp-127.0.0.1-8009-1" daemon prio=10 tid=0x00002aaabde44000 nid=0x106c waiting on condition [0x000000004d2b7000]
   java.lang.Thread.State: TIMED_WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x000000078f4a3140> (a java.util.concurrent.Semaphore$FairSync)
	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:196)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1011)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1303)
	at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:383)
	at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.getConnection(InternalManagedConnectionPool.java:193)
	at org.jboss.resource.connectionmanager.JBossManagedConnectionPool$BasePool.getConnection(JBossManagedConnectionPool.java:747)
	at org.jboss.resource.connectionmanager.BaseConnectionManager2.getManagedConnection(BaseConnectionManager2.java:404)
	at org.jboss.resource.connectionmanager.TxConnectionManager.getManagedConnection(TxConnectionManager.java:424)
	at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:496)
	at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941)
	at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89)
	...

Resolution

For EAP 6/7 review the datasource subsystem in the standalone*.xml or domain*.xml or for EAP 4/5 review the *-ds.xml:

  • Increase the value for max-pool-size for the connection pool (the default is 20)
  • See the diagnostics section below for details on checking for connection usage and leak issues.
  • In rare cases, it may be sufficient to increase the value for blocking-timeout-millis (default is 30000 milliseconds or 30 seconds).
    • The blocking-timeout-millis element within the datasource timeout settings (see docs/schema/jboss-as-datasources_1_*.xsd) indicates the maximum time in milliseconds that a thread will block while waiting for a connection before throwing an exception.
    • Note that this setting only impacts the block while waiting for an existing connection, and will not impact the block when creating a new connection takes an inordinately long time.
    • This is not usually feasible to resolve IJ000655 errors encountered by end users but may be suitable to address errors encountered during batch processing

Root Cause

  • The IJ000453 exception (in this case) results from the IJ000655 exception
  • The IJ000655: No managed connections available within configured blocking timeout exception (or the EAP 5 and earlier equivalent) is usually seen due to 1 of 3 reasons:
  1. The datasource connection pool has not been tuned (e.g. max-pool-size and blocking-timeout-millis) correctly for the maximum load on the application.
  2. The application is leaking connections because it is not closing them and thereby returning them to the pool.
  3. Threads with connections to the database are hanging and holding on to the connections or slow performance (e.g. due to CPU or memory usage issues) may delay (otherwise rapid) return of connections by worker threads and lead to pool exhaustion.

Diagnostic Steps

  • For EAP 6/7, see this article for an overview of troubleshooting connection issues.
  • For EAP 4/5
    • The MaxConnectionsInUse attribute of the ManagedConnectionPool MBean (in the jmx-concole), for the datasource in question, suggests connection pool size requirements for the application (if this number is close to the configured max-pool-size, the max-pool-size may need to be increased)
    • See How to check which application is using JDBC connections in JBoss EAP.
  • Analyze the JBoss server log in the Access Labs app Log Reaper to verify the issue is occurring and at what frequency.
  • Generate a series of thread dumps, spaced 30 to 60 seconds apart, and check if there are long running treads that may be hanging on to connections.
  • Evaluate system performance (Garbage Collection activity, CPU usage, system memory usage, etc.)
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.