Connections are not closed after 'idle-timeout-minutes' in EAP

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 7
  • 6

Issue

• The datasource idle-timeout-minutes property has been set to a value > 0 (e.g. 15 minutes)
• The number of connections open (as reported by the database server) does not seem to decrease

Resolution

• Check for leaked connections using the CachedConnectionManager or Leak Dumper and correct any leaks identified
• For EAP 6.4.2 and later, the filo property may be used to alter the connection selection strategy

Root Cause

The following are known reasons why connections may not time out

• Connections reserved by application components are not subject to timeout by JBoss. Connections obtained by DataSource.getConnection() cannot be timed out by JBoss until after they have been returned to the pool (by calling Connection.close()) and remain unused in the pool for idle-timeout-minutes. Connection status is InUse between DataSource.getConnection() and Connection.close() even if no database command is active because these connections are owned by application code.
• Leaked connections will never be timed out.
  • This is an extended case of the above - connections which are retrieved by application components which call DataSource.getConnection() but which are never returned to the pool (by calling Connection.close()), can never be timed out by JBoss.
  • The use of the prefill property in older EAP 6 releases may lead to connection leaks.
  • The use of the use-strict-min property in older EAP 6 releases may leak to connection leaks.
• When use-strict-min is set to true, connections below the min-pool-size will not be timed out.
• The connection pool in JBoss EAP 6.1 - 6.4 uses a "least recently used" connection selection strategy for getConnection() requests
  • As long as the number of connection requests¹ within idle-timeout-minutes is greater than or equal to the number of connections active in the pool, no connections will time out.

See also How does idle-timeout-minutes work in a datasource?

Diagnostic Steps

• Monitor the JBoss EAP pool statistics over time

  • Determine if the number of "active" connections (counts both in-use connections reserved by application components and free connections in the pool) matches the number reported by the database server.
    • If the number of connections reported by the database server is lower than the number reported by JBoss EAP, this may suggest that there is a connection leak.
    • If the number of connections reported by the database server is higher than the number reported by JBoss EAP, this may also suggest a connection leak (such as the those related to use of the prefill or use-strict-min properties noted above).
  • If the number of "in-use" connections remains high (over time) and the number does not match expected level of activity within the system, this may suggest that there a connection leak.

• Enable pool trace to capture details about the activity of the IdleRemover thread

• If it can be verified (e.g. using pool trace) that there are connection requests (e.g. periodically during the idle-timeout-minutes period) but the origin of/reason for the requests is not known, a stack trace can be generated for getConnection() requests using a Byteman rule like the below:

  RULE java.sql.DataSource.getConnection
  INTERFACE java.sql.DataSource
  METHOD getConnection
  IF true
  # Optionally, a command like the below will limit the stack frames to be printed (reducing verbosity in logging)
  # DO traceStack("[BYTEMAN] ", 15);
  DO traceStack("[BYTEMAN] ");
  ENDRULE
  

  • A rule such as the above can be deployed dynamically (e.g. during an idle period when activity is occurring infrequently) - see the bmsubmit command used to deploy rules. Byteman rules may serve as a more targeted (e.g. to specific operations) and flexible alternative to pool trace facilities in some cases.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.