How to check which application is using JDBC connections in JBoss EAP 4/5

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP)
    • 4.3
    • 5.x

Issue

  • Need some Info regarding DB connections. We have multiple apps sharing the same Jboss Instance, we would like to know how we can find out which application is using which Database in that particular instance, dynamically.

  • How to enable the cached connection manager to track connections that are not being closed by an application?

  • How to Enable the logging to show connection leaks

Resolution

There's no way to see what is using database connections at an application level.  However, connections can be tracked at a more granular level using the CachedConnectionManager and/or CachedConnectionValve.  Follow these steps:

  • Ensure the CachedConnectionValve is installed and CachedConnectionManager debugging is enabled in order to show where the connection leak is coming from. The CachedConnectionValve is installed by default in EAP.  CachedConnectionManager debugging depends on the server profile (enabled  in default and all, disabled in production).
  • Configuration details are as follows:

EAP 4.3 jbossjca-service.xml:

              <mbean code="org.jboss.resource.connectionmanager.CachedConnectionManager" 
                     name="jboss.jca:service=CachedConnectionManager">
                <depends optional-attribute-name="TransactionManagerServiceName">jboss:service=TransactionManager</depends>
            
                <!-- Enable connection close debug monitoring -->
                <attribute name="Debug">true</attribute>

EAP 5 jca-jboss-beans.xml:


                  <bean name="CachedConnectionManager" class="org.jboss.resource.connectionmanager.CachedConnectionManager">
                
                      <!-- Whether to track unclosed connections and close them -->
                     <property name="debug">true</property>

Debugging is enabled as follows:

  • EAP 4.x

    • CachedConnectionValve: JBOSS_HOME/server/$PROFILE/deploy/deploy/jboss-web.deployer/server.xml
    • CachedConnectionManager is configured in  JBOSS_HOME/server/$PROFILE/deploy/jbossjca-service.xml . 

  • EAP 5.x

    • CachedConnectionValve:  JBOSS_HOME/server/$PROFILE/deploy/deploy/jbossweb.sar/server.xml
    • CachedConnectionManager is configured in JBOSS_HOME/server/$PROFILE/deploy/jca-jboss-beans.xml.

  • Search server.log for the following:

          INFO [CachedConnectionManager] Closing a connection for you. Please close them yourself

  • Review the associated stack trace to find the code that's not releasing connections.

  • Identify the code which is leaking the connection.  By using the CachedConnectionManager from the JMX Console:

    1. Log into the JMX Console (e.g. http://jboss-host:8080/jmx-console).
    2. Scroll down to the "jboss.jca" section.
    3. Click the "service=CachedConnectionManager" link.
    4. Scroll down and invoke the "listInUseConnections" operation.
    5. A stack trace for each in-use connection will be printed. (Only if CachedConnectionManager is in debug mode. This will tell you what code called getConnection().)

  • Take subsequent readings from listInUseConnections() after a 10 second interval and figure out from the trace which application code is holding the connections for longer periods.  This will help in detecting the application code which may be leaking the connections.

  • Look at the database connections from the database side to see what SQL is running on the connections.  Make sure there is not any long running SQL or recursion bugs in the SQL that could be causing the connections to never be released.

  • For more verbose details on what is happening with the connections, you can enable TRACE log for the package org.jboss.resource.connectionmanager in conf/jboss-log4j.xml

    <category name="org.jboss.resource.connectionmanager">
      <priority value="TRACE"/>
   </category>

Please note that setting "Debug" to "true" as previously explained will impact performance (potentially considerably).  The performance penalty is the trade-off for connection tracking at this level.

Also consider just using a datasource per application and then using the JMX Console to look at the statistics of each of the datasources (via the associated ManagedConnectionPool MBean).  There would be no performance penalty for this.

Diagnostic Steps

The OQL query below can be used (in Eclipse MAT) to examine the pool population (verified in EAP 5.2)

SELECT clf.jndiName.toString() AS jndiName, maxUsedConnections AS maxUsed,
        checkedOut.map.size AS inUse, cls.size AS idle,
        (connectionCounter.created - connectionCounter.destroyed) AS open,
        connectionCounter.created AS created, connectionCounter.destroyed AS destroyed
FROM org.jboss.resource.connectionmanager.InternalManagedConnectionPool
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.