Deadlock during 'EntirePool' flush on MySQL connection error in EAP 6

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP) 6
  • MySQL1
1

Other datasource types may exhibit similar issues

Issue

A datasource pool is configured with <flush-strategy>EntirePool</flush-strategy>

  1. time=1: On thread-one, an in-use connection (JDBCMysqlXAConnection - which is locked within MySQL code) detects a failure and notifies its event listeners
  2. time=2: On thread-two, during connection validation, an invalid connection is detected and the thread synchronously locks and begins flushing the pool
  3. time=3: On thread-one, an XAManagedConnection instance, on connection error notification, synchronously attempts to flush the pool and blocks on the pool lock
  4. time=4: On thread-two, the in-progress pool flush attempts to close the JDBCMysqlXAConnection owned by thread-one and blocks on the connection lock

Resolution

  • Use the flush-strategy1 IdleConnections instead of EntirePool with MySQL
  • The IdleConnections strategy will flush only those connections which are not in-use by application components.
    • Other connections will be safely closed, one at a time, when subsequently accessed (e.g. during validation).
  • Future releases will include the Content from issues.jboss.org is not included.AllGracefully flush strategy which should address such issues2
1

See also Flush Strategies
2: This will not be available in the EAP 6 release line

Root Cause

  • MySQL JDBC4MysqlXAConnection code holds an internal connection lock when notifying listeners of a connection failure
  • JBoss org.jboss.jca.adapters.jdbc.xa.XAManagedConnection, on notification of a failure synchronously attempts to flush the pool
  • A second JBoss thread has detected an invalid connection (through explicit test) and has already begun to flush the pool
  • The in-progress flush blocks waiting for the lock held on the locked connection by a thread blocked waiting to flush the pool
  • The deadlock is between a MySQL connection lock (maintained by MySQL code when it calls error event listeners) and the JBoss pool lock held during pool flush()

Diagnostic Steps

Deadlocked Thread Stacks Example

"Thread-One"
  at org.jboss.jca.core.connectionmanager.pool.AbstractPool.flush(AbstractPool.java:315)
  - waiting to lock <0x0000000781621fd0> (a org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject)
  at org.jboss.jca.core.connectionmanager.listener.AbstractConnectionListener.connectionErrorOccurred(AbstractConnectionListener.java:367)
  at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.connectionErrorOccurred(TxConnectionListener.java:445)
  at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.broadcastConnectionError(BaseWrapperManagedConnection.java:647)
  at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.broadcastConnectionError(XAManagedConnection.java:199)
  at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection$1.connectionErrorOccurred(XAManagedConnection.java:87)
  at com.mysql.jdbc.jdbc2.optional.MysqlPooledConnection.callConnectionEventListeners(MysqlPooledConnection.java:248)
  - locked <0x000000079ff923e8> (a com.mysql.jdbc.jdbc2.optional.JDBC4MysqlXAConnection)
  at com.mysql.jdbc.jdbc2.optional.WrapperBase.checkAndFireConnectionError(WrapperBase.java:61)
  at com.mysql.jdbc.jdbc2.optional.PreparedStatementWrapper.executeQuery(PreparedStatementWrapper.java:858)
  at org.jboss.jca.adapters.jdbc.CachedPreparedStatement.executeQuery(CachedPreparedStatement.java:107)
  ...

"Thread-Two":
  at com.mysql.jdbc.jdbc2.optional.ConnectionWrapper.close(ConnectionWrapper.java:830)
  - waiting to lock <0x000000079ff923e8> (a com.mysql.jdbc.jdbc2.optional.JDBC4MysqlXAConnection)
  at com.mysql.jdbc.jdbc2.optional.ConnectionWrapper.close(ConnectionWrapper.java:466)
  at com.mysql.jdbc.jdbc2.optional.JDBC4ConnectionWrapper.close(JDBC4ConnectionWrapper.java:72)
  at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.destroy(BaseWrapperManagedConnection.java:494)
  at org.jboss.jca.adapters.jdbc.xa.XAManagedConnection.destroy(XAManagedConnection.java:225)
  at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.doDestroy(SemaphoreArrayListManagedConnectionPool.java:819)
  at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.flush(SemaphoreArrayListManagedConnectionPool.java:568)
  at org.jboss.jca.core.connectionmanager.pool.AbstractPool.flush(AbstractPool.java:323)
  - locked <0x0000000781621fd0> (a org.jboss.jca.core.connectionmanager.pool.strategy.PoolBySubject)
  at org.jboss.jca.core.connectionmanager.listener.AbstractConnectionListener.connectionErrorOccurred(AbstractConnectionListener.java:367)
  at org.jboss.jca.core.connectionmanager.listener.TxConnectionListener.connectionErrorOccurred(TxConnectionListener.java:445)
  at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.broadcastConnectionError(BaseWrapperManagedConnection.java:647)
  at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnection.checkValid(BaseWrapperManagedConnection.java:522)
  at org.jboss.jca.adapters.jdbc.xa.XAManagedConnectionFactory.matchManagedConnections(XAManagedConnectionFactory.java:497)
  at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreArrayListManagedConnectionPool.getConnection(SemaphoreArrayListManagedConnectionPool.java:284)
  at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getTransactionNewConnection(AbstractPool.java:495)
  at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:374)
  at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:329)
  at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:368)
  at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:464)
  at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:139)
  ...
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.