Deadlock in ManagedConnectionFactory
Environment
- JBoss Enterprise Application Platform (EAP) 5.1.2 and earlier
Issue
- Our JBoss server hangs and becomes unresponsive. From the thread dump, the root cause appears to be threads deadlocked calling Properties.equals() in ManagedConnectionFactory code, for example:
"http-0.0.0.0-8080-1": at java.util.Hashtable.size(Hashtable.java:206) - waiting to lock <0x0000000779ebe9c0> (a java.util.Properties) at java.util.Hashtable.equals(Hashtable.java:742) - locked <0x0000000779ea2ff8> (a java.util.Properties) at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.equals(XAManagedConnectionFactory.java:524) at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory$SubjectActions.run(BaseWrapperManagedConnectionFactory.java:640) at java.security.AccessController.doPrivileged(Native Method) at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory$SubjectActions.addMatchingProperties(BaseWrapperManagedConnectionFactory.java:655) at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.getConnectionProperties(BaseWrapperManagedConnectionFactory.java:504) - locked <0x0000000779fccef0> (a org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory) at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.checkIdentity(BaseWrapperManagedConnection.java:522) at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnection.getConnection(BaseWrapperManagedConnection.java:283) at org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:505) at org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:941) at org.jboss.resource.adapter.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:89) ..."JCA PoolFiller":
at java.util.Hashtable.get(Hashtable.java:333)
- waiting to lock <0x0000000779ea2ff8> (a java.util.Properties)
at java.util.Hashtable.equals(Hashtable.java:755)
- locked <0x0000000779ebe9c0> (a java.util.Properties)
at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.equals(XAManagedConnectionFactory.java:524)
at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory$SubjectActions.run(BaseWrapperManagedConnectionFactory.java:640)
at java.security.AccessController.doPrivileged(Native Method)
at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory$SubjectActions.addMatchingProperties(BaseWrapperManagedConnectionFactory.java:655)
at org.jboss.resource.adapter.jdbc.BaseWrapperManagedConnectionFactory.getConnectionProperties(BaseWrapperManagedConnectionFactory.java:504)
- locked <0x0000000779ce3178> (a org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory)
at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.getXAManagedConnection(XAManagedConnectionFactory.java:443)
at org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory.createManagedConnection(XAManagedConnectionFactory.java:409)
- locked <0x0000000779ce3178> (a org.jboss.resource.adapter.jdbc.xa.XAManagedConnectionFactory)
at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.createConnectionEventListener(InternalManagedConnectionPool.java:639)
at org.jboss.resource.connectionmanager.InternalManagedConnectionPool.fillToMin(InternalManagedConnectionPool.java:547)
at org.jboss.resource.connectionmanager.PoolFiller.run(PoolFiller.java:74)
at java.lang.Thread.run(Thread.java:662
Resolution
- Upgrade to EAP 5.2.0
Root Cause
- These threads deadlock because they concurrently call Properties.equals() which has an inherrent deadlock bug
Diagnostic Steps
- Troubleshoot using thread dumps and additional steps mentioned in Java application unresponsive
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.