Tomcat comes to a halt with threads waiting on GenericObjectPool

Solution Verified - Updated

Environment

  • JBoss Enterprise Web Server (EWS)
  • Tomcat 6
  • Oracle

Issue

Tomcat became unresponsive and thread dumps showed many threads waiting to lock a org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.  The waiting threads look like:

"ajp-8009-1" daemon prio=10 tid=0x0896e400 nid=0x380e waiting for monitor entry [0x612b9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
     at org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:781)
     - waiting to lock <0x75df1678> (a org.apache.tomcat.dbcp.pool.impl.GenericObjectPool)
     at org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
     at org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)

The thread holding the lock looks like:

"ajp-8009-2" daemon prio=10 tid=0x08dadc00 nid=0x3673 runnable [0x62bab000]
   java.lang.Thread.State: RUNNABLE
     at java.net.SocketInputStream.socketRead0(Native Method)
     at java.net.SocketInputStream.read(SocketInputStream.java:146)
     at oracle.net.ns.Packet.receive(Packet.java:283)
     at oracle.net.ns.DataPacket.receive(DataPacket.java:103)
     at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:230)
     at oracle.net.ns.NetInputStream.read(NetInputStream.java:175)
     at oracle.net.ns.NetInputStream.read(NetInputStream.java:100)
     at oracle.net.ns.NetInputStream.read(NetInputStream.java:85)
     at oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:123)
     at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:79)
     at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1122)
     at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1099)
     at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
     at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
     at oracle.jdbc.driver.T4C7Ocommoncall.doOLOGOFF(T4C7Ocommoncall.java:61)
     at oracle.jdbc.driver.T4CConnection.logoff(T4CConnection.java:498)
     - locked <0x7c45fc88> (a oracle.jdbc.driver.T4CConnection)
     at oracle.jdbc.driver.PhysicalConnection.close(PhysicalConnection.java:3931)
     - locked <0x7c45fc88> (a oracle.jdbc.driver.T4CConnection)
     at org.apache.tomcat.dbcp.dbcp.DelegatingConnection.close(DelegatingConnection.java:159)
     at org.apache.tomcat.dbcp.dbcp.PoolableConnection.reallyClose(PoolableConnection.java:95)
     at org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.destroyObject(PoolableConnectionFactory.java:301)
     at org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.addObjectToPool(GenericObjectPool.java:946)
     at org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.returnObject(GenericObjectPool.java:917)
     - locked <0x75df1678> (a org.apache.tomcat.dbcp.pool.impl.GenericObjectPool)
     at org.apache.tomcat.dbcp.dbcp.PoolableConnection.close(PoolableConnection.java:80)
     - locked <0x7c45fc00> (a org.apache.tomcat.dbcp.dbcp.PoolableConnection)
     at org.apache.tomcat.dbcp.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.close(PoolingDataSource.java:180)
  • Issue about connection pool configuration. We have sometime issues on our JBoss web server based on tomcat 6 and JVM 1.6.0_45.Particularly we noticed lost of connection with the db (Oracle 11.2.04) with subsequent hang of Tomcat (JbossWS).

Resolution

  • Check for firewall/network issues interfering with communication to Oracle, or Oracle db performance issues stalling responses.
  • Try upgading to the latest Oracle driver and engage Oracle for further assistance.
  • Note that dbcp is not a very high performance pooling implementation. As noted by the Content from people.apache.org is not included.following, the entire pool is locked when threads fetch from the pool or place connections in it, thus allowing for heavy thread blockage with any database connection creation/destruction/validation slowdown.

Root Cause

The thread holding the lock is trying to close out a connection from the  database connection pool.   To do so, it must lock the pool object.    But this thread is continuously stuck in oracle code, executing  SocketInputStream.socketRead0 and trying to close out the connection.    Thus the thread hangs on the lock and prevents any other threads from  using the GenericObjectPool as they need to, causing them all to hang as  well.  The most likely cause of this thread hanging in SocketInputStream.socketRead0 and failing to close this connection is communication issues with the Oracle db (network issues, firewalls, etc.).

Diagnostic Steps

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.