Performance of Oracle datasource validation degrades in EAP 7.3.10 or 7.4.2+

Environment

• JBoss Enterprise Application Platform (EAP)
  • 7.3.10
  • 7.4.2+

Issue

• We saw a performance degradation after updating to EAP 7.4.2+ when using connection validation with Oracle datasources. If validation is disabled, the impact is not seen. But with validation, thread dumps show validation attempts stalled in new thread creations and many generic threads pending in TERMINATED status:

"default task-7" #1783 prio=5 os_prio=0 cpu=22397.71ms elapsed=1025.12s tid=0x0000562e01ea7800 nid=0x555e runnable  [0x00007f6790e35000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.Thread.start0(java.base@11.0.17/Native Method)
	at java.lang.Thread.start(java.base@11.0.17/Thread.java:798)
	- locked <0x00000006828cb6f8> (a java.lang.Thread)
	at oracle.jdbc.driver.PhysicalConnection.pingDatabase(PhysicalConnection.java:5746)
	at oracle.jdbc.driver.PhysicalConnection.isValid(PhysicalConnection.java:10285)
	at oracle.jdbc.driver.PhysicalConnection.isValid(PhysicalConnection.java:10257)
	at org.jboss.jca.adapters.jdbc.extensions.novendor.JDBC4ValidConnectionChecker.isValidConnection(JDBC4ValidConnectionChecker.java:58)
	at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.isValidConnection(BaseWrapperManagedConnectionFactory.java:1372)
	at org.jboss.jca.adapters.jdbc.BaseWrapperManagedConnectionFactory.getInvalidConnections(BaseWrapperManagedConnectionFactory.java:1179)
	at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:382)
	at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:640)
	at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:605)
	at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:624)
	at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:440)
	at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789)
	at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:151)
...
"Thread-2084840" #2086761 daemon prio=5 os_prio=0 cpu=0.16ms elapsed=0.11s tid=0x0000562df58e4800 nid=0x43ff runnable  [0x0000000000000000]
   java.lang.Thread.State: TERMINATED

"Thread-2085081" #2087001 daemon prio=5 os_prio=0 cpu=0.20ms elapsed=0.11s tid=0x0000562e05502000 nid=0x4401 runnable  [0x0000000000000000]
   java.lang.Thread.State: TERMINATED

"Thread-2084833" #2086754 prio=5 os_prio=0 cpu=0.20ms elapsed=0.10s tid=0x0000562e027de800 nid=0x4404 runnable  [0x0000000000000000]
   java.lang.Thread.State: TERMINATED

"Thread-2084830" #2086751 prio=5 os_prio=0 cpu=0.21ms elapsed=0.10s tid=0x0000562e0548f000 nid=0x4407 runnable  [0x0000000000000000]
   java.lang.Thread.State: TERMINATED

Resolution

• Change your datasource to use org.jboss.jca.adapters.jdbc.extensions.oracle.OracleLegacyValidConnectionChecker for its valid-connection-checker class. This provides the same implementation as 7.4.1 to perform the ping without a timeout so this can be a suitable solution to have the same validation behavior without the timeout or created thread overhead from the Oracle driver validation.
• Another means to avoid the degradation impact directly on response times would be to switch from on match validation to only background validation instead.
• Or if using a java agent, consider disabling the java agent to improve speed of thread creation and destruction.

Root Cause

• With EAP 7.3.10/7.4.2+, OracleValidConnectionChecker was improved to use Oracle's connection isValid method with a default 5 second ping timeout. Oracle driver's handle this timeout with an additionally forked temporary timeout thread so any slowness in thread creation can then cause notable degradation in performance with heavy activity checking out and validating connections.
  • This content is not included.JBJCA-1421
• A java agent in particular can cause notable slowness in thread creation and destruction like this.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.