Java thread stack overflow caused by Oracle JDBC driver

Solution Verified - Updated

Environment

  • Oracle JDBC thin (Java) driver version 11.2.0.1.0
  • Oracle JDBC OCI driver 11.2
  • Oracle/Sun JDK 1.6.0_u13

Issue

  • JBoss without this data source starts up normally, however when it is added to the deploy directory  JBoss fails to start, with this error in the console output:
    11:52:05,720 INFO  [ConnectionProviderFactory] Initializing connection 
provider: 
org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider
./run.sh:
 line 210: 18206 Segmentation fault      "$JAVA" $JAVA_OPTS 
-Djava.endorsed.dirs="$JBOSS_ENDORSED_DIRS" -classpath 
"$JBOSS_CLASSPATH" org.jboss.Main "$@"
  • The following error in the Linux /var/log/messages file:
    kernel: java[17601]: segfault at 00000000498a2ca8 rip 00002aac2144459d rsp 00000000498a2c90 error 6
  • java.lang.StackOverflowError

  • After one node in the cluster crashes, the other nodes crash in quick succession.

  • JVM crash after switching to Oracle thin jdbc driver 
    #  SIGSEGV (0xb) at pc=0x063966c3, pid=11885, tid=638315408
# Problematic frame:
# V  [libjvm.so+0x3966c3]
#
Current thread (0x0a013c00):  JavaThread "main" [_thread_in_vm, id=11897, stack(0x2606e000,0x260bf000)]

    siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x00000000
    

    Stack: [0x2606e000,0x260bf000],  sp=0x260ba1c8,  free space=304k
    
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    
V  [libjvm.so+0x3966c3]
    
C  [libocijdbc11.so+0x4d67]  Java_oracle_jdbc_driver_T2CConnection_t2cSetSessionTimeZone+0x57
    
j  oracle.jdbc.driver.T2CConnection.t2cSetSessionTimeZone(JLjava/lang/String;)I+0
    
j  oracle.jdbc.driver.T2CConnection.logon()V+825
    
j  oracle.jdbc.driver.PhysicalConnection.<init>(Ljava/lang/String;Ljava/util/Properties;Loracle/jdbc/driver/OracleDriverExtension;)V+323
    
j  oracle.jdbc.driver.T2CConnection.<init>(Ljava/lang/String;Ljava/util/Properties;Loracle/jdbc/driver/OracleDriverExtension;)V+4
    
j  oracle.jdbc.driver.T2CDriverExtension.getConnection(Ljava/lang/String;Ljava/util/Properties;)Ljava/sql/Connection;+34
    
j  oracle.jdbc.driver.OracleDriver.connect(Ljava/lang/String;Ljava/util/Properties;)Ljava/sql/Connection;+296
    
j  org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(Ljavax/security/auth/Subject;Ljavax/resource/spi/ConnectionRequestInfo;)Ljavax/resource/spi/ManagedConnection;+113
    
.. ..
    

    Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
    
j  oracle.jdbc.driver.T2CConnection.t2cSetSessionTimeZone(JLjava/lang/String;)I+0
    
j  oracle.jdbc.driver.T2CConnection.logon()V+825
    
j  oracle.jdbc.driver.PhysicalConnection.<init>(Ljava/lang/String;Ljava/util/Properties;Loracle/jdbc/driver/OracleDriverExtension;)V+323
    
j  oracle.jdbc.driver.T2CConnection.<init>(Ljava/lang/String;Ljava/util/Properties;Loracle/jdbc/driver/OracleDriverExtension;)V+4
    
j  oracle.jdbc.driver.T2CDriverExtension.getConnection(Ljava/lang/String;Ljava/util/Properties;)Ljava/sql/Connection;+34
    
j  oracle.jdbc.driver.OracleDriver.connect(Ljava/lang/String;Ljava/util/Properties;)Ljava/sql/Connection;+296
    
j  org.jboss.resource.adapter.jdbc.local.LocalManagedConnectionFactory.createManagedConnection(Ljavax/security/auth/Subject;Ljavax/resource/spi/ConnectionRequestInfo;)Ljavax/resource/spi/ManagedConnection;+113

Resolution

Upgrade to the latest Oracle JDBC driver.

Root Cause

The thread stack size is being exceeded due to a bug in the Oracle JDBC thin driver.

Diagnostic Steps

  • Ask what Oracle JDBC driver version is being used.
  • Does the issue go away if the datasource is not deployed?
  • On Linux + Sun JDK the issue results in a JVM crash instead of java.lang.StackOverflowError.
  • For JVM crashes, verify a fatal error log was not created. One is not known to be created with this issue manifests as a JVM crash.
  • For JVM crashes, get  a core dump when the issue happens and analyze the backtrace and corresponding jstack output to see what the JVM was doing at the time of the crash. See Java application down due to JVM crash. Verify Oracle JDBC driver code is being executed when the crash happens: 
    Thread 14410: (state = IN_NATIVE)
 - java.net.SocketOutputStream.socketWrite0(java.io.FileDescriptor, byte[], int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.net.SocketOutputStream.socketWrite(byte[], int, int) @bci=44, line=92 (Compiled frame)
 - java.net.SocketOutputStream.write(byte[], int, int) @bci=4, line=136 (Compiled frame)
 - oracle.net.ns.DataPacket.send(int) @bci=144, line=199 (Compiled frame)
 - oracle.net.ns.NetOutputStream.flush() @bci=15, line=211 (Compiled frame)
 - oracle.net.ns.NetInputStream.getNextPacket() @bci=41, line=227 (Compiled frame)
 - oracle.net.ns.NetInputStream.read(byte[], int, int) @bci=33, line=175 (Compiled frame)
 - oracle.net.ns.NetInputStream.read(byte[]) @bci=5, line=100 (Compiled frame)
 - oracle.net.ns.NetInputStream.read() @bci=5, line=85 (Compiled frame)
 - oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket() @bci=11, line=122 (Compiled frame)
 - oracle.jdbc.driver.T4CSocketInputStreamWrapper.read() @bci=18, line=78 (Compiled frame)
 - oracle.jdbc.driver.T4CMAREngine.unmarshalUB1() @bci=11, line=1179 (Compiled frame)
 - oracle.jdbc.driver.T4CMAREngine.unmarshalSB1() @bci=1, line=1155 (Compiled frame)
 - oracle.jdbc.driver.T4CTTIfun.receive() @bci=11, line=279 (Compiled frame)
 - oracle.jdbc.driver.T4CTTIfun.doRPC() @bci=40, line=186 (Compiled frame)
 - oracle.jdbc.driver.T4C8Oall.doOALL(boolean, boolean, boolean, boolean, boolean, byte, int, byte[], int, oracle.jdbc.driver.Accessor[], int, oracle.jdbc.driver.Accessor[], int, byte[], char[], short[], int, oracle.jdbc.driver.DBConversion, byte[], java.io.InputStream[][], byte[][][], oracle.jdbc.oracore.OracleTypeADT[][], oracle.jdbc.driver.OracleStatement, byte[], char[], short[], oracle.jdbc.driver.T4CTTIoac[], int[], int[], int[], oracle.jdbc.driver.NTFDCNRegistration) @bci=728, line=521 (Compiled frame)
 - oracle.jdbc.driver.T4CPreparedStatement.doOall8(boolean, boolean, boolean, boolean, boolean) @bci=754, line=205 (Compiled frame)
 - oracle.jdbc.driver.T4CPreparedStatement.executeForDescribe() @bci=52, line=861 (Compiled frame)
 - oracle.jdbc.driver.OracleStatement.executeMaybeDescribe() @bci=104, line=1145 (Interpreted frame)
 - oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout() @bci=147, line=1267 (Interpreted frame)
 - oracle.jdbc.driver.OraclePreparedStatement.executeInternal() @bci=106, line=3449 (Compiled frame)
 - oracle.jdbc.driver.OraclePreparedStatement.executeQuery() @bci=41, line=3493 (Compiled frame)
 - oracle.jdbc.driver.OraclePreparedStatementWrapper.executeQuery() @bci=4, line=1491 (Compiled frame)
 - org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeQuery() @bci=44, line=342 (Compiled frame)
...
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.