JVM crash triggered by libtcnative

Solution Verified - Updated 16 Jun 2025

---

Environment

• JBoss Enterprise Application Platform (EAP) 5
• JBoss Enterprise Web Server (EWS)
  • Tomcat
• JBoss Native

Issue

* Our application is the google implementation for the Comet protocol. This is just a technique for allowing a web server to push data to a browser (also known as Ajax Push). JVM crashed on startup with the following when using tcnative

    >Stack: [0x000000004d485000,0x000000004d586000],  sp=0x000000004d582200,  free space=3f40000000000000018k
    Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
    C  [libtcnative-1.so+0x1459a]  Java_org_apache_tomcat_jni_Socket_sendbb+0x5a
    
    Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
    j  org.apache.tomcat.jni.Socket.sendbb(JII)I+0
    j  org.apache.coyote.ajp.AjpAprProcessor.flush()V+22
    j  org.apache.coyote.ajp.AjpAprProcessor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+66
    j  org.apache.coyote.Response.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+31
    j  org.apache.catalina.connector.OutputBuffer.doFlush(Z)V+94
    j  org.apache.catalina.connector.OutputBuffer.flush()V+2
    j  org.apache.catalina.connector.CoyoteOutputStream.flush()V+4
    j  net.myapp.gwt.comet.server.deflate.DeflaterOutputStream.flush()V+15
    j  net.myapp.gwt.comet.server.impl.CountOutputStream.flush()V+11
    j  sun.nio.cs.StreamEncoder.implFlush()V+15
    j  sun.nio.cs.StreamEncoder.flush()V+12
    j  java.io.OutputStreamWriter.flush()V+4
    j  net.myapp.gwt.comet.server.impl.CometServletResponseImpl.flush()V+25
    j  net.myapp.gwt.comet.server.impl.CometServletResponseImpl.terminate()V+12
    j  net.myapp.gwt.comet.server.impl.CometServletResponseImpl.tryTerminate()V+1
    j  net.myapp.gwt.comet.server.impl.CometServletResponseImpl.initiate()V+143
    j  net.myapp.gwt.comet.server.impl.IEHTMLFileCometServletResponse.initiate()V+1
    j  net.myapp.gwt.comet.server.CometServlet.doCometImpl(Lnet/zschech/gwt/comet/server/impl/CometServletResponseImpl;)V+1
    j  net.myapp.gwt.comet.server.CometServlet.doGet(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+90
    j  javax.servlet.http.HttpServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+35

• Tomcat native crashes destroying an APR connection when using java imaging:

Stack: [0x00007f5261f6d000,0x00007f526206e000],  sp=0x00007f526206c580,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libapr-1.so.0+0x188d0]  signed char+0x20

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.apache.tomcat.jni.Pool.destroy(J)V+0
j  org.apache.tomcat.util.net.AprEndpoint$Sendfile.add(Lorg/apache/tomcat/util/net/AprEndpoint$SendfileData;)Z+92
J  org.apache.coyote.http11.Http11AprProcessor.process(J)Lorg/apache/tomcat/util/net/AprEndpoint$Handler$SocketState;
j  org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(J)Lorg/apache/tomcat/util/net/AprEndpoint$Handler$SocketState;+39
j  org.apache.tomcat.util.net.AprEndpoint$Worker.run()V+155
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub

• Or

Stack: [0x00007f239c2a7000,0x00007f239c3a8000],  sp=0x00007f239c3a65d0,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libapr-1.so.0.3.9+0x188bd]  signed char+0xd

[error occurred during error reporting (printing native stack), id 0xb]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  org.apache.tomcat.jni.Socket.destroy(J)V+0
j  org.apache.tomcat.util.net.AprEndpoint$Worker.run()V+128
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub

• Tomcat native crashes setting timeouts on a socket when used in conjunection with AWT:

Stack: [0x00007fd9ff9fa000,0x00007fd9ffafb000],  sp=0x00007fd9ffaf9750,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [libtcnative-1.so+0x12a5b]  Java_org_apache_tomcat_jni_Socket_timeoutSet+0x2b
j  org.apache.tomcat.util.net.AprEndpoint.setSocketOptions(J)Z+63
j  org.apache.tomcat.util.net.AprEndpoint$Worker.run()V+153
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub
V  [libjvm.so+0x4e14a0]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x1e0
V  [libjvm.so+0x711a39]  os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x19
V  [libjvm.so+0x4e0de6]  JavaCalls::call_virtual(JavaValue*, KlassHandle, symbolHandle, symbolHandle, JavaCallArguments*, Thread*)+0x126
V  [libjvm.so+0x4e0e97]  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, symbolHandle, symbolHandle, Thread*)+0x47
V  [libjvm.so+0x574c00]  thread_entry(JavaThread*, Thread*)+0xa0
V  [libjvm.so+0x81c491]  JavaThread::run()+0x121
V  [libjvm.so+0x712eef]  java_start(Thread*)+0x13f

Resolution

• If you have a need for javax.imageio.stream, then you shouldn't use tcnative, but rather the Java connectors. Disabling APR connectors in JBoss EWS Tomcat

  • Setting the org.apache.catalina.connector.RECYCLE_FACADES system property to true can be used as a potential workaround to a crash triggered by imageio finalizers. With org.apache.catalina.connector.RECYCLE_FACADES set to true, JBossWeb/Tomcat will automatically clear the CoyoteOutputStream itself upon request completion and dereference the APR OutputBuffer from the CoyoteOutputStream. This should prevent imageio from hitting the crash when it is finalized because the call will not reach the APR socket for misuse.

• If there is a need for the Comet protocol, use the Comet processor that ships  Tomcat/JBossWeb

• Consider changing connectors in your server.xml to use another protocol instead of apr (e.g. protocol="org.apache.coyote.http11.Http11NioProtocol).

Root Cause

tcnative is not supported with those streams because they "steal" the stream descriptors from servlet io streams.

The problem is in any kind of stream that detaches the Socket from the Servlet which is

1. Violation of Servlet spec
2. Crashes the Tomcat Native

When using third party solution to read/write directly to the client connection, which is a violation of the spec, Tomcat (APR connector) cannot inform that third party that it closed the socket. One would need to modify the servlet spec for that.
The third party will be unaware that the connection was closed, and write to the invalidated socket causing a JVM core.

For example, Spring async multithreaded request handling is not safe with the APR connector.

Diagnostic Steps

• Check hs_err_pid files for a "Java2D Disposer" thread or awt/java image libraries loaded

SBR

• JVM & Diagnostics
• Webservers

Product(s)

• Red Hat JBoss Enterprise Application Platform
• Red Hat JBoss Web Server

Components

• jbossas
• tomcat5
• tomcat6

Category

• Troubleshoot

Tags

• active_directory
• jboss
• jvm
• tomcat

---

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.