Java application unresponsive
Environment
- Java
- Red Hat JBoss Enterprise Application Platform (EAP)
- 8.x
- 7.x
- 6.x
- 5.x
- 4.x
- Red Hat JBoss Fuse
- 6.0
- Tomcat
- 6.0
- 5.5
Issue
- JBoss freezes and brings down our production environment.
- JBoss unresponsive.
- JBoss crashing.
- JBoss hangs.
- JBoss CPU usage 100%.
- GC thrashing.
- High load.
- High CPU.
- CPU 100% utilization.
- Slow performance.
- Serious performance issues with API calls
- Hung JBoss threads.
- JBoss goes to hung state.
- Server stuck.
- Load on Server
- Slow response from application.
- Throughout is quite low about 30-40MBps when application runs in
JBossdomain mode. - Application deployed on EAP become slow, need to restart the service and then application is running normal, But after a while it is run slowly again.
Resolution
- Rewrite application code to use less memory.
- Resolve delays upstream or in external systems.
- Remove instrumentation.
- Decrease memory pressure to avoid swapping.
- Ensure the
-XX:+TieredCompilationJVM option is not used - See Resolutions for the documents listed in the Root Cause section.
Root Cause
- Excessive garbage collection (GC). A use case in the application code requires a large amount of memory (e.g. many objects, a memory leak, etc.) and is pushing the JVM to its limits.
- Requests are delayed upstream.
- JBoss is waiting on an external system.
- Instrumentation overhead.
- System is swapping
- If a virtual environment, consider This content is not included.Unaccounted memory usage when running Red Hat Enterprise Linux as a VMware guest
- Known issues related to unresponsiveness:
- General
- Java garbage collection long pause times
- Java application high CPU
- JBoss slows after 24 hours for no apparent reason
- Use of symbolic links leads to large disk usage in JBoss EAP
- High CPU in "VM Thread" in Java application
- Server becomes unresponsive with many java tasks attempting to run at the same time on RHEL 4.
- "/dev/random" hangs in java program when server is inactive
- Deadlock in JBoss' StatefulSessionFilePersistenceManager when passivating stateful session beans
- Java application periodic high latency / processing times due to NUMA page reclaim on RHEL
- Java threads stalled in InetAddress.getLocalHost
- Java threads backed up in XPathContext initialization and DTMManager.newInstance
- DNS lookups slow in Java due to poor response to IPv6 AAAA records
- JBoss EAP 6 startup/shutdown doesn't complete
- JBoss hangs on startup in JNDI lookup
- HTTPS access to admin console in EAP6 is very slow when DNS is down
- Upgrade from RHEL 6.5 to 6.6 causes Java application to hang
- com.wily.introscope.agent.probe.net.ManagedDatagramSocket monitor contention
- JBoss becomes unresponsive with many running threads that have no stack
- Many threads are waiting on org.jboss.as.weld.ejb.SessionObjectReferenceImpl.getBusinessObject method
- EAP 6 EJB client deadlock
- JBoss is unresponsive with threads hanging in HttpURLConnection
- HTTPS connection creation bottlenecked by sun.security.util.MemoryCache
- LDAP security domain causes degraded performance on EAP 7
- Remote EJB client hang when load increases in EAP 7
- JVM hung from incomplete safepoint pause
- Picketbox Login module is called many times after upgrading from JBoss EAP 6 to JBoss EAP 7
- Java app contention on sun.net.www.http.KeepAliveCache lock
- EJB timer cancellations or removals face slowness when many EJB timers are present
- Deadlock during EJB client discovery leads to high thread stalls on org.jboss.remoting3.ConnectionInfo.getConnection
- Java application becomes slow in FileHandler.
- Logging
- Deadlock in Apache log4j
- Changing jboss-log4j.xml caused JBoss to go unresponsive
- Log4j and javax.mail are deadlocking
- JBoss becomes unresponsive with threads blocking on org.apache.log4j.spi.RootLogger
- JBoss becomes unresponsive with threads blocking in org.jboss.logmanager.handlers.WriterHandler
- Excess time spent in audit logging
- JBoss deadlocks due to log4j serialization
- JBoss deadlocks on PrintStream and Category
- JBoss stalls with many threads blocking in AsyncHandler
- JBoss deadlocks in LogManager
- JBoss EAP deadlocks on a java.util.logging.ConsoleHandler and java.io.PrintStream
- JBoss EAP deadlocks on a log4j RollingFileAppenders and java.io.PrintStream
- Log4j calls are slow in getStackTrace calls
- Transactions JCA & SQL
- JDBC connection pool running out of connections in JBoss EAP
- JBoss process hangs attempting to get a new connection from Oracle
- Java/JBoss hangs creating and authenticating connections to the database
- Deadlock in ManagedConnectionFactory
- JCA dead-lock in JBoss EAP
- JBoss hangs in database queries
- JVM hangs at the JCA PoolFiller Thread
- Threads stalled in T2CConnection.t2cCreateState
- Getting new database connections takes longer than expected in JBoss EAP 6.0/6.1
- Only single thread can create JDBC connection at once in JBoss EAP 5.2.0
- Logic performing several hundred queries takes significantly more time in Hibernate 4
- Performance of Oracle datasource validation degrades in EAP 7.4.2+
- EAP 7.4.9+ degrades performance with increased CPU in statement overhead
- Messaging
- JMS becomes unresponsive with threads waiting for MessagingPostOffice's ReaderLock
- JMS thread hangs on socketWrite during client message delivery
- JBoss deadlocks between MicroRemoteClientInvoker.establishLease and Client.notifyListeners
- JBoss Messaging stalls startup synchronizing on an aop AspectManager
- Message publisher stops when sending messages to HornetQ in JBoss EAP
- Remoting
- Classloading
- High CPU or many blocked threads in BaseClassLoader.getResourcesLocally
- JBoss becomes slow/unresponsive with many threads waiting on a BaseClassLoader
- JBoss is slow with threads heavily blocked in BaseClassLoader.doLoadClass()
- System hangs with a lot of threads blocked at java.lang.Class.forName0(Native Method) with no indicator of where the lock is held
- JBoss Deadlock in org.jboss.util.NestedThrowable
- Threads Blocked in
org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader - EAP 6 EJB threads bottleneck in the classloader when another nodes joins the cluster
- Struts contention on WebCtxLoader
- EAP 6 classloader deadlocks on JRockit JVM
- JBoss Clustering & Cache
- Web Frameworks
- JBossWeb/Tomcat/Undertow
- What happens if requests exceeds specified maxThreads on JBoss Web HTTP Connector
- JBoss server slows and hangs with many threads blocked in JspServletWrapper.service()
- Tomcat comes to a halt with threads waiting on GenericObjectPool
- JBoss unresponsive due to http connector threads blocked in org.apache.naming.resources.ProxyDirContext.cacheLoad()
- JBoss is unresponsive because the JBossWeb pool is exhausted in socketRead states
- JBoss web requests become slow when using a queueless executor pool
- NIO EventPoller thread dies from uncaught NPE, leading to unresponsiveness
- Tomcat request hangs in ChunkedInputFilter.parseChunkHeader
- Deadlock in WsRemoteEndpointImplServer.onWritePossible
- Deadlock between WsRemoteEndpointImplServer.onWritePossible and Http11NioProcessor.writeEvent
- Deadlock between WsSession.onClose and HttpEventImpl.close
- EAP 6 NIO deadlocks during blocking executor exhaustion
- Degraded performance in EAP 7.1 from ImportedClassELResolver
- JWS 5 Tomcat deadlocks across several request threads in java NIO calls
- JBoss unresponsive and logging 'UT000124: renegotiation timed out' errors
- JBoss threads hung in Http2StreamSinkChannel and AbstractFramedStreamSinkChannel.awaitWritable
- WebServices
- General
Diagnostic Steps
-
Review application logging at the time of the issue.
-
Determine if there is anything upstream of the application that could be the main cause (e.g. JBoss fronted by Apache and mod_jk). Test if the issue goes away when accessing the application directly (e.g. JBoss on port 8080).
-
Determine if the application is waiting on an external system or data store to become available:
-
Verify that the JVM is really unresponsive and in fact did not crash. Verify that the Java process is still alive (e.g. using jps -lv, ps. etc.). Sometimes the term "unresponsive" is used when really JBoss is down due to the JVM crashing. See Java application down due to JVM crash.
-
Verify that the Java process is still running (R or S state) by
ps auxcommand. For example,jstack -F <pid>makes the target Java process "trace stop" (T) state. -
Check the JVM options to see if any instrumenting agents are being specified with the -javaagent option. For example:
-javaagent:/opt/jboss/wily/Agent.jar.
Agents can add overhead and cause issues. Test with the agent removed, if possible.
-
Check the JVM options to see if any debugging instrumentation is enabled. For example:
-Xrunjdwp:transport=dt_socket,address=8787,server=y,suspend=n
Debugging adds a lot of overhead and should not be enabled in production. Test with debugging removed.
-
Determine if the bottleneck is the result of garbage collection activity. Enable garbage collection logging, and analyze it for the time period when the issue happens:
-
Determine if there is high cpu when the issue happens. If there is high cpu, or it is unknown, see Java application high CPU. By "high CPU" we mean sustained usage over 80%.
-
When the issue happens, get a series of thread dumps:
-
Analyze the thread dumps (How do I analyze a Java thread dump or javacore?):
- Are there any monitors that do not have a locking thread, which indicates the monitor is locked in native code, typically due to GC activity?
- Is there monitor contention or deadlocks?
-
On JBoss/Tomcat, determine if the http/ajp thread pools used are exhausted.
-
Check for the messages like the following on JBoss indicating a maxed out thread pool:
INFO [JIoEndpoint] Maximum number of threads (10) created for connector with address /127.0.0.1 and port 8080
-
-
Check the number of current busy threads from the http/ajp pool being used. Log in to the jmx-console and look for a link like the following for the connector in question:
name=ajp-127.0.0.1-8009,type=ThreadPool
Follow the instructions in How to Access JBossWeb mbeans and their Related statistics in JBoss EAP6? and check the currentThreadsBusy attribute here.
How many threads are there in the dumps from this connector? Does it reach the maxThreads level?
-
If this is the case, either more threads are needed for the load or bugs are resulting in threading issues (deadlocks, lock contention, etc.) to stall the connector threads and exhaust the pool.
-
When does the unresponsiveness happen? After a period of time, after a particular use case, or does it appear random?
-
Has this application been in production a long time and only now this issue is happening, or was the application fairly recently deployed in production?
-
If JBoss Enterprise Application Platform (EAP) 5, check to see if symlinks are used, as it is a known issue that use of symbolic links leads to large disk usage due to Virtual File System (VFS) bug. See Use of symbolic links leads to large disk usage in JBoss EAP.
-
Test with logging set to ERROR to rule out logging issues.
-
If the issue results in slow http responses, log request time on the front end and back end to "see" the delay or to narrow it down to a layer. See How do I enable friendly access log times in Apache and JBoss?.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.