How to diagnose ClassNotFoundException / NoClassDefFoundError and other classloading issues in JBoss EAP 8 / 7 / 6 with Tattletale

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 8
    • 7
    • 6

Issue

  • How to diagnose ClassNotFoundException issues in JBoss Enterprise Application Platform (JBoss EAP)?
  • How to diagnose NoClassDefFoundError issues in JBoss EAP ?
  • How to diagnose classloading issues in JBoss EAP?
  • How to diagnose ClassCastException issues in JBoss EAP?
  • How to diagnose LinkageError issues in JBoss EAP?
  • How to diagnose NoSuchMethodError issues in JBoss EAP?
  • How to diagnose UnsupportedClassVersionError issues in JBoss EAP?
  • How to diagnose java.lang.InstantiationError issues in JBoss EAP?
  • Our application is failing with a ClassNotFoundException : 
...
Caused by: java.lang.ClassNotFoundException: com.jboss.examples.MyClass from [Module "deployment.application.war:main" from Service Module Loader]
...
  • Application is unresponsive. While hitting with request , the application is responding with connection timed out . When we are removing our xfire-all-1.2.6.jar the application is working fine but as soon as we again add the mentioned jar it again shows the same response that is connection timed out.
49 00:06:47,820 ERROR [stderr] (http-/10.10.10.74:80-1) Caused by: java.lang.NoSuchMethodException: com.abc.jkl.xyz.ClassMap()

Resolution

Please run the attached JBoss Tattletale jar, the current version is "tattletale.jar". JBoss Tattletale is a project started on jboss.org that generates reports about the classes and dependencies of a jar/war/ear.

If there are custom modules in the $JBOSS_HOME/modules directory, run Tattletale on the custom modules as well.

To run the report:

  1. Download the tattletale.jar attached to this article

  2. Then execute Tattletale as shown below, with the application as the first parameter and the name of the output for the report as the second parameter (it defaults to report.zip if none is specified).

$ java -Xmx512m -jar tattletale.jar yourapplication.war yourapplication-report.zip

For EAP 6

$ java -Xmx512m -jar tattletale.jar -p eap64 yourapplication.war yourapplication-report.zip

For EAP 8

  • To list all available 'profiles', tattletale can be executed with the '-l' option:
$ java -jar tattletale_0.jar -l                                                                                                                                                                         
All profiles: [ee10, eap80, jdk17, jdk11, jdk8, ee6, ee8, eap74, eap64]
  • If you're using JDK 11 with EAP 8, use the following command 
$ java -Xmx512m -jar tattletale.jar -p eap80 yourapplication.war yourapplication-report.zip
  • If you're using JDK 17 with EAP 8, use the following command
$ java -jar tattletale_0.jar -p jdk17,ee10,eap80 <application> tattletale-report.zip
  1. A report zipped folder will be generated, which contains several reports as html pages. If you are running Tattletale for a support case, please attach the zipped report to the support case.

A summary report outlining potential problems is contained in problematic-archives-summary sub-folder of the zipped report. Check and confirm from its reports that expected classes currently facing classloader errors are provided or unnecessary, duplicate classes are not provided. If classes are properly provided but still seeing such errors with a java agent (like AppDynamics), consider testing with the java agent disabled as these have been seen to have classloader impacts.

NOTE: Tattletale looks for classes in the application, if there are no classes available in the application then it would make sense that it would not generate a report, hence there must be a class in your application while creating tattletale report.

Related Solutions

Root Cause

Typically there should be a ClassNotFoundException that preceeds NoClassDefFoundError

Attached is JBoss Tattletale zip with some additional reports added to look for duplicated classes. Tattletale is a 3rd party tool and not supported as part of JBoss EAP.

Note: the support for JBoss Modules does not take into account implicit dependencies that have already been added and does not take into account the modules in the particular JBoss EAP being used, so it can be a starting place, but one should review the output for correctness.

For more information on the uses of Tattletale, please refer to the This content is not included.Migration Guide of EAP 6.

Diagnostic Steps

  • Run the tattletale.jar too on the application(s)
  • unzip the report.zip and open the report/index.html in a web browser
  • Review the full server.log and locate the first classloading exception.
  • For the given classloading exception review the diagnostic steps below and check the section below and look at the report mentioned to locate the issue.

ClassNotFoundException

ClassNotFoundException and NoClassDefFoundError typically are thrown because the class is not in the classpath, which could be because of the packaging of the application or the classloader isolation settings specified. Look at the Tattletale Report: Class Location to locate where it is packaged and compare to the isolation settings & Java EE classpath defined for the particular deployment type.

Note: if the class does not exist in the application and the java source file does not list the class, decompile the class using javap or other decompiler and see if the class definition matches the source. Some IDEs can generate invalid classes when the classpath is missing classes, they could generate a class with invalid class references and if those get deployed it will cause unusual ClassNotFoundExceptions such as a class name with no package.

With Javap look for Unresolved compilation problems or other messages in the class definition:

javap -c -classpath my-deployment.jar com.jboss.examples.FileUploadBean  | less

NoClassDefFoundError

NoClassDefFoundError is thrown when the JVM tries to load a definition of a class and no definition can be found. NoClassDefFoundError is typically caused by a ClassNotFoundException or a ExceptionInInitializerError. After the initial ClassNotFoundException / ExceptionInInitializerError, the JVM will just throw a NoClassDefFoundError and will not show the original cause.

Could not initialize class

When a NoClassDefFoundError is thrown with the message Could not initialize class ..., it typically indicates it was caused by a ExceptionInInitializerError which is caused by uncaught exceptions being thrown from static blocks / variables. It is recommended to use a try / catch and handle exceptions that might be thrown from static blocks or static fields.

java.lang.NoClassDefFoundError: Could not initialize class example.MyClass
Example of static blocks / fields that can cause NoClassDefFoundError and Byteman
public class MyClass {

  private static MyObject = OtherObject.init(...);  // if this throws an exception and is not caught, then the class will fail to initialize

  static {
     AnotherObject.init(); // if this throws an exception and is not caught, then the class will fail to initialize
  }
}

The JVM verbose class logging does not help in debugging NoClassDefFoundError where the original exception is not being logged. Attached is byteman-no-class-def-error.zip which contains a Byteman script that works similar to AOP and will inject code to log a stacktrace when ExceptionInInitializerError or ClassNotFoundException occurs. To try to locate the root cause exception with byteman-no-class-def-error.zip.
Follow this procedure to investigate it with Byteman:

  1. This will create a byteman directory in your $JBOSS_HOME/

    unzip -d $JBOSS_HOME/ byteman-no-class-def-error.zip

  2. Edit $JBOSS_HOME/bin/standalone.conf and add these lines at the end of the file (make sure to add them outside of the if ... fi block)
    Set this property -DNoClassDefFoundErrorClass to the class which is mentioned in the NoClassDefFoundError exception message

    BYTEMAN_HOME=$JBOSS_HOME/byteman/
JAVA_OPTS="$JAVA_OPTS -javaagent:/$BYTEMAN_HOME/lib/byteman.jar=script:$BYTEMAN_HOME/scripts    /NoClassDefFoundError.btm,boot:$BYTEMAN_HOME/lib/byteman.jar,boot:$BYTEMAN_HOME/lib/byteman-helpers.jar,listener:true,port:9091 -Dorg.jboss.byteman.transform.all=true -Dorg.jboss.byteman.compileToBytecode=true -DNoClassDefFoundErrorClass=example.MyClass"

  3. Start JBoss and reproduce the NoClassDefFoundError

  4. Search the log for BYTEMAN-NCDFE, if found it will log a stacktrace below it, showing the class that caused the ClassNotFoundException or ExceptionInInitializerError.

ExceptionInInitializerError

This error occurs when there is an unexpected exception thrown in a static initializer such as a static code block or static variable initialization.

ClassCastException

ClassCastException typically is thrown because a class is loaded in two different classloaders, and an object is being created from one classloader and passed to an application using the other. Look at the Tattletale Report: Class Location to see if the API/IMPL class is packaged more than once in the application. Look at the Tattletale Report: Packaged JDK / J2EE Classes, Packaged JBoss Classes to see if JBoss is packaging the API / IMPL and check your isolation settings.

LinkageError

LinkageError typically is from a mismatch between an API jar & Implementation jar. Look at Tattletale Report: Packaged JDK / J2EE Classes, Packaged JBoss Classes if the classses involved are JBoss or JavaEE classes. Look at the
Class Location report to see if the classes are packaged in more than one location.

NoSuchMethodError

NoSuchMethodError typically occurs when a class tries to invoke a method on a class that does not exist. This can occur if the application is compiled against one version of a class but then at runtime is using a different version of the class. Another way this can occur is when an application is packaging multiple versions of classes. If multiple versions of a class are packaged in the application, then the wrong version could be loaded, because the operating system does not return jars/classes in a particular order and which ever class gets loaded first will be used, since a class definition can only be loaded once in a particular classloader.

report/multiplejarspackage/index.html - if an application is packaging multiple versions of a class, this report will show which packages are being duplicated and which jars are duplicating them. Removing wrong versions of the jars/classes will resolve the issue.

Example of duplicated jars/classes in an application. Below it shows that DefaultListableBeanFactory is loaded from spring-beans-4.1.0.RELEASE.jar and it expects OrderComparator to have the method getPriority(Ljava/lang/Object;)Ljava/lang/Integer. Looking in the Tattletale report to find the jars packaging org.springframework.core.OrderComparator, there will be duplicate versions of the class. Then the wrong version(s) of the jar / class needs to be removed to resolve the issue.

Caused by: java.lang.NoSuchMethodError: org.springframework.core.OrderComparator.getPriority(Ljava/lang/Object;)Ljava/lang/Integer;
  at org.springframework.beans.factory.support.DefaultListableBeanFactory.getPriority(DefaultListableBeanFactory.java:1227) [spring-beans-4.1.0.RELEASE.jar:4.1.0.RELEASE]

IllegalAccessError: tried to access method ... from class ...

This exception typically means a class tried to access a method that it does not have access to, such as a class trying to access a package level method or package level class. The cause is likely using a library which is split in to multiple jars and then packaging multiple versions of the jars. Or having the jars which need to be in the same classloader in different classloaders causing Java to throw this error because the class trying to access a package level method or package level class is not allowed since the classes are in different classloaders.

UnsupportedClassVersionError

UnsupportedClassVersionError thrown because JBoss is being run with an older version of Java than JBoss was compiled on. JBoss EAP 6.x must be run on JDK 6 or JDK 7.

IncompatibleClassChangeError: Implementing class

This error suggests multiple versions of a class being packaged which are incompatible with each other. Only the compatible classes should be packaged, the other version of the class should be removed.

InstantiationError

Content from docs.oracle.com is not included.java.lang.InstantiationError An InstantiationError occurs when code tries to create an abstract class or interface, which is invalid and would be caught at compile time unless the code was compiled and then a different class version was used at runtime, thus looking for duplicate classes in the application or mismatched versions in jars would show where the problem is.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.