Using Byteman to troubleshoot Java issues

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
  • Java
  • Byteman
  • Automated testing or production runtime issue

Issue

  • Troubleshoot runtime issues or perform automated testing without modifying code
  • An exception is printed in server.log; however, there is no stacktrace
  • Find out if a method is executed

Resolution

Byteman is a tool to inject Java code into your running application with no need to recompile. Byteman hooks in as a JVMTI agent and has often a very low overhead. 1

Download Byteman and resources

Download Byteman from Content from byteman.jboss.org is not included.the Downloads page release. It is recommended to use the latest 4.x or 3.x releases. Especially, some early releases (< 4.0.3, < 3.0.13) had a bug that led to failure in routing traceStack() output to JBoss EAP server.log, and the bug has been fixed as This content is not included.BYTEMAN-370.

Once downloaded, please unzip the binary zip file to your local file system, for example /path/to/byteman-download-4.0.21/ and declare BYTEMAN_HOME system environment variable, for example, on Linux:

export BYTEMAN_HOME=/path/to/byteman-download-4.0.21/

Documentation and other resources can be found in Content from byteman.jboss.org is not included.the Docs page as well as Content from github.com is not included.Byteman Github project page. Example Byteman scripts are also included in the downloaded binary release byteman-download-<version>/sample folder.

Byteman versions vs. JDK versions

  • See What JDK version does byteman require for details.
  • In short, we recommend using the latest Byteman 4.x release for JDK 8+ users, and the latest Byteman 3.x release for old JDK 6/7 users. (Note that Byteman 4.0.0+ won't work with JDK 6, and Byteman 4.0.19+ won't work with JDK 7.)
  • You might also want to know which Byteman version added support for newer JDK versions. See "Table of Which Byteman versions added support for newer JDK versions" in the solution for details.

Activate Byteman and enable the rules

Enable Byteman for JBoss server persistently

Byteman rules can be enabled in a persistent way so that these configurations live across Java/JBoss restart. This is often the simplest option for configuration.

  1. Create a text file and put Byteman rules in, rename the file with .btm extension for example, /my/path/examplescript.btm
  2. Add JVM parameters below to enable Byteman and restart Java/JBoss server instance

  • For JBoss EAP 5 and JBoss EAP 6/7 in standalone mode:

    Enable the Byteman instrumentation by adding the following Java VM option to JAVA_OPTS in run.conf/run.conf.bat (JBoss EAP 5) or standalone.conf/standalone.conf.bat (JBoss EAP 6/7). For example, here is example of configuration snippet from JBoss EAP 7 standalone.conf:

    ... ...
    if [ "x$JAVA_OPTS" = "x" ]; then
          JAVA_OPTS="-Xms1303m -Xmx1303m -XX:MetaspaceSize=96M -XX:MaxMetaspaceSize=256m -Djava.net.preferIPv4Stack=true"
... ...
          JAVA_OPTS="$JAVA_OPTS -javaagent:/path/to/byteman/lib/byteman.jar=script:/my/path/examplescript.btm,sys:/path/to/byteman/lib/byteman.jar  -Dorg.jboss.byteman.transform.all=true"
... ...

  • For JBoss EAP 6/7 in domain mode:

    Add the javaagent configuration in the <jvm> options for the <server> or <server-group> in domain.xml or host.xml: 

     <jvm name="default">
   <java-agent value="/path/to/byteman/lib/byteman.jar=script:/my/path/examplescript.btm,sys:/path/to/byteman/lib/byteman.jar"/>   
 </jvm>
<--- optional jvm flags that can help --->
             <jvm-options>
                 <option value="-Dorg.jboss.byteman.transform.all=true"/>
                 <option value="-Dorg.jboss.byteman.verbose"/>
                 <option value="-Dorg.jboss.byteman.debug=true"/>
             </jvm-options>
Adding a custom Byteman Helper to the classpath

JBoss Modules by default will expose classes in the package org.jboss.byteman , the simplest way is to have a custom byteman helper package be org.jboss.byteman, otherwise the JBOSS_MODULES_SYSTEM_PKGS will need to include the package name where the custom byteman helper is located. JBOSS_MODULES_SYSTEM_PKGS is a comma separated list of packages.

$JBOSS_HOME/bin/standalone.conf or standalone.conf.bat or standalone.conf.ps1

if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then
   JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman"
fi

Next include the custom byteman helper(s) inside of a jar which gets added to the system classpath similar to the byteman.jar, for example:

              JAVA_OPTS="$JAVA_OPTS -javaagent:/path/to/byteman/lib/byteman.jar=script:/my/path/examplescript.btm,sys:/path/to/byteman/lib/byteman.jar,sys:/path/to/byteman/lib/custom-byteman-helpers.jar  -Dorg.jboss.byteman.transform.all=true"

Install Byteman and submit rules dynamically

Activating Byteman rules dynamically can be useful for troubleshooting issues in production environment against running Java/JBoss processes. Rules can be activated and deactivated dynamically without requiring to restart Java/JBoss process. This may require greater effort for successful deployment than the persistent approach mentioned above.

  1. Create a text file and put Byteman rules in, rename the file with .btm extension for example, /my/path/examplescript.btm

  2. Identify the process ID (PID) of the Java application process or JBoss EAP server process, <pid>

  3. Install Byteman on this specific process (Run Byteman using the same OS user as you run Java application/JBoss server)

    $  $BYTEMAN_HOME/bin/bminstall.sh -Dorg.jboss.byteman.verbose=true -Dorg.jboss.byteman.debug=true -Dorg.jboss.byteman.transform.all=true <pid>

  4. Activate the rules:

    $  $BYTEMAN_HOME/bin/bmsubmit.sh  /my/path/examplescript.btm

  5. Deactivate all rules after troubleshooting:

    $  $BYTEMAN_HOME/bin/bmsubmit.sh  -u

Additional Notes:

  • The above example uses the "sys" - System class loader. This is sufficient for instrumenting end-user application code. If you wish to instrument JVM classes, such as java.net.Socket, then you should use the "boot" - Bootstrap class loader. If you are getting NoClassDefFoundError when using "sys", switch to "boot" and see if the NoClassDefFoundError errors persist.
  • If you are running JBoss EAP 6 or another program based on JBoss Modules, ensure that "-Djboss.modules.system.pkgs=org.jboss.byteman" is added to your Java VM options in standalone.conf or domain.conf.
  • Please refer to Using Byteman to troubleshoot Java issues in JBoss Fuse or A-MQ for Byteman troubleshooting with JBoss Fuse or A-MQ.

Example Byteman Rules

  • Generate a stacktrace (send to standard out stdout) whenever the org.jboss.security.identity.plugins.SimpleRole(String roleName, Role parent) constructor is called.

    • This rule uses method traceStack(String prefix, int maxFrames) to print stacktrace on exiting constructor method

    • "prefix" is a message printed in front of the stacktrace and "maxFrames" limits the max stack size getting printed.

      RULE trace SimpleRole constructor
CLASS org.jboss.security.identity.plugins.SimpleRole
METHOD <init>(String,Role)
AT EXIT
IF TRUE
DO traceStack("-----Byteman Rule is executed: printing stacktrace", 500) 
ENDRULE

  • Generate a stacktrace to view the calling method whenever the getParameter or getInputStream is called.

    • This Byteman script was designed specifically to capture offending code that calls the getInputStream and related methods prematurely in a POST request which will cause future calls to getParameter to fail. The effect here is that parameters will be seen as null in this situation. See the following knowledge solution for a further discussion on this: Why is the POST body unavailable to my servlet when the RequestDumperValve is enabled?

          RULE check getParameter
    CLASS org.apache.catalina.connector.Request
    METHOD getInputStream
    AT ENTRY
    IF $0.parametersParsed
    DO System.err.println("-----Byteman Rule is executed: attempting to get inputstream when parameters already parsed")
    ENDRULE

    RULE check parseParameters
    CLASS org.apache.catalina.connector.Request
    METHOD parseParameters
    AT ENTRY
    IF $0.usingInputStream
    DO System.err.println("-----Byteman Rule is executed: attempting to get parameters when input stream already read")
    ENDRULE

  • Create a timer on entering a method to track the method execution time, print the time spent when exiting method.

    • The symbol ^ in front of the class name requests that the rule is injected down the class hierarchy into any implementation defined by a subclass of com.abc.BusinessService.

    • BIND is used to create variables and bind/assign their values.

    • this:Object is to specify the variable being created with name this and type Object.

    • timerCreated:boolean is to specify the variable being created with name timerCreated and type boolean.

    • $0 is a special value and used by Byteman to reference a Thread object which triggered this rule (current thread).

    • createTimer method can be called to create a new Timer associated with input parameter (A String object in this example). createTimer returns true if a new Timer was created and false if a Timer associated with o already exists.

    • deleteTimer method can be called to delete the Timer associated with input parameter.

          RULE com.abc.BusinessService#processRequest - START
    CLASS ^com.abc.BusinessService
    METHOD processRequest
    AT ENTRY
    BIND
        this:Object = $0;
        timerCreated:boolean = createTimer(this.toString()+"processRequest");
    IF TRUE
        DO System.out.println("[BYTEMAN] " + this.toString() + " processRequest called"); 
    ENDRULE

    RULE com.abc.BusinessService#processRequest - END
    CLASS ^com.abc.BusinessService
    METHOD processRequest
    AT EXIT
    BIND
        this:Object = $0;
        timeSpent:long = getElapsedTimeFromTimer(this.toString()+"processRequest");
    IF TRUE
        DO System.out.println("[BYTEMAN] " + this.toString() + " processRequest took " + timeSpent + " ms");deleteTimer(this.toString()+"processRequest");
    ENDRULE

  • Print stacktrace of the unhandled exception in org.jboss.seam.Component.forName() method

    • An AT EXCEPTION EXIT specifier identifies the point where a method returns control back to its caller via unhandled exceptional control flow.

    • Special variable $^ is valid in an AT EXCEPTION EXIT rule and is bound to the throwable being returned from the method via exceptional control flow. Its type is Throwable.

      RULE printStackTrace
CLASS org.jboss.seam.Component
AT EXCEPTION EXIT
METHOD forName
IF TRUE
DO
    System.out.println(formatStack("---------[BYTEMAN-1] Exception caught:\n"));
    $^.printStackTrace();
ENDRULE

  • Print stacktrace and extra caller (Thread) information

    • This set of rules was created to troubleshoot SSL certificate related issues by printing more details (caller stacktrace and thread information)

    • Rules can be attached to interfaces using INTERFACE keyword

    • ^ special character in front of javax.net.ssl.X509KeyManager make the rule not only effective to classes directly implement this X509KeyManager interface, but also effective to those subclasses which extends the parent implementation class.

          RULE log key manager - chooseClientAlias
    INTERFACE ^javax.net.ssl.X509KeyManager
    METHOD chooseClientAlias
    AT ENTRY
    IF TRUE
    DO
        System.out.println("Printing stacktrace for chooseClientAlias - " + Thread.currentThread().getName());
        traceStack();
    ENDRULE

    RULE log key manager - chooseServerAlias
    INTERFACE ^javax.net.ssl.X509KeyManager
    METHOD chooseServerAlias
    AT ENTRY
    IF TRUE
    DO
        System.out.println("Printing stacktrace for chooseServerAlias - " + Thread.currentThread().getName());
        traceStack();
    ENDRULE
1

Note: Any agent will have an overhead, Byteman makes very specific, Content from developer.jboss.org is not included.highly localized changes and incurs little overhead in doing so, however the overhead is completely dependent upon the rules in play, what the rules do, and the resulting number of invocations on that rule with the given application load and behaviour.

Root Cause

Byteman is a free and open-source tool that can be used to modify bytecode in a running application for use with testing and debugging without changing the source. Red Hat Support sometimes uses Byteman for troubleshooting deployed customer applications, but it is not a shipped or supported product. If you need help or to report an issue, see the Content from byteman.jboss.org is not included.The Byteman Community.

Diagnostic Steps

Note: If encountering an error like the below when starting a Java process with Byteman, verify that the Byteman folder, jar and input script exist in the path and the user running the process has permission to access them:

Error opening zip file or JAR manifest missing : /opt/byteman-download-3.0.10/lib/byteman.jar
Error occurred during initialization of VM
agent library failed to init: instrument
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.