Removing traces of failed transactions from the object store in JBoss EAP

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
  • 8
  • 7
  • 6
  • 5

Issue

  • Strange log entries regarding XA transactions

  • Following warning in JBoss EAP 5 based server log

      WARN  [Thread-9] () com.arjuna.ats.jta.logging.loggerI18N - [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] [com.arjuna.ats.internal.jta.resources.arjunacore.norecoveryxa] Could not find new XAResource to use for recovering non-serializable XAResource < 131075, 29, 27, 4945551024848484848495857544997585210155515252545658491025049551024848484848495857544997585210155515252545658491025051 > 
  WARN  [Thread-9] () com.arjuna.ats.jta.logging.loggerI18N - [com.arjuna.ats.internal.jta.resources.arjunacore.noresource] [com.arjuna.ats.internal.jta.resources.arjunacore.noresource] No XAResource to recover < 131075, 29, 27, 4945551024848484848495857544997585210155515252545658491025049551024848484848495857544997585210155515252545658491025051 >

  • Warnings generated by the JBoss TM XA recovery manager while trying to recover an XA transaction in EAP 6

      WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a2c0a0d:-140df526:50d1ddd3:ac883, node_name=1, branch_uid=0:ffff0a2c0a0d:-140df526:50d1ddd3:ac95c, subordinatenodename=null, eis_name=java:jboss/eis/Connection >, heuristic: TwoPhaseOutcome.FINISH_OK, product: ActiveMQ/5.7.0, jndiName: java:jboss/eis/Connection com.arjuna.ats.internal.jta.resources.arjunacore.XAResourceRecord@1325877 >
  WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=0:ffff0a2c0a0d:-140df526:50d1ddd3:ac883, node_name=1, branch_uid=0:ffff0a2c0a0d:-140df526:50d1ddd3:ac95c, subordinatenodename=null, eis_name=java:jboss/eis/Connection >
  • Every 2 minutes WARN messages are logged.

Resolution

JBoss EAP 8 , 7 , 6

via CLI

  • Standalone Mode:
    In EAP 6+, there is a proper documented way of removing stale transaction records from the store. This works with all transaction store formats i.e. os file, hornetq aio and jdbc.

    • Log into jboss-cli

    • Refresh the transaction store

        [standalone@localhost:9999 transactions] /subsystem=transactions/log-store=log-store/:probe()

    • List all transactions

        [standalone@localhost:9999 transactions] ls /subsystem=transactions/log-store=log-store/transactions

  0:ffff0a2c0a0d:-140df526:50d1ddd3:ac883   0:ffff0a2c0a0d:-140df526:50d1ddd3:ac88d

    • Then delete the one that is causing the warning

        [standalone@localhost:9999 transactions] /subsystem=transactions/log-store=log-store/transactions=0\:ffff0a2c0a0d\:-140df526\:50d1ddd3\:ac883:delete()

      {"outcome" => "success"}

  • Domain Mode:

    • Login to CLI and Refresh the transaction store

        /host=master/server=server-name/subsystem=transactions/log-store=log-store:probe()

    • Then delete one that is causing the warning

        /host=master/server=server-name/subsystem=transactions/log-store=log-store/transactions=0\:ffff0a212190\:-86fc9d3\:5236a39a\:b58b:delete()

    By default, the transaction store is physically located in ${JBOSS_HOME}/<profile>/data/tx-object-store. Manual deletions from this location are not recommended.

via the console


Removal of transactions from the object-store is also available via the JBoss Management Console. In Standalone mode, navigate to `Runtime -> Status -> Subsystems -> Transactions Logs`. Select `Probe`, highlight the offending Transactions, then select `delete`.

JBoss EAP 5

In EAP 5 it is mainly a manual process of removing the debris from the store. This is done by converting the transaction Xid into a file name, finding the file in the store and removing it.

  • The transaction store may be found at $JBOSS_HOME/jboss-as/server/<profile>/data/tx-object-store/HashedActionStore/defaultStore/Recovery/TransactionStatusManager.
  • The conversion of transaction Xid into a file name is done by changing the ":" into "_".
  • It is fine to delete $JBOSS_HOME/jboss-as/server/$PROFILE/data/tx-object-store/HashedActionStore/defaultStore/Recovery/TransactionStatusManager folder as long as the saved transactions do not need to be recovered.

User can delete the records from the tx store, but you will likely want to find the reason these WARNs are appearing in the first place.

Root Cause

The warning is usually caused by misconfiguration of XA resources. But under very rare conditions it may be caused when XA transaction fails and its remains can not be recovered by the recovery process. In this situation the warning will be logged repeatedly in the log file.

There are mainly 4 reasons why one may see those warnings in the log file:

  1. Misconfiguration of XA Recovery manager
  2. Problems with one of the XA Resource Managers
  3. JBossTM crash.
  4. Server is brought down using kill -9 command.

The first two should be relatively easy to deal with since usually the warning should be accompanied by additional errors that should provide some information as to why the warning may be appearing in the log file.

The 3rd is more difficult to deal with since the log file will contain only warnings and no additional information as to what may be causing it.
The 3rd situation may arise because there is a very small window in which the JBossTM may crash after committing all transactions but before performing its house keeping.
This leaves some debris in the transaction manager store area which would be read upon Jboss server restart. At that point the recovery manager will attempt to recover the XA transactions based on what is in its store. Unfortunately, the records are incomplete - hence the warning.
That means that the warning will be logged until the debris is removed from the store.

Diagnostic Steps

Trouble deleting? Please make sure all special characters are escaped:

ls /subsystem=transactions/log-store=log-store/transactions
<tx-id 1>   <tx-id 2>
/subsystem=transactions/log-store=log-store/transactions=0\:nnnnnnnnnn\:-xxxxxxxx:yyyyyyy\:zzzz:delete()
'yyyyyyy\:zzzz:delete' is not a valid operation name.

Note the : before yyyyyyy was not escaped.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.