JBoss ON server does not define transaction node identifier resulting in JBAS010153, ARJUNA016037, or ARJUNA016038 warnings

Solution Verified - Updated

Environment

  • Red Hat JBoss Operations Network (ON)
    • 3.2
    • 3.3.0
    • 3.3.1
    • 3.3.2
    • 3.3.3
    • 3.3.4
    • 3.3.5
    • 3.3.6
    • 3.3.7
    • 3.3.8
    • 3.3.9
    • 3.3.10
  • Multiple JBoss ON servers configured in high-availability (HA)

Issue

  • Server log contains the following warnings every few minutes

      WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016037: Could not find new XAResource to use for recovering non-serializable XAResource XAResourceRecord < resource:null, txid:< formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=<tx-id>, node_name=1, branch_uid=<branch-id, subordinatenodename=null, eis_name=java:jboss/datasources/RHQDS >, heuristic: TwoPhaseOutcome.FINISH_OK...
  WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016038: No XAResource to recover < formatId=131077, gtrid_length=29, bqual_length=36, tx_uid=<tx-id>, node_name=1, branch_uid=<branch-id>, subordinatenodename=null, eis_name=java:jboss/datasources/RHQDS >

  • Server logs the following warning on startup

      WARN  [org.jboss.as.txn] (ServerService Thread Pool -- 57) JBAS010153: Node identifier property is set to the default value. Please make sure it is unique.

  • Database reports long standing lock contention issues between RHQADMIN sessions running updates from 2 or more JBoss ON servers

Resolution

Update to JBoss ON 3.3 Update 11 or later and define a node identifier value by setting jboss.tx.node.id to a unique value that is 23 characters or less in rhq-server.properties. For details, see Defining a Unique Node Identifier in Your JBoss Operations Network High Availability System. Please note that jboss.tx.node.id must be unique on each server in the high-availability system.

Root Cause

A unique transaction node identifier is necessary if multiple application servers will be sharing the same datasource configuration. This allows the transaction manager to identify and associate in process or aborted transactions with a specific server.

In the case of JBoss ON, a unique node identifier is not defined and therefore the default identifier value of 1 is used. When multiple servers are active, they are all utilizing the same node identifier of 1. This results in the transaction reaper of one server to discover the transactions from other servers utilizing the global transaction manager. However, the transactions are unknown to the local server resulting in warnings regarding missing transactions.

This issue has been identified as This content is not included.Red Hat Bugzilla 1127359.

Diagnostic Steps

  • Has a unique transaction node identifier been defined for each JBoss ON server in the HA configuration?

    By default, the JBoss ON server installer and start scripts do not handle this. You can check the current attribute value utilizing jboss-cli:

      "${RHQ_SERVER_HOME}"/jbossas/bin/jboss-cli.sh --connect --controller=localhost:6999 --command='/subsystem=transactions:read-attribute(name=node-identifier,include-defaults=true)'

    If the returned value is 1 it indicates the default value is in use. If the same value is returned by two or more JBoss ON servers in the cluster, this issue applies.

  • Review server.log for a warning on startup that indicates the default value is in use:

      JBAS010153: Node identifier property is set to the default value. Please make sure it is unique.

    If this warning is present for more than one JBoss ON server, this issue applies.

  • Review server.log for repeated warnings with message identifier ARJUNA016037 or ARJUNA016038. These messages will include the node identifier value as node_name=<node identifier>. If the node_name value from two or more servers in the cluster is the same, this issue applies.

SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.