EJB chain failed to commit the distributed XA transaction if the EJB's are deployed on more than 2 EAP7 servers

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 7.1
  • 7.2
• Deploying applications on 3 different servers handling an EJB chain (Server 1 -> Server 2 -> Server 3)

Issue

• If an application is using several different servers, the invocation seems correct but the final commit of the XA transaction will fail with WFTXN0034, JBREM000308, ARJUNA016027, ARJUNA016009, ARJUNA016045. If the applications are all deployed to the 2nd server, then the use-case works fine.

• Transaction recovery fails and throws an Exception if applications are deployed on different instances:

    3:30:46,524 WARN  [com.arjuna.ats.jta] (Periodic Recovery) ARJUNA016027: Local XARecoveryModule.xaRecovery got XA exception XAException.XAER_RMFAIL: javax.transaction.xa.XAException: WFTXN0034: Failed to acquire a connection for this operation
    	at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.getPeerIdentityXA(RemotingRemoteTransactionPeer.java:94)
    	at org.wildfly.transaction.client.provider.remoting.RemotingRemoteTransactionPeer.recover(RemotingRemoteTransactionPeer.java:201)
    	...
    Caused by: javax.security.sasl.SaslException: Authentication failed: none of the mechanisms presented by the server (DIGEST-MD5) are supported
    	... more
  

Resolution

Upgrade to EAP 7.2 CP 4 or later.

This issue can happen on EAP 7.1 or 7.2 server if the applications are deployed to different servers and more than 1 server is involved in the distributed EJB transaction.
The issue was originally tracked by Content from issues.jboss.org is not included.JBEAP-17163 and it was fixed in 7.2 CP4.

Until 7.2.4 there are workarounds:

• Keep the EJB together so that the invocation is always local
• Add the following, which needs to be applied for all server instances which invoke remote EJB's
  • create standalone/configuration/wildfly-config.xml with the following content (you may change the user.name and password)

        <configuration>
            <authentication-client xmlns="urn:elytron:1.0">
                <authentication-rules>
                    <rule use-configuration="default"/>
                </authentication-rules>
                <authentication-configurations>
                    <configuration name="default">
                        <set-user-name name="jbossRemote"/>
                        <credentials>
                            <clear-password password="topSecret"/>
                        </credentials>
                    </configuration>
                </authentication-configurations>
            </authentication-client>    
        </configuration>

• add the user to all involved EAP instance
• add `-Dwildfly.config.url=-Dwildfly.config.url=standalone/configuration/wildfly-config.xml to the startup parameter by
  • adding it to the standalone.conf
  • using the parameter as start option for standalone.sh
  • setting the JVM properties for the server inside the domain configuration

Root Cause

The remoting communication will not use the correct security context to process the prepare/commit for the transaction. The EJB method invocation will work as expected and the methods are called for each server, but if the transaction needs to commit the communication between the servers will fail with one of the following causes:

org.wildfly.security.auth.AuthenticationException: JBREM000308: Authentication failed (no mechanisms left), tried: (none)
javax.security.sasl.SaslException: Authentication failed: none of the mechanisms presented by the server (DIGEST-MD5) are supported

The issue can happen in different use-cases

• If the Transaction should be recovered and the applications are deployed to different server
  • the automatic recovery will fail after restart
• If there is an EJB chain across 3 servers
  • The invocation from server1->server2->server3 using the remoting will succeed
  • The XA transaction prepare will cause an Exception at server2 as the remoting from server2->server3 will not use the correct security context. Therefore you'll find a transaction failure error message within the log files of server1 and server2.

Diagnostic Steps

• EJB application uses more than 1 server
  • the recovery will fail with SASL Exception
• EJB application use more than 2 server
  • the EJB invocation from server1->server2->server3 will succeed but the 2P XA commit prepare will fail at server2 (to server3) as here the remoting will not use the correct security context.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.