EAP 7.4 JSF application get stuck and causes state session timeout

Environment

• Red Hat Enterprise Application Platform
  • 7.2.x+
  • JSF deployment (Mojarra is the default ref impl for EAP 7)

Issue

EAP 7.4 deployment of JSF gets stae session timeouts caused by deadlock occurs on Session access through JSF code in EAP 7.4.10:

Thread A - getAttribute

"default task-13" #338 prio=5 os_prio=0 tid=0x0000563de8c0c800 nid=0xb975d waiting for monitor entry [0x00007f120419e000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.wildfly.clustering.web.cache.session.fine.FineSessionAttributes.getAttribute(FineSessionAttributes.java:151)
    - waiting to lock <0x00000007b0407ae0> (a java.util.HashMap)
    at org.wildfly.clustering.web.undertow.session.DistributableSession.getAttribute(DistributableSession.java:207)
    at io.undertow.servlet.spec.HttpSessionImpl.getAttribute(HttpSessionImpl.java:122)
    at com.sun.faces.context.AlwaysPuttingSessionMap.put(AlwaysPuttingSessionMap.java:32)
    at com.sun.faces.context.AlwaysPuttingSessionMap.put(AlwaysPuttingSessionMap.java:23)
    at com.sun.faces.application.view.ViewScopeManager.processPostConstructViewMap(ViewScopeManager.java:327)
    - locked <0x00000007c3997bb0> (a java.util.Collections$SynchronizedMap)

Thread B - marshalling

"default task-10" #335 prio=5 os_prio=0 tid=0x0000563dea910000 nid=0xb975a waiting for monitor entry [0x00007f1203d5e000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at java.util.Collections$SynchronizedMap.writeObject(Collections.java:2693)
    - waiting to lock <0x00000007c3997bb0> (a java.util.Collections$SynchronizedMap)
    at sun.reflect.GeneratedMethodAccessor89.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.jboss.marshalling.reflect.JDKSpecific$SerMethods.callWriteObject(JDKSpecific.java:206)
    at org.jboss.marshalling.reflect.SerializableClass.callWriteObject(SerializableClass.java:215)

Resolution

This is a bug where Mojarra's JSF detects the application is clustered (as in distributed) and redundantly does some operations such as setAttributes. In as much as this is the spec behavior this might cause locking with EAP behavior (details on Root Cause).
This is being tracked on the jira: This content is not included.WFLY-18708 to disable the JSF's auto-detection mechanism/auto-distributable behavior via

Workarounds

There are a few workarounds for this matter:

1. Set the application for non-distributable (via removing the distributable tag on the web.xml)
2. Set the cache replication as REPEATABLE_READ/BATCH - so there won't be any multiple access (cause multiple access will return timeout) - As the solution How to configure and tune the session replication for EAP explains, REPEATABLE_READ/BATCH does not allow multiple access.
3. Set com.sun.faces.enableDistributable via listener code, and not via web.xml file descriptor (see each one below).

In table:

Note: Workaround c above: disabling via web.xml requires a fix on JSF to work, Content from github.com is not included.Mojarra 347.

Via web.xml

In some instances the workaround via change on web.xml won't work because Mojarra's bootstrap ignores Content from github.com is not included.any value specified:

<context-param>
    <param-name>com.sun.faces.enableDistributable</param-name>
    <param-value>false</param-value>
</context-param>

The above will not work given Mojarra ignores the web.xml.

Via code by creating a listener:

1. Define a ServletContextListener that overrides the com.sun.faces.enableDistributable flag. e.g.

package foo.bar;

import jakarta.faces.context.FacesContext;
import jakarta.servlet.ServletContext;
import jakarta.servlet.ServletContextEvent;
import jakarta.servlet.ServletContextListener;

import com.sun.faces.config.WebConfiguration;
import com.sun.faces.config.InitFacesContext;

public class DisableDistributableListener implements ServletContextListener {

    @Override
    public void contextInitialized(ServletContextEvent event) {
        ServletContext context = event.getServletContext();
        FacesContext facesContext = new InitFacesContext(context);
        try {
            // Disable counter-productive "distributable" logic from Mojarra implementation
            WebConfiguration.getInstance(context).setOptionEnabled(WebConfiguration.BooleanWebContextInitParameter.EnableDistributable, false);
            context.setAttribute(WebConfiguration.BooleanWebContextInitParameter.EnableDistributable.getQualifiedName(), Boolean.FALSE);
        } finally {
            facesContext.release();
        }
    }
}

2. Update web.xml to run the listener after the listener used by Mojarra. e.g.

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="https://jakarta.ee/xml/ns/jakartaee"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="https://jakarta.ee/xml/ns/jakartaee https://jakarta.ee/xml/ns/jakartaee/web-app_6_0.xsd"
         version="6.0">
    <distributable/>
    <listener>
        <listener-class>com.sun.faces.config.ConfigureListener</listener-class>
    </listener>
    <listener>
        <listener-class>foo.bar.DisableDistributableListener</listener-class>
    </listener>
</web-app>

Root Cause

JSF Bug

Mojarra is Oracle's JSF reference implementation. Mojarra includes code that detects a distributable application and makes changes to the session. These changes interfere with EAP's session management and can lead to deadlocks on the session.
There is an option to disable this behaviour in Mojarra via the applications web.xml, however, this disablement is not being honored by Mojarra, to be fixed on Content from github.com is not included.PR-Mojarra 5347.

There is also an upstream jira This content is not included.JBEAP-26003 / This content is not included.JBEAP-26002 solve this problem by disabling the JSF mechanism in EAP 7, by disabling the distributable logic in Mojarra upon detecting the application is distributable.

EAP specific

As explained in the main solution How to configure and tune the session replication for EAP, EAP session replication code is in between session.getAttribute/session.setAttribute and the underlying cache.get and cache.put calls.
In this case, when the app calls session.getAttribute EAP keeps track of it, and at the end of the request makes a cache.put call to trigger replication, which is why it does not need an explicit session.setAttribute call since it's going to replicate regardless.
Above is EAP 7 specific, other server containers will necessarily need the setAttribute call at their own discretion.

Diagnostic Steps

1. See the cache configuration has READ_COMMITTED/NONE, which allows multiple access.
2. Verify the thread dumps for the dead-locking pattern above.
3. Consider the workarounds above.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.