StackOverflowError in JBoss Serialization

Environment

• JBoss Enterprise Application Platform (EAP)

Issue

• We see a StackOverflowError thrown from JBoss serialization attempts. We see this particularly for large custom data structures being serialized with many recursive child elements. Serialization calls like below repeat:

...
	at org.jboss.marshalling.river.RiverMarshaller.writeKnownObject(RiverMarshaller.java:696)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:217)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1080)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
	at org.jboss.marshalling.river.RiverMarshaller.writeKnownObject(RiverMarshaller.java:696)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:217)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1080)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
	at org.jboss.marshalling.river.RiverMarshaller.writeKnownObject(RiverMarshaller.java:696)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:217)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1080)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
	at org.jboss.marshalling.river.RiverMarshaller.writeKnownObject(RiverMarshaller.java:696)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:217)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteFields(RiverMarshaller.java:1143)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1101)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteSerializableObject(RiverMarshaller.java:1080)
	at org.jboss.marshalling.river.RiverMarshaller.doWriteObject(RiverMarshaller.java:268)
	at org.jboss.marshalling.river.RiverMarshaller.writeKnownObject(RiverMarshaller.java:696)
...

Resolution

• The recursive serialization's stack use naturally grows in line with the size of your datastructure in its current implementation so you would need to scale up the thread stack setting (-Xss) as needed to support larger data structures. Increasing the thread stack to meet the needs of your recursive serialization for your expected datastructure sizes would be safe so long as you have the required memory for the thread stacks (Xss value * threadCounts) on top of the heap needed for the JVM heap, metaspace, code cache, etc.
• Otherwise, consider re implementing the involved datastructure or its serialization (custom writeObject/readObject calls) to avoid recursion. Take for example, the JDK Content from github.com is not included.LinkedList implementation; it serializes all entries iteratively without recursion:

    private void writeObject(java.io.ObjectOutputStream s)
        throws java.io.IOException {
        // Write out any hidden serialization magic
        s.defaultWriteObject();

        // Write out size
        s.writeInt(size);

        // Write out all elements in the proper order.
        for (Node<E> x = first; x != null; x = x.next)
            s.writeObject(x.item);
    }

    /**
     * Reconstitutes this {@code LinkedList} instance from a stream
     * (that is, deserializes it).
     */
    @SuppressWarnings("unchecked")
    @java.io.Serial
    private void readObject(java.io.ObjectInputStream s)
        throws java.io.IOException, ClassNotFoundException {
        // Read in any hidden serialization magic
        s.defaultReadObject();

        // Read in size
        int size = s.readInt();

        // Read in all elements in the proper order.
        for (int i = 0; i < size; i++)
            linkLast((E)s.readObject());
    }

• So the resulting serialization call stack should not run into any stack size limits as its call depth is only like below regardless of the number of entries:

  ->list.writeObject
       ->entry.writeObject

Root Cause

• This is a simple recursive scaling issue. Without an efficient serialization implementation, a custom data structure with many children elements will serialize by recursively calling writeObject of each next child iteratively. The larger the data structure and longer list of recursive children, then the deeper its recursive call stack will grow through its serialization.
• Without an efficient serialization implementation, the recursive data structure's serialization call likely has an O(n) depth where n is the number of entries:

  ->recursiveStructure.writeObject
        ->1st entry.writeObject
            ->2nd entry.writeObject
                ->3rd entry.writeObject
                    ...
                        ->nth entry.writeObject

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.