How to Filter Incoming Serialization Data in JBoss EAP 8 / 7.4

Solution Verified - Updated

Issue

  • How to Filter Incoming Serialization Data in JBoss EAP 8.x ?
  • How to Filter Incoming Serialization Data in JBoss EAP 7.4
  • We want to configure Java serialization filtering (https://docs.oracle.com/en/java/javase/11/core/serialization-filtering1.html) for an enterprise application. Clients connect to stateless session beans via the JBoss EJB client API, with the 'remote+http' protocol.
  • Where can we find documentation on JBoss serialization filtering?
  • How can serialization filtering for session beans be configured? I.e. can we allow/deny classes beyond the default configuration?

Resolution

With EAP 7.4 Update 13 and later, JBoss Marshalling will honor the JEP290 jdk.serialFilter filter setting.

  • If JVM-wide serial filter is set, it would be honored by both the native ObjectInputStreams and JBoss Marshalling unmarshallers.
  • If developer wants to set filter for specific unmarshaller, they can do so by setting MarshallingConfiguration.setUnmarshallingFilter() method. Underlying ObjectInputStream would be configured to use serial filter that delegates decisions to the configured unmarshalling filter.

The jdk.serialFilter can be configured in:

  • The JVM system property jdk.serialFilter
  • $JAVA_HOME/jre/lib/security/java.security

If the system property is set, it supersedes the value set in java.security.

Also, with EAP 7.4 Update 13 and later, EAP will default to the -Djdk.serialFilter to this configuration below to help protect against the heap exhaustion CVE-2023-3171.

JDK_SERIAL_FILTER="maxbytes=10485760;maxdepth=128;maxarray=100000;maxrefs=300000"

Users can change the default serial filter using an environment variable: JDK_SERIAL_FILTER, the value of this environment variable should be valid patterns defined by JEP290. Users can also disable the jdk.serialFilter by defining the environment variable DISABLE_JDK_SERIAL_FILTER to true. JDK_SERIAL_FILTER can be defined in $JBOSS_HOME/bin/standalone.conf / standalone.conf.bat , domain.conf / domain.conf.bat. This JDK_SERIAL_FILTER sets the JVM system property jdk.serialFilter when starting the EAP JVM.

Similarly set the environment variable DISABLE_JDK_SERIAL_FILTER=true in the $JBOSS_HOME/bin/standalone.conf / standalone.conf.bat , domain.conf / domain.conf.bat / Deployment or DeploymentConfig environment variable to disable the default JDK_SERIAL_FILTER. By default the jdk.serialFilter JVM argument is added through standalone.sh / domain.sh script only if DISABLE_JDK_SERIAL_FILTER is not set and the jdk.serialFilter JVM argument is not configured already.

#Snippet from standalone.sh / domain.sh that adds the default jdk.serialFilter JVM argument

    if [ "x$DISABLE_JDK_SERIAL_FILTER" = "x" -a "x$JDK_FILTER_SET" = "x" ]; then
        JDK_SERIAL_FILTER="${JDK_SERIAL_FILTER:-maxbytes=10485760;maxdepth=128;maxarray=100000;maxrefs=300000}"
        PREPEND_JAVA_OPTS="$PREPEND_JAVA_OPTS -Djdk.serialFilter=\"$JDK_SERIAL_FILTER\""
    fi

The $JAVA_HOME/jre/lib/security/java.security contains comments explaining the jdk.serialFilter configuration:

# A filter is configured as a sequence of patterns, each pattern is either
# matched against the name of a class in the stream or defines a limit.
# Patterns are separated by ";" (semicolon).
# Whitespace is significant and is considered part of the pattern.
#
# If the system property jdk.serialFilter is also specified, it supersedes
# the security property value defined here.
#
# If a pattern includes a "=", it sets a limit.
# If a limit appears more than once the last value is used. 
# Limits are checked before classes regardless of the order in the sequence of patterns.
# If any of the limits are exceeded, the filter status is REJECTED.
# 
#   maxdepth=value - the maximum depth of a graph
#   maxrefs=value  - the maximum number of internal references
#   maxbytes=value - the maximum number of bytes in the input stream
#   maxarray=value - the maximum array length allowed
# 
# Other patterns, from left to right, match the class or package name as
# returned from Class.getName. 
# If the class is an array type, the class or package to be matched is the element type.
# Arrays of any number of dimensions are treated the same as the element type.
# For example, a pattern of "!example.Foo", rejects creation of any instance or
# array of example.Foo.
#
# If the pattern starts with "!", the status is REJECTED if the remaining pattern
#   is matched; otherwise the status is ALLOWED if the pattern matches.
# If the pattern ends with ".**" it matches any class in the package and all subpackages.
# If the pattern ends with ".*" it matches any class in the package.
# If the pattern ends with "*", it matches any class with the pattern as a prefix.
# If the pattern is equal to the class name, it matches.
# Otherwise, the status is UNDECIDED.
# 
#jdk.serialFilter=pattern;pattern

Related Solutions

Diagnostic Steps

  • If you see rejections as a result of this filter, then the JDK provides additional filter logging that may be used for details about the rejected values. Note that EAP 7.4.13+ or 8.0.2+ are needed to enable this logging per DEBUG logging for java.io.serialization cannot be enabled in EAP 7. You can enable that in your logging subsystem with a logger added like below:
            <logger category="java.io.serialization">
                <level name="TRACE"/>
            </logger>
  • So then you can test with a filter limit and see the results and which values get rejected in a message like below. Note that the java.io.InvalidClassException: filter status: REJECTED exception you see may be on one server (the data sender) while the below DEBUG message on the rejection is on another (the data receiver) because the data sender logs the exception that it received from the receiver. So be sure to enable and check the trace for all involved JVMs:
ObjectInputFilter REJECTED: class com.package.ClassName, array length: -1, nRefs: 1, depth: 1, bytes: 77, ex: n/a
  • And then consider if you would wish to allow such a value and adjust the filter settings to meet details of the value as indicated in the message like above.
SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.