EAP 7 logging mechanism and customization in OCP 4

Solution Verified - Updated 18 Jun 2024

Environment

Red Hat JBoss Enterprise Application Platform (EAP)
- 7.x
Red hat OpenShift Container Platform (OCP)
- 4.x

Issue

How does EAP 7 logging mechanism work?
How to customize EAP 7 image logging?

Resolution

The default EAP 7.4 will bring the server logs in the pod logs - instead of a a separated file server.log:

        <subsystem xmlns="urn:jboss:domain:logging:8.0">
            <console-handler name="CONSOLE"> <!-- console handler === CONSOLE -->
                <formatter>
                    <named-formatter name="COLOR-PATTERN"/> <!-- handler CONSOLE has one formatter COLOR-PATTERN -->
                </formatter>
            </console-handler> <!-- console handler === CONSOLE -->
            <logger category="com.arjuna">
                <level name="WARN"/>
            </logger>
            <logger category="io.jaegertracing.Configuration">
                <level name="WARN"/>
            </logger>
            <logger category="org.jboss.as.config">
                <level name="DEBUG"/>
            </logger>
            <logger category="sun.rmi">
                <level name="WARN"/>
            </logger>
            <root-logger>
                <level name="INFO"/>
                <handlers>
                    <handler name="CONSOLE"/> <!-- ROOT logger points to handler === CONSOLE -->
                </handlers>
            </root-logger>
            <formatter name="COLOR-PATTERN"> <!-- COLOR-PATTERN is defined as below -->
                <pattern-formatter pattern="%K{level}%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%e%n"/>
            </formatter>
            <formatter name="OPENSHIFT">
                <json-formatter>
                    <exception-output-type value="formatted"/>
                    <key-overrides timestamp="@timestamp"/>
                    <meta-data>
                        <property name="@version:" value="1"/>
                    </meta-data>
                </json-formatter>
            </formatter>
        </subsystem>

Comparing with the default EAP 7:

        <subsystem xmlns="urn:jboss:domain:logging:8.0">
            <console-handler name="CONSOLE"> <!-- handler CONSOLE has one formatter COLOR-PATTERN -->
                <level name="INFO"/>
                <formatter>
                    <named-formatter name="COLOR-PATTERN"/>
                </formatter>
            </console-handler>
            <periodic-rotating-file-handler name="FILE" autoflush="true"> <!-- handler FILE has one formatter PATTERN -->
                <formatter>
                    <named-formatter name="PATTERN"/>
                </formatter>
                <file relative-to="jboss.server.log.dir" path="server.log"/>
                <suffix value=".yyyy-MM-dd"/>
                <append value="true"/>
            </periodic-rotating-file-handler>
            <logger category="com.arjuna">
                <level name="WARN"/>
            </logger>
            <logger category="io.jaegertracing.Configuration">
                <level name="WARN"/>
            </logger>
            <logger category="org.jboss.as.config">
                <level name="DEBUG"/>
            </logger>
            <logger category="sun.rmi">
                <level name="WARN"/>
            </logger>
            <root-logger>
                <level name="INFO"/>
                <handlers>
                    <handler name="CONSOLE"/>  <!-- ROOT logger points to handler === CONSOLE -->
                    <handler name="FILE"/>  <!-- ROOT logger points to handler === FILE -->
                </handlers>
            </root-logger>

EAP 7 standalone (`standalone.xml`) vs EAP 7 image (`standalone-openshift.xml`):

The difference is simple: in OCP the root-logger points only to CONSOLE handler and has level INFO:

### Default EAP 7 image standalone-openshift.xml
            <root-logger>
                <level name="INFO"/>
                <handlers>
                    <handler name="CONSOLE"/> <!-- ROOT logger points to handler === CONSOLE -->
                </handlers>
            </root-logger>
### Default EAP 7 standalone.xml:
            <root-logger>
                <level name="INFO"/>
                <handlers>
                    <handler name="CONSOLE"/>  <!-- ROOT logger points to handler === CONSOLE -->
                    <handler name="FILE"/>  <!-- ROOT logger points to handler === FILE -->
                </handlers>
            </root-logger>

Defining logging mechanism

The process to defined logging is: on the ROOT logger, define a handler, and on this handler set the level and the formatter:

ROOT logger > handler > formatter

Customizing logging

In EAP 7 images, one can set LOGGER_CATEGORIES=io.undertow:TRACE for example to set trace logging level for undertow or other system - this feature is there since EAP 7.

Example:

envs:
  - name: "LOGGER_CATEGORIES"
    example: "com.my.package:TRACE,com.my.other.package:ALL"
    description: Comma separated list of <logger-name>:<level>

Example:

### set:
$ oc set env dc/eap-app LOGGER_CATEGORIES=io.undertow:TRACE,org.jgroups.protocols.DNS_PING:TRACE
deploymentconfig.apps.openshift.io/eap-app updated
### confirm:
$ oc get dc/eap-app -o yaml | grep -A2 LOGGER_CATEGORIES
        - name: LOGGER_CATEGORIES
          value: io.undertow:TRACE,org.jgroups.protocols.DNS_PING:TRACE

This can be set on the deployment/deploymentconfig or on the helm charts.
The environment variable is executed/used at runtime when /opt/eap/bin/opensift-launch.sh is called.

Other options: A second and third way to change the logs, one can use a custom standalone-openshift.xml or one use the jboss-cli command in a actions.cli and call it from a postconfigure.sh file, as described in here.
Note that doing any change via jboss-cli will be ephemeral and will be vanished the next restart.
Meaning one can do the process (add trace logs) as thoroughly described here for us and it won't be persistent == ephemeral;

Enabling server.log file in EAP 7 in OCP 4

To enable a server.log file, use a rotating file handler:

/subsystem=logging/size-rotating-file-handler=SIZEDFILE:add(file={path="server.log",relative-to="jboss.server.log.dir"},rotate-size=10m,max-backup-index=100)
/subsystem=logging/root-logger=ROOT:add-handler(name=SIZEDFILE)
/subsystem=logging/root-logger=ROOT:remove-handler(name=FILE) ---> this is optional if the image doesn't bring FILE handler

This will do the following configuration changes:

 <size-rotating-file-handler name="SIZEDFILE">
                <file relative-to="jboss.server.log.dir" path="server.log"/>
                <rotate-size value="10m"/>
                <max-backup-index value="100"/>
            </size-rotating-file-handler>
...
            <root-logger>
                <level name="INFO"/>
                <handlers>
                    <handler name="CONSOLE"/>
                    <handler name="SIZEDFILE"/>
                </handlers>
            </root-logger>

For a non-rotating logger, the command is simpler FILE configuration - /subsystem=logging/file-handler=SIZEDFILE:add(file ...

Root Cause

EAP 7.4 image doesn't write the server logs in a server.log file, one needs to see the pod logs, as explained on Troubleshooting JBoss EAP 7 issues in OCP 3/4.
To define a logger, define a handler with a formatter and a level of logging, define a formatter with name and format, and set the root logger to that handler:

root logger > handler > formatter

As a consequence of the above - and the CONSOLE handler explained above, setting JAVA_OPTS_APPEND=jboss.server.log.dir = <location> will not make the server.log be written into a custom location. To accomplish this, follow the resolution above, by changing the rooter-logger.

For more details on customization of the handler see Configuring size-rotating-file-handler in JBoss EAP 6/7.

Diagnostic Steps

To show the logging mechanism (handler) is being used - use jboss-cli.sh - example below shows the root-logger points to the console handler:

$ oc rsh $pod_name
[$pod_name]$ /opt/eap/bin/jboss-cli.sh -c
[standalone@localhost:9990 /] /subsystem=logging/root-logger=ROOT:read-resource()
{
    "outcome" => "success",
    "result" => {
        "filter" => undefined,
        "filter-spec" => undefined,
        "handlers" => ["CONSOLE"],
        "level" => "INFO"
    }
}

Confirm the setting of LOGGER_CATEGORIES:

INFO Found env LOGGER_CATEGORIES, configuring....
INFO Configuring logger category io.undertow with level TRACE
INFO Configuring logger category org.jgroups.protocols.DNS_PING with level TRACE
INFO Access log is disabled, ignoring configuration.
INFO Configuring the server using embedded server
INFO Duration: 2145 milliseconds
INFO Running jboss-eap-7/eap73-openjdk11-runtime-openshift-rhel8 image, version 7.3.10

SBR

JBoss Base AS

Product(s)

Red Hat JBoss Enterprise Application Platform

Components

jbossas

Category

Customize or extend

Tags

eap
jboss

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.