Enabling hotrod and REST access logs in Red Hat Datagrid 8.x

Environment

• Red Hat Datagrid
  • 8.x

Issue

• How to enable hotrod access logs in JDG 8.x?
• How to enable REST access logs in JDG 8.x
• Is restart necessary after configuring changes for enabling access logs in RHDG 8.x server?

Resolution

Hot Rod and REST endpoints can record all inbound client requests as log entries with the following categories:

However, access logs for Hot Rod and REST endpoints are disabled by default. Therefore, to enable either logging category, set the level to TRACE in the Data Grid logging configuration file $RHDG_HOME/${infinispan.server.root}/conf/log4j2.xml, in local deployment for instance.

For Hotrod access logs, follow below example:

#### Change from INFO to TRACE as below:
<Logger name="org.infinispan.HOTROD_ACCESS_LOG" additivity="false" level="TRACE">
   <AppenderRef ref="HR-ACCESS-FILE"/>
</Logger>

For REST access logs, follow below example:

 <Logger name="org.infinispan.REST_ACCESS_LOG" additivity="false" level="TRACE">
      <AppenderRef ref="REST-ACCESS-FILE"/>
    </Logger>

After making the changes above, restart the RHDG server.
This will create hotrod-access.log and rest-access.log in the default path: $RHDG_HOME/${infinispan.server.root}/log

And the default format for access logs is as follows:

`%X{address} %X{user} [%d{dd/MMM/yyyy:HH:mm:ss Z}] "%X{method} %m %X{protocol}" %X{status} %X{requestSize} %X{responseSize} %X{duration}%n`

The preceding format creates log entries such as the following:

127.0.0.1 - [DD/MM/YYYY:HH:MM:SS +0000] "PUT /rest/v2/caches/default/key HTTP/1.1" 404 5 77 10

Via CLI commands

For setting via CLI command, one can do:

#Hot rod access logs
logging set --level=TRACE org.infinispan.HOTROD_ACCESS_LOG
#Rest access logs
logging set --level=TRACE org.infinispan.REST_ACCESS_LOG

For more information see 8.23. LOGGING(1) - in Chapter 8 - Command Reference RHDG 8. This can be done in OCP environment, however it will not be persistent when the pod scaled-down or terminated, for instance - specifically for OCP see the section below:

In OCP 4 environment:

To specify logging configuration do so in the spec.logging section in the Infinispan CR - as described in Red Hat Data Grid8.0Running Data Grid on OpenShiftChapter 11. Monitoring Data Grid Logs, see below:

spec:
  ...
  logging: 
    categories: 
      org.infinispan: debug 
      org.jgroups: debug
      org.infinispan.REST_ACCESS_LOG: trace
      org.infinispan.HOTROD_ACCESS_LOG: trace

The above procedure will be persistent, since it is on the Infinispan CR directly, not on the pod.

Root Cause

Interpreting Hot Rod Access Logs:

Default access logs

The default pattern is ("%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c] %m%throwable%n"/>) and results on (03:19:15,552 TRACE (non-blocking-thread--p2-t2) [org.infinispan.HOTROD_ACCESS_LOG] /-/-). And this means:

-5 means 5 spaces.

Setting an access logs pattern:

<PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c] %X{address} %X{user} [%d{dd/MMM/yyyy:HH:mm:ss Z}] %X{method} %m %X{protocol}  %X{status} %X{requestSize} %X{responseSize} %X{duration}%n"/> 

This will result in:

org.infinispan.HOTROD_ACCESS_LOG] 127.111.0.11 - [07/Apr/2023:22:59:36 +0000] PUT /counter.war/[B0x010403002D9801028A01270A2560C801..[50] HOTROD/3.0  OK 81 6 2

And the interpretation of it is given by:

How to apply the access log pattern:

Apply the following log4j.xml file:

            <PatternLayout pattern="%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p (%t) [%c] %X{address} %X{user} [%d{dd/MMM/yyyy:HH:mm:ss Z}] %X{method} %m %X{protocol}  %X{status} %X{requestSize} %X{responseSize} %X{duration}%n"/>
...
          <Logger name="org.infinispan.HOTROD_ACCESS_LOG" additivity="true" level="TRACE">
             <AppenderRef ref="HR-ACCESS-FILE"/>
          </Logger>
        </Loggers>

For Operator, see solution Using custom configuration in DG 8 via Operator.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.