ClusterLogForwarder is not sending full audit logs to external rsyslog

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Red Hat OpenShift Logging (RHOL)
  • 5
  • 6
• Fluentd
• Vector
• ClusterLogForwarder
• Syslog protocol

Issue

• Enabled ClusterLogForwarder to send audit logs to external rsyslog log aggregator, but full audit logs are not appearing.

• Checking the OpenShift audit logs on the node using oc node-logs, full log entries are shown.

• The output send from the ClusterLogForwarder, only the following is shown:

  fluentd: type=ANOM_PROMISCUOUS msg=audit(xxxxxxxxxx.989:205): dev=vethxxxxxxxx prom=0 old_prom=256 auid=4xxxxxxx5 uid=800 gid=801 ses=4xxxxxxx 5AUID="unset" UID="openvswitch" GID="xxxxxfs"
  

• The collector is not forwarding the logs to external rsyslog server when RFC5424 is used with ClusterLogForwarder custom resource.

• The payloadKey syslog parameter is configured to message to capture audit messages.

Resolution

Most of the audit data is being dropped because of the payloadKey config:

    payloadKey: message

Remove the payloadKey entry to allow the full log to arrive. The OpenShift Logging operator will redeploy the collector pods with the new configuration.

Root Cause

As per the syslog parameters documentation:

Configuring the `payloadKey` parameter prevents other parameters from being forwarded to the syslog.

Looking at the logs in Kibana from a default deployment, other parts of the log actually show significantly more information than the message field. For instance, the annotations.authorization.k8s.io/reason key shows:

RBAC: allowed by ClusterRoleBinding "system:openshift:operator:kube-controller-manager-operator" of ClusterRole "cluster-admin" to ServiceAccount "kube-controller-manager-operator/openshift-kube-controller-manager-operator"

Diagnostic Steps

• Verify the audit logs are being generated:

  $ oc adm node-logs <node> --path openshift-apiserver/audit.log
  

• Check the ClusterLogForwarder configuration:

  $ oc get clusterlogforwarder instance -n openshift-logging -o yaml
  [...]
     type: syslog
     syslog:
       [...]
       payloadKey: message        <==== This should not be here
       [...]
  

• Try using a default configuration to verify if the logs appear in Kibana.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.