Multi log forwarder feature use cases

The goal of this article is to detail some uses cases for the multi log forwarder feature released with Red Hat OpenShift Logging 5.8.

Prerequisites

• Red Hat OpenShift Logging 5.8 or newer
• Multi Log Forwarder must run the collector type: Vector. This means, the clusterLogging CR not having the name of "instance" must have the spec.collection.type: vector

For what's not the multi log forwarder feature?

It's not for delegating to the OpenShift Platform users/project admins the log collection. The OpenShift admin should have the control of the log collection.

A security warning exists in the Red Hat Knowledge article "Why don't delegate to project admins the management of the Logging collectors in RHOCP 4".

For what can the multi log forwarder feature be used?

1. For reducing the number of log duplicated when migrating from Fluent to Vector
2. For testing Vector meanwhile Fluentd is used as production
3. For isolating inputs from outputs for avoiding the described in the article "Collector stops of log forwarding when one of the outputs reaches the queue size limit in RHOCP 4"

Other use cases exist, but not all are listed here.

Scenario 1. For reducing the number of log duplicated when migrating from Fluent to Vector

This scenario is detailed in the article "Migrating the log collector from fluentd to vector reducing the number of logs duplicated in RHOCP 4"

Scenario 2. For testing Vector meanwhile Fluentd is used as production

Current Legacy implementation with Fluentd

The current configuration is using the Legacy implementation. As not clusterLogForwarder, then, all the application and infrastructure logs are collected:

$ oc -n openshift-logging get clusterlogging instance -o yaml
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  creationTimestamp: "2024-07-01T12:32:18Z"
  generation: 2
  name: instance
  namespace: openshift-logging
  resourceVersion: "656845"
  uid: 2e7d3785-8411-4eb5-b0f1-9437f055f71f
spec:
  collection:
    type: fluentd
    resources:
      limits:
        cpu: 200m
        memory: 1024Mi
      requests:
        cpu: 200m
        memory: 256Mi

--- OUTPUT OMMITED ---

Create the clusterRoleBinding

NOTE: for all the details, read Log Forwarding

Create the serviceAccount to be used in the Log Forwarder:

$ oc -n openshift-logging create sa fluentd2vector

Starting in Red Hat Logging v5.8, three new clusterRoles exists, one for each type of log to be desired to collect:

• collect-application-logs: allows to collect application logs
• collect-infrastructure-logs: allows to collect infrastructure logs
• collect-audit-logs: allows to collect audit logs

In this example, it will be created the clusterRoleBinding for giving permissions to collect the application and insfructure logs. In case that also collecting the audit logs, it will be needed to create the clusterRoleBinding to the clusterRole: collect-audit-logs

$ oc create clusterrolebinding collect-app-logs-fluentd2vector --clusterrole=collect-application-logs --serviceaccount openshift-logging:fluentd2vector
$ oc create clusterrolebinding collect-infra-logs-fluentd2vector --clusterrole=collect-infrastructure-logs --serviceaccount openshift-logging:fluentd2vector

Verify that the clusterRolebinding are created:

$ oc get clusterrolebinding collect-app-logs-fluentd2vector 
NAME               ROLE                                   AGE
collect-app-logs   ClusterRole/collect-application-logs   6m1s 

$ oc get clusterrolebinding collect-infra-logs-fluentd2vector 
NAME                          ROLE                                      AGE
collect-infrastructure-logs   ClusterRole/collect-infrastructure-logs   4m59s

Create the clusterLogging CR

Create a clusterLogging CR for setting the resources: limits and requests and in case that needed, the nodeSelector and/or taints. In this example, it will be called the ClusterLogging CR: migration:

$ cat << EOF | oc create -f - 
apiVersion: logging.openshift.io/v1
kind: ClusterLogging
metadata:
  name: migration
  namespace: openshift-logging
spec:
  collection:
    type: vector
    resources:
      limits:
        cpu: 200m
        memory: 1024Mi
      requests:
        cpu: 200m
        memory: 256Mi
EOF

Create the clusterLogForwarder CR for collecting the same type logs that used by Fluentd. The clusterLogForwarder must have the same name that the clusterLogging CR. In this example is migration and log forwarding to an output where it's not the default for verifying that it works:

$ cat << EOF |oc create -f -
apiVersion: "logging.openshift.io/v1"
kind: ClusterLogForwarder
metadata:
  name: migration 
  namespace: openshift-logging 
spec:
  serviceAccountName: fluentd2vector
  outputs:
  - name: receiver
    type: "http"
    url: http://receiver.example.com:1234
  pipelines: 
   - name: all
     inputRefs:
     - application
     - infrastructure
     outputRefs:
     - receiver
EOF

Scenario 3. For isolating inputs from outputs for avoiding the described in the article "Collector stops of log forwarding when one of the outputs reaches the queue size limit in RHOCP 4"

As consequence of having issues for log forwarding to one of the outputs/destinations defined in the clusterLogForwarder and if the action to take when the buffer gets full is block (read the article "Tuning log payloads and delivery in RHOCP 4" for details for Vector), Fluentd and Vector block reading logs for all the outputs. This means that no logs are sent to all the outputs defined in the same clusterLogForwarder. Similar to the described in the article "Fluentd stops of log forwarding when one of the outputs reaches the totalLimitSize in RHOCP 4".

For avoiding this situation, it could be possible to have separated clusterLogForwarder where the clusterLogForwarder CR instance is used for sending to the Red Hat Log Store and the other clusterLogForwarder CR configured for log forwarding to other outputs.

This has a penalty, and it's that if the logs sent to the Red Hat Log Store and to the custom output are the same, for example, application logs, they will be read twice. Once for each collector from each clusterLogForwarder configured.