How to change the log level in the Vector in RHOCP 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Red Hat OpenShift Logging (RHOL)
  • 5
  • 6
• Vector

Issue

• Is it possible to change the log level in the collector type Vector?
• How can the log level being changed in Vector?

Resolution

The possible values that the Vector log level can take are: The possible values are trace, debug, info, warn, error and off.

Considerations:

• the name of the clusterLogging/clusterLogForwarder CR is instance
• The Vector Log Level will be set to debug
• The namespace is openshift-logging

The values of name clusterLogging/clusterLogForwarder CR, Vector Log Level and namespace could be different in the environment, then, the commands should be modified for them.

In RHOL 5.9 and older versions

The next steps are for doing troubleshooting and it's needed to go through Unmanaged status. Read considerations about the Unmanaged status in the documentation section Unsupported configurations.

Note: For Multi CLF (Cluster Log Forwarder) configuration, refer to the pre-requisites described in solution/How to configure collector resources for Multi Log Forwarder. Once in place, apply the steps from this solution to the target ClusterLogging CR.

Step 1: Temporarily set ClusterLogging (CL) Custom Resource (CR) to Unmanaged

$  oc -n openshift-logging patch clusterlogging/instance -p '{"spec":{"managementState": "Unmanaged"}}' --type=merge

Step 2: Set the Vector Log Level

Edit the daemonset:

$ oc -n openshift-logging edit ds/collector

And add/modify in spec.template.spec.containers.env the environment variable to define the log level desired to set to Vector:

...
    spec:
      containers:
      - args:
        - /usr/bin/run-vector.sh
        command:
        - sh
        env:
        - name: VECTOR_LOG  <---------- the next two lines set an environment variable called VECTOR_LOG and with value debug
          value: debug

Step 3: Verify the collectors are running with the log level desired

After applying the change in Step 2, the collectors pods should be restarted automatically taking the environment variable VECTOR_LOG. Verify the collectors have restarted:

$ oc get pods -l component=collector -n openshift-logging

Verify the collector pod is using the log level debug:

$ collector=$(oc get pods -l component=collector -o jsonpath='{.items[0].metadata.name}' -n openshift-logging)
$ oc logs $collector -c collector -n openshift-logging|grep -c DEBUG
1643

Return ClusterLogging (CL) Custom Resource (CR) to Managed

$ oc -n openshift-logging patch clusterlogging/instance -p '{"spec":{"managementState": "Managed"}}' --type=merge

In RHOL 6

Step 1: Set the Vector Log Level

Patch the clusterLogForwarder CR and add the annotation with the desired log level. Once it's done, the collector pods will be restarted automatically by the ClusterLogging Operator. In the next example command, the log level will be set to debug:

$ oc patch obsclf/instance --type='json' --patch="[{'op': 'add', 'path': '/metadata/annotations', 'value': {'observability.openshift.io/log-level': 'debug'}}]" -n openshift-logging

Step 2: Verify the collectors are running with the log level desired

Verify the ClusterLogging Operator has modified the collector daemonset setting the Vector Log Level desired:

$ oc get ds/instance -o yaml -n openshift-logging |grep -A 1 VECTOR_LOG

After changing the Vector Log Level, the collectors pods should be restarted automatically taking the environment variable VECTOR_LOG. Verify the collectors have restarted:

$ oc get pods -l app.kubernetes.io/component=collector -n openshift-logging

Verify the collector pod is using the log level debug:

$ collector=$(oc get pods -l app.kubernetes.io/component=collector -o jsonpath='{.items[0].metadata.name}' -n openshift-logging)
$ oc logs $collector -c collector -n openshift-logging|grep -c DEBUG
1643

Step 3. Return to the original Vector Log Level

Remove the annotation added for setting the Vector Log Level:

$ oc patch obsclf/instance --type='json' --patch="[{'op': 'remove', 'path': '/metadata/annotations', 'value': {'observability.openshift.io/log-level'}}]" -n openshift-logging
clusterlogforwarder.observability.openshift.io/collector patched

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.