Collector in CrashLoopBackOff when defined an output that not used in the pipelines in RHOL 6

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Red Hat OpenShift Logging (RHOL)
  • 6
• Vector

Issue

• Collector pods are going in CrashLoopBackOff state when there is unused outputs defined in the ClusterLogForwarder(CLF) in RHOL6, but not an error visible in the clusterLogForwarder custom resource status section:

  $ oc get pods -l app.kubernetes.io/component=collector -n openshift-logging
  NAME              READY   STATUS             RESTARTS       AGE
  collector-796xd   0/1     CrashLoopBackOff   33 (85s ago)   144m
  collector-bkswb   0/1     CrashLoopBackOff   33 (86s ago)   144m
  collector-gjct7   0/1     CrashLoopBackOff   33 (56s ago)   144m
  collector-htq46   0/1     CrashLoopBackOff   33 (32s ago)   144m
  collector-rnw5w   0/1     CrashLoopBackOff   33 (72s ago)   144m
  

• Collector pods throw the error=Transform "<normalized output name> " has no inputs has no inputs error:

  $ oc logs collector-796xd 
  Creating the directory used for persisting Vector state /var/lib/vector/openshift-logging/collector
  Starting Vector process...
  2025-02-20T18:24:16.695224Z ERROR vector::cli: Configuration error. error=Transform "output_rsyslog_parse_encoding" has no inputs
  

Resolution

Red Hat investigated this issue in bug report:

If this issue still occurs in the environment after updating, open a support case in the Red Hat Customer Portal referring to this solution.

Workaround

Review the outputs defined in the clusterLogForwarder custom resource not used in the pipelines and:

• or remove the not used output
• or use the output in the pipeline for log forwarding logs to it

Root Cause

The operator could deploy the collector with output configurations that were not referenced by any inputs leading to the collector pods to be in CrashLoopBackOff.

Diagnostic Steps

Verify that the collector pods are in CrashLoopBackOff status:

$ oc get pods -l app.kubernetes.io/component=collector -n openshift-logging
NAME              READY   STATUS             RESTARTS       AGE
collector-796xd   0/1     CrashLoopBackOff   33 (85s ago)   144m
collector-bkswb   0/1     CrashLoopBackOff   33 (86s ago)   144m
collector-gjct7   0/1     CrashLoopBackOff   33 (56s ago)   144m
collector-htq46   0/1     CrashLoopBackOff   33 (32s ago)   144m
collector-rnw5w   0/1     CrashLoopBackOff   33 (72s ago)   144m

Verify that the collector pods are throwing the error=Transform "<normalized output name>" has no inputs error:

 $ oc logs collector-796xd 
Creating the directory used for persisting Vector state /var/lib/vector/openshift-logging/collector
Starting Vector process...
2025-02-20T18:24:16.695224Z ERROR vector::cli: Configuration error. error=Transform "output_rsyslog_parse_encoding" has no inputs

Verify if there are any unused outputs defined in the CLF configuration. In the below example syslog is defined in the outputs section, but never used in the pipelines:

apiVersion: observability.openshift.io/v1
kind: ClusterLogForwarder
metadata:
  name: collector
  namespace: openshift-logging
spec:
  managementState: Managed
  outputs:
    - lokiStack:
        authentication:
          token:
            from: serviceAccount
        target:
          name: logging-loki
          namespace: openshift-logging
      name: default-lokistack
      tls:
        ca:
          configMapName: openshift-service-ca.crt
          key: service-ca.crt
      type: lokiStack
    - name: rsyslog
      syslog:
        facility: auth
        rfc: RFC3164
        severity: informational
        url: 'udp://syslog.example.com:514'
      type: syslog
  pipelines:
    - inputRefs:
        - audit
      name: syslog
      outputRefs:
        - default-lokistack
    - inputRefs:
        - infrastructure
        - application
      name: logging-loki
      outputRefs:
        - default-lokistack
    - inputRefs:
        - application
      name: container-logs
      outputRefs:
        - default-lokistack
  serviceAccount:
    name: collector

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.