Fluentd pods not able to send application logs to kibana

Solution Verified - Updated

Environment

  • OpenShift Container Platform 3.6
  • logging-fluentd-v3.6.173.0.21

Issue

Fluent pods are not able to send the application logs to elasticsearch server. But send the system logs to the elasticsearch. Able to see the other logs in kibana. I deployed the hello-openshift as the application pod and it is running fine.
Logs are accessible either from the OpenShift Web Console -> Pod -> Logs or by using oc logs <pod>

Resolution

This problem was investigated in This content is not included.bugzila#1494612. It is resolved with errata RHBA-2018:0113. The following steps are workarounds for the affected version but it is strongly recommended to instead update the logging stack as described in the errata.

Workaround

  • Remove logging stack

  • Set the following variable in Ansible inventory file:
    3.6

    openshift_logging_image_version=v3.6.173.0.5

    3.7 

    openshift_logging_image_version=v3.7.23

  • Install logging

  • If any fluentd pods show below error:

" error_class="TypeError" error="no implicit conversion of nil into String" plugin_id=""
  • Stop fluentd:
# oc label node $nodename logging-infra-fluentd-
  • Delete stale buffer files on that node in: /var/lib/fluentd
  • Start fluentd
# oc label node $nodename logging-infra-fluentd=true

Alternate Workaround


It seems to affect only `v3.6.173.0.21` therefore until this problem is resolved users should be able to use the previous version as **workaround**

  1. Set the ansible variable in the inventory file: openshift_logging_image_version=v3.6.173.0.5

  2. Install logging

Diagnostic Steps

  • Check logging-fluentd versions, affected ones are v3.6.173.0.21
$ oc get po -l component=fluentd -o 'go-template={{range $pod := .items}}{{if eq $pod.status.phase "Running"}}{{range $container := $pod.spec.containers}}oc exec -c {{$container.name}} {{$pod.metadata.name}} -- find /root/buildinfo -name Dockerfile-openshift* | grep -o logging.* {{"\n"}}{{end}}{{end}}{{end}}' | bash -
logging-fluentd-v3.6.173.0.21
logging-fluentd-v3.6.173.0.21
  • Check logs for logging-fluentd pods and search for errors similar to these ones:
2017-10-06 20:57:28 +0800 [warn]: temporarily failed to flush the buffer. next_retry=2017-10-06 20:57:30 +0800 error_class="TypeError" error="no implicit conversion of nil into String" plugin_id="object:1863278"
  2017-10-06 20:57:28 +0800 [warn]: suppressed same stacktrace
2017-10-06 20:57:30 +0800 [warn]: temporarily failed to flush the buffer. next_retry=2017-10-06 20:57:34 +0800 error_class="TypeError" error="no implicit conversion of nil into String" plugin_id="object:1863278"
  2017-10-06 20:57:30 +0800 [warn]: suppressed same stacktrace
2017-10-06 20:57:34 +0800 [warn]: temporarily failed to flush the buffer. next_retry=2017-10-06 20:57:42 +0800 error_class="TypeError" error="no implicit conversion of nil into String" plugin_id="object:1863278"
  2017-10-06 20:57:34 +0800 [warn]: suppressed same stacktrace
2017-10-06 20:57:42 +0800 [warn]: temporarily failed to flush the buffer. next_retry=2017-10-06 20:57:58 +0800 error_class="TypeError" error="no implicit conversion of nil into String" plugin_id="object:1863278"
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.