BUFFER_QUEUE_LIMIT variable in logging-fluentd DaemonSet is not honoured

Environment

OCP 3.x

Issue

After increasing the fluentd loglevel, we have found that fluentd pod is not honoring the env variable BUFFER_QUEUE_LIMIT, because of this, we can see the following error:

2018-06-07 11:47:02 +0200 [debug]: buffer queue is full. Wait 1 second to re-emit events
2018-06-07 11:47:04 +0200 [debug]: buffer queue is full. Wait 1 second to re-emit events
2018-06-07 11:47:05 +0200 [debug]: buffer queue is full. Wait 1 second to re-emit events
2018-06-07 11:47:06 +0200 [debug]: buffer queue is full. Wait 1 second to re-emit events
2018-06-07 11:47:07 +0200 [debug]: buffer queue is full. Wait 1 second to re-emit events
2018-06-07 11:47:08 +0200 [debug]: buffer queue is full. Wait 1 second to re-emit events

Resolution

Increase value of variable FILE_BUFFER_LIMIT to, at least, the same value set for BUFFER_SIZE_LIMIT.

Root Cause

When calculating BUFFER_QUEUE_LIMIT, we Content from github.com is not included.check the disk size as well as the FILE_BUFFER_LIMIT and pick the smaller value, which is divided by BUFFER_SIZE_LIMIT.

This issue has been reported in This content is not included.BZ 1588430.

Diagnostic Steps

• Check variables set within the fluetnd pod and confirm:

pods logging-fluentd-85pt2, container fluentd-elasticsearch
K8S_HOST_URL=https://kubernetes.default.svc.cluster.local
ES_HOST=logging-es
ES_PORT=9200
ES_CLIENT_CERT=/etc/fluent/keys/cert
ES_CLIENT_KEY=/etc/fluent/keys/key
ES_CA=/etc/fluent/keys/ca
OPS_HOST=logging-es
OPS_PORT=9200
OPS_CLIENT_CERT=/etc/fluent/keys/cert
OPS_CLIENT_KEY=/etc/fluent/keys/key
OPS_CA=/etc/fluent/keys/ca
JOURNAL_SOURCE=
JOURNAL_READ_FROM_HEAD=
BUFFER_QUEUE_LIMIT=1024
BUFFER_SIZE_LIMIT=32m
# FLUENTD_CPU_LIMIT from resource field fluentd-elasticsearch/limits.cpu
# FLUENTD_MEMORY_LIMIT from resource field fluentd-elasticsearch/limits.memory
FILE_BUFFER_LIMIT=256Mi

• Set fluetnd in debug mode and restart the fluentd pods. You should see how the value set for BUFFER_QUEUE_LIMIT is not being honored.

  <label @OUTPUT>
    <match **>
      @type copy
      <store>
        @type elasticsearch
        host logging-es
        port 9200
        scheme https
        target_index_key viaq_index_name
        id_key viaq_msg_id
        remove_keys viaq_index_name
        user fluentd
        password xxxxxx
        client_key /etc/fluent/keys/key
        client_cert /etc/fluent/keys/cert
        ca_file /etc/fluent/keys/ca
        type_name com.redhat.viaq.common
        reload_connections false
        reload_on_failure false
        flush_interval 1s
        max_retry_wait 300
        disable_retry_limit true
        buffer_type file
        buffer_path /var/lib/fluentd/buffer-output-es-config
        buffer_queue_limit 8    <<-------------------------------------
        buffer_chunk_limit 33554432
        buffer_queue_full_action block
        write_operation create
        request_timeout 2147483648
      </store>
    </match>
  </label>

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.