Tuning log payloads and delivery in RHOCP 4

Environment

• Red Hat OpenShift Container Platform (RHOCP)
  • 4
• Red Hat OpenShift Logging (RHOL)
  • 5.9
• Collector
• Vector

Issue

• What is the meaning of AtLeastOnce and AtMostOnce?
• What are the default values of compression, maxWrite, minRetryDuration and maxRetryDuration?
• What's the retry policy?
• What happens when the buffer gets full?

Resolution

Disclaimer: Links contained herein to external website(s) are provided for convenience only. Red Hat has not reviewed the links and is not responsible for the content or its availability. The inclusion of any link to an external website does not imply endorsement by Red Hat of the website or their entities, products or services. You agree that Red Hat is not responsible or liable for any loss or expenses that may result due to your use of (or reliance on) the external site or content.

Default values

Let's translate the ClusterLogForwarder's variables name to Vector's variables name:

Review for each Vector Content from vector.dev is not included.sink the default values. For instance, for the Content from vector.dev is not included.HTTP sink the default values are:

What's the difference between AtLeastOnce and AtMostOnce?

As it's now known, the atMostOnce means buffer.type: memory and AtLeastOnce means buffer.type: disk. Then, what's the real meaning of this? Read the upstream article Content from vector.dev is not included."Guarantees" for having more information and taking a better decision about what delivery (buffer.type) choose.

But, as it's read in the upstream documentation for the HTTP sink, Content from vector.dev is not included.buffer.type means:

delivery: AtLeastOnce or buffer.type: disk: Events are buffered on disk. This is less performant, but more durable. Data that has been synchronized to disk will not be lost if Vector is restarted forcefully or crashes. Data is synchronized to disk every 500ms.
delivery: AtMostOnce or buffer.type: memory: Events are buffered in memory. This is more performant, but less durable. Data will be lost if Vector is restarted forcefully or crashes.

It's important to highlight that AtLeastOnce means that the event received by a Vector component is delivered at least once.

How many events will be stored? What's the size of the buffer?

It depends on what type of delivery (buffer.type) mode is used. Depending on it, it's needed to review the default values for the Content from vector.dev is not included.outputs (sink) where are log forwarded the logs as Red Hat is not introducing different values of the used by default for the upstream Vector tool.

Let's take again the example of the HTTP sink and review the Content from vector.dev is not included.buffer definition

When the delivery: AtMostOnce - buffer.type: memory is used, then, the size of the buffer is disposed by the value of the Vector's variable Content from vector.dev is not included.buffer.max_events. The default value is 500.

When the delivery: AtLeastOnce - buffer.type: disk is used, then, the size of the buffer is disposed by the value of Vector variable Content from vector.dev is not included.buffer.max_size. The default is 256 megabytes. Red Hat is Content from github.com is not included.hard coding this value for being the same value that Vector has.

These values are per output/Vector sink. Then, if having two different outputs defined in the ClusterLogForwarder each output would have its specific buffer and delivery definition.

What happens when the buffer gets full?

It's needed to remember that the Red Hat Logging stack is not a This page is not included, but the link has been rewritten to point to the nearest parent document.guaranteed log sink, then, it's always possible to miss/lose some logs.

Then, what happens when the buffer get's full?

When setting delivery: AtMostOnce is translated to the Vector's value: Content from github.com is not included.buffer.when_full: drop_newest and the buffer gets full, the newest events are discarded.

When setting delivery: AtLeastOnce or not set the clusterLogForwarder.spec.outputs[].delivery, it is translated to the Vector's value: Content from github.com is not included.buffer.when_full: block. Then, when the buffer to disk gets full, the reading of more logs for all the outputs is blocked, meaning that not more events log forwarder to any output defined in the same clusterLogForwarder instance, similar to the described in the article "Fluentd stops of log forwarding when one of the outputs reaches the totalLimitSize in RHOCP 4". The log forwarding is resumed from the same point as when it stopped. In case that the logs were rotated meanwhile blocked the read, then, those logs will be lost.

NOTE
Currently, Vector has not the option drop_older as Fluentd has. This is requested to be implemented in This content is not included."Enhance sink buffering to all dropping oldest messages".

What's the retry policy and the number of retries?

The retry policy depends on the type of output/sink. Let's continue analyzing the behaviour of the [HTTP sink], but it should be analyzed the behaviour per output/sink.

Review the Content from vector.dev is not included."Retry policy" considering that the first retry should be after waiting the minRetryDuration time (retry_initial_backoff_secs) of 1 second and the next retries when waiting the maximum amount of time to wait before retries: maxRetryDuration (request.retry_max_duration_secs).

The number of times that retried an event before being dropped is defined by the Vector's variable request.retry_attempts, it's 9.223372036854776e+18.

What's the retry policy for the syslog output?

The syslog sink is really a Content from vector.dev is not included.socket sink where syslog is just a payload format implemented by Red Hat as Vector upstream doesn't support log forward to syslog.

The number of retries relays and the time between retries rely on the OS socket layer.

Does it exist a way of when the buffer gets full not getting stopped of log forwarding to the rest of the outputs defined?

The solution should be to isolate the pipelines using the multi log forwarder feature available starting in RHOL 5.8.

What compression algorithm should be used?

• the most compression provides the algorithm less network bandwidth, and slower performance in the collectors and in the log store
• the less compression provides the algorithm, more network bandwidth and faster performance in the collectors and in the log store

OTHER LINKS OF INTEREST

Vector Content from vector.dev is not included.Buffering model.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.