Vector cpu and memory usage is increasing over the time in RHOCP 4

Solution Verified - Updated

Environment

  • Red Hat OpenShift Container Platform (RHOCP)
    • 4
  • Red Hat OpenShift Logging (RHOCP)
    • 5.6
    • 5.7

Issue

  • The cpu and memory usage of Vector is always increasing over the time

Resolution

This issue has been reported to Red Hat engineering. It is being tracked in a private Bug and delivered a fix in RHOL 5.7.6 through errata RHSA-2023:4933.

If this issue still occurs in the environment after updating, check the article Log collector vector has high memory consumption in RHOCP 4 or open a This content is not included.new support case with Red Hat Support.

Root Cause

The internal metrics collected by Vector are never expired. Then, the cpu and memory usage by Vector keeps always increasing. In Vector 0.28 is exposed the variable expire_metrics_secs for setting the time to expirse the metrics, but this doesn't exist in older versions.

Diagnostic Steps

Go to the OCP Console > Observe > Metrics and run the query below as described in the Querying metrics documentation observing that its value is always increasing leading to more cpu and memory:

vector_internal_metrics_cardinality_total
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.