CPU throttling on Data Grid pods in OCP 4

Solution Verified - Updated

Environment

  • Red Hat Data Grid
    • 8.x
  • Red Hat Openshift Container Platform
    • 4.x

Issue

  • CPU throttling on Data Grid pods in OCP 4
  • Effects and workarounds for CPU throttling on Data Grid pods in OCP 4

Resolution

CPU throttling impacts the performance of the DG threads, but it does not cause the DG to crash. In other others CPU throttling will not finish off the pods directly. However, it will impact the performance in scenarios especially with many threads, let's say DG deployments without limits - that's the reason we recommend always settings limits in the DG containers - even Java in general.

One option could be to adjust the CPU limits (and ratio) in the configuration for the affected pods in order to alleviate the CFS throttling impact - see ratio definitions on solution How to define memory and cpu resource via limits/requests on DG 8 Operator.

In case of crash happening/impacting, see the solution Troubleshoot options for Data Grid pod crash.

Finally, the solution CPU Throttling even when the container does not reach its CPU Limit brings details on the impact of throttling even when the limits are not reached. The main article Red Hat build of OpenJDK container awareness for Kubernetes tuning covers tuning details for Kubernetes.

Root Cause

CPU throttling

The throttling should occur when the application's threads have used up its quota (limit), given the CPU quota the process is preempted. Depending on the kernel version this plays less of a role though.
OCP might use CFS’s quota mechanism to implement process limits, and therefore impose a limit on the threads as a consequence, regardless of cpu and memory settings. However, above all else, it will not kill an DG pod directly (it is not Host's OOMKill).

Diagnostic Steps

  1. See Grafana's CPU plot
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.