Red Hat build of OpenJDK container awareness for Kubernetes tuning

This article is important because throughout the web, there are several articles about performance tuning and optimization on Kubernetes and they might be mistaken or plain simply wrong when specifically talking about Java workloads.

In this matter, Red Hat cannot do peer-review/investigations of 3rd party articles. Although users can take 3rd party tuning guides as references, Red Hat recommends that its products use official links and guides should be taken into consideration. This avoids situations such as conflict of information and miscommunications.

In terms of Java deployment the article post This content is not included.How to use Java container awareness in OpenShift 4 covers the core of the concepts and should be enough to learn how container awareness works. In a nutshell:

• Red Hat's OpenJDK will only abide by cgroups limits meaning it will ignore container requests.
• Setting a container without cgroups limits will make Java use the OCP node as an upper limit for GC and thread calculation.
• Requests are used by Kubelet, not by the Java application
• Not setting limits in Java workloads inside a cluster is unwise and should be avoided. That's the point of container-aware settings, to make the Java container abide by them
• cgroups is an API, so the application can use it as a reference as they want. it is not mandatory to use both requests and limits.
• setting a different value of requests and limits will make the QoS Burstable, whereas QoS will be Guaranteed in case both values match. Only requests result in BestEffort Qos. By not setting limits you are imposing the QoS to be BestEffort.
• Red Hat's stance on that is to keep using over-provisioning but to set QoS guaranteed (so avoid over-provisioning) for critical workloads, thus allocating specific resources for those applications.
• Red Hat build of OpenJDK will have its limits written in stone. It is not going to be dynamic.
• Overprovisioning is possible and useful via QoS Burstable and QoS Best Effort for example, but should not be taken for granted and applied without due diligence and proper testing
• CPU throttling is a kernel feature, and won't be preventable necessarily by setting all applications in a OCP Node to have QoS such as BestEffort

By not setting container limits, Red Hat's OpenJDK will use the OCP Node limits in terms of memory and CPU usage, which is used for thread pooling calculation and GC threads. Because requests are null and void for Red Hat's OpenJDK deployments it won't change the number of CPUs taken into consideration at runtime.

Some web articles, aiming to avoid CPU throttling or focusing on over-provisioning, might have been uninformed and/or missing the key aspect of how Java container awareness is important for deployment inside a container, and allowing not fully container aware deployments inside a container creates more problems than solves them.
Deploying an application not fully container aware (i.e. without the container limits) will have the following consequences:

• If the memory limit is not set, JVM will use a percentage of the OCP Node, which will then ballon and cause cgroups OOMkills most likely
• If the CPU limits are not set, JVM will use the OCP Node as the upper limit for thread pool calculations and GC threads/usage from the get-go, except it is still inside a container and it will have that container time inside the kernel.

Therefore, as extensively explained, without container awareness, it can be problematic precedence because it causes a decoupling between the container settings and the application inside the container, this deployment just happens to be Java. In other words, some articles forget to disclaim/take into consideration the impact of not setting cgroups limits and doing this decoupling (container <->java setting), turning the OCP cluster unstable and unpredictable. Basically, without a limit, the user is allowing a non-scoped application application inside an OCP Node at the expense of other applications, which can be very problematic.
In conclusion remarks, worse than having CPU throttling, which can be avoided in several ways such as a larger ratio of requests to limits, is to have non-container deployments and explode the whole OCP node. The application, without boundaries, will eventually starves out the other resources in the pod because just for one tuning focus taking the container as a VM.

In summary:

• CPU throttling can affect any container deployed in an OCP cluster, so long as it has CPU limits. This is caused by kernel.
• QoS is a kubelet setting and won't prevent the CPU throttling and are defined as follows: Burstable QoS: limits different than requests. Guaranteed QoS: when both values match. BestEffort QoS: Only requests.
• Leaving no limits defined, makes the OCP node be the maximum limit: CPU and memory and will prevent CPU throttling. However, it will make the BestEffort QoS.
• The order of eviction is given by: BestEffort QoS ->Burstable QoS -> Guaranteed QoS.
• For Cgroups v2 see the article This content is not included.How does cgroups v2 impact Java in OpenShift 4?, where Java that is not cgroups v2 complaint will required manual setting: Xmx, and CPU boundaries manually set, otherwise it will use the Host ones (behaving as non-container aware).