High CPU: Competing Java parallel garbage collection threads

Solution Verified - Updated

Environment

  • JBoss
  • OpenJDK
  • Oracle JDK
  • cgroups (e.g. Docker) container

Issue

  • JBoss is hanging due to 400% CPU utilization.
  • JBoss app servers are spinning (100% CPU utilization).

Resolution

  • Explicitly set -XX:ParallelGCThreads to the number of CPUs assigned to the virtual environment or a fraction of the CPUs based on the number of other services running on the shared hardware.
  • Use the -XX:ActiveProcessorCount=N flag introduce in JDK8u191 to override the number of CPUs the JVM detects and allow the JVM to assign threading based on the override value (e.g. -XX:ActiveProcessorCount=4).
    *Upgrade to JDK8u191+ so the JVM can set appropriate default values like threading based on container limits, not host limits.

Root Cause

  • If -XX:ParallelGCThreads is set too high, the garbage collection threads will compete with one another.
  • The JVM is not able to set appropriate default values for GC threading when multiple JVMs are running on the same hardware.
  • The JVM version does not include functionality to set appropriate default values for GC threading when running in a container environment.

Diagnostic Steps

  • Follow Java application high CPU to determine the cause of the high CPU.
  • Check JVM options to see if any parallel collectors are being defined (e.g. -XX:+UseParNewGC, -XX:+UseParallelGC, -XX:+UseParallelOldGC).
  • Check garbage collection (GC) logging to see if any parallel collectors are being used by default.
  • Check that the number of CPUs available to the JVM does not conflict with the default or explicitly defined -XX:ParallelGCThreads setting. Make sure the setting makes sense when considering any virtual environments or other services that might be running on the same physical box competing for CPU.
  • The default number of GC threads for +UseParallelGC is to set it to the total number of hardware threads. [For +UseConcMarkSweepGC/+UseParNewGC](For +UseConcMarkSweepGC/+UseParNewGC) the default is computed as: (ncpus <= 8) ? ncpus : 3 + ((ncpus * 5) / 8) (where ncpus is the # cores on the platform or in the pset of the JVM).
  • Are multiiple JVMs or containers running on the same hardware?
  • Does the JVM include support for setting default values based on container limits, not host limits (e.g. JDK8u191+)?
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.