Concurrent access to WeakHashMap in ConcurrentCache causes infinite loop and 100% CPU in JBoss

Solution Verified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP) 
    • 4.2
    • 4.3
    • 5.x

Issue

  • High CPU usage with threads stuck in java.util.WeakHashMap.put() and java.util.WeakHashMap.get(). For example:

      "ajp-146.215.106.42-8009-38" daemon prio=10 tid=0x00002aab6425d800 nid=0x135b runnable [0x0000000048c14000]
  java.lang.Thread.State: RUNNABLE
      at java.util.WeakHashMap.get(WeakHashMap.java:355)
      at org.apache.el.util.ConcurrentCache.get(ConcurrentCache.java:24)
      at org.apache.el.lang.ExpressionBuilder.createNodeInternal(ExpressionBuilder.java:90)

  "ajp-146.215.106.42-8009-18" daemon prio=10 tid=0x00002aab643ea000 nid=0x538d runnable [0x00000000458fd000]
  java.lang.Thread.State: RUNNABLE
      at java.util.WeakHashMap.put(WeakHashMap.java:405)
      at java.util.WeakHashMap.putAll(WeakHashMap.java:518)

  • We are facing HIGH CPU utilization in one of the Application server nodes in production environment. We currently have two application server nodes running on jboss-eap-5.1. However we are seeing High CPU usage on only one node.

Resolution

This issue is caused by a concurrency bug in the expression language parser.

Root Cause

This is an instance of the issue described in High CPU due to multiple Java threads accessing HashMap simultaneously

EAP 4.3 bug report: Content from issues.jboss.org is not included.JBPAPP-5342
EAP 5 bug report: Content from issues.jboss.org is not included.JBPAPP-5343

Diagnostic Steps

See Java application high CPU

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.