High CPU in AbstractClassLoaderValue.computeIfAbsent calls

Solution Unverified - Updated

Environment

  • OpenJDK 11+
  • JBoss Enterprise Application Platform (EAP) 7.x
  • Apache CXF

Issue

  • Moving to java 11+, we are see that we can reach a state with excessively high CPU and heavy thread blocking in jdk.internal.loader.AbstractClassLoaderValue.computeIfAbsent calls like below. We have to restart the app to address it:
"default task-8214" #1201642 prio=5 os_prio=0 cpu=124605.58ms elapsed=842.18s tid=0x0000558a4e944000 nid=0x1d712f runnable  [0x00007f131adf2000]
   java.lang.Thread.State: RUNNABLE
	at java.util.concurrent.ConcurrentHashMap.putVal(java.base@11.0.22/ConcurrentHashMap.java:1012)
	at java.util.concurrent.ConcurrentHashMap.putIfAbsent(java.base@11.0.22/ConcurrentHashMap.java:1541)
	at jdk.internal.loader.AbstractClassLoaderValue.computeIfAbsent(java.base@11.0.22/AbstractClassLoaderValue.java:194)
	at java.lang.reflect.Proxy.getProxyConstructor(java.base@11.0.22/Proxy.java:415)
	at java.lang.reflect.Proxy.newProxyInstance(java.base@11.0.22/Proxy.java:1009)
	at org.apache.cxf.binding.soap.SOAPBindingUtil.getProxy(SOAPBindingUtil.java:92)
	at org.apache.cxf.binding.soap.SOAPBindingUtil.getSoapBody(SOAPBindingUtil.java:223)
	at org.apache.cxf.binding.soap.SOAPBindingUtil.getSoapBody(SOAPBindingUtil.java:214)
	at org.apache.cxf.binding.soap.SoapBindingFactory.initializeMessage(SoapBindingFactory.java:760)
	at org.apache.cxf.binding.soap.SoapBindingFactory.initializeBindingOperation(SoapBindingFactory.java:710)
	at org.apache.cxf.binding.soap.SoapBindingFactory.createBindingInfo(SoapBindingFactory.java:686)
	at org.apache.cxf.wsdl11.WSDLServiceBuilder.buildBinding(WSDLServiceBuilder.java:492)
	at org.apache.cxf.wsdl11.WSDLServiceBuilder.buildServices(WSDLServiceBuilder.java:360)
	at org.apache.cxf.wsdl11.WSDLServiceBuilder.buildServices(WSDLServiceBuilder.java:209)
	at org.apache.cxf.wsdl11.WSDLServiceFactory.create(WSDLServiceFactory.java:161)
	at org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.buildServiceFromWSDL(ReflectionServiceFactoryBean.java:407)
	at org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.initializeServiceModel(ReflectionServiceFactoryBean.java:527)
	at org.apache.cxf.wsdl.service.factory.ReflectionServiceFactoryBean.create(ReflectionServiceFactoryBean.java:262)
	- locked <0x00000007abe1e438> (a org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean)
	at org.apache.cxf.jaxws.support.JaxWsServiceFactoryBean.create(JaxWsServiceFactoryBean.java:199)
	at org.apache.cxf.frontend.AbstractWSDLBasedEndpointFactory.createEndpoint(AbstractWSDLBasedEndpointFactory.java:103)
	at org.apache.cxf.frontend.ClientFactoryBean.create(ClientFactoryBean.java:91)
	at org.apache.cxf.frontend.ClientProxyFactoryBean.create(ClientProxyFactoryBean.java:159)
	- locked <0x00000007abe7a928> (a org.apache.cxf.jaxws.JaxWsProxyFactoryBean)
	at org.apache.cxf.jaxws.JaxWsProxyFactoryBean.create(JaxWsProxyFactoryBean.java:142)
	- locked <0x00000007abe7a928> (a org.apache.cxf.jaxws.JaxWsProxyFactoryBean)
	at org.apache.cxf.jaxws.ServiceImpl.createPort(ServiceImpl.java:492)
	at org.jboss.wsf.stack.cxf.client.ProviderImpl$JBossWSServiceImpl.createPort(ProviderImpl.java:580)
	at org.apache.cxf.jaxws.ServiceImpl.getPort(ServiceImpl.java:358)
	at org.apache.cxf.jaxws.ServiceImpl.getPort(ServiceImpl.java:349)
	at javax.xml.ws.Service.getPort(Service.java:139)

Resolution

  • Downgrade to java 8 as a workaround.
  • Adjust application code to limit concurrency to AbstractClassLoaderValue.computeIfAbsent and/or addrss any exceptions being thrown by the Memoizer.get calls in the proxy creation attempt. If a CXF Web Service client application, then this could be achieved by caching and reusing the created WS Service ports for highest possible efficiency and to limit the amount of times such code calls would be hit overall. Likely exceptions being raised from the CXF proxy creation attempt to possibly lead to high CPU can be avoided with a WEB-INF/jboss-deployment-structure.xml like below:
<jboss-deployment-structure>
  <deployment>
       <dependencies>
         <module name="org.apache.cxf.impl" />
      </dependencies>
  </deployment>
</jboss-deployment-structure>

Root Cause

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.