Thread exhaustion (OutOfMemoryError) with CXF AutomaticWorkQueueImpl as main suspect

Environment

• Red Hat JBoss Enterprise Application Platform (EAP)
  • 6.x

Issue

JVM is failing with an OutOfMemoryError due to thread exhaustion. After taking a thread dump, most threads appear to be stuck like this:

"default-workqueue-1" daemon prio=10 tid=0x00007f1049759800 nid=0x3a95 waiting on condition [0x00007f1054232000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000091180068> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at org.apache.cxf.workqueue.AutomaticWorkQueueImpl$AWQThreadFactory$1.run(AutomaticWorkQueueImpl.java:353)
	at java.lang.Thread.run(Thread.java:724)

How to we solve this?

Resolution

Share one javax.xml.ws.Service object by creating a static member:

private static Service service = null;

This would need to be initialized in a synchronized block to ensure it's only created once:

synchronized(ContainingClass.class) {
  if(service == null) {
    service = Service.create(wsdl, serviceNS);
  }
}

Each invocation of the client would then call service.getProxy(...) in a separate synchronized block. Using the client interface from my example:

  HelloWorld port = null;
  synchronized(service) {
    port = service.getPort(HelloWorld.class);
  }

In addition to solving the OutOfMemoryError issue, the overall performance of client invocations should improve since creating the Service object is an expensive operation and this solution removes the creation of that object except for the very first invocation.

Root Cause

• javax.xml.ws.Services are recreated consistently, in turn recreating underlying CXF executors each time.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.