JBoss threads looping in KeyAffinityServiceImpl.getKeyForAddress

Solution Verified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP) 6.4.3 and earlier with dist mode replication

Issue

  • We see increased CPU with threads persisting in the following:

"ajp-localhost/127.0.0.1:8009-512" daemon prio=10 tid=0x00007f1d9c784000 nid=0x3468 waiting on condition [0x00007f1d2f9f9000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000006089540f8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
	at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
	at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
	at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:361)
	at org.infinispan.affinity.KeyAffinityServiceImpl.getKeyForAddress(KeyAffinityServiceImpl.java:141)

Resolution

  • Apply 6.4 CP4 or later
  • Or as a workaround use repl mode cache replication instead of dist
  • If you continue to see threads persist in KeyAffinityServiceImpl.getKeyForAddress, check the state of the KeyAffinityService Thread Pool thread that fills the key queue. You may see it blocked in random reads like so, and if you do consider switching to urandom per "/dev/random" hangs in java program when server is inactive:
"KeyAffinityService Thread Pool -- 1" prio=10 tid=0x00007fed3ce94800 nid=0x7fd2 runnable [0x00007fed2a623000]
   java.lang.Thread.State: RUNNABLE
	at java.io.FileInputStream.readBytes(Native Method)
	at java.io.FileInputStream.read(FileInputStream.java:272)
	at sun.security.provider.NativePRNG$RandomIO.readFully(NativePRNG.java:202)
	at sun.security.provider.NativePRNG$RandomIO.ensureBufferValid(NativePRNG.java:264)
	at sun.security.provider.NativePRNG$RandomIO.implNextBytes(NativePRNG.java:278)
	- locked <0x00000005780249d0> (a java.lang.Object)
	at sun.security.provider.NativePRNG$RandomIO.access$200(NativePRNG.java:125)
	at sun.security.provider.NativePRNG.engineNextBytes(NativePRNG.java:114)
	at java.security.SecureRandom.nextBytes(SecureRandom.java:466)
	- locked <0x0000000579af7870> (a java.security.SecureRandom)
	at org.apache.catalina.session.ManagerBase.generateSessionId(ManagerBase.java:585)
	at org.jboss.as.web.session.DistributableSessionManager.createSessionId(DistributableSessionManager.java:817)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getKey(DistributedCacheManager.java:466)
	at org.jboss.as.clustering.web.infinispan.DistributedCacheManager.getKey(DistributedCacheManager.java:66)
	at org.infinispan.affinity.KeyAffinityServiceImpl$KeyGeneratorWorker.generateKeys(KeyAffinityServiceImpl.java:280)
	at org.infinispan.affinity.KeyAffinityServiceImpl$KeyGeneratorWorker.run(KeyAffinityServiceImpl.java:253)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:745)
	at org.jboss.threads.JBossThread.run(JBossThread.java:122)

Root Cause

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.