HTTPS connection creation bottlenecked by sun.security.util.MemoryCache

Environment

• Java

Issue

• Our java application makes many outgoing HTTPS calls and eventually threads are bottlenecked in sun.security.util.MemoryCache for HTTPS handshakes:

   java.lang.Thread.State: BLOCKED (on object monitor)
	at sun.security.util.MemoryCache.put(Cache.java:361)
	- waiting to lock <0x00000004d0869d48> (a sun.security.util.MemoryCache)
	at sun.security.ssl.SSLSessionContextImpl.put(SSLSessionContextImpl.java:178)
	at sun.security.ssl.ClientHandshaker.serverFinished(ClientHandshaker.java:1250)
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:376)
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
	- locked <0x00000004dad2b4e8> (a sun.security.ssl.SSLSocketImpl)
	- locked <0x00000004dad2b4a0> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
	- locked <0x00000004dad2b598> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)

• The lock owner is persisting for long amounts of time here:

   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap$TreeNode.find(HashMap.java:1878)
	at java.util.HashMap$TreeNode.find(HashMap.java:1874)
	at java.util.HashMap$TreeNode.find(HashMap.java:1874)
	at java.util.HashMap$TreeNode.find(HashMap.java:1874)
	at java.util.HashMap$TreeNode.find(HashMap.java:1874)
	at java.util.HashMap$TreeNode.getTreeNode(HashMap.java:1886)
	at java.util.HashMap.removeNode(HashMap.java:824)
	at java.util.HashMap.remove(HashMap.java:799)
	at sun.security.util.MemoryCache.emptyQueue(Cache.java:299)
	at sun.security.util.MemoryCache.put(Cache.java:361)
	- locked <0x00000004d0869d48> (a sun.security.util.MemoryCache)
	at sun.security.ssl.SSLSessionContextImpl.put(SSLSessionContextImpl.java:178)
	at sun.security.ssl.ClientHandshaker.serverFinished(ClientHandshaker.java:1250)
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:376)
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
	- locked <0x00000004dacced50> (a sun.security.ssl.SSLSocketImpl)
	- locked <0x00000004dacced08> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
	- locked <0x00000004daccee00> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)

Resolution

• Ensure you are on java 8u211 for reduced session cache collisions
• Limit the cache via the javax.net.ssl.sessionCacheSize system property

Root Cause

• Content from bugs.openjdk.java.net is not included.JDK-8203190
• This content is not included.BZ-1710095
• The ssl session cache is unbounded by default. Its entries are soft references, so they can be released as needed depending upon available heap, but this cache could still grow larger than desired. This performance concern becomes more prominent the larger the cache is.

Diagnostic Steps

• Check the size of the sun.security.util.MemoryCache from a heap dump at the issue time

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.