Java Finalizer queue has many instances of sun.security.ssl.SSLSocketImpl

Solution Unverified - Updated

Environment

  • JBoss Enterprise Application Platform (EAP) 6.x, 7.x, 8.x
  • Java application using SSLSocket like an application acting as HTTPS client

Issue

  • The application becomes unresponsive, and the following error is round in the JBoss server log:
java.lang.OutOfMemoryError: GC overhead limit exceeded
  • Inspection of the heap dump shows retention in a single instance of java.lang.ref.Finalizer with many instances of sun.security.ssl.SSLSocketImpl in the queue.

Resolution

  • Decrease creation of new ssl connections
  • The ssl session cache is unbounded by default until JDK 11, and it defaults to 20480 since JDK 12+ by Content from bugs.openjdk.org is not included.JDK-8210985. Its entries are soft references, so they can be released as needed, but the overhead could still grow larger than desired. So it may help to limit the ssl session cache to a reasonable size via the javax.net.ssl.sessionCacheSize system property.
  • Note that SSLSocketImpl is referenced by Finalizer until Finalizer is invoked by garbage collection. So, even if your application closes SSLSocket correctly, it can remain in heapdump if you have enough room in heap and garbage collection activity is low. It can be discarded correctly after garbage collection happens, so it's not a problem in this case.

Root Cause

  • There is code that does not explicitly close a connection, resulting in connections being closed by the Finalizer. This can take a long time (e.g. connection timeout, waiting for the JVM to schedule the Finalizer, etc.), resulting in sun.security.ssl.SSLSocketImpl buildup/retention.

Diagnostic Steps

See Java heap retention in java.lang.ref.Finalizer.

Select the sun.security.ssl.SSLSocketImpl objects in the heap dump where the host (peerHost in JDK 11+) is not null: (1) View incoming references to see what code is creating the connections. (1) View host values (peerHost since JDK 11+) to find the destination IP address for clues what code is creating the connections. For example, the following Object Query Language (OQL) in the Eclipse Memory Analyzer tool:

Until JDK 8:

select * from sun.security.ssl.SSLSocketImpl x where x.host != null

Since JDK 11+:

select * from sun.security.ssl.SSLSocketImpl x where (x.peerHost != null)

SELECT toString(peerHost) FROM sun.security.ssl.SSLSocketImpl x WHERE (x.peerHost != null)
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.