OutOfMemoryError due to many instances of java.lang.ref.Finalizer after upgrading OpenJDK 1.6

Solution Unverified - Updated

Environment

  • Red Hat JBoss Enterprise Application Platform (EAP)
    • 5.2.0
  • OpenJDK 1.6
    • java-1.6.0-openjdk-1.6.0.0-4.1.13.1.el5_10
    • 3.1.13.1
  • Red Hat Enterprise Linux (RHEL)
    • 5
    • 6

Issue

  • OpenJDK 1.6 problems after updating
  • OutOfMemoryError: Java heap space after upgrading OpenJDK 1.6 to version 1.6.0.0-3.1.13.1.el5_10.
  • Instances of java.lang.ref.Finalizer loaded by <system class loader> looking like the root cause of the OutOfMemoryError.
  • After upgrading OpenJDK 6 1.42.1.11.14 to version 3.1.13.1, JBoss starts to consume more memory up until a java.lang.OutOfMemoryError: Java heap space appears on the server log.
  • A JBoss application with AD backed LDAP authentication is periodically leaking file descriptors. This occurs when java-1.6.0-openjdk-1.6.0.0-3.1.13.1.el5_10 is installed. Downgrading to java-1.6.0-openjdk-1.6.0.0-1.40.1.11.11.el5_9 appears to fix the issue. Are there any known problem regarding that openjdk release? Note that the file descriptor leak results in an OOM exception once the file descriptor limit is reached.

Resolution

Ideally the open file descriptor(s) will not be causing an OOM error to happen. There is a registered bug about this happening, as mentioned in the Root Cause section.

  • On RHEL 5, one can downgrade to java-1.6.0-openjdk-1.6.0.0-1.42.1.11.14.el5_10 .
  • On RHEL 6, upgrade to java-1.6.0-openjdk-1.6.0.0-5.1.13.3.el6_5 or later
  • The bug was fixed in the latest release of java-1.6.0-openjdk package released. The release announcement is here, http://rhn.redhat.com/errata/RHSA-2014-0408.html.
    So one can also upgrade to java-1.6.0-openjdk-1.6.0.0-5.1.13.3.el5_10 as mentioned in the above errata link.

Root Cause

Diagnostic Steps

Check the OutOfMemoryError thread:

java.lang.Thread @ 0xe4a44e40                                                                                        |ajp-127.0.0.1-8409-9|          112 |       354,344 |org.jboss.web.tomcat.service.WebCtxLoader$ENCLoader @ 0xdd65c1d8|true
|- at java.lang.OutOfMemoryError.<init>()V (OutOfMemoryError.java:42)
|- at java.nio.CharBuffer.wrap([CII)Ljava/nio/CharBuffer; (CharBuffer.java:367)
|- at java.nio.CharBuffer.wrap([C)Ljava/nio/CharBuffer; (CharBuffer.java:390)
|- at java.lang.StringCoding$StringDecoder.decode([BII)[C (StringCoding.java:156)
|- at java.lang.StringCoding.decode(Ljava/lang/String;[BII)[C (StringCoding.java:191)
|- at java.lang.String.<init>([BIILjava/lang/String;)V (String.java:451)
|- at java.util.jar.Attributes.read(Ljava/util/jar/Manifest$FastInputStream;[B)V (Attributes.java:418)
|- at java.util.jar.Manifest.read(Ljava/io/InputStream;)V (Manifest.java:251)
|- at sun.security.util.SignatureFileVerifier.processImpl(Ljava/util/Hashtable;)V (SignatureFileVerifier.java:205)
|- at sun.security.util.SignatureFileVerifier.process(Ljava/util/Hashtable;)V (SignatureFileVerifier.java:193)
|- at java.util.jar.JarVerifier.processEntry(Lsun/security/util/ManifestEntryVerifier;)V (JarVerifier.java:296)
|- at java.util.jar.JarVerifier.update(I[BIILsun/security/util/ManifestEntryVerifier;)V (JarVerifier.java:207)
|- at java.util.jar.JarFile.initializeVerifier()V (JarFile.java:342) 
|- at java.util.jar.JarFile.getInputStream(Ljava/util/zip/ZipEntry;)Ljava/io/InputStream; (JarFile.java:410) 
|- at sun.net.www.protocol.jar.JarURLConnection.getInputStream()Ljava/io/InputStream; (JarURLConnection.java:162)
|- at java.net.URL.openStream()Ljava/io/InputStream; (URL.java:1031)
|- at java.lang.ClassLoader.getResourceAsStream(Ljava/lang/String;)Ljava/io/InputStream; (ClassLoader.java:1170)
|- at javax.xml.parsers.SecuritySupport$4.run()Ljava/lang/Object; (SecuritySupport.java:94)
|- at java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object; (Native Method)

Get a heap dump when the issue happens with the -XX:+HeapDumpOnOutOfMemoryError option and analyze the heap dump.

Verify the heap retention is due to a large number of java.lang.ref.Finalizer objects.

Look at the threads (e.g. using the Eclipse Memory Analyzer tool Thread Overview and Stacks function) and see if there are a number of "Timer-#" threads (e.g. Timer-0, Timer-1, Timer-2, etc.) with paths to GC roots showing them coming from the sun.java2d.Disposer class:

Class Name                                                           | Ref. Objects | Shallow Heap | Ref. Shallow Heap | Retained Heap
---------------------------------------------------------------------------------------------------------------------------------------
class sun.java2d.Disposer @ 0xe4b1e708 System Class, Native Stack    |       37,271 |           24 |         1,490,840 |         2,184
'- records java.util.Hashtable @ 0xe4b1e900                          |       37,271 |           48 |         1,490,840 |         2,160
   '- table java.util.Hashtable$Entry[47] @ 0xe5728b40               |       37,271 |          208 |         1,490,840 |         2,112
      |- [0] java.util.Hashtable$Entry @ 0xe588bae8                  |       36,193 |           32 |         1,447,720 |            64
      |  '- key java.lang.ref.PhantomReference @ 0xe5728cf0          |       36,193 |           32 |         1,447,720 |            48
      |     '- discovered java.lang.ref.PhantomReference @ 0xe5728d10|       36,193 |           32 |         1,447,720 |            48
      |        '- discovered java.lang.ref.Finalizer @ 0xe5728d30    |       36,193 |           40 |         1,447,720 |           128
---------------------------------------------------------------------------------------------------------------------------------------
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.