JVM crash in awt.dll called from sun.awt.image.ImagingLib.transformBI() when out of native memory

Environment

• 32 bit Java
• Red Hat JBoss Enterprise Application Platform (EAP)

Issue

The JVM crashes in a thread with the following frames:

Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  [awt.dll+0xa8ed7]

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  sun.awt.image.ImagingLib.transformBI(Ljava/awt/image/BufferedImage;Ljava/awt/image/BufferedImage;[DI)I+0
j  sun.awt.image.ImagingLib.filter(Ljava/awt/image/BufferedImageOp;Ljava/awt/image/BufferedImage;Ljava/awt/image/BufferedImage;)Ljava/awt/image/BufferedImage;+194

Resolution

• Switch to 64 bit Java
• As a work-around, free up more native memory by reducing heap size, perm gen size, or thread stack size

Root Cause

• This is a Content from bugs.sun.com is not included.bug in the JRE code. When ImagingLib.transformBI calls run into memory constraints, it can result in a crash.
• Even if it were fixed, there is a reasonable chance that the JVM would crash shortly after due to being out of native memory.
• Using 32 bit limits the process to only 2 gb on Windows and so it is especially prone to these issues as 1 gb or more is typically already used by JBoss for heap and/or perm

Diagnostic Steps

• Is 32 bit Java in use?
• Check if the crash is in awt.dll called frim ImageingLib code.
• If on x86, does EAX=0x00000000?

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.