JBoss EAP native memory leak with mutual authentication on RHEL6 with OpenSSL

Environment

• Red Hat Enterprise Linux (RHEL) 6
• Red Hat JBoss Enterprise Application Platform (EAP) 7
• Mutual authentication using OpenSSL
• OpenSSL installed from the RHEL6 repository.

Issue

• Java process memory consumption increasing at about 1G per day until killed by the Out Of Memory (OOM) killer.
• When not specifying the openssl provider (and using Java's JSSE one), the issue does not happen.
• The issue also does not happen on RHEL7.

Resolution

Update OpenSSL to the JBoss Core Services (JBCS) version. Check it in this This content is not included.link.

A workaround is to disable the openssl provider (and use Java's JSSE).

Root Cause

The leak is specific to RHEL6 OpenSSL package, which is not tested or supported in combination with EAP 7. Only the OpenSSL library version 1.0.2 or greater provided by JBoss Core Services is supported. Check more about it in documentation page.

• This content is not included.JBEAP-20232

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.