Java application using PKCS11 provider leaks memory from Java_sun_security_pkcs11_wrapper_PKCS11_C_1DeriveKey

Solution Unverified - Updated

Environment

  • OpenJDK 1.7.0.45
  • Oracle JDK
  • PKCS11 security provider

Issue

JBoss slowly increases RSS memory usage, but the Java heap is not being used up.

Resolution

Do not use the PKCS11 security provider (e.g. SunJSSE for FIPS compliance).

If it was enabled by default, by the java-1.7.0-openjdk package on RHEL, upgrade to java-1.7.0-openjdk-1.7.0.45-2.4.3.4.el6_5 or later which disables it.

Security Providers:

A workaround is to use the G1 collector and immediately flush softly referenced objects with the following JVM options:

-XX:+UseG1GC -XX:SoftRefLRUPolicyMSPerMB=1

Root Cause

The following bugs are relevant:

The use of phantom references to manage native resources, the PKCS11 keys will not be freed until several weak reference processing cycles occur. This can take a long time with the throughput collectors.

The java-1.7.0-openjdk RHEL package enabled it for some releases, but was disabled again. See This content is not included.This content is not included.https://bugzilla.redhat.com/show_bug.cgi?id=1036813.

Diagnostic Steps

Capture a heap dump, and look at the sun.security.pkcs11.SessionKeyRef class(es). If the contained reference list is large but the reference queue small, it suggests those JDK bugs are the cause.

Run the application under valgrind, and check for memory being allocated by Java_sun_security_pkcs11_wrapper_PKCS11_C_1DeriveKey.

To reproduce the issue on RHEL tomcat:

  1. Start with a fresh tomcat install:
yum tomcat install

  1. Download the tomcat sample webapp (sample.war) from here:
    Content from tomcat.apache.org is not included.Content from tomcat.apache.org is not included.https://tomcat.apache.org/tomcat-7.0-doc/appdev/sample/sample.war

  2. Copy sample.war to /var/lib/tomcat/webapps/.

  3. Install nss:

yum install nss
  1. Edit /etc/alternatives/java_sdk/jre/lib/security/java.security:
*Add this as first security provider:
security.provider.1=sun.security.pkcs11.SunPKCS11 /path/to/nss_pkcs11_fips.cfg

*Increment existing, remaining security provider numbers (e.g. security.provider.1 --> security.provider.2, etc.).

*Change this line:
security.provider.5=com.sun.net.ssl.internal.ssl.Provider

to this:
security.provider.5=com.sun.net.ssl.internal.ssl.Provider  SunPKCS11-nss-fips
  1. Create /etc/alternatives/java_sdk/jre/lib/security/nss_pkcs11_fips.cfg with the following content:
name = nss-fips
nssLibraryDirectory = /usr/lib64
nssSecmodDirectory = /opt/db
nssDbMode = readOnly
nssModule = fips
  1. Create db:
mkdir -p /opt/db
modutil -create -dbdir /opt/db
chmod a+r /opt/db/*.db 
modutil -fips true -dbdir /opt/db
modutil -changepw "NSS FIPS 140-2 Certificate DB" -dbdir /opt/db

If you don't choose a password that meets FIPS password requirements, you will get the following error:

ERROR: Unable to change password on token "NSS FIPS 140-2 Certificate DB".

You should be able to use RedHat~2016 as the password.

  1. Create certificate:
certutil -S -k rsa -n jbossweb  -t "u,u,u" -x -s "CN=localhost, OU=MYOU, O=MYORG, L=MYCITY, ST=MYSTATE, C=MY" -d /opt/db
  1. Configure the tomcat SSL connector to use the PKCS11 keystore. Edit /etc/tomcat/server.xml and add the following connector (near where the existing port 8442 connector that is commented out is located):
<Connector port="8443" protocol="org.apache.coyote.http11.Http11Protocol"
  maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
  clientAuth="false" sslProtocol="TLS"
  keystorePass="RedHat~2016"       
keystoreType="PKCS11"
  ciphers="SSL_RSA_WITH_3DES_EDE_CBC_SHA,SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_DHE_DSS_WITH_AES_128_CBC_SHA,TLS_DHE_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_DHE_DSS_WITH_AES_256_CBC_SHA,TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDH_RSA_WITH_AES_128_CBC_SHA,TLS_ECDH_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA,TLS_ECDH_anon_WITH_AES_128_CBC_SHA,TLS_ECDH_anon_WITH_AES_256_CBC_SHA"/>
  1. Start tomcat:
service tomcat start
  1. Copy the following to a command line to run the ab tool that makes requests:
while true; do ab -c 10 -n 9999999 -f TLS1 https://localhost:8443/sample/;done
  1. Observe tomcat process size using top.

References:
[1]Run Tomcat with NSS on RHEL 7
[2]How do I make JBoss EAP use FIPS 140-2 compliant cryptography?

SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.