Invalid padding exceptions cause mod_cluster proxied requests over HTTPS to 502

Environment

• JBoss Enterprise Application Platform (EAP)
  • 5.2.0+
  • 6.0.0+
• Java 1.7_u6 and later

Issue

• We're seeing random 502 failures with requests proxied through mod_cluster to JBoss over HTTPS:

[debug] ssl_engine_kernel.c(1881): OpenSSL: Read: SSLv3 read finished A
[debug] ssl_engine_kernel.c(1900): OpenSSL: Exit: failed in SSLv3 read finished A
[info] [client 127.0.0.1] SSL Proxy connect failed
[info] SSL Library Error: 336151568 error:14094410:SSL routines:SSL3_READ_BYTES:sslv3 alert handshake failure
[info] [client 127.0.0.1] Connection closed to child 0 with abortive shutdown (server localhost:443)
[error] (502)Unknown error 502: proxy: pass request body failed to 127.0.0.1:8443 (127.0.0.1)

• We have acquired the fix to the prior mod_cluster 502 bug (Content from issues.jboss.org is not included.JBPAPP-10409, Content from issues.jboss.org is not included.JBPAPP6-1170) yet still see this issue.
• SSL debug logging from JBoss shows the falling exception accompanies these 502s:

INFO  [stdout] (http-127.0.0.1:8443-1) 248, 178, 17, 86, 119, 76, 2, 163, 65, http-127.0.0.1:8443-1, IOException in getSession():  javax.net.ssl.SSLHandshakeException: Invalid padding

Resolution

• Configure your connector to use another cipher instead of a Diffie-Hellman-based cipher (such as RSA):

    <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" secure="true">
        <ssl ... cipher-suite="SSL_RSA_WITH_3DES_EDE_CBC_SHA"/>
    </connector>
  

• Alternatively, you can disable DH ciphers httpd/mod_cluster end for proxy connections:

    SSLProxyCipherSuite ALL:!DH:...
  

• Downgrade your JDK to 1.7_u5 or earlier if you need to Diffie-Hellman-based ciphers until an fix is issued in a later JDK update.

• If using the OpenJDK from RHEL 6, upgrade to java-1.7.0-openjdk-1.7.0.25-2.3.10.4.el6_4

Root Cause

• It looks like a bug causing this with Diffie-Hellman-based ciphers was introduced in JDK 1.7 update 6.
• The following Oracle forum thread has more information on this:
  • Content from forums.oracle.com is not included.https://forums.oracle.com/forums/thread.jspa?messageID=10875177&tstart=0
• SSL handshakes randomly fail on Java 7
• Note that BouncyCastle had a similar error as described in SSL handshakes randomly fail when using BouncyCastle

Diagnostic Steps

• Enable SSL handshake debug logging on JBoss with the system property below, reproduce, and check for Invalid padding exceptions in DH-cipher handshakes as seen above:

-Djavax.net.debug=ssl,handshake

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.