SSL Proxy connect failed and mod_cluster returns "Bad Gateway" HTTP ErrorCode 502

Solution Verified - Updated

Environment

  • Apache httpd 2.2.15
  • Red Hat Enterprise Linux(RHEL) 6.2
  • mod_cluster
    • 1.0.10.GA_CP02
    • 1.2.3.Final
  • JBoss Enterprise Application Platform (EAP)
    • 5.1.2
    • 6.0.0

Issue

  • We have Apache/mod_cluster loadbalancing traffic to JBoss over SSL. After a longer pause with no requests (e.g. in the morning), many requests fail with 502 "Bad Gateway" errors. Apache debug logging shows it occurs it is triggered from a failure in the SSL handshake:

      [debug] ssl_engine_kernel.c(1902): OpenSSL: Read: SSLv2/v3 read server hello A
  [debug] ssl_engine_kernel.c(1926): OpenSSL: Exit: error in SSLv2/v3 read server hello A
  [info] [client 127.0.0.1] SSL Proxy connect failed
  [info] SSL Library Error: 336032754 error:140773F2:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert unexpected message
  [info] [client 127.0.0.1] Connection closed to child 0 with abortive shutdown (server localhost:443)
  [error] (502)Unknown error 502: proxy: pass request body failed to localhost:8443 (127.0.0.1)

  • On jboss.org, I found Content from community.jboss.org is not included.the following thread

  • After multiple requests, the error is gone and everything works fine. Just after a longer pause the error comes back. Also, restarting Apache appears to clear up the 502s until another period of idleness.

Resolution

  • Upgrade to the mod_cluster Apache module bits and jar provided by EAP 5.2.0 or 6.0.1
  • As a workaround, you can disable all keepalive and connection reuse between the Apache/mod_cluster and JBoss tiers:

    • Set maxKeepAliveRequests="1" on your JBoss HTTP connector in your server.xml to disable JBoss keepalive

    • Add the following to the root of your Apache config (outside of virtual hosts) to globally disable proxy keepalive and connection reuse:

        SetEnv proxy-nokeepalive 1
  SetEnv proxy-initial-not-pooled 1

    • Set smax to 0 in your mod-cluster-jboss-beans to ensure the mod_cluster balancer will close out all idle proxy connections:

        <bean name="ModClusterService" class="org.jboss.modcluster.ModClusterService" mode="On Demand">
      ...
      <property name="smax">0</property>
  </bean>
  • If you continue to see 502s after acquiring this bug fix and are using Java 7, see Invalid padding exceptions cause mod_cluster proxied requests over HTTPS to 502 for another potential cause.

Root Cause

Diagnostic Steps

  • Set LogLevel to debug on Apache and check for errors in the handshake as noted to cause the 502

  • Enable SSL debug logging on JBoss:

      -Djavax.net.debug=all

    This can be quite weighty and so may not be desired in production. Check the debug logging for timeouts (which would likely be in the handshake) like the following:

      19:49:08,329 INFO  [STDOUT] (localhost-8443-1) localhost-8443-1, setSoTimeout(600000) called
  ...
  19:59:08,466 INFO  [STDOUT] (localhost-8443-1) localhost-8443-1, handling exception: java.net.SocketTimeoutException: Read timed out

  • Gather thread dumps before and after such timeouts to confirm these threads are going from a handshake state and timing out to return back idle to the pool

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.