foreman-proxy log has many "OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0 state=SSLv2/v3 read client hello A /usr/share/ruby/openssl/ssl.rb:280:in `accept'" error

Solution Verified - Updated

Environment

  • Red Hat Enterprise Linux
  • Red Hat Satellite 6

Issue

  • Foreman proxy response to requests slowly and frequently get request timeout.
  • Foreman proxy service is crashed and has no longer respond to any request.
  • Satellite or Capsule /var/log/foreman-proxy/proxy.log file reported many "OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0 state=SSLv2/v3 read client hello A /usr/share/ruby/openssl/ssl.rb:280:in `accept'" error.

Resolution

  • Stop security scanning for all Capsules or put HTTP(S) proxy with DoS protection in front of each Capsule.

For more KB articles/solutions related to Red Hat Satellite 6.x SSL Certificates Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x SSL Certificates Issues

Root Cause

  • It may be a security scanner(such as Nexpose) which overloads the Foreman proxy (port 9090 and 8000) with thousands of requests and therefore it is unavailable for normal request processing. In other words, the Foreman proxy is undergoing DoS attack. A bugzilla This content is not included.bugzilla has been raised regarding foreman-proxy crashed after overloaded by huge amount of requests.

Diagnostic Steps

Either take and analyze tcpdump (that reveals all such activity but needs it requires some basic knowledge of TCP(dump)), or enable foreman-proxy debugs that shall log the activity as well (but not all such activity).

Option to take tcpdump:

Run on the affected system:

tcpdump -i any -s 0 port 9090 -w /tmp/port9090.cap

and wait until few occurrences of the error appears. Then stop the dump, analyze it via tshark or wireshark and spot expectedly behaviour like:

  • 3way TCP handshake (normal establishing of TCP connection)
  • almost immediately followed by FIN packet from the client, and further proper connection closure

Check client IP address of that activity and identify what process/program on that system triggers it.

Option to enable debug logs:

  • Set the debug level of Foreman proxy to DEBUG by editing "/etc/foreman-proxy/settings.yml"
# Uncomment and modify if you want to change the log level
# WARN, DEBUG, ERROR, FATAL, INFO, UNKNOWN
:log_level: DEBUG           <=========== Set this to DEBUG
  • Restart Foreman proxy
systemctl restart foreman-proxy
  • Wait for 24 hours to collect enough logs.
  • Check the /var/log/foreman-proxy/proxy.log file. You should see a lot of lines similar to below. 
D, [<DATE>T<HH>:23:14.791872 ] DEBUG -- : accept: 10.11.12.133:<SRC PORT>
D, [<DATE>T<HH>:23:14.792204 ] DEBUG -- : close: 10.11.12.133:<SRC PORT>
E, [<DATE>T<HH>:23:45.509579 ] ERROR -- : OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0 state=SSLv2/v3 read client hello A
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
D, [<DATE>T<HH>:24:14.033209 ] DEBUG -- : accept: 10.11.12.133:<SRC PORT>
D, [<DATE>T<HH>:24:14.033527 ] DEBUG -- : close: 10.11.12.133:<SRC PORT>
E, [<DATE>T<HH>:24:45.927289 ] ERROR -- : OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0 state=SSLv2/v3 read client hello A
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
D, [<DATE>T<HH>:25:14.081065 ] DEBUG -- : accept: 10.11.12.133:<SRC PORT>
D, [<DATE>T<HH>:25:14.081432 ] DEBUG -- : close: 10.11.12.133:<SRC PORT>
E, [<DATE>T<HH>:25:45.607392 ] ERROR -- : OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0 state=SSLv2/v3 read client hello A
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
D, [<DATE>T<HH>:26:14.427429 ] DEBUG -- : accept: 10.11.12.133:<SRC PORT>
D, [<DATE>T<HH>:26:14.427846 ] DEBUG -- : close: 10.11.12.133:<SRC PORT>
E, [<DATE>T<HH>:26:46.775176 ] ERROR -- : OpenSSL::SSL::SSLError: SSL_accept SYSCALL returned=5 errno=0 state=SSLv2/v3 read client hello A
        /usr/share/ruby/openssl/ssl.rb:280:in `accept'
MORE..
  • Based on the example log above, we can see that a host with IP address "10.11.12.133" is scanning Foreman proxy every minute.
  • Investigate the host with IP address "10.11.12.133" and find out why it is scanning Foreman proxy.
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.