[Satellite6] synchronizing >800 repos to a Capsule fails with connection errors

Environment

• Red Hat Satellite or Proxy 6.2
• Red Hat Enterprise Linux (RHEL) 6 or 7

Issue

• having over 800 repositories in a Lifecycle Environment (L.E.)
• trying to synchronize a Capsule assigned to that L.E. fails - expected error code is either:
  • wrong status line: "CONNECT capsule01.example.com:443 HTTP/1.1" (Net::HTTPBadResponse)
  • Server broke connection (RestClient::ServerBrokeConnection)

Resolution

Increase ulimit of open files to

• at least 5 * REPOS on any Capsule
• at least 5 * REPOS * CAPSULES on the Satellite

where REPOS is maximal number of repositories that can be synchronized to a Capsule concurrently, and CAPSULES is the number of Capsules.

To configure it on RHEL6: add to /etc/security/limits.conf:

apache - nofile 10000

To configure it on RHEL7, add to the /usr/lib/systemd/system/httpd.service, to the end of [Service] section:

LimitNOFILE=10000

To apply it on either RHEL6 or RHEL7, restart httpd service.

For more KB articles/solutions related to Red Hat Satellite 6.x Capsule Sync Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Capsule Sync Issues

Root Cause

Synchronizing repositories is executed concurrently. A repository sync to a Capsule consists of a series of mainly-sequential HTTPS requests between httpd processes on the Satellite and the Capsule. For one Capsule and one repo, there can be up to 5 concurrent HTTP requests / concurrently opened TCP connections.

Synchronizing sufficiently high repos in parallel thus hits ulimit of open files (with default value 1024) of httpd process, either on the Satellite or on a Capsule.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.