Host registration with Red Hat Satellite fails with error 502 timeout

Environment

• Red Hat Satellite 6
• Red Hat Capsule 6

Issue

• The host registration with Satellite timeout because of multiple system facts being uploaded by the host.
• Reverse lookup time exceeds apache timeout on satellite/capsule

Resolution

Issue with concurrent facts upload.

• Add entries for undesired facts in the exclude parameter to remove facts from collection : Log in Satellite UI -> Administer -> Settings -> Provisioning -> Exclude pattern for facts stored in satellite

• Execute below command on Satellite machine to cleanup extra facts:

   # foreman-maintain maintenance-mode start   
   # foreman-rake interfaces:clean --trace
   # foreman-rake facts:clean --trace
   # foreman-maintain maintenance-mode stop
  

Be aware of the following

  - Satellite is put into maintenance mode during the procedure, causing client activity (WebUI, Hosts requests) is blocked
  - The rake script can take significant amount of time - depending on number of facts or interfaces to clean, it can run up to several hours. 

Reverse lookup query time exceeds apache timeout.

• Set correct nameserver in file /etc/resolv.conf .

For more KB articles/solutions related to Red Hat Satellite 6.x Client Subscription Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Client Subscription Issues

Root Cause

• Multiple facts are uploaded to satellite during registration process causing timeout.

Diagnostic Steps

• Below error is observed in /var/log/rhsm/rhsm.log file on client host machine:

     2021-01-01 12:00:00,193 [ERROR] subscription-manager:9177:MainThread @managercli.py:182 - Server error attempting a POST to /rhsm/consumers?owner=ORG&activation_keys=Activation_Key returned status 502
     Traceback (most recent call last):
     File "/usr/lib64/python2.6/site-packages/subscription_manager/managercli.py", line 1113, in _do_command
       type=self.options.consumertype
      File "/usr/lib64/python2.6/site-packages/rhsmlib/services/register.py", line 80, in register
       type=type
      File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 966, in registerConsumer
       return self.conn.request_post(url, params)
      File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 693, in request_post
       return self._request("POST", method, params, headers=headers)
      File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 716, in _request
       info=info, headers=headers)
      File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 595, in _request
       self.validateResponse(result, request_type, handler)
      File "/usr/lib64/python2.6/site-packages/rhsm/connection.py", line 660, in validateResponse
       handler=handler)
     RemoteServerException: Server error attempting a POST to /rhsm/consumers?owner=ORG&activation_keys=Activation_Key returned status 502
  

If issue is because of concurrent facts upload.

• Execute below commands on the host server to check the generated facts:.

  # subscription-manager facts
   
  # cat /etc/rhsm/facts/*
  

• Check the stored facts on Satellite server using following command:

   # su - postgres -c 'psql foreman -c " select * from fact_names;"'
  

Reverse lookup time exceeds apache timeout.

• By default satellite perform reverse-lookup query for registering host IP address and apache on satellite/capsule is set to 60 seconds.

• Execute below command on the satellite to check how much time is required for satellite to make reverse-lookup query

  # foreman-rake console << EOF
      Benchmark.measure {  pp Resolv::DNS.new.getnames("IP-address of -client ") }
      EOF
  

  Please note, it is not necessary for DNS to do reverse-lookup but, it has to respond within timeout.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.