Satellite 6 How To Troubleshoot Capsule Synchronization Issue

Environment

• Red Hat Satellite 6.x

Issue

• The Capsule Synchronization has thrown an error or has been running for a very long time without any progress.

• If there is no error related to the Synchronization Task(s) and you simply want to validate the content that was synced, you may also refer to the following:
  How to verify all expected content in a Lifecycle Environment has been successfully synchronized to a Capsule?

Diagnostic Steps

• Confirm the version of Satellite and Capsule are matching

  • Both need to be on the same minor version, example: 6.x.y

      [root@mysatellite ~]# rpm -q satellite
      satellite-6.9.2.1-1.el7sat.noarch
  
      [root@mycapsule ~]# rpm -q satellite-capsule
      satellite-capsule-6.9.2.1-1.el7sat.noarch
  
  

• If versions are not the same then please update outdated components and rerun satellite-installer. Please refer to the appropriate documentation:

  • Satellite 6.7
  • Satellite 6.8
  • Satellite 6.9

• Check if the all the Satellite services are up and running

  • Inspect attentively the output of the foreman-maintain service status command on both the Satellite and the Capsule to ensure no service is reporting an error

  
      [root@mysatellite ~]# foreman-maintain service status
  
      [root@mycapsule ~]# foreman-maintain service status
  
  

• Check/verify ssl connection between Satellite and Capsule servers

  
      [root@mysatellite ~]#  curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem --cacert /etc/foreman/proxy_ca.pem https://capsule.example.com:9090/features
  
      ["logs","pulpnode","puppet","puppetca","templates","tftp"]
  
  
      [root@mycapsule ~]#  curl --cert /etc/foreman-proxy/foreman_ssl_cert.pem --key /etc/foreman-proxy/foreman_ssl_key.pem --cacert /etc/foreman-proxy/foreman_ssl_ca.pem https://satellite.example.com:9090/features
  
      ["discovery","dynflow","logs","openscap","pulp","puppet","puppetca","ssh","tftp"]
  
  

• Verify the ports are open

  • As per Installation Guide, it is mandatory to have the required tcp port open
    • From Satellite to Capsule

  
      [root@mysatellite ~]# nc -v mycapsule.example.com 443
      Ncat: Version 6.40 ( http://nmap.org/ncat )
      Ncat: Connected to 192.168.20.137:443.
  
      [root@mysatellite~]# nc -v mycapsule.example.com 9090
      Ncat: Version 6.40 ( http://nmap.org/ncat )
      Ncat: Connected to 192.168.20.137:9090.
  
  

• From Capsule to Satellite:

  
      [root@mycapsule~]# nc -v mysatellite.example.com 443
      Ncat: Version 6.40 ( http://nmap.org/ncat )
      Ncat: Connected to 192.168.20.2:443.
  
      [root@mycapsule~]# nc -v mysatellite.example.com 5646
      Ncat: Version 6.40 ( http://nmap.org/ncat )
      Ncat: Connected to 192.168.20.2:5646.
  
      [root@mycapsule ~]# nc -v mysatellite.example.com 5647
      Ncat: Version 6.40 ( http://nmap.org/ncat )
      Ncat: Connected to 192.168.20.2:5647.
  
  

• Ensure the /etc/hosts file is correctly configured

  • On both Satellite and Capsule, ensure that any hostname entries in /etc/hosts is configured with the proper IP address

• Have at least one Lifecycle Environment and an Organization assigned

  • Note: Avoid assigning Library to a Capsule

• Inspect the Capsule Synchronization Task

  • Identify the ongoing Capsule Synchronization Task
  • Under Monitor > Tasks, you may use the following filter to identity the ongoing task:

  
      (label = Actions::Katello::ContentView::CapsuleGenerateAndSync or label =  Actions::Katello::Repository::CapsuleGenerateAndSync or label = Actions::Katello::CapsuleContent::Sync ) and state = running
  
  

• Click on the identified task to open its details

• Click on the Running Steps tab and pay attention to the current State:

• Waiting for Pulp to finish the task
  The Capsule has picked up the task and should be downloading the packages
  [root@mycapsule ~]# ls -lRt /var/cache/pulp/
[root@mycapsule ~]# tail -100f /var/log/messages|grep pulp
[root@mycapsule ~]# tail -100f /var/log/httpd/pulp-https_access_ssl.log

• Waiting for Pulp to start the task
  The Capsule might be processing other tasks
  In case some pulp tasks are stuck, proceed with the following:

  • Install pulp-admin on the Capsule
    Red Hat Satellite 6 : How to deploy and use pulp-admin ?
    [root@mycapsule ~]# pulpversion=$(rpm -q pulp-server --queryformat "%{VERSION}")
[root@mycapsule ~]# yum install pulp-admin-client-${pulpversion} pulp-rpm-admin-extensions.noarch pulp-rpm-handlers.noarch

• Then run this script to make sure all pulp tasks are killed:

#!/bin/bash

pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)

STATE=""
for TASK in `pulp-admin -u admin -p $pulpAdminPassword tasks list | egrep '^Task Id:|^State:' | sed -e 's,^Task Id: ,,' -e 's,^State: ,,'`; do
        if [ "$STATE" = "" ]; then
                STATE=$TASK
        else
                if [ $STATE != Successful ] && [ $STATE != Cancelled ] && [ $STATE != Failed ]; then
                        pulp-admin -u admin -p $pulpAdminPassword tasks details --task-id=$TASK
                        pulp-admin -u admin -p $pulpAdminPassword tasks cancel --task-id=$TASK
                fi
                STATE=""
        fi
done

• Remove pulp orphans:

   [root@mycapsule ~]# pulpAdminPassword=$(grep ^default_password /etc/pulp/server.conf | cut -d' ' -f2)
   [root@mycapsule ~]# pulp-admin -u admin -p $pulpAdminPassword orphan list
   [root@mycapsule ~]# pulp-admin -u admin -p $pulpAdminPassword orphan remove --all
  

For more KB articles/solutions related to Red Hat Satellite 6.x Capsule Sync Issues, please refer to the Consolidated Troubleshooting Article for Red Hat Satellite 6.x Capsule Sync Issues

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.