Yum commands and registration to Red Hat Satellite or Capsule fail due to SSL-related errors.

Solution Verified - Updated

Environment

  • Red Hat Satellite 6
  • Red Hat Capsule 6
  • Custom SSL Certificates

Issue

  • Registration to Red Hat Satellite or Red Hat Capsule is failing with certificate key usage inadequate for attempted operation.

  • Where to look if there is an issue with SSL certificates or connectivity over HTTPS with Red Hat SatelliteorRed Hat Capsule`?

  • How to verify and troubleshoot whether the SSL certificate installed on the Client systems are matching with Red Hat satellite or Red Hat Capsule?

  • When registering systems to Red Hat Satellite/Capsule using subscription-manager fails with Unable to verify server's identity: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) .

  • After updating custom SSL certificates on the Satellite and the Capsule, subscription-manager fails with the below error.

    Unable to verify server's identity: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
    
  • After updating custom SSL certificates on Red Hat Satellite 6, the dnf commands issued from content hosts are encountering the following errors:

     - Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://satellitetest.example.com/pulp/repos/Test/Library/content/dist/rhel8/8/x86_64/appstream/os/repodata/repomd.xml [SSL certificate problem: self signed certificate in certificate chain]
     Error: Failed to download metadata for repo 'rhel-8-for-x86_64-appstream-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
    
       - Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://capsule.example.com/pulp/content/organization/Library/content/dist/rhel8/8/x86_64/appstream/os/repodata/repomd.xml [SSL certificate problem: unable to get local issuer certificate]
    

Resolution

  • By default, Red Hat Satellite 6 uses a self-signed SSL certificate to enable encrypted communications between Satellite Server, external Capsule Servers, and all hosts.

  • SSL certificate signed by an external/Internal Certificate Authority can be configured on Red Hat Satellite/Capsule. Refer to our solution for the steps.

  • To check whether Red Hat Satellite 6 has default self-signed or Custom aka Third-party SSL certificates configured, and to know the Location and path on the Satellite server. Refer to How to check Custom SSL certificate information after Satellite 6.x is installed?

        cat /etc/foreman-installer/scenarios.d/satellite-answers.yaml |grep -iE 'server_key|server_cert_req|server_ca_cert|server_cert' 
    
        server_cert: /root/sat_cert/satellite_cert.pem
        server_key: /root/sat_cert/satellite_cert_key.pem
        server_cert_req: 
        server_ca_cert:  /root/sat_cert/ca_cert_bundle.pem
    

    If the above command output, shows no path or filename, then your Red Hat Satellite 6 is configured with default self-signed certificates.

  • If the previous commands returns the paths, check the details of Custom/External SSL certificates installed on the Satellite 6 server with the help of the below commands.

    To check the server certificates:

        openssl crl2pkcs7 -nocrl -certfile /root/sat_cert/satellite_cert.pem | openssl pkcs7 -text -print_certs |egrep '(Issuer:|Subject:|CA:|DNS:|Digital|Not Before|Not After|keyid|serial:|TLS)'
    
          Issuer: DC=AA, DC=BB, CN=PKI Class Intermediate-CA1   
                Not Before: Jun 21 21:47:00 2023 GMT
                Not After : Jun 21 21:47:00 2025 GMT
          Subject: C=XYZ, ST=Moon, L=Earth, O=IT, OU=Data Center, CN=satellite.example.com 
    

    Check the expiration date of your SSL certificate by reviewing the "Not After" field.

    To check the CA hierarchy/chain:

        openssl crl2pkcs7 -nocrl -certfile /root/sat_cert/ca_cert_bundle.pem | openssl pkcs7 -text -print_certs |egrep '(Issuer:|Subject:|CA:|DNS:|Digital|Not Before|Not After|keyid|serial:|TLS)'
    
           Issuer: CN=PKI Class RootCA   
                Not Before: Oct 14 17:29:07 2015 GMT
                Not After : Oct  6 17:29:07 2045 GMT                                                             
           Subject: DC=AA, DC=BB, CN=PKI Class Intermediate-CA1               
    
           Issuer: CN=PKI Class RootCA    
               Not Before: Jul  6 17:38:11 2015 GMT
               Not After : Jun 26 17:38:11 2055 GMT
           Subject: CN=PKI Class RootCA           
    
  • According to the Red Hat Satellite 6 documentation, server certificate should be having X. 509 v3 Key Usage and Extended Key Usage extensions with required values. Refer to katello-cert-check utility failing on certificate does not allow for the Digital Signature key usage.

    X509v3 Key Usage: critical
        Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment
    X509v3 Extended Key Usage: 
        TLS Web Server Authentication, TLS Web Client Authentication
    

    [Errno 14] curl#60 - "Certificate key usage inadequate for attempted operation." observed when server certificate does not have Key Usage and those values.

  • The satellite-installer tool generates a bootstrap RPM package called [katello-ca-consumer-latest.noarch.rpm]. This RPM package is used to deploy on a client system in the context of a Red Hat Satellite or External Capsule setup. The purpose of this package is to facilitate the configuration of default self-signed certificates or certificates signed by an External/Internal Certificate Authority (CA).

  • katello-ca-consumer package has a script which executed on a client after installing the package, and it makes changes to the values in rhsm.conf and places the CA certificate into /etc/rhsm/ca/katello-server-ca.pem

  • Once the client starts communicating to Red Hat Satellite or Red Hat Capsule over SSL it uses /etc/rhsm/ca/katello-server-ca.pem for authentication purposes.

  • By installing the katello-ca-consumer-latest.noarch.rpm package on a client system, the necessary certificate and trust configurations are established. This enables secure communication between the client and the Red Hat Satellite server or External Capsule.

  • If any changes, renewals, or updates are made to the SSL certificates in a Satellite or Capsule configuration, or if old certificates are still present on a content host, it can lead to communication failures between the satellite/capsule and the client. This typically happens in the following scenarios:

    • When you are updating default self-signed certificates to Internal/External CA-signed certificates.
    • When you are renewing Existing CA-signed certificates from one CA to another CA.
  • In the above scenarios, it is necessary to redeploy the CA certificate chain on all the hosts including the external capsules. This can be performed using either of the following processes.

    1. Refresh the CA certificate chain on the client using Foreman CA Refresh feature.

      Note: This feature is only available on Satellite 6.16 and above.

    2. Update/Install the latest katello-ca-consumer package on the clients.

    ```
    curl --insecure --output katello-ca-consumer-latest.noarch.rpm https://<Satellite_Or_Capsule_FQDN>/pub/katello-ca-consumer-latest.noarch.rpm
    yum localinstall katello-ca-consumer-latest.noarch.rpm 
    ```
    
    
    _Note: katello-ca-consumer package is currently in deprecated state and is expected to be removed in a future version of Satellite._
    
  • To avoid communication failures, it is recommended to keep the SSL certificates properly managed and ensure that all systems involved are updated with the latest certificates when changes or updates occur in the Satellite/Capsule configuration. This will help maintain secure and uninterrupted communication between the Satellite/Capsule and the clients.

Root Cause

  • subscription-manager and yum commands are failing on the below error.

        # subscription-manager register --org="Test" --activationkey="RHEL-server"
        Unable to verify server's identity: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
    
    
        # subscription-manager refresh
        Network error, unable to connect to the server. Please see /var/log/rhsm/rhsm.log for more information.
    
        2022-05-12 20:27:11,195 [ERROR] subscription-manager:1213883:MainThread @entcertlib.py:121 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
    Traceback (most recent call last):
        -- snip--
        self._sslobj.do_handshake()
        ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897)
    
  • yum repolist getting the below error on a client connected to the Red Hat Satellite server.

Content from satellite.example.com is not included.https://satellite.example.com/pulp/repos/Test/Dev/RHEL_7/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml:%20[Errno%2014]%20curl#60%20-%20%22Peer's%20certificate%20issuer%20has%20been%20marked%20as%20not%20trusted%20by%20the%20user.%22

  • curl command failed on the below error when tried to check the Red Hat Satellite connection over port 443

     # curl https://<Satellite_Or_Capsule_FQDN>:443 --cacert /etc/rhsm/ca/katello-server-ca.pem
     curl: (60) Peer's certificate issuer has been marked as not trusted by the user.
    
  • The content host did not have the latest katello-ca-consumer package from the Red Hat Satellite or Capsule, which resulted in the SSL error.

Diagnostic Steps

  • Check the Date and Time on the Client are correctly set using timedatectl.

  • To verify the CA certificates installed on a content-host match with Red Hat Satellite or Red Hat Capsule’s installedCA` certificate.

    • Use openssl command to see the details of /etc/rhsm/ca/katello-server-ca.pem from content-host.

           md5sum /etc/rhsm/ca/katello-server-ca.pem
      
           openssl crl2pkcs7 -nocrl -certfile  /etc/rhsm/ca/katello-server-ca.pem  | openssl pkcs7 -text -print_certs |grep -iE "Serial Number:|Issuer:|Subject:|Not After :|serial:|X509v3 Authority Key Identifier:|X509v3 Subject Key Identifier:" -A 2
      
    • Verify the CA certificates installed on the Red Hat Satellite or Red Hat Capsule server, which must be matched with certificates installed on content-host.

           md5sum /etc/pki/katello/certs/katello-server-ca.crt
      
           openssl crl2pkcs7 -nocrl -certfile  /etc/foreman-proxy/foreman_ssl_ca.pem  | openssl pkcs7 -text -print_certs |grep -iE "Serial Number:|Issuer:|Subject:|Not After :|serial:|X509v3 Authority Key Identifier:|X509v3 Subject Key Identifier:" -A 2
      
  • IMPORTANT NOTE: Review the output of the above openssl command and check the expiration date of your SSL certificate by examining the Not After field.

  • Check the katello-ca-consumer version installed on the host and the latest version on the Satellite/Capsule is the same:

    • On the Satellite/Capsule:
    ls -lht /var/www/html/pub/
    
    • On the content host:
    rpm -qa | grep katello-ca-consumer
    
  • Check for a proxy or transparent proxy settings added in the configuration.

    echo $http_proxy $https_proxy $HTTP_PROXY $HTTPS_PROXY
    env |grep -i proxy
    cat /etc/rhsm/rhsm.conf | grep proxy
    cat /etc/yum.conf |grep -i "proxy"
    
  • Verify if there are any network devices such as proxies or firewalls in your network that are not intercepting SSL traffic between the content host and the Red Hat Satellite/Capsule.

    tracepath -p 443 <Satellite_Or_Capsule_FQDN>
    openssl  s_client -connect  <Satellite_Or_Capsule_FQDN>:443 -CAfile /etc/rhsm/ca/katello-server-ca.pem
    curl -v https://<Satellite_Or_Capsule_FQDN>:443 --cacert /etc/rhsm/ca/katello-server-ca.pem
    curl -v --proxy-user user:password --proxy proxy.example.com:8080  https://<Satellite_Or_Capsule_FQDN>:443 --cacert /etc/rhsm/ca/katello-server-ca.pem
    
  • It is essential to ensure that SSL traffic between the content host and the Satellite/Capsule is not intercepted or blocked by any network devices. Proxies or firewalls that intercept SSL traffic can potentially cause communication issues by inspecting or modifying the SSL packets.

SBR
Product(s)
Components
Category

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.