Yum commands and registration to Red Hat Satellite or Capsule fail due to SSL-related errors.
Environment
- Red Hat Satellite 6
- Red Hat Capsule 6
- Custom SSL Certificates
Issue
-
Registration to Red Hat Satellite or Red Hat Capsule is failing with certificate key usage inadequate for attempted operation.
-
Where to look if there is an issue with
SSLcertificates or connectivity overHTTPSwith Red Hat SatelliteorRed Hat Capsule`? -
How to verify and troubleshoot whether the
SSLcertificate installed on the Client systems are matching withRed Hat satelliteorRed Hat Capsule? -
When registering systems to
Red Hat Satellite/Capsuleusingsubscription-managerfails with Unable to verify server's identity: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) . -
After updating custom
SSL certificateson the Satellite and the Capsule,subscription-managerfails with the below error.Unable to verify server's identity: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) -
After updating custom SSL certificates on
Red Hat Satellite6, the dnf commands issued from content hosts are encountering the following errors:- Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://satellitetest.example.com/pulp/repos/Test/Library/content/dist/rhel8/8/x86_64/appstream/os/repodata/repomd.xml [SSL certificate problem: self signed certificate in certificate chain] Error: Failed to download metadata for repo 'rhel-8-for-x86_64-appstream-rpms': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried- Curl error (60): Peer certificate cannot be authenticated with given CA certificates for https://capsule.example.com/pulp/content/organization/Library/content/dist/rhel8/8/x86_64/appstream/os/repodata/repomd.xml [SSL certificate problem: unable to get local issuer certificate]
Resolution
-
By default,
Red Hat Satellite6 uses aself-signed SSLcertificate to enable encrypted communications between Satellite Server, external Capsule Servers, and all hosts. -
SSLcertificate signed by an external/Internal Certificate Authority can be configured on Red Hat Satellite/Capsule. Refer to our solution for the steps. -
To check whether
Red Hat Satellite6 has default self-signed or Custom aka Third-party SSL certificates configured, and to know theLocationand path on the Satellite server. Refer to How to check Custom SSL certificate information after Satellite 6.x is installed?cat /etc/foreman-installer/scenarios.d/satellite-answers.yaml |grep -iE 'server_key|server_cert_req|server_ca_cert|server_cert' server_cert: /root/sat_cert/satellite_cert.pem server_key: /root/sat_cert/satellite_cert_key.pem server_cert_req: server_ca_cert: /root/sat_cert/ca_cert_bundle.pemIf the above command output, shows no path or filename, then your
Red Hat Satellite6 is configured with default self-signed certificates. -
If the previous commands returns the paths, check the details of
Custom/External SSL certificatesinstalled on the Satellite 6 server with the help of the below commands.To check the server certificates:
openssl crl2pkcs7 -nocrl -certfile /root/sat_cert/satellite_cert.pem | openssl pkcs7 -text -print_certs |egrep '(Issuer:|Subject:|CA:|DNS:|Digital|Not Before|Not After|keyid|serial:|TLS)' Issuer: DC=AA, DC=BB, CN=PKI Class Intermediate-CA1 Not Before: Jun 21 21:47:00 2023 GMT Not After : Jun 21 21:47:00 2025 GMT Subject: C=XYZ, ST=Moon, L=Earth, O=IT, OU=Data Center, CN=satellite.example.comCheck the expiration date of your SSL certificate by reviewing the "Not After" field.
To check the CA hierarchy/chain:
openssl crl2pkcs7 -nocrl -certfile /root/sat_cert/ca_cert_bundle.pem | openssl pkcs7 -text -print_certs |egrep '(Issuer:|Subject:|CA:|DNS:|Digital|Not Before|Not After|keyid|serial:|TLS)' Issuer: CN=PKI Class RootCA Not Before: Oct 14 17:29:07 2015 GMT Not After : Oct 6 17:29:07 2045 GMT Subject: DC=AA, DC=BB, CN=PKI Class Intermediate-CA1 Issuer: CN=PKI Class RootCA Not Before: Jul 6 17:38:11 2015 GMT Not After : Jun 26 17:38:11 2055 GMT Subject: CN=PKI Class RootCA -
According to the
Red Hat Satellite6 documentation, server certificate should be having X. 509 v3Key UsageandExtended Key Usageextensions with required values. Refer to katello-cert-check utility failing on certificate does not allow for the Digital Signature key usage.
X509v3 Key Usage: critical Digital Signature, Non Repudiation, Key Encipherment, Data Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication, TLS Web Client Authentication[Errno 14] curl#60 - "Certificate key usage inadequate for attempted operation." observed when server certificate does not have
Key Usageand those values. -
The
satellite-installertool generates a bootstrap RPM package called [katello-ca-consumer-latest.noarch.rpm]. This RPM package is used to deploy on a client system in the context of a Red Hat Satellite or External Capsule setup. The purpose of this package is to facilitate the configuration of default self-signed certificates or certificates signed by an External/Internal Certificate Authority (CA). -
katello-ca-consumer package has a script which executed on a client after installing the package, and it makes changes to the values in rhsm.conf and places the CA certificate into /etc/rhsm/ca/katello-server-ca.pem
-
Once the client starts communicating to Red Hat Satellite or Red Hat Capsule over SSL it uses /etc/rhsm/ca/katello-server-ca.pem for authentication purposes.
-
By installing the
katello-ca-consumer-latest.noarch.rpmpackage on a client system, the necessary certificate and trust configurations are established. This enables secure communication between the client and theRed Hat Satelliteserver or ExternalCapsule. -
If any changes, renewals, or updates are made to the SSL certificates in a Satellite or Capsule configuration, or if old certificates are still present on a content host, it can lead to communication failures between the satellite/capsule and the client. This typically happens in the following scenarios:
- When you are updating default self-signed certificates to Internal/External CA-signed certificates.
- When you are renewing Existing CA-signed certificates from one CA to another CA.
-
In the above scenarios, it is necessary to redeploy the CA certificate chain on all the hosts including the external capsules. This can be performed using either of the following processes.
-
Refresh the CA certificate chain on the client using Foreman CA Refresh feature.
Note: This feature is only available on Satellite 6.16 and above.
-
Update/Install the latest
katello-ca-consumerpackage on the clients.
``` curl --insecure --output katello-ca-consumer-latest.noarch.rpm https://<Satellite_Or_Capsule_FQDN>/pub/katello-ca-consumer-latest.noarch.rpm yum localinstall katello-ca-consumer-latest.noarch.rpm ``` _Note: katello-ca-consumer package is currently in deprecated state and is expected to be removed in a future version of Satellite._ -
-
To avoid communication failures, it is recommended to keep the SSL certificates properly managed and ensure that all systems involved are updated with the latest certificates when changes or updates occur in the Satellite/Capsule configuration. This will help maintain secure and uninterrupted communication between the Satellite/Capsule and the clients.
Root Cause
-
subscription-manager and yum commands are failing on the below error.
# subscription-manager register --org="Test" --activationkey="RHEL-server" Unable to verify server's identity: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) # subscription-manager refresh Network error, unable to connect to the server. Please see /var/log/rhsm/rhsm.log for more information. 2022-05-12 20:27:11,195 [ERROR] subscription-manager:1213883:MainThread @entcertlib.py:121 - [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) Traceback (most recent call last): -- snip-- self._sslobj.do_handshake() ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:897) -
yum repolist getting the below error on a client connected to the
Red Hat Satelliteserver.
-
curl command failed on the below error when tried to check the
Red Hat Satelliteconnection over port 443# curl https://<Satellite_Or_Capsule_FQDN>:443 --cacert /etc/rhsm/ca/katello-server-ca.pem curl: (60) Peer's certificate issuer has been marked as not trusted by the user. -
The content host did not have the latest katello-ca-consumer package from the Red Hat Satellite or Capsule, which resulted in the SSL error.
Diagnostic Steps
-
Check the Date and Time on the Client are correctly set using timedatectl.
-
To verify the
CAcertificates installed on acontent-hostmatch withRed Hat SatelliteorRed Hat Capsule’s installedCA` certificate.-
Use
opensslcommand to see the details of/etc/rhsm/ca/katello-server-ca.pemfromcontent-host.md5sum /etc/rhsm/ca/katello-server-ca.pem openssl crl2pkcs7 -nocrl -certfile /etc/rhsm/ca/katello-server-ca.pem | openssl pkcs7 -text -print_certs |grep -iE "Serial Number:|Issuer:|Subject:|Not After :|serial:|X509v3 Authority Key Identifier:|X509v3 Subject Key Identifier:" -A 2 -
Verify the
CAcertificates installed on theRed Hat SatelliteorRed Hat Capsuleserver, which must be matched with certificates installed oncontent-host.md5sum /etc/pki/katello/certs/katello-server-ca.crt openssl crl2pkcs7 -nocrl -certfile /etc/foreman-proxy/foreman_ssl_ca.pem | openssl pkcs7 -text -print_certs |grep -iE "Serial Number:|Issuer:|Subject:|Not After :|serial:|X509v3 Authority Key Identifier:|X509v3 Subject Key Identifier:" -A 2
-
-
IMPORTANT NOTE: Review the output of the above openssl command and check the expiration date of your SSL certificate by examining the Not After field.
-
Check the
katello-ca-consumerversion installed on the host and the latest version on the Satellite/Capsule is the same:- On the Satellite/Capsule:
ls -lht /var/www/html/pub/- On the content host:
rpm -qa | grep katello-ca-consumer -
Check for a proxy or transparent proxy settings added in the configuration.
echo $http_proxy $https_proxy $HTTP_PROXY $HTTPS_PROXY env |grep -i proxy cat /etc/rhsm/rhsm.conf | grep proxy cat /etc/yum.conf |grep -i "proxy" -
Verify if there are any network devices such as proxies or firewalls in your network that are not intercepting SSL traffic between the content host and the
Red Hat Satellite/Capsule.tracepath -p 443 <Satellite_Or_Capsule_FQDN> openssl s_client -connect <Satellite_Or_Capsule_FQDN>:443 -CAfile /etc/rhsm/ca/katello-server-ca.pem curl -v https://<Satellite_Or_Capsule_FQDN>:443 --cacert /etc/rhsm/ca/katello-server-ca.pem curl -v --proxy-user user:password --proxy proxy.example.com:8080 https://<Satellite_Or_Capsule_FQDN>:443 --cacert /etc/rhsm/ca/katello-server-ca.pem -
It is essential to ensure that SSL traffic between the content host and the Satellite/Capsule is not intercepted or blocked by any network devices. Proxies or firewalls that intercept SSL traffic can potentially cause communication issues by inspecting or modifying the SSL packets.
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.