Common networking issues while accessing Red Hat Subscription Management (RHSM)
Your network routes, proxy server, and firewall will need to allow network connections from your Red Hat Enterprise Linux based systems to the various URLs and ports used by Red Hat Subscription Management (RHSM). When you run the yum or subscription-management command, a network request is made to one or more of your system's registered package repositories or the subscription management service hosted by Red Hat.
If your route or proxy or firewall are not configured to allow these requests, yum and similar client or administration commands such as subscription-manager will fail and one or more of the following errors may be reported:
- Network error, unable to connect to server. Please see /var/log/rhsm/rhsm.log for more information.
- error: [Errno 111] Connection refused.
- [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 407 Proxy Authentication Required.
- Unable to verify server's identity: (104, 'Connection reset by peer').
- gaierror: (-3, 'Temporary failure in name resolution').
- [Errno 256] No more mirrors to try.
- error: Tunnel connection failed: 403 Forbidden.
- [Errno 14] PYCURL ERROR 7 - "couldn't connect to host".
- Errno 14] curl#35 - "Encountered end of file" Trying other mirror.
- Unable to verify server's identity: ('_ssl.c:602: The handshake operation timed out',)
- [Errno 14] curl#35 - "TCP connection reset by peer"
- Subscription-manager command fails with the error "Tunnel connection failed: 407 authenticationrequired"
- Red Hat Subscription Manager Commands Return 'Invalid Credentials'.
- Unable to verify server's identity: certificate verify failed.
- SSLError: certificate verify failed.
- Unable to verify server's identity: [SSL: PARSE_TLSEXT] parse tlsext (_ssl.c:579).
- Unable to verify server's identity: unexpected eof.
- Unable to verify server's identity: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)
- [Errno 14] problem making ssl connection.
- [Errno 14] curl#60 - "Peer's certificate issuer has been marked as not trusted by the user."
- [Errno 14] curl#56 - "Callback aborted" Trying other mirror.
- Error during registration: Unable to connect to: proxy.example.com:8080 [Errno -2] Name or service not known
- Unable to connect to: proxy.example.com:8080 [Errno -2] Name or service not known
- Proxy error, unable to connect to proxy server.
- error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error
Some of the most common causes of these connectivity failures and issues are:
- The system is not able to access the public network or reach the internet.
- Firewall or proxy is not properly configured to access Red Hat Subscription Manager (RHSM) and its Content Delivery Network (CDN).
- Incorrect or missing configuration in
/etc/rhsm/rhsm.conf. - The system's clock is incorrect.
- Firewall interrupts the TLS/SSL connection and uses a self-signed certificate. This will result in
subscription-managerfailing due to the unexpected certificate. - A proxy server or other network access control software is disassembling and reassembling SSL certificates. This is typically done so that HTTPS sessions can be cached and results in the peer's certificate issuer being different than expected.
Review the logs for network errors and traceback
subscription-manager and yum commands will typically log status and failure messages to /var/log/rhsm/rhsm.log. Review the log file for hints regarding what is actually failing and the specific root error.
You can also test basic connectivity using curl or openssl as a means to assist with troubleshooting.
Firewall configuration considerations
It is necessary to allow the following host names and ports on the corporate firewall to enable yum and subscription-manager to access Red Hat subscription services and Content Delivery Network:
- [https] subscription.rhn.redhat.com:443
- [https] subscription.rhsm.redhat.com:443
- [https] cdn.redhat.com:443
- [https] *.akamaiedge.net:443
- [https] *.akamaitechnologies.com:443
NOTE: It is not recommended that IP addresses be specified as the hosts are typically load balanced and the IP addresses will change frequently. Red Hat utilizes the Content from www.akamai.com is not included.Akamai Content Delivery Network (CDN). However, if your firewall is unable to use host name filtering, Red Hat provides a pool of IP addresses that should provide CDN delivery.
If the corporate firewall is performing SSL inspection the following changes will be needed:
- Disable SSL inspections on *.redhat.com.
- Whitelist Red Hat's CA Certificate redhat-uep.pem in Firewall's Certificate so these CAs will not get conflicted while receiving traffic via *.redhat.com.
Common URLs (hosts and ports)
- This content is not included.https://subscription.rhn.redhat.com/
- This content is not included.https://subscription.rhsm.redhat.com/
- This content is not included.https://cdn.redhat.com/
How to check connectivity
When troubleshooting connectivity issues you can check basic connectivity using the curl or openssl commands.
Using curl command
To check basic connectivity using curl the command syntax will be:
curl -v <URL> --cacert /etc/rhsm/ca/redhat-uep.pem
If your system is behind a proxy, you will need to provide your proxy server host address and credentials using curl's --proxy-user and --proxy arguments:
curl -v --proxy-user <PROXY_USER>:<PROXY_PASSWORD> --proxy <PROXY_HOST>:<PROXY_PORT> <URL> --cacert /etc/rhsm/ca/redhat-uep.pem
You should test connectivity to all the common URLs. For example:
# curl -v https://subscription.rhn.redhat.com --cacert /etc/rhsm/ca/redhat-uep.pem
# curl -v https://subscription.rhsm.redhat.com --cacert /etc/rhsm/ca/redhat-uep.pem
# curl -v https://cdn.redhat.com --cacert /etc/rhsm/ca/redhat-uep.pem
The output from the curl command should help you identify your issue.
Example output from failed connection using curl
[root@server]# curl -v https://subscription.rhsm.redhat.com --cacert /etc/rhsm/ca/redhat-uep.pem
* About to connect() to subscription.rhsm.redhat.com port 443 (#0)
* Trying xx:xxxx::xxxx:xxxx...
* Connected to subscription.rhsm.redhat.com (xx:xxxx::xxxx:xxxx) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/rhsm/ca/redhat-uep.pem
CApath: none
* NSS error -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
* Closing connection 0
curl: (35) TCP connection reset by peer
Example output from successful connection using curl
[root@server ~]# curl -v https://subscription.rhsm.redhat.com --cacert /etc/rhsm/ca/redhat-uep.pem
* About to connect() to subscription.rhsm.redhat.com port 443 (#0)
* Trying xx:xxxx::xxxx:xxxx...
* Connected to subscription.rhsm.redhat.com (xx:xxxx::xxxx:xxxx) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/rhsm/ca/redhat-uep.pem
CApath: none
* NSS: client certificate not found (nickname not specified)
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
* subject: E=ca-support@redhat.com,CN=subscription.rhsm.redhat.com,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US
* start date: May 22 14:00:55 2019 GMT
* expire date: May 21 14:00:55 2021 GMT
* common name: subscription.rhsm.redhat.com
* issuer: E=ca-support@redhat.com,CN=Red Hat Entitlement Operations Authority,OU=Red Hat Network,O="Red Hat, Inc.",ST=North Carolina,C=US
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: subscription.rhsm.redhat.com
> Accept: */*
>
* HTTP 1.0, assume close after body
< HTTP/1.0 404 Not Found
< Server: BigIP
* HTTP/1.0 connection set to keep alive!
< Connection: Keep-Alive
< Content-Length: 0
<
* Connection #0 to host subscription.rhsm.redhat.com left intact
[root@server ~]#
Using openssl command
To check basic connectivity using openssl the command syntax will be:
openssl s_client -connect <HOST>:443 -CAfile /etc/rhsm/ca/redhat-uep.pem
If your system is behind a proxy, you will need to provide your proxy server host address using openssl s_client's -proxy argument:
openssl s_client -proxy <PROXY_HOST>:<PROXY_PORT> -connect <HOST>:443 -CAfile /etc/rhsm/ca/redhat-uep.pem
You should test connectivity to all the hosts from the list of common URLs. For example:
# openssl s_client -connect subscription.rhn.redhat.com:443 -CAfile /etc/rhsm/ca/redhat-uep.pem
# openssl s_client -connect subscription.rhsm.redhat.com:443 -CAfile /etc/rhsm/ca/redhat-uep.pem
# openssl s_client -connect cdn.redhat.com:443 -CAfile /etc/rhsm/ca/redhat-uep.pem
The output from the openssl command should help you identify your issue.
Example output from failed connection using openssl
# openssl s_client -connect subscription.rhn.redhat.com:443 -CAfile /etc/rhsm/ca/redhat-uep.pem
CONNECTED(00000003)
depth=2 C = US, ST = North Carolina, L = Black, O = "Organization Name, Inc.", OU = IT, CN = Organization Name, emailAddress = user@Organization.Name.com
verify return:1
depth=1 C = US, ST = North Carolina, L = Black, O = "Organization Name, Inc.", OU = IT, CN = Organization.Name.com
verify return:1
depth=0 C = US, ST = North Carolina, O = "Red Hat, Inc.", OU = Red Hat Network, CN = subscription.rhn.redhat.com, emailAddress = customerservice@redhat.com
verify return:1
---
Certificate chain
0 s:/C=US/ST=North Carolina/O=Red Hat, Inc./OU=Red Hat Network/CN=subscription.rhn.redhat.com/emailAddress=customerservice@redhat.com
i:/C=US/ST=North Carolina/L=Organization Name/O=Organization Name, Inc./OU=IT/CN=Organization.Name.com
1 s:/C=US/ST=North Carolina/L=Organization Name/O=Organization Name, Inc./OU=IT/CN=Organization.Name.com
i:/C=US/ST=North Carolina/L=Organization Name/O=Organization Name, Inc./OU=IT/CN=Organization CA Root Certificate/emailAddress=user@Organization.Name.com
2 s:/C=US/ST=North Carolina/L=Organization Name/O=Organization Name, Inc./OU=IT/CN=Organization CA Root Certificate/emailAddress=user@Organization.Name.com
i:/C=US/ST=North Carolina/L=Organization Name/O=Organization Name, Inc./OU=IT/CN=Organization CA Root Certificate/emailAddress=user@Organization.Name.com
Example output from successful connection using openssl
# openssl s_client -connect subscription.rhn.redhat.com:443 -CAfile /etc/rhsm/ca/redhat-uep.pem
CONNECTED(00000003)
depth=2 C = US, ST = North Carolina, L = Raleigh, O = "Red Hat, Inc.", OU = Red Hat Network, CN = Entitlement Master CA, emailAddress = ca-support@redhat.com
verify return:1
depth=1 C = US, ST = North Carolina, O = "Red Hat, Inc.", OU = Red Hat Network, CN = Red Hat Entitlement Operations Authority, emailAddress = ca-support@redhat.com
verify return:1
depth=0 C = US, ST = North Carolina, O = "Red Hat, Inc.", OU = Red Hat Network, CN = subscription.rhn.redhat.com, emailAddress = customerservice@redhat.com
verify return:1
Certificate chain
0 s:/C=US/ST=North Carolina/O=Red Hat, Inc./OU=Red Hat Network/CN=subscription.rhn.redhat.com/emailAddress=customerservice@redhat.com
i:/C=US/ST=North Carolina/O=Red Hat, Inc./OU=Red Hat Network/CN=Red Hat Entitlement Operations Authority/emailAddress=ca-support@redhat.com
Verify the configuration in rhsm.conf
If your system accesses the public network using a proxy, verify the proxy_hostname and proxy_port are properly defined in /etc/rhsm/rhsm.conf. If your proxy requires authentication, verify the proxy_user and proxy_password values as well.
$ grep -E '^\s*(proxy_hostname|proxy_scheme|proxy_port|proxy_user|proxy_password)' /etc/rhsm/rhsm.conf
proxy_hostname = myproxy.example.com
proxy_scheme = http
proxy_port = 8080
proxy_user = proxy_username
proxy_password = proxy_password
Check the hostname, prefix, and baseurl values in /etc/rhsm/rhsm.conf are correct and the host names can be resolved.
$ grep -E '^\s*(hostname|prefix|baseurl)' /etc/rhsm/rhsm.conf
hostname = subscription.rhsm.redhat.com
prefix = /subscription
baseurl= https://cdn.redhat.com
NOTE:
hostnamecan be eithersubscription.rhsm.redhat.comorsubscription.rhn.redhat.com.
$ ping -c 1 subscription.rhsm.redhat.com
PING subscription.rhsm.redhat.com (209.132.183.108) 56(84) bytes of data.
...
You can also use dig to get a complete DNS record for the target host. If the output indicates the host name is invalid or an IP address cannot be resolved, you will need to verify and correct your DNS configuration.
Verify DNS configuration
If any of the target host names can not be properly resolved, review the contents of /etc/resolve.conf to confirm that the nameserver definition and order is correct.
Verify the system has the correct date and time
The system clock must be current.
-
Verify the time zone is set correctly:
$ grep -E '^\s*ZONE' /etc/sysconfig/clock -
Verify the proper date and time are reported and the proper time zone offset is applied:
$ date
If the system's time is off by more than a few seconds, you should verify that the system's time service client is able to communiate with its peers and that they are reporting accurate time with minimal jitter.
-
Red Hat Enterprise Linux 5 and 6:
# ntpq -p -
Red Hat Enterprise Linux 7:
-
Check sources:
# chronyc sources -
Check jitter:
# chronyc tracking
-
Verify Enabled Repositories and Release Versions
Occasionally, a repo that does not match the OS version can be accidentally enabled which could produce a similar following error when executing yum repolist or yum check-update:
failure: repodata/repomd.xml from rhel-7-server-rpms: [Errno 256] No more mirrors to try.
https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found
In the event of the above error, review enabled repositories and release versions.