[Satellite 6] Publishing new version of content view fails with error "ProxyAPI::ProxyException: ERF12-4115 [ProxyAPI::ProxyException]"

Solution Verified - Updated

Environment

  • Red Hat Satellite 6

Issue

  • Publishing new version of content view fails with error as below,
ProxyAPI::ProxyException: ERF12-4115 [ProxyAPI::ProxyException]: Unable to get classes from Puppet for puppet-environment-name ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy https://satellite.example.com:9090/puppet

ProxyAPI::ProxyException: ERF12-4115 [ProxyAPI::ProxyException]: Unable to get classes from Puppet for puppet-environment-name ([RestClient::Forbidden]: 403 Forbidden) for proxy https://satellite.example.com:9090/puppet"

"ProxyAPI::ProxyException: ERF12-4115 [ProxyAPI::ProxyException]: Unable to get classes from Puppet for puppet-environment-name ([Errno::ECONNREFUSED]: Connection refused - connect(2)) for proxy https://satellite.example.com:9090/puppet"

ProxyAPI::ProxyException: ERF12-2749 
 [ProxyAPI::ProxyException]: Unable to get environments from Puppet
 ([OpenSSL::SSL::SSLError]: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verif...) for proxy https://satellite.example.com:9090/puppet

ERF12-4115 [ProxyAPI::ProxyException]: Unable to get environments from for puppet-environment-name ([OpenSSL::SSL::SSLError]: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verif...) for proxy https://satellite.example.com:9090/puppet

ERF12-4115 [ProxyAPI::ProxyException]: Unable to get environments from Puppet puppet-environment-name ([OpenSSL::SSL::SSLError]: hostname does not match the server certificate) for Proxy https://satellite.example.com:9090/puppet
  • Looking in /var/log/foreman-proxy/proxy.log it shows,
ERROR -- : Failed to show puppet classes: Failed to query Puppet find environments API (503): This website is under heavy load
We're sorry, too many people are accessing this website at the same time. We're working on this problem. Please try again later.
ip-address - - [07/Jan/2016 16:49:35] "GET /puppet/environments/puppet-environment-name/classes HTTP/1.1" 406 259 0.3676

OR

ERROR -- : Failed to show puppet classes: Permission denied - /etc/puppet/modules/cis/manifests/init.pp
ip-address - - [07/Oct/2015 09:01:45] "GET /puppet/environments/puppet-environment-name/classes HTTP/1.1" 406 92 0.4038

Resolution

  • There can be multiple reasons for this problem, check below things, in below steps replace satellite.example.com with correct Fully Qualified Domain Name(FQDN) of Satellite system,

    • Start foreman-proxy service if it is not running,
    # service foreman-proxy restart
# systemctl restart foreman-proxy
    • Edit file /etc/foreman-proxy/settings.yml and under the section trusted_hosts add Satellite's Fully Qualified Domain Name(FQDN) and restart foreman-proxy service,
    # the hosts which the proxy accepts connections from
# commenting the following lines would mean every verified SSL connection allowed
:trusted_hosts:
  - satellite.example.com
    • Correct entry for Satellite's Fully Qualified Domain Name(FQDN) inside /etc/hosts, replace Satellite's correct ipaddress with satellite-ip-address, 
    satellite-ip-address satellite.example.com

    • Correct DNS forward and reverse entries for Satellite's Fully Qualified Domain Name(FQDN).

    • Confirm passenger services are not blocked by selinux.

    • If /var/log/foreman-proxy/proxy.log logs having messages as This website is under heavy load then it is good idea to tune passenger parameters, refer Satellite performance tuning guide for same.

    • Correct ownership and permissions of directories and files in /etc/puppetlabs/code/environments. Ownership of everything should be apache:apache and permissions of directories should be 755.

    # chown apache:apache /etc/puppetlabs/code/environments/* -R
# for dir in $(find /etc/puppetlabs/code/environments -type d); do  chmod 755 $dir;  done
    • Verify that the puppet CA has not expired. If it has expired, follow the steps in this solutions article. To check the date,
    # openssl x509 -in /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem -dates | head -n2
# openssl x509 -in /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem -noout -startdate -enddate -subject -issuer
# openssl x509 -in  /etc/puppetlabs/puppet/ssl/certs/ca.pem -noout -startdate -enddate -subject -issuer
# openssl verify -CAfile  /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem /etc/puppetlabs/puppet/ssl/certs/`hostname -f`.pem
    • For Satellite 6.1 check if file /usr/share/gems/gems/passenger-4.0.18/ruby_extension_source/extconf.rb is present on the system, confirm its coming from Red Hat provided package and not from upstream package. If file /usr/share/gems/gems/passenger-4.0.18/ruby_extension_source/extconf.rb is not from Red Hat then move it to backup location and restall package rubygem-passenger and restart services,
    # rpm -qi `rpm -qf /usr/share/gems/gems/passenger-4.0.18/ruby_extension_source/extconf.rb`
Packager    : Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>
Vendor      : Red Hat, Inc.

# yum reinstall rubygem-passenger
# katello-service restart
    • If this is happening on Satellite 6.1 then from Satellite system backup all directories and files recursively from /etc/pki/katello-certs-tools/{private,certs}. Remove all RPM packages that provides the files certificates under /etc/pki/katello-certs-tools/{private,certs} because the installer is not re-installing the RPMs with the new CA information. After this re-run Satellite 6.1 installer,
    # for i in $(ls /etc/pki/katello-certs-tools/certs/*); 
    do
        rpm -qf $i >> /tmp/pkgs
    done
# yum remove `cat /tmp/pkgs | sort | uniq` -y
# find  /etc/pki/katello-certs-tools/* -type f -exec rm -f {} \; && rm -f /tmp/pkgs
# katello-installer --certs-update-server --certs-update-all

For more KB articles/solutions related to Red Hat Satellite 6.x Content View Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Content View Issues

Root Cause

  • The foreman-proxy service is not running.
  • The foreman-proxy is dependent upon proper forward and reverse DNS records. Currently, Satellite installer does not notify the user if there are issues with forward and reverse DNS. This is currently being tracked in This content is not included.Bugzilla 1243566 - katello-installer should do dns forward reverse validation
  • Default Apache httpd Passenger module (phusionpassenger) parameters may be not enough for Satellite 6's installation under heavy load.
  • Satellite's DNS records are incorrect or Satellite has different hostname and ipaddress entry inside /etc/hosts.
  • Incorrect permissions or ownership of puppet module's init.pp file.
  • Incorrect permissions or ownership of puppet module directories or files under /etc/puppetlabs/code/environments.
  • On Satellite 6.1 old certificates are present under /etc/pki/katello-certs-tools directory.
  • Expired Puppet CA certificate

Diagnostic Steps

  • Check dynflow console of failed task to see errors.
  • Check /var/log/foreman-proxy/proxy.log for errors.
  • Check Satellite systems DNS entries, check if there is any valid answer from DNS server,
# dig -x satellite.example.com

  • Check for ownership and permissions of directories and files under /etc/puppetlabs/code/environments. Ownership should be apache:apache and permissions should be 755.

  • Check if below command succeeds (it mimics the foreman request to foreman-proxy):

curl -v --cacert /etc/foreman/proxy_ca.pem --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem https://$(hostname -f):9090/puppet/environments
SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.