Puppet on client registered with Red Hat Satellite 6 fails with "Could not retrieve catalog from remote server: SSL_CTX_use_PrivateKey: key values mismatch"

Environment

• Red Hat Satellite 6
• puppet

Issue

• Puppet on client registered with Red Hat Satellite 6 fails with "Could not retrieve catalog from remote server: SSL_CTX_use_PrivateKey: key values mismatch"

Feb 13 11:18:40 hostname puppet-agent[24441]: Unable to fetch my node definition, but the agent run will continue:
Feb 13 11:18:40 hostname puppet-agent[24441]: SSL_CTX_use_PrivateKey: key values mismatch
Feb 13 11:18:40 hostname puppet-agent[24441]: (/File[/var/lib/puppet/lib]) Failed to generate additional resources using 'eval_generate': SSL_CTX_use_PrivateKey: key values mismatch
Feb 13 11:18:40 hostname puppet-agent[24441]: (/File[/var/lib/puppet/lib]) Could not evaluate: Could not retrieve file metadata for puppet://satellite6.example.com/plugins: SSL_CTX_use_PrivateKey: key values mismatch
Feb 13 11:18:40 hostname puppet-agent[24441]: (/File[/var/lib/puppet/lib]) Wrapped exception:
Feb 13 11:18:40 hostname puppet-agent[24441]: (/File[/var/lib/puppet/lib]) SSL_CTX_use_PrivateKey: key values mismatch
Feb 13 11:18:41 hostname puppet-agent[24441]: Could not retrieve catalog from remote server: SSL_CTX_use_PrivateKey: key values mismatch
Feb 13 11:18:41 hostname puppet-agent[24441]: Using cached catalog
Feb 13 11:18:41 hostname puppet-agent[24441]: Local environment: "production" doesn't match server specified environment "KT_Default_Organization_test_RHEL_7_4", restarting agent run with environment "KT_Default_Organization_test_RHEL_7_4"
Feb 13 11:18:41 hostname puppet-agent[24441]: Could not retrieve catalog from remote server: SSL_CTX_use_PrivateKey: key values mismatch
Feb 13 11:18:41 hostname puppet-agent[24441]: Using cached catalog
Feb 13 11:18:41 hostname puppet-agent[24441]: Finished catalog run in 0.05 seconds
Feb 13 11:18:41 hostname puppet-agent[24441]: Could not send report: SSL_CTX_use_PrivateKey: key values mismatch

Resolution

• Regenerate the puppet cert for the client on which this issue is observed.
• First remove the cert form the satellite server by running the below command on the Red Hat Satellite 6 Server:

    # puppet cert clean <Client HOSTNAME>    # works on puppet 5 or older
    # puppetserver ca clean --certname <Client HOSTNAME>    # works since puppet 6

• Now run the below command on the clients running puppet 3:

    # rm -rf /var/lib/puppet/ssl
    # service puppet restart
    # puppet agent --test --debug

For clients running puppet 4 run the below commands:

  # rm -rf /etc/puppetlabs/puppet/ssl/*     
  # service puppet restart
  # puppet agent --test --debug

• Sign ssl certificate for the puppet client from the Satellite v 6 server webui:

 - Login to Satellite webui
 - Go to `Infrastructure` -> `Capsules` 
 - Click on `Certificates`, go to an entry for the client system hostname with Sign button available for it. 
 - Click on `Sign` button.

For more KB articles/solutions related to Red Hat Satellite 6.x Puppet Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Puppet Issues

Root Cause

• The Puppet cert on the client got corrupted so needs to be regenerated.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.