Remote Execution fails from Capsule server with message Error initializing command: Net::SSH::AuthenticationFailed - Authentication failed for user root@client.example.com in Red Hat Satellite 6.5.
Environment
- Red Hat Satellite 6.5 or newer
- Red Hat Satellite Capsule 6.5 or newer
Issue
-
Post upgrading Satellite and Capsule server to 6.5, remote execution fails on some or all hosts connected with capsule server.
-
Remote execution works fine via Satellite server but fails from capsule server with the message below, even after the ssh keys were copied properly.
Error initializing command: Net::SSH::AuthenticationFailed - Authentication failed for user root@client.example.com
Resolution
-
Make sure that
/usr/share/foreman-proxy/.sshis a softlink to directory/var/lib/foreman-proxy/sshon the capsule server.# ls -ld /usr/share/foreman-proxy/.ssh lrwxrwxrwx. 1 root root 26 Jun 23 13:00 /usr/share/foreman-proxy/.ssh -> /var/lib/foreman-proxy/ssh -
Please check the Diagnostic Steps before proceeding with the solution described below to fix the issue with softlinks.
-
When two different directories will be identified instead of, one being merely a soft link to the other directory, follow the steps below to re-create the proper structure.
# mkdir /root/use_share_foreman-proxy_.ssh_bkup # cp -pr /usr/share/foreman-proxy/.ssh/* /root/use_share_foreman-proxy_.ssh_bkup/ # cd /usr/share/foreman-proxy/ # rm -rf .ssh # ln -s /var/lib/foreman-proxy/ssh .ssh # ls -ld /usr/share/foreman-proxy/.ssh # for i in httpd dynflowd foreman-proxy smart_proxy_dynflow_core 'dynflow-sidekiq@*.service'; do systemctl restart $i; done
(the latest command will attempt to restart differently named service for different Sat versions - it is assumed either service as missing on either Sat version)
-
Refresh the features of the capsule from Satellite GUI, so that the new key is replaced with the old key inside foreman database.
GUI --> Infrastructure --> Capsule --> Click on the Capsule name for which SSH keys were fixed --> Click on "Refresh Features" -
Perform the Remote Execution again and verify that it is working as expected.
-
If required, re-distribute the SSH-Keys from the capsule to the affected hosts after fixing the directory structure.
-
Also if it's required to use the key present inside
/usr/share/foreman-proxy/.ssh, move those keys inside/var/lib/foreman-proxy/sshand then create the soft-link properly. -
Verify if there is any issue with the host registration. Reinstall katello-ca-consumer package and re register if required.
-
If this resolution is not applicable and the remote execution fails, despite fixing the ssh keys, contact This content is not included.Red Hat Technical Support.
For more KB articles/solutions related to Red Hat Satellite 6.x Remote Execution Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Remote Execution Issues
Root Cause
-
From Satellite\Capsule 6.4 and 6.5 onwards, there is no separate directory present by the name
/usr/share/foreman-proxy/.ssh, instead it's a soft link to the directory/var/lib/foreman-proxy/sshand actual ssh-keys being used are present inside this directory.# ls -ld /usr/share/foreman-proxy/.ssh lrwxrwxrwx. 1 root root 26 Jun 23 13:00 /usr/share/foreman-proxy/.ssh -> /var/lib/foreman-proxy/ssh # ls -ld /var/lib/foreman-proxy/ssh drwxr-x---. 2 foreman-proxy foreman-proxy 4096 Jun 20 2018 /var/lib/foreman-proxy/ssh # ls -l /var/lib/foreman-proxy/ssh/ total 32 -rw------- 1 foreman-proxy foreman-proxy 1675 Sep 19 2018 id_rsa_foreman_proxy -rw-r--r-- 1 foreman-proxy foreman-proxy 431 Sep 19 2018 id_rsa_foreman_proxy.pub -rw-r--r-- 1 foreman-proxy foreman-proxy 22295 May 30 13:37 known_hosts
Diagnostic Steps
-
Verify on Capsule server whether
/usr/share/foreman-proxy/.sshis a soft link to the directory/var/lib/foreman-proxy/sshor not.# ls -ld /usr/share/foreman-proxy/.ssh drwx------. 2 foreman-proxy foreman-proxy 85 Dec 23 2018 /usr/share/foreman-proxy/.ssh # ls -ld /var/lib/foreman-proxy/ssh/ drwxr-x---. 2 foreman-proxy foreman-proxy 66 Jun 20 2018 /var/lib/foreman-proxy/ssh/ -
If found two separate directories then verify if they contain the same SSH keys or different ones.
# cat /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCcFllRsiatOVesKwTSVMjb75GDaLrJ3MdGk6q9ZahAi4TSy5JELaa1KFiotkJfdYjduArNX4tkuuXILBp39J0fguoCk2cmP3wnZ5is6DKF3/rJbCHIIYK8XWnCtA2KAbFmJshzi1oT5YTfYulO/n3d1edACgPatKFM/FfBjayIORSqE94IAm0p6IaNo0QOHEi/rWNsMRVt/H9BxtPXb+CKArQpJVthlFICvppi9MUJWtb8SqFqPoqjuz0FhABjOs3F+WHsExRtHNK8BlPzUUR9fQN/+2QQvCx5yoCAeDvpX0xVL+fK+i+g7J3f9M70cucrbKrI2VbP36/YKeq5uvBN foreman-proxy@capsule.example.com # cat /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy.pub ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDtilOcscgLM+ozTLhSvzSsaaCjEvNJ/f9nkDWJgybpGF9nCH6qVtMY/bjCP0oQvHQwwGZkvAMZTppOQzEFpYTA56eUcm5eyW4eYng/sBiApf5icNz+z5Ntvwk5S7PwjfxtSL+8dmB5hwLFcvfT7gqC4aCahIZuK7ZNXaynoKsVuBwg6WkkGwQstF5gu52cOQKkKNHWXsLYUTfWYhmAw9AkiV3vM3JRPj1gGolg13awPnkQpj/OCx1e2W67rtpqbn8YXJzylbfwbzKSf5i+DgWaZlA6R396rJYE6mkqperwNtJETWcybZNRot9qr3JbR09jpJIO46k+wso67GVi4g93 foreman-proxy@capsule.example.com -
Execute below command on the affected client host and Verify which SSH-key from above, was shared with the client host.
# grep capsule.example.com .ssh/authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDtilOcscgLM+ozTLhSvzSsaaCjEvNJ/f9nkDWJgybpGF9nCH6qVtMY/bjCP0oQvHQwwGZkvAMZTppOQzEFpYTA56eUcm5eyW4eYng/sBiApf5icNz+z5Ntvwk5S7PwjfxtSL+8dmB5hwLFcvfT7gqC4aCahIZuK7ZNXaynoKsVuBwg6WkkGwQstF5gu52cOQKkKNHWXsLYUTfWYhmAw9AkiV3vM3JRPj1gGolg13awPnkQpj/OCx1e2W67rtpqbn8YXJzylbfwbzKSf5i+DgWaZlA6R396rJYE6mkqperwNtJETWcybZNRot9qr3JbR09jpJIO46k+wso67GVi4g93 foreman-proxy@capsule.example.com
This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.