Remote Execution job fails with error "no such identity: .ssh/id_rsa_foreman_proxy" in Red Hat Satellite 6

Environment

Satellite 6.9

Issue

• When executing an ansible-based remote execution job, It fails with the following error:

  fatal: [hostname.example.com]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: no such identity: .ssh/id_rsa_foreman_proxy: 
  No such file or directory\r\nPermission denied (publickey,gssapi-keyex,gssapi-with-mic,password,keyboard-interactive).", "unreachable": true}
  

Resolution

• Ensure that the scenario described in this article 4282171 is not applicable and if it is, fix the same by following the resolution steps.

• Ensure that the value of ansible_ssh_private_key_file of the parameter in Satellite settings is blank.

  # hammer settings set --name ansible_ssh_private_key_file --value ""
  

• If the environment was installed before Satellite 6.5 and has been upgraded to 6.9, verify that /usr/share/foreman-proxy/.ssh is a symlink to /var/lib/foreman-proxy/ssh

  # ls -ldZ /usr/share/foreman-proxy/.ssh 
  lrwxrwxrwx. root root unconfined_u:object_r:usr_t:s0   /usr/share/foreman-proxy/.ssh -> /var/lib/foreman-proxy/ssh
  

  • If it's not then remediate the scenario by following the resolution steps described in this article 4282171.

• Ensure that the satellite-answers.yaml file contains the correct values of the following parameters as displayed below.

  # grep ssh_identity_dir /etc/foreman-installer/scenarios.d/satellite-answers.yaml -A1
    ssh_identity_dir: "/var/lib/foreman-proxy/ssh"
    ssh_identity_file: id_rsa_foreman_proxy
  

  • If there are some misconfigurations present, please fix the same and re-execute the satellite-installer command to apply the changes.

• Feel free to contact This content is not included.Red Hat Technical Support for further assistance on this problem.

For more KB articles/solutions related to Red Hat Satellite 6.x Remote Execution Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Remote Execution Issues

Root Cause

• The system survived upgrades without fixing remote execution directory changes as what we should witness for a newer satellite installation.

• The private key entry for ansible_ssh_private_key_file parameter was pointing to .ssh/id_rsa_foreman_proxy file which does not exists.

Diagnostic Steps

• Verify if only ansible-based jobs are failing or non-ansible type remote execution jobs are failing as well.

• Check and compare this output from the output of your satellite.

  # ls -ldZ /usr/share/foreman-proxy/.ssh /var/lib/foreman-proxy/ssh /usr/share/foreman-proxy/.ssh/* /var/lib/foreman-proxy/ssh/*
  lrwxrwxrwx. root          root          system_u:object_r:usr_t:s0       /usr/share/foreman-proxy/.ssh -> /var/lib/foreman-proxy/ssh
  -rw-------. foreman-proxy foreman-proxy unconfined_u:object_r:var_lib_t:s0 /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy
  -rw-r--r--. foreman-proxy foreman-proxy unconfined_u:object_r:var_lib_t:s0 /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy.pub
  -rw-r--r--. foreman-proxy foreman-proxy unconfined_u:object_r:var_lib_t:s0 /usr/share/foreman-proxy/.ssh/known_hosts
  drwx------. foreman-proxy foreman-proxy system_u:object_r:var_lib_t:s0   /var/lib/foreman-proxy/ssh
  -rw-------. foreman-proxy foreman-proxy unconfined_u:object_r:var_lib_t:s0 /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy
  -rw-r--r--. foreman-proxy foreman-proxy unconfined_u:object_r:var_lib_t:s0 /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy.pub
  -rw-r--r--. foreman-proxy foreman-proxy unconfined_u:object_r:var_lib_t:s0 /var/lib/foreman-proxy/ssh/known_hosts
  

  • Verify that /usr/share/foreman-proxy/.ssh is a softlink to /var/lib/foreman-proxy/ssh directory.

  • Verify the SELinux labels for the directories and SSH keypairs.

  • Verify if both of the directories in question have the same SSH keypairs present.

• Execute the following command to identify if any SELinux denials were logged related to accessing the SSH private or public key files.

  # ausearch -m AVC,USER_AVC -ts today
  

• Identify the value of "ansible_ssh_private_key_file" parameter in satellite settings. It should ideally be blank.

  
  # hammer settings list --search "ansible_ssh_private_key_file" --fields "name,full name,value"
  

* Check the value for `ssh_identity_key_file` inside `/etc/foreman-proxy/settings.d/remote_execution_ssh.yml` file. The expected output should be as displayed.

grep "ssh_identity_key_file" /etc/foreman-proxy/settings.d/remote_execution_ssh.yml

:ssh_identity_key_file: /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.