Remote execution tasks using scripts fail, whereas remote execution tasks using Ansible roles/playbooks complete successfully for the same host(s) on Red Hat Satellite 6.15

Solution Verified - Updated

Environment

  • Red Hat Satellite 6.15.2
  • Red Hat Satellite 6.15.3
  • Red Hat Satellite 6.15.4

Issue

  • Any Remote execution job in a newly provisioned host via Red Hat Satellite 6.15 fails with the following error.

  • Remote execution jobs using Ansible roles/playbooks complete successfully, whereas remote execution jobs using script and run against the same host(s) fail with the following error:

    Output:
\---
proxy_task_id: 99557edd-8c01-4974-b2c6-60ae18d042a0
proxy_output:
  result:
  - output_type: debug
    output: "Error initializing command: RuntimeError - Could not establish connection
      to remote host using any available authentication method, tried publickey\n\nAuthentication
      method 'publickey' failed with:\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\n@
      \   WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @\r\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\r\nIT
      IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!\r\nSomeone could be eavesdropping
      on you right now (man-in-the-middle attack)!\r\nIt is also possible that a host
      key has just been changed.\r\nThe fingerprint for the ECDSA key sent by the
      remote host is\nSHA256:5ZRN7em1Zg180A+RHjcr1QHiDCUIriRVzHcGsaA7iuY.\r\nPlease
      contact your system administrator.\r\nAdd correct host key in /usr/share/foreman-proxy/.ssh/known_hosts
      to get rid of this message.\r\nOffending ECDSA key in /usr/share/foreman-proxy/.ssh/known_hosts:25\r\nECDSA
      host key for host.example.com has changed and you have requested strict checking.\r\nHost
      key verification failed.\r\n"
    timestamp: 1723718281.6434834
    runner_id: bd1c00ad-623c-4ce0-bb1b-7779d060cdcc
  exit_status: EXCEPTION

  • Satellite task details for the host in question show the below error:

    Error initializing command: RuntimeError - Could not establish connection to remote host using any available authentication method, tried password, publickey
   2:

   3:
Authentication method 'password' failed with:
   4:
Exiting, you have requested strict checking.
   5:
Host key verification failed.
   6:

   7:
Authentication method 'publickey' failed with:
   8:
Exiting, you have requested strict checking.
   9:
Host key verification failed.
  10:
Exit status: EXCEPTION
  11:
StandardError: Job execution failed

Resolution

  • As a workaround, comment out or remove the affected host key from the known_hosts file, or rename the file entirely:

    • Option 1: Remove or comment out the existing entry:

      # vi /usr/share/foreman-proxy/.ssh/known_hosts

      Locate the line corresponding to the target host and delete or comment it.

    • Option 2: Rename the entire file:

      # mv /usr/share/foreman-proxy/.ssh/known_hosts /usr/share/foreman-proxy/.ssh/known_hosts.bak

      This allows the SSH connection to re-establish trust and proceed with remote execution.

For more KB articles/solutions related to Red Hat Satellite 6.x Remote Execution Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Remote Execution Issues

Root Cause

  • Satellite tries to keep track which capsules were used to run remote execution jobs for which hosts, and if satellite determines it is the first execution, it tries to remove the known host keys from /usr/share/foreman-proxy/.ssh/known_hosts before trying to connect to the host.

  • This host keys cleanup mechanism works as follows:

    • If the first remote execution job run against a specific host uses script, then the cleanup mechanism works. i.e., it checks /usr/share/foreman-proxy/.ssh/known_hosts on the Satellite (or Capsule) server, and if an old host key is stored there, the cleanup mechanism removes it.
    • If the first remote execution job run against a specific host uses Ansible, the hosts key cleanup is not performed. This still counts as an execution through a proxy, and when a remote execution job using a script is executed, the hosts key cleanup does not take place because it is not the first remote task executed against the host.

  • The issue described here occurs when the first remote execution job run against a specific host uses Ansible. When this occurs, the old host key is not removed from /usr/share/foreman-proxy/.ssh/known_hosts on the Satellite (or Capsule) server. In this case, if a remote execution job using script is then run against the host, it fails due to the old host key still present in /usr/share/foreman-proxy/.ssh/known_hosts on the Satellite (or Capsule) server, reporting the error referred to in the Issue description section of this knowledge base solution article.

  • When a host is provisioned (or re-provisioned) via Satellite, default Ansible roles run as part of the provisioning. This means that the first remote execution job run against the host uses Ansible, and next remote execution jobs run against the same host afterwards will fail.

SBR
Product(s)
Components
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.