Satellite remote execution job to external Capsule fails with Error initializing command Net::SSH::AuthenticationFailed - Authentication failed for user root@capsule01.example.org

Environment

• Red Hat Satellite
  • 6.8
  • 6.9

Issue

• Running a remote execution job from the Satellite UI to an external Capsule failed.

    Error initializing command: Net::SSH::AuthenticationFailed - Authentication failed for user root@capsule01.example.org
    Exit status: EXCEPTION
  

• The failed job added an entry to /var/log/secure file on the external Capsule.

    capsule01 sshd[2198]: Connection closed by 198.51.100.29.
  

Resolution

• Red Hat investigated this issue in bug report This content is not included.RHBZ#1873241 and delivered a fix in Satellite 6.11.0 through errata RHSA-2022:5498. If this issue still occurs in your environment after updating, open a support case in the Red Hat Customer Portal referring to this solution.

• To workaround this issue:

  • Workaround 1 (preferred): Add the subnet of the external Capsule in Satellite - Infrastructure -> Subnets.
  • Workaround 2: Add all the external Capsules' foreman-proxy pub keys to the other external Capsules' /root/.ssh/authorized_keys files.

For more KB articles/solutions related to Red Hat Satellite 6.x Remote Execution Issues, refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Remote Execution Issues

Root Cause

• The cause of the failure was due to the remote execution job being routed through a different external Capsule (capsule02), instead of the intended external Capsule (capsule01).

• In this environment, 198.51.100.29 was not the IP address of the Satellite server (satellite01) but the IP of another external Capsule (capsule02) registered with Satellite.

• Actual remote execution job workflow:

    Satellite server (satellite01 - 198.51.100.27) --> Different External Capsule (capsule02 - 198.51.100.29) --> External Capsule (capsule01 - 198.51.100.28)
  

• Expected remote execution job workflow:

    Satellite server (satellite01 - 198.51.100.27) --> External Capsule (capsule01 - 198.51.100.28)
  

• The different external Capsule (capsule02) was unable to authenticate to capsule01.

• By default, Satellite automatically selects a remote execution Capsule based on the subnet of a registered host.

• This requires the subnet of that host to be created in Satellite Infrastructure -> Subnets as a pre-requisite.

• In the absence of a subnet in Infrastructure -> Subnets, Satellite automatically selects the remote execution Capsule(s) and the user does not have control of the external Capsule(s) selected.

Diagnostic Steps

• Pre-requisite: Satellite infrastructure with multiple external Capsules and subnets not configured in the Satellite UI.

• Add the Satellite server's foreman-proxy pub key to an external Capsule's /root/.ssh/authorized_keys file.

    [root@satellite01 ~]# ssh-copy-id -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy.pub root@capsule01.example.org
  

• Test SSH connection to the host through Satellite CLI.

    [root@satellite01 ~]# ssh -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy root@capsule01.example.org 'df'
  

• Run a remote job from the Satellite UI Content -> Job Templates.

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.