Remote execution fails after upgrading Satellite to 6.11 with error "Unable to create directory on remote system" and "exit code: 255".

Environment

• Red Hat Satellite 6
• Red Hat Capsule 6

Issue

• Remote execution fails with below error:

  Error initializing command: RuntimeError - Unable to create directory on remote system /var/tmp/foreman-ssh-cmd-xxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: exit code: 255
  

Resolution

1. Remove below line from /etc/ssh/ssh_config and check configuration files under directory /etc/ssh/ssh_config.d/ on Satellite server and Capsule server.

    ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h
   

2. Restart sshd service:

   # systemctl restart sshd
   

3. Perform remote execution now.

For more KB articles/solutions related to Red Hat Satellite 6.x Remote Execution Issues, please refer to the Red Hat Satellite Consolidated Troubleshooting Article for Red Hat Satellite 6.x Remote Execution Issues

Root Cause

• Below entry prevents connection to machine without reverse address:

  ProxyCommand /usr/bin/sss_ssh_knownhostsproxy -p %p %h
  

Diagnostic Steps

1. Error in foreman task export:

   - output_type: debug
     output: "Error initializing command: RuntimeError - Unable to create directory
       on remote system /var/tmp/foreman-ssh-cmd-xxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx:
       exit code: 255\n ssh_exchange_identification: Connection closed by remote host\r\n"
   exit_status: EXCEPTION
   

2. Error in /var/log/foreman/production.log:

   Processing by ForemanTasks::Api::TasksController#callback as HTML
   Parameters: {"callback"=>{"task_id"=>"<task-id>", "step_id"=><id>}, "data"=>{"result"=>[{"output_type"=>"debug", "output"=>"Error initializing command: RuntimeError - Unable to create directory on remote system /var/tmp/foreman-ssh-cmd-xxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx: exit code: 255\n ssh_exchange_identification: Connection closed by remote host\r\n", "timestamp"=><timestamp>}], "runner_id"=>"ab5aa69f-4feb-4d17-9a78-810cb1819aec", "exit_status"=>"EXCEPTION"}, "task"=>{}}
   Completed 200 OK in 22ms (Views: 0.4ms | ActiveRecord: 3.8ms | Allocations: 3168)
   Started GET "/job_invocations/job-id>?page=1&per_page=20" for <host-ip> at <date> +0300
   Processing by JobInvocationsController#show as JSON
   

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.