Running an Ansible role from Satellite, a subset of the target content hosts report: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host \"host.fqdn\". Make sure this host can be reached over ssh: ", "unreachable": true}

Solution Verified - Updated

Environment

Red Hat Satellite 6.13

Issue

  • The affected hosts work just fine and are reachable from Satellite.

  • The affected hosts encounter the error below when running ansible roles from Satellite server against them:

    fatal: [host.fqdn]: UNREACHABLE! => {"changed": false, "msg": "Data could not be sent to remote host 
\"host.fqdn\". Make sure this host can be reached over ssh: ", "unreachable": true}
14:
PLAY RECAP *********************************************************************
15:
host.fqdn  : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0   ignored=0
16:
Exit status: 1
17:
StandardError: Job execution failed

Resolution

  • Follow the steps in the Diagnostic Steps section of this KB solution.
  • If the outcome from the diagnostic steps you perform matches with that from the Diagnostic Steps section:
  1. Check the connectivity with the NFS server.
  2. Reboot the affected host(s) and make sure that the NFS volume is mounted properly and accessible.

Root Cause

The Ansible roles run against host are failing due to a connectivity issue with NFS server 'fqdn.nfs.server'.

Diagnostic Steps

  • Let's try to collect more debug information by adding the following line to either /etc/ansible/ansible.cfg or /usr/share/foreman-proxy/.ansible.cfg on the Satellite server

    debug=1

    Rerun Ansible roles against one of the content hosts

    PLEASE NOTE : if you use /usr/share/foreman-proxy/.ansible.cfg, Ansible spawned by Satellite ignores configuration in /etc/ansible/ansible.cfg

  • Checking the /var/log/messages into the host:

    Aug 13 00:01:47 hostname python3[1104231]: ansible-ansible.legacy.setup Invoked with gather_subset=['all'] 
gather_timeout=10 filter=[] fact_path=/etc/ansible/facts.d
Aug 13 00:01:48 hostname kernel: ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm
Aug 13 00:01:50 hostname kernel: nfs: server fqdn.nfs.server not responding, timed out
Aug 13 00:01:58 hostname python3[1104231]: ansible-ansible.legacy.setup [WARNING] Timeout exceeded when 
getting mount info for /admin_mnt/test_nfs
....
Aug 18 08:51:40 hostname kernel: nfs: server fqdn.nfs.server not responding, timed out
Aug 18 08:51:43 hostname kernel: ceph: get_quota_realm: ino (1.fffffffffffffffe) null i_snap_realm

  • Looking at the output from the 'mount' command:

    fqdn.nfs.server:/rbd/test_nfs on /admin_mnt/test_nfs type nfs4
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.