Troubleshoot automation controller

In cases where your configuration is set up to run an automation controller node behind a load balancer or proxy as "HTTP only", and you only want to access it without SSL/TLS (for troubleshooting, for example), you must add the following settings in the custom.py file located at /etc/tower/conf.d of your automation controller instance:

SESSION_COOKIE_SECURE = False
CSRF_COOKIE_SECURE = False

If you change these settings to false it enables automation controller to manage cookies and login sessions when using the HTTP protocol. You must do this on every node of a cluster installation.

To apply the changes, run:

automation-controller-service restart

If you experience intermittent job failures or delays, particularly with the following symptoms:

• The running ansible process received a shutdown signal
• Worker did not produce events or streaming was aborted
• Job reaped due to instance shutdown
• Nodes toggling between Available and Unavailable states
• 503 Service Unavailable or 403 Forbidden errors
• Database connection failures (psycopg.OperationalError)
• Kernel messages: audit: backlog limit exceeded
• Unexplained CPU load spikes or system freezes lasting 60+ seconds

Investigate external security and monitoring tools on your Ansible Automation Platform infrastructure nodes. Consult your security team to whitelist Ansible Automation Platform execution paths and consider reducing the verbosity of audit logging on those systems.

Diagnostic checklist

• Review antivirus/EDR logs for Ansible Automation Platform process quarantine or termination events
• Check monitoring agent CPU and memory overhead on Ansible Automation Platform nodes
• Examine auditd logs for excessive system call logging during job failures
• Verify Ansible Automation Platform paths are excluded from real-time security scanning
• Monitor disk I/O during job execution for bottlenecks caused by audit logging
• Consult Red Hat Solution Red Hat Solution 7144721 if issues persist

• Ensure that the playbook is valid YML and can be parsed by Ansible.
• Ensure that the permissions and ownership of the project path (/var/lib/awx/projects) is set up so that the "awx" system user can view the files. Run the following command to change the ownership:

chown awx -R /var/lib/awx/projects/

• Ensure that all supervisor services are running through supervisorctl status.
• Ensure that the /var/ partition has more than 1 GB of space available. Jobs do not complete with insufficient space on the /var/ partition.
• Run automation-controller-service restart on the automation controller server.

If you continue to have issues, run sosreport as root on the automation controller server, then file a Content from support.ansible.com is not included.support request with the result.

Instances have been reported where reusing the external database during subsequent installation of nodes causes installation failures.

Example

You perform a clustered installation. Then, you need to do this again and perform a second clustered installation reusing the same external database, only this subsequent installation failed.

When setting up an external database that has been used in a prior installation, you must manually clear the database used for the clustered node before any additional installations can succeed.