Ansible Automation Platform 2.5 jobs fail with "The running ansible process received a shutdown signal" due to system hangs from audit backlog limit and security agent activity

Solution Verified - Updated

Environment

  • Red Hat Ansible Automation Platform (AAP) 2.5
  • Red Hat Enterprise Linux (RHEL) 8 or 9
  • Third-party security agents: Microsoft Defender for Endpoint (MDATP), Trend Micro Deep Security (ds_agent).

Issue

Following an upgrade to AAP 2.5, multiple scheduled and ad-hoc jobs fail concurrently with one or more of the following symptoms:

  • Job Errors:

    • The running ansible process received a shutdown signal.
    • Worker did not produce events or streaming was aborted, check execution node health.
    • Job reaped due to instance shutdown.

  • UI/Platform Symptoms:

    • Automation Controller nodes intermittently toggle between Available and Unavailable in the Web UI.
    • Intermittent 503 Service Unavailable or 403 Forbidden errors when accessing the platform.
    • Database connection errors in tower.log: psycopg.OperationalError: connection failed: server closed the connection unexpectedly.

  • System Logs:

    • /var/log/messages or the serial console shows: audit: backlog limit exceeded.
    • High CPU load spikes and frozen system behavior for 60+ seconds.

Resolution

Optimize Audit Configuration

  • Modify the persistent audit rules to increase the buffer (depending on system memory tweak this) and reduce the backlog_wait_time duration:

  • Increase the backlog limit to handle spikes in event volume:

    -b 16384

  • Reduce the backlog wait time (the default is often 60000ms/60s, which is long enough to kill heartbeats):

    --backlog_wait_time 10000

  • Apply changes:

    # augenrules --load

    NOTE: If rules are immutable (-e 2) comment out this line and node would require a reboot to apply this change.

Configure Security Agent Exclusions

  • Exclude high-activity AAP directories from real-time anti-malware scanning. AAP job execution model creates/deletes thousands of temporary files (rootless podman containers would be created with each job launch) in these paths, which triggers excessive scanning and auditing:

    • /var/lib/awx/ (Includes .local/share/containers/)
    • /tmp/receptor/
    • /var/run/

Cleanup Noise Generating Rules

  • Identify which audit rules are flooding the buffer:

    # aureport --summary -k

    NOTE: If a specific endpoint agent (like MDATP) is generating thousands of events, remove/exclude its specific rules file from /etc/audit/rules.d/ and reload the configuration.

Root Cause

  • Excessive auditing by third-party endpoint security agents filled the kernel audit backlog.
  • Under default configurations, this triggers a kernel-level process suspension (backlog_wait_time) that freezes the system.
  • This suspension causes the Automation Controller processes to miss internal heartbeats, leading the cluster to believe the node is offline and reaping the active jobs.

Diagnostic Steps

  • Search the system logs and kernel ring buffer for evidence of audit buffer overflows:

     # grep -i "backlog limit exceeded" /var/log/messages
 # dmesg | grep -i "audit"

    NOTE: If you see audit: backlog limit exceeded, the system is likely hanging while waiting for the audit daemon to catch up.

  • Use aureport to find which audit keys or processes are generating the most traffic:

    # aureport --summary -k

NOTE: In the case of MDATP, you may see thousands of events tied to the mdatp key within a very short window if an excessive audit rules are configured or this endpoint is auditing the AAP Processes and its directories. (e.g., thousands of events in a short interval).

  • Check the Automation Controller logs for missed schedules, which indicate the application was frozen or unable to process tasks:

    # egrep -i 'error|warning' /var/log/tower/tower.log

    Example output: WARNING awx.main.dispatch.periodic Missed 21 schedules of task_manager

  • Check for Database Connection Drops, Look for closed connection errors in the dispatcher or tower logs, which often occur as a side effect of the system hang:

    # grep "psycopg.OperationalError" /var/log/tower/tower.log
SBR
Product(s)
Category
Tags

This solution is part of Red Hat’s fast-track publication program, providing a huge library of solutions that Red Hat engineers have created while supporting our customers. To give you the knowledge you need the instant it becomes available, these articles may be presented in a raw and unedited form.