Manage live event streams to the UI

By default, automation controller streams live events to the user interface (UI) for jobs that are running.

About this task

Events are sent to any node where there is a UI client subscribed to a job. This task is expensive, and becomes more expensive as the number of events that the cluster is producing increases and the number of control nodes increases, because all events are broadcast to all nodes regardless of how many clients are subscribed to particular jobs.

Procedure

To reduce the overhead of displaying live events in the UI, administrators can choose to either:
  • Disable live streaming events.
  • Reduce the number of events shown per second or before truncating or hiding events in the UI.

Results

When you disable live streaming of events, they are only loaded on hard refresh to a job’s output detail page. When you reduce the number of events shown per second, this limits the overhead of showing live events, but still provides live updates in the UI without a hard refresh.

Disable live streaming events

You can disable live streaming events in automation controller to reduce system load or for troubleshooting purposes.

Procedure

Disable live streaming events by using one of the following methods:
  1. In the API, set UI_LIVE_UPDATES_ENABLED to False.
  2. In the navigation panel, select Settings > Automation Execution > System.
    1. Click Edit.
    2. Set the Enable Activity Stream option to Off.

Settings to modify rate and size of events

If your system generates a large number of events, the live streaming of events to the user interface (UI) can cause performance issues. You can disable live streaming of events or reduce the number of events that are displayed in the UI by modifying the following settings.

If you cannot disable live streaming of events because of their size, reduce the number of events that are displayed in the UI. You can use the following settings to manage how many events are displayed:

Settings available for editing in the UI or API:

  • EVENT_STDOUT_MAX_BYTES_DISPLAY: Maximum amount of stdout to display (as measured in bytes). This truncates the size displayed in the UI. The default is 1024 bytes. Minimum = 0.
  • MAX_WEBSOCKET_EVENT_RATE: Number of events to send to clients per second. Default is 30 messages/second. Minimum = 0 (no limit).

Either

Navigate to Settings > Job settings or Settings > System in the automation controller web interface. The fields EVENT_STDOUT_MAX_BYTES_DISPLAY and MAX_WEBSOCKET_EVENT_RATE appear as editable fields there.

or

Use a PATCH request to the automation controller settings API endpoint: PATCH https://<controller-host>/api/v2/settings/jobs/ { "EVENT_STDOUT_MAX_BYTES_DISPLAY": 2048, "MAX_WEBSOCKET_EVENT_RATE": 20 } or for individual settings: PATCH https://<controller-host>/api/v2/settings/system/ Authentication is required. Use either an OAuth2 token or session auth with an admin user.

Settings available by using file based settings:

  • MAX_UI_JOB_EVENTS: Number of events to display before truncating.This setting hides the rest of the events in the list. Default value is 4000. Minimum = 100. Set hidden=True.
  • MAX_EVENT_RES_DATA: The maximum size of the ansible callback event’s "res" data structure. The "res" is the full "result" of the module. When the maximum size of ansible callback events is reached, then the remaining output will be truncated. Default value is 700000 bytes.
  • LOCAL_STDOUT_EXPIRE_TIME: The amount of time before a stdout file is expired and removed locally. They can be regenerated on demand if downloaded again. The default value is 25992000 seconds (30 days).

These settings are not exposed in the UI or API. They must be set in the automation controller configuration file, typically by creating or editing a custom settings file such as /etc/tower/conf.d/custom.py (or the equivalent path in a containerized or operator-based deployment):

MAX_UI_JOB_EVENTS = 4000

MAX_EVENT_RES_DATA = 700000

LOCAL_STDOUT_EXPIRE_TIME = 25992000

After changing file-based settings, restart automation controller services for them to take effect.