Review agent activity logs

You can review agent activity logs to monitor AI agent behavior, inspect LLM interaction metrics, and identify out-of-bounds agent activity. Agent activity logs capture metadata about agent execution, LLM API calls, and tool invocations.

Before you begin

  • You have a role with audit log read permissions in Automation Orchestrator.
  • You are logged in to the Automation Orchestrator web console.
  • At least one automation workflow with agent nodes has been executed.

Procedure

  1. Navigate to the agent activity log view.

    In the Automation Orchestrator web console, click Observability > Audit Logs.

    Filter the event_category to Agent interaction or Large language model (LLM) interaction to display only agent-related events.

  2. Filter agent activity logs to focus on a specific agent, automation run, or time period.

    Use the following filter criteria:

    • Agent name — filter by the agent_type field to isolate events from a specific agent type, such as orchestrator or generic_agent.
    • Automation run — filter by execution_id to view all agent activity within a single workflow execution.
    • Invocation — filter by invocation_id to trace a single agent invocation from creation through completion.
    • Time range — set start and end timestamps to scope the review to a specific period.
  3. Inspect LLM interaction metadata to evaluate agent performance.

    For each llm_interaction event, review the following key fields:

    • Model version — the model_name field identifies which LLM model was used for the interaction.
    • Interaction type — the interaction_type field shows whether the call was standard, structured_output, or extraction.
    • Tool usage — the tools_available and tool_calls_made fields show how many tools were provided to the LLM and how many it invoked.
    • Response status — the status field shows whether the LLM call succeeded, returned an empty response, or produced an error.
    • Fallback behavior — the fallback_strategy_used field indicates when the platform fell back to an alternative strategy after a primary LLM call failed.
    Note:

    Automation orchestrator records only metadata about LLM interactions. Full prompt and response content is not stored in audit events.

  4. Identify out-of-bounds agent behavior.

    Look for the following indicators of abnormal agent activity:

    • Excessive tool calls — a high value in tool_calls_made relative to tools_available can indicate inefficient agent behavior or tool selection issues.
    • Repeated errors — multiple consecutive llm_interaction events with status: error for the same invocation_id indicate persistent LLM provider issues.
    • Fallback invocations — frequent fallback_strategy_used values indicate that the primary LLM call path is failing regularly.
    • Agent failures — agent_execution events with status: failed indicate agent-level failures that may affect workflow outcomes.
    • Context planning anomalies — context_planning events with status: failed indicate issues with retrieval-augmented context preparation.

Results

You have reviewed agent activity logs and identified any performance issues or out-of-bounds behavior. Use these findings to adjust agent configurations, investigate LLM provider issues, or refine workflow designs.