Build a workflow to remediate web service failures

Build a workflow that detects when an NGINX web service fails, uses a task agent to analyze the failure, requests approval from the on-call team, and restarts the service through an Ansible Automation Platform job template.

This process uses a specific scenario: Prometheus Alertmanager detects HTTP 5xx errors from an NGINX service and forwards the alert through Event-Driven Ansible to automation orchestrator.

Prerequisites

Before you build the web service remediation workflow, verify that the following services and credentials are configured.

  • An Ansible Automation Platform credential is configured in automation orchestrator. Navigate to Credentials to verify or create one.
  • An LLM provider credential is configured in automation orchestrator.
  • An Event-Driven Ansible controller instance is connected to your Ansible Automation Platform deployment.
  • Prometheus Alertmanager is configured to detect NGINX service failures and send webhook alerts to Event-Driven Ansible. For more information, see the Event-Driven Ansible documentation.
  • A remediation job template exists in Ansible Automation Platform that restarts the NGINX service on target hosts. For example, a playbook that runs systemctl restart nginx or podman restart nginx-container.
  • A project is available to contain your workflow.

Add an Event-Driven Ansible trigger

Add an Event-Driven Ansible trigger to start the workflow when a webhook receives an alert payload.

Procedure

  1. Navigate to the Workflows page and click Create workflow.
  2. Enter a name for the workflow, such as NGINX failure remediation.
  3. Select the project.
  4. Click Add step > Triggers > Event-Driven Ansible.
  5. In the Webhook path field, enter nginx-service-alerts.

    Automation orchestrator generates a webhook URL. Copy this URL. You need it to configure the Event-Driven Ansible rulebook activation.

  6. In the Request body field, define a schema to validate incoming alert payloads.

    This ensures the workflow only starts when the payload includes the required fields. Include every field that downstream steps reference.

    {
  "type": "object",
  "required": ["host", "severity", "service_name", "alert_name"],
  "properties": {
    "host": {
      "type": "string",
      "description": "Hostname or IP of the affected server"
    },
    "severity": {
      "type": "string",
      "enum": ["warning", "critical"]
    },
    "service_name": {
      "type": "string",
      "description": "Name of the failing service"
    },
    "alert_name": {
      "type": "string",
      "description": "Name of the Prometheus alert rule that fired"
    },
    "log_snippet": {
      "type": "string",
      "description": "Recent log output from the failing service"
    }
  }
}
  7. Click Create or Update to save the step and add it to the canvas.

Add a task agent step to analyze the failure

Add a task agent step that uses a large language model (LLM) to analyze the failure details from the trigger payload. The step produces a structured summary for the approver.

Procedure

  1. Click Add step > Task Agent.
  2. In the Name field, enter Analyze NGINX failure.
  3. In the Model dropdown, select a model from your LLM provider integration.

    After you select a model, click Set up connection and select your LLM provider credential.

  4. In the Prompt field, enter the following instructions.

    Template expressions inject the alert data from the trigger payload:

    You are an SRE assistant analyzing a web service failure.

A monitoring alert has fired with the following details:
- Host: ${trigger.host}
- Service: ${trigger.service_name}
- Alert: ${trigger.alert_name}
- Severity: ${trigger.severity}
- Recent logs: ${trigger.log_snippet}

Analyze the alert and log data. Determine whether the service can be
safely restarted or whether the failure indicates a deeper issue that
requires manual investigation.

If a restart is safe, set recommended_action to "restart".
If manual investigation is needed, set recommended_action to "escalate".
Explain your reasoning in the analysis_summary field.
  5. In the Tools field, select All tools to give the task agent access to available Model Context Protocol (MCP) server tools.
  6. In the Response schema field, click the expand icon and define the expected output structure: 
    {
  "type": "object",
  "required": ["recommended_action", "analysis_summary", "confidence"],
  "properties": {
    "recommended_action": {
      "type": "string",
      "enum": ["restart", "escalate"],
      "description": "The recommended remediation action"
    },
    "analysis_summary": {
      "type": "string",
      "description": "Explanation of the root cause analysis and reasoning"
    },
    "confidence": {
      "type": "string",
      "enum": ["high", "medium", "low"],
      "description": "Confidence level in the recommendation"
    }
  }
}
  7. Click Create or Update to save the step and add it to the canvas.

Add an approval step for production remediation

Add an approval step that pauses the workflow and notifies on-call engineers to review the task agent analysis before the remediation job runs.

Procedure

  1. Click Add step > Approval.
  2. In the Name field, enter Approve service restart.
  3. In the Approver users field, enter the usernames of the on-call engineers who can approve production changes.

    Press Enter after each username.

  4. In the Message field, enter a message that presents the task agent analysis to the approver.

    Use template expressions to reference the task agent output:

    NGINX service failure detected on ${trigger.host}.

AI Analysis: ${analyze_nginx_failure.output.analysis_summary}

Recommended action: ${analyze_nginx_failure.output.recommended_action}
Confidence: ${analyze_nginx_failure.output.confidence}

Approve to restart the NGINX service on the affected host.
Reject to escalate for manual investigation.

    Replace analyze_nginx_failure with the actual step ID assigned to the task agent step on the canvas. To find the step ID, click the step and check the details panel header.

  5. Set the Decision window to 30 minutes.

    If no approver responds within this window, the fallback decision takes effect.

  6. In the Fallback decision field, select Reject so that unapproved remediations do not run automatically.
  7. Click Create or Update to save the step and add it to the canvas.

Add an Ansible Automation Platform job template step to restart the service

Add an Ansible Automation Platform job template step that runs a playbook to restart the failed NGINX service.

Procedure

  1. Click Add step > AAP Execution > Launch AAP job template.
  2. In the Name field, enter Restart NGINX service.
  3. In the Integration field on the Parameters pane, select an Ansible Automation Platform integration from the dropdown.

    After you select an integration, a credential status appears below the field. If a credential is already configured, its name appears. If no credential is configured, click Set up connection and select a credential from the dropdown. To create a new credential without leaving the step configuration, select Create new credential at the top of the dropdown.

  4. In the Organization field, select the organization that contains your remediation job templates.
  5. In the Job template field, select the job template that restarts the NGINX service.

    Alternatively, toggle Use input variables and enter an expression such as nginx-restart or a dynamic value from an earlier step.

  6. Optional: In the Limit field, restrict execution to the affected host by entering the expression ${trigger.host}.
  7. Optional: In the Extra variables field, click the expand icon and add context from the trigger and analysis: 
    {
  "target_host": "${trigger.host}",
  "service_name": "${trigger.service_name}",
  "severity": "${trigger.severity}",
  "analysis_summary": "${analyze_nginx_failure.output.analysis_summary}",
  "approved_by": "${approve_service_restart.decided_by}"
}
  8. Click Create or Update to save the step and add it to the canvas.

Connect and test the workflow

Connect the workflow steps on the canvas and run a test to verify the end-to-end flow.

Procedure

  1. Arrange the steps in order on the canvas: Event-Driven Ansible > Analyze NGINX failure > Approve service restart > Restart NGINX service.
  2. Connect each step by clicking the + icon on a step and selecting the next step.

    Alternatively, drag from the + icon to an existing step.

  3. For the approval step, connect the Approved output handle to the Restart NGINX service step.
  4. Click Save to save the workflow.
  5. Test the workflow using either mock data or click Run to test the workflow with live data from the predecessor steps.

Configure the Event-Driven Ansible controller

Configure the Event-Driven Ansible controller to forward Prometheus alerts to the automation orchestrator webhook.

Procedure

  1. In your Event-Driven Ansible controller, create or update a rulebook activation to forward matching Prometheus alerts to the webhook URL you copied when you added the Event-Driven Ansible trigger.
  2. Ensure the rulebook includes a rule that matches the NGINX alert name and sends the payload with the fields defined in the input schema: host, severity, service_name, alert_name, and log_snippet.

Verify and publish the workflow

Test the completed workflow with a simulated alert, then publish it for production use.

Procedure

  1. Simulate an NGINX service failure in your monitoring environment to trigger a Prometheus alert, or manually send a test payload to the webhook URL: 
    {
  "host": "web-prod-03.example.com",
  "severity": "critical",
  "service_name": "nginx",
  "alert_name": "NginxHighErrorRate",
  "log_snippet": "2026/06/10 14:32:01 [error] upstream timed out (110: Connection timed out) while connecting to upstream"
}
  2. Navigate to the workflow run history and confirm that a new run started.
  3. Click each step in the run to verify the following:
    • The Event-Driven Ansible trigger received the alert payload.
    • The task agent step produced a structured analysis with a recommended action of restart or escalate.
    • The approval step shows a pending approval request with the AI analysis in the message.
  4. Navigate to Approvals and approve the request.
  5. Return to the run history and confirm that the Ansible Automation Platform job template step completed with a job_status of successful.
  6. Verify on the target host that the NGINX service restarted and is serving traffic.
  7. Click the options menu (⋮) in the top toolbar and select Verify workflow.

    Confirm that the validation results pane shows no errors or warnings.

  8. In the top toolbar, click Publish workflow to activate the workflow in production and begin accepting incoming events.