Understand internal TLS

Automation orchestrator uses mutual TLS (mTLS) to encrypt and authenticate communication between backend, worker, Temporal, and UI components. Each connection requires both a valid certificate and a trusted certificate authority (CA).

How internal TLS works

These deployments participate in internal mTLS:

Table 1. Component roles in internal mTLS
Component Role How TLS is used
Backend API server Serves HTTPS with its own certificate; enforces client certificate authentication on port 8000; presents client certificate when connecting to Temporal
Worker Workflow executor Presents client certificate when connecting to Temporal
Background worker Internal maintenance tasks Presents client certificate when connecting to Temporal
Temporal Workflow engine Serves gRPC with TLS; requires client certificate authentication from backend, worker, and background worker connections
UI (nginx) Web interface proxy Presents its own client certificate when proxying requests to the backend over HTTPS; verifies the backend server certificate against the internal CA

Backend, worker, background worker, Temporal, and UI each mount three files at two paths:

Table 2. Certificate mount paths
Mount path Contents
/certs/ca/ca.crt Internal CA certificate, used to verify peer certificates
/certs/service/tls.crt Service certificate
/certs/service/tls.key Service private key

Backend, worker, background worker, and Temporal present their service certificates as both server and client identity. The UI nginx proxy uses the service certificate as a client certificate when connecting to the backend. The UI uses the internal CA to verify the backend server certificate.

The operator sets the following environment variables on backend, worker, and background worker pods to enable mTLS:

Table 3. mTLS environment variables
Environment variable Value
APP_S2S_TLS_ENABLED true
APP_S2S_TLS_CA_CERT_PATH /certs/ca/ca.crt
APP_S2S_TLS_CERT_PATH /certs/service/tls.crt
APP_S2S_TLS_KEY_PATH /certs/service/tls.key

On the backend only, the operator also sets APP_S2S_TLS_CN_ALLOWLIST to ["backend.ao.svc","worker.ao.svc","background-worker.ao.svc","temporal.ao.svc"]. That JSON array lists client certificate common names (CNs) that receive service identity. The UI CN is excluded so proxied user requests authenticate with a JWT.

Temporal does not use these environment variables. The operator configures Temporal TLS through a server ConfigMap that references /certs/service/tls.crt, /certs/service/tls.key, and /certs/ca/ca.crt with requireClientAuth: true.

Temporal requires client certificate authentication on its gRPC frontend. Backend, worker, and background worker pods present their service certificates as client credentials when connecting to Temporal.

The backend enforces CERT_REQUIRED on its HTTPS port (8000). Every connection to this port must present a valid client certificate signed by the internal CA. Kubelet liveness, readiness, and startup probes present a valid client certificate when connecting to the backend port.

Important:

External monitoring or health-check tools that connect directly to the backend port are rejected unless they present a valid client certificate. If you use third-party monitoring that targets the backend service, configure it to present a certificate signed by the internal CA, or route it through the UI proxy instead.

Default behavior: operator-generated certificates

When you omit the spec.tls section from the custom resource, the operator generates all certificates automatically. You do not need to configure anything for internal TLS to work.

The operator generates:

  • One self-signed CA — Rivest-Shamir-Adleman (RSA) 2048-bit key, 100-year validity, with the common name (CN) automation-orchestrator-internal-ca.
  • Five service certificates — one per component, each signed by the internal CA with RSA 2048-bit keys and 100-year validity.

Each service certificate includes both serverAuth and clientAuth extended key usages (EKU). This allows every component to act as both a TLS server and a TLS client.

Subject Alternative Names (SANs)

Each service certificate includes DNS SANs for the Kubernetes Service FQDN, short name, and localhost, plus the IP SAN 127.0.0.1. For the required DNS SANs per component, see "Provide your own internal TLS certificates" in Additional resources.

Auto-generated Secrets

The operator creates six Kubernetes Secrets: one for the internal CA and one for each of the five service components (backend, worker, background worker, Temporal, and UI). For the complete list of auto-generated TLS secrets and their keys, see "Internal TLS" in Additional resources.

The operator labels each auto-generated Secret with aap.ansible.com/managed-secret: "true", app.kubernetes.io/component: tls, app.kubernetes.io/instance: {name}, app.kubernetes.io/name: automation-orchestrator, and app.kubernetes.io/managed-by: automation-orchestrator-operator. The operator also sets an owner reference to the custom resource. When you delete the custom resource, Kubernetes garbage-collects the auto-generated Secrets automatically.

User-managed certificates

You can replace the operator-generated certificates with your own. Replace the default certificates when your organization requires a specific CA or must integrate with an existing public key infrastructure (PKI).

For step-by-step instructions, see Additional resources.

Certificate change detection

The operator monitors every referenced Secret by computing a SHA-256 checksum. It stores the checksum as a pod template annotation on each affected deployment. When a Secret changes, the checksum changes and Kubernetes triggers a rolling restart of the affected pods.

The operator also emits Kubernetes events when it detects a TLS Secret update during reconciliation:

  • InternalCACertChanged
  • BackendTLSCertChanged
  • WorkerTLSCertChanged
  • BackgroundWorkerTLSCertChanged
  • TemporalTLSCertChanged
  • UITLSCertChanged

You can view these events with:

$ oc get events -n namespace --field-selector reason=BackendTLSCertChanged

TLSReady status condition

The operator reports the health of internal mTLS through a TLSReady status condition on the AutomationOrchestrator custom resource. TLSReady also appears as a printcolumn, so oc get automationorchestrator shows the TLS status without requiring a jsonpath query.

The operator sets TLSReady=True when all TLS Secrets (CA and service certificates) exist and contain valid keys. The reason field identifies the certificate source:

Table 4. TLSReady reason values
Reason Message When
CustomerCertificates Using customer-provided TLS certificates spec.tls.caSecretRef is set
SelfSignedCertificates Internal mTLS is using operator-generated certificates. These certificates are not automatically rotated. To enable automated rotation, configure spec.tls with certificates managed by cert-manager, Vault, or your organization's PKI. spec.tls is omitted orcaSecretRef is not set

The operator sets TLSReady=False when a TLS Secret is missing or contains invalid keys. The condition message identifies the specific Secret that is unhealthy. The reason is TLSSecretGenerationFailed when auto-generation fails.

Check the TLSReady condition:

$ oc get automationorchestrator name -n namespace \
    -o jsonpath='{.status.conditions[?(@.type=="TLSReady")]}'  | jq .

The operator emits Kubernetes events when the TLSReady condition transitions between states. You can view these events with:

$ oc get events -n namespace --field-selector reason=TLSReady