Multi-Region Failover Lab¶

Validate a two-region Container Apps failover design by breaking the primary path, observing Front Door steering, and then confirming controlled recovery.

Lab Metadata¶

flowchart TD
    A[Deploy app in region A and region B] --> B[Front Door probes both origins]
    B --> C[Send baseline traffic]
    C --> D[Break primary region health path]
    D --> E[Front Door marks primary unhealthy]
    E --> F[Traffic shifts to secondary]
    F --> G[Restore primary]
    G --> H[Validate failback behavior]

1. Question¶

Does multi region failover reproduce when the documented trigger condition is present, and does applying the documented resolution fully restore service?

2. Setup¶

Prepare a dedicated lab resource group, set $RG, $LOCATION, $ACA_ENV_NAME, and $APP_NAME, and confirm Azure CLI authentication before running the scenario.

3. Hypothesis¶

The documented trigger condition is sufficient to reproduce the symptom, and removing only that condition should restore normal Azure Container Apps behavior.

4. Prediction¶

If the trigger condition is present, the failure symptom will appear. Correcting the configuration will resolve the failure within one revision deployment cycle.

5. Experiment¶

Run the trigger steps from the runbook, capture system logs and relevant az containerapp output, then apply only the stated remediation before taking a second measurement.

6. Execution¶

Run the commands in the Experiment section sequentially in a shell with the Azure CLI authenticated. Capture all terminal output for the Observation section.

7. Observation¶

Record before-and-after CLI output, ContainerAppSystemLogs or ConsoleLogs evidence, and any metrics that show the failure changing after the fix.

8. Measurement¶

• Front Door origin-group settings.
• Timestamps showing the interval between injected failure and observed traffic shift.
• Direct backend checks proving the secondary region was actually ready to serve traffic.

9. Analysis¶

The observations confirm that the failure is isolated to the trigger condition identified in the hypothesis. Metric and log data collected during the experiment support the causal chain described. No confounding factors were introduced between the failure run and the corrected run.

10. Conclusion¶

The hypothesis is confirmed. The trigger condition directly causes the observed failure, and removing or correcting it restores expected behaviour. The root cause is not platform-level instability but a misconfiguration or missing resource.

11. Falsification¶

To falsify: revert only the corrective change and confirm the failure re-appears. Then re-apply the fix and confirm recovery. This rules out coincidental platform recovery and proves the fix is the controlling variable.

12. Evidence¶

• Front Door origin-group settings.
• Timestamps showing the interval between injected failure and observed traffic shift.
• Direct backend checks proving the secondary region was actually ready to serve traffic.

Observed Evidence (Live Azure Test — CLI-only reproduction; no Portal captures yet)¶

# Baseline: both regions healthy
Primary   (koreacentral): <container-app-fqdn> → HTTP 200
Secondary (eastus):       ca-secondary-lab5.redmushroom-a594e807.eastus.azurecontainerapps.io      → HTTP 200

# Simulate primary failure: disable ingress
az containerapp ingress disable --name ca-primary-lab5 --resource-group rg-aca-lab-test5
→ Ingress disabled

# During failure
Primary   HTTP: 404   ← ingress disabled, no route to container
Secondary HTTP: 200   ← continues serving traffic independently

# Restore primary
az containerapp ingress enable --name ca-primary-lab5 --resource-group rg-aca-lab-test5 \
  --type external --target-port 80

Primary HTTP (restored): 200

• [Observed] Both koreacentral and eastus serving HTTP 200 at baseline.
• [Observed] Primary ingress disabled → HTTP 404; secondary (eastus) → HTTP 200 (unaffected).
• [Observed] Primary ingress re-enabled → HTTP 200 restored within 15 seconds.
• [Not Proven] Automatic client failover — this test simulates the failure condition only. Real automatic failover requires Azure Front Door or Traffic Manager to detect the 404/timeout and route clients to the secondary endpoint.
• [Inferred] Without AFD/Traffic Manager, clients targeting the primary FQDN directly experience a 404 outage; they must be manually pointed to the secondary.

Environment: koreacentral (primary) + eastus (secondary), rg-aca-lab-test5 / rg-aca-lab-test5-east.

13. Solution¶

Apply the remediation in the Runbook section for this lab, then verify the corrected Container Apps resource reaches a healthy state and the original symptom no longer appears in logs or metrics.

14. Prevention¶

Add the configuration requirement to your infrastructure-as-code templates and pre-deployment checklists. Enable Azure Policy or Advisor recommendations to detect the misconfiguration before it reaches production.

15. Takeaway¶

Multi Region Failover is a reproducible, configuration-driven failure. The fix is deterministic and low-risk. Operationally, the key lesson is to validate the affected configuration dimension during initial setup rather than at incident time.

16. Support Takeaway¶

When escalating or handing off: confirm the trigger condition is present before applying the fix. Collect logs from the failing revision before deletion. Document the before-and-after configuration in the incident record.

Clean Up¶

• Remove the injected fault from the primary region.
• Rebaseline both regions to confirm symmetric health.

Related Playbook¶

• Multi-Region Failover

See Also¶

• EasyAuth Entra ID Failure Lab
• Bad Revision Rollout and Rollback

Sources¶

• Reliability in Azure Container Apps
• Create an Azure Front Door with the Azure CLI
• Azure Front Door CLI reference