Zone Redundancy Is Best-Effort Lab¶

Test the operator assumption that zoneRedundant=true plus minReplicas=N guarantees N replicas land in N Availability Zones. Statistically falsify or confirm the platform's behavior using parallel multi-min deployments and a deterministic perturbation control.

Lab Metadata¶

flowchart TD
    A[Zone-redundant Container Apps env] --> B[app-min2 min=max=2]
    A --> C[app-min3 min=max=3]
    A --> D[app-min6 min=max=6]
    E[Audit Job cron 5m] --> F[ARM REST list replicas]
    F --> G[ContainerAppConsoleLogs_CL<br/>ReplicaInventorySample JSON]
    B --> H[ContainerAppSystemLogs_CL<br/>ContainerTerminated / Started]
    C --> H
    D --> H
    I[trigger.sh --perturb restart] --> J[az containerapp revision restart]
    J --> H
    K[trigger.sh --load --client no-retry/retry-backoff] --> L[Local stdout JSON<br/>LoadEnd success/fail totals<br/>not ingested into LAW]
    G --> M[KQL pack Q1-Q4, Q6, Q7<br/>H0a falsification]
    H --> M
    L --> N[Local analysis<br/>H0b client-visible failure rate]

1. Question¶

Does a zone-redundant Container Apps environment, configured with minReplicas == maxReplicas == N and explicit resource requests, produce clustered multi-replica churn events for the same app over a 24-hour baseline window without operator action — and when those events occur, do they correlate with client-visible 503 failures?

The question is framed as two pre-registered hypotheses (Section 3) so the lab is falsifiable from observation alone, not from interpretation.

2. Setup¶

Region selection¶

Pick a region that supports both Container Apps workload profiles and Availability Zones. Verified options: koreacentral, eastus, westeurope, japaneast. Cross-check the current matrix in Reliability in Azure Container Apps before deploying.

Required environment variables¶

export RG="rg-aca-zr-lab"
export LOCATION="koreacentral"
export SUBSCRIPTION_ID="<subscription-id>"

az account set --subscription "$SUBSCRIPTION_ID"
az extension add --name containerapp --upgrade

Deploy the lab¶

cd labs/zone-redundancy-best-effort
./deploy.sh
./verify.sh

Build the audit image¶

The default auditImage parameter is a placeholder (mcr.microsoft.com/azure-cli:2.83.0) that emits a single notice JSON so deployment succeeds before the custom image is built. To produce real ReplicaInventorySample samples:

ACR="<your-acr>.azurecr.io"
az acr build --registry "$(basename "$ACR" .azurecr.io)" \
    --image "zr-lab/audit:latest" \
    ./audit

az deployment group create \
    --resource-group "$RG" \
    --template-file ./infra/main.bicep \
    --parameters ./infra/main.parameters.json \
    --parameters auditImage="${ACR}/zr-lab/audit:latest" \
    --parameters auditAcrName="$(basename "$ACR" .azurecr.io)"

3. Hypothesis¶

Two pre-registered null hypotheses. Both are stated before running any perturbation; the experiment is structured to keep these definitions immutable for the duration of the run.

H0a — Platform does not produce clustered multi-replica churn¶

In a zone-redundant Container Apps environment with fixed minReplicas=N (N in {2,3,6}) and explicit resource requests, the platform does not produce clustered multi-replica churn for the same app within any 60-second window over a 24-hour baseline period without operator action.

Clustered churn is defined operationally as:

Two or more replicas of the same app are observed in ContainerTerminated state within a single 60-second bin in ContainerAppSystemLogs_CL.

H0b — Clustered churn does not correlate with client-visible 503 spikes¶

When clustered multi-replica churn occurs (regardless of cause), the rate of HTTP 503 responses observed at the ingress for that app does not spike above the baseline rate during the 60-second window covering the churn event, when the calling client uses default no-retry behavior.

Pre-registered analysis plan¶

To prevent post-hoc reframing once data is in hand, the lab commits to these decisions before the first measurement:

1. Primary metric for H0a: count of clustered-churn events per app over the 24-h baseline window from KQL Q3. A non-zero count for any app falsifies H0a.
2. Primary metric for H0b: client-visible failure rate from the single trigger.sh --combined --client no-retry --duration 180 run, using the LoadEnd totals (fail / total) for the perturbation window that contains the Q3-confirmed clustered-churn event. Any fail >= 1 during that window falsifies H0b for this lab run. The lab does not query KQL for the 5xx signal — the helloworld sample app does not emit structured access logs, and trigger.sh output is not ingested into Log Analytics. See "Limitations" in Section 12.
3. Secondary metrics (descriptive only, not used to confirm/refute):
   • MaxReplacementFraction per app from Q7 (how much of the app was replaced in one event).
   • RecoverySecs per event from Q4.
   • Difference in failure rate between --client no-retry and --client retry-backoff runs (L2 mitigation effect).

4. Stopping rule: collect a single 24-h baseline window, then execute exactly three perturbation events against app-min3 in sequence, recording the exact UTC timestamp of each event into the committed evidence corpus:

   1. One supplemental restart-all run — trigger.sh --perturb restart with no client load. Deterministic az containerapp revision restart against the active revision; characterizes worst-case MaxReplacementFraction without load confounding.
   2. One combined no-retry run of 180 s — trigger.sh --combined --client no-retry --duration 180. Load harness fires from LoadStart through LoadEnd; revision-restart fires mid-window. This is the H0b primary metric (Section 3, item 2).
   3. One combined retry-backoff run of 180 s — trigger.sh --combined --client retry-backoff --duration 180. Quantifies the L2 mitigation effect.

   Do not extend the run on the basis of early results. This stopping rule supersedes any earlier "exactly three perturbation runs per client variant" framing; the lab's pre-registered design intentionally trades multi-run replication for tight scope and committed evidence corpus, per the design rationale in issue #204. 5. What this lab does NOT measure: (a) per-replica AZ placement — the Azure Container Apps management plane does not expose it, and IMDS inside a Container Apps replica is unsupported; any zone-related conclusion is [Inferred] from clustered-churn patterns, never [Observed] per-replica zone data. (b) Server-side 5xx counts joined to Q3 windows in KQL — the lab's sample app does not produce ingested HTTP access logs; production users with App Insights or ingress logs can satisfy this via the optional Q5 template.

4. Prediction¶

If the platform delivers the strict "N replicas in N zones, zero clustered churn" interpretation many operators assume, then:

• H0a is not falsified — Q3 returns zero rows during the 24-h baseline.
• H0b is not falsified — during the single trigger.sh --combined --client no-retry --duration 180 run on app-min3, LoadEnd.fail == 0.
• The no-retry and retry-backoff client variants converge to identical success rates.

If the platform delivers best-effort distribution (the published MS Learn behavior), then:

• H0a may be falsified — Q3 returns at least one row, most likely on app-min2 and app-min3 rather than app-min6.
• H0b is likely falsified for the no-retry client during perturbation; retry-backoff measurably narrows the gap.
• MaxReplacementFraction is highest (closest to 1.0) on app-min2, where every clustered churn is more likely to take out the whole app.

5. Experiment¶

Phase 1 — Baseline (24 hours, no operator action)¶

./verify.sh
date -u +"%Y-%m-%dT%H:%M:%SZ" > /tmp/zr-baseline-start.txt

# Wait 24 hours. Do not touch the apps, env, or audit Job.
# Optional: tail the audit output via az containerapp job execution list.

date -u +"%Y-%m-%dT%H:%M:%SZ" > /tmp/zr-baseline-end.txt

Phase 2 — Perturbation (controlled — three runs total against app-min3)¶

Per the stopping rule pre-registered in Section 3 item 4, the perturbation phase consists of exactly three runs against app-min3, executed in sequence with their exact UTC timestamps recorded in the committed evidence corpus. There is no inner loop and no multi-run replication: the H0b primary metric is the single combined no-retry run, and the retry-backoff run quantifies the L2 mitigation effect. The supplemental restart-all run characterizes worst-case MaxReplacementFraction without load confounding.

Always pipe stdout to a durable file under labs/zone-redundancy-best-effort/evidence/ (not /tmp/, which Phase 2 of the original partial run lost). Each filename embeds the variant tag and a UTC timestamp so KQL Q6 can isolate the windows.

EV=labs/zone-redundancy-best-effort/evidence
TS() { date -u +"%Y%m%d%H%M%S"; }

# Run 1 (supplemental) — restart-only, no load. Worst-case MaxReplacementFraction.
./trigger.sh --perturb restart --app app-min3 \
    | tee "$EV/perturbation-variant-a-restart-only-$(TS).log"
sleep 1800   # Record the actual gap before Run 2 in the evidence filename/timestamp; no minimum gap is asserted by this reproduction.

# Run 2 (H0b primary) — combined no-retry, 180 s. Single restart fires mid-window.
./trigger.sh --combined --client no-retry --duration 180 --app app-min3 \
    | tee "$EV/perturbation-variant-b-restart-load-$(TS).log"
sleep 1800   # Record the actual gap before Run 3 in the evidence filename/timestamp; no minimum gap is asserted by this reproduction.

# Run 3 (L2 mitigation measurement) — combined retry-backoff, 180 s.
./trigger.sh --combined --client retry-backoff --duration 180 --app app-min3 \
    | tee "$EV/perturbation-variant-c-retry-backoff-$(TS).log"

Phase 3 — (Optional) Cross-app comparison¶

Repeat one full perturbation pair against app-min2 and app-min6 to populate Q7 with three subject apps.

6. Execution¶

Run Phase 1 continuously for 24 hours with no other lab activity in $RG. For Phase 2, run the three perturbation events in sequence; each event is timestamped and analyzed as a distinct perturbation window in the committed evidence corpus. Capture every command's stdout into labs/zone-redundancy-best-effort/evidence/ (per the Phase 2 runbook in Section 5 — not /tmp/, which the lab's original partial run lost) so Q6's PerturbWindows datatable can be hand-populated from the committed PerturbationStart timestamps after the fact.

7. Observation¶

After Phase 1 + 2 + (optional) 3, run the KQL pack queries in order:

Record the exact numbers in the Observed Evidence subsection of Section 12.

8. Measurement¶

The lab's primary metrics, derived directly from the pre-registered KQL queries and from trigger.sh local stdout:

• [Measured] BaselineChurnEvents per app over 24 h (from Q3, filtered to the baseline window from Q6).
• [Measured] PerturbationChurnEvents per app and client variant (from Q3 + Q6).
• [Measured] MaxReplacementFraction per minReplicas (from Q7).
• [Measured] RecoverySecs p50 / p95 across all events (from Q4).
• [Measured] Client-visible failure rate per variant, computed locally from trigger.sh --client {no-retry,retry-backoff} stdout (LoadEnd.fail / LoadEnd.total). This is the H0b primary metric — see Section 3.
• [Inferred] H0b verdict tying client-visible failure rate during a perturbation window to a Q3-confirmed clustered-churn event. The inference is necessary because trigger.sh stdout is not joined to Q3 in KQL.

9. Analysis¶

Combine the measurements with the pre-registered analysis plan:

• H0a evaluation: If BaselineChurnEvents == 0 for all three apps, H0a is not falsified for this 24-h window. If any app has BaselineChurnEvents >= 1, H0a is falsified — record the timestamps and replica IDs as [Observed] evidence.
• H0b evaluation: From the single trigger.sh --combined --client no-retry --duration 180 run, take the LoadEnd.fail / LoadEnd.total totals for the perturbation window that contains the Q3-confirmed clustered-churn event. If fail >= 1, H0b is falsified. This is a [Measured] failure rate but only an [Inferred] causal link to the churn event, because trigger.sh stdout and Q3 are joined manually by timestamp, not in KQL.
• Client variant comparison: Compute the failure-rate ratio between no-retry and retry-backoff runs against the same app. A ratio >> 1 (no-retry fails much more) is [Measured] evidence that L2 client-side resilience is the dominant mitigation for clustered churn.
• Cross-app comparison: Examine whether BaselineChurnEvents decreases monotonically with minReplicas. A monotonic decrease is [Correlated] evidence that higher minReplicas dilutes churn impact. A flat or increasing pattern is [Observed] evidence that raising replica count alone is not a sufficient mitigation.

10. Conclusion¶

State the conclusion in three buckets, one per evidence level, citing only the pre-registered measurements:

• Confirmed (or falsified) hypotheses: H0a and H0b verdict, with the specific Q3 numbers and trigger.sh LoadEnd totals that drove the decision.
• Confirmed secondary effects: L2 retry impact, cross-app pattern.
• Open / not measurable: per-replica AZ placement (always [Not Proven] for this lab regardless of result).

11. Falsification¶

The lab is built to be falsifiable in two complementary directions:

1. Negative-control rigor: A baseline window with BaselineChurnEvents == 0 for all three apps falsifies the working hypothesis that the platform produces clustered churn. The lab reports zero baseline rows as [Observed] evidence that no clustered-churn event was detected in this 24-hour window, and [Inferred] evidence that any platform-driven churn is below this run's detection floor for this configuration.
2. Positive-control rigor: trigger.sh --perturb restart is a deterministic restart of the active revision. Q3 must return a row covering the perturbation timestamp; if it does not, the audit pipeline or Q3 itself is broken and no downstream conclusion is valid. Re-run Phase 1 first.

12. Evidence¶

Required CLI evidence (always collected)¶

Required KQL evidence¶

Run Mass-Reschedule KQL pack Q1-Q4, Q6, Q7. Q5 is optional — only meaningful if you wire App Insights (or another ingested 5xx source) to the subject apps; the default sample image does not emit ingested 5xx telemetry and the lab's H0b verdict comes from trigger.sh stdout (see Section 3, item 2).

Optional setup: Custom subject-app image + Azure Monitor workbook¶

This lab ships two opt-in components for richer evidence beyond the default helloworld image. The infrastructure for both components is committed under labs/zone-redundancy-best-effort/; measured evidence (AppRequests rows for Q5 + C9/C13 Portal captures) will be added in a follow-up reproduction run.

• Custom subject-app image at labs/zone-redundancy-best-effort/apps/ (Python Flask + Azure Monitor OpenTelemetry Distro). Override the appImage parameter on az deployment group create to deploy — for a private ACR image, you must also pass appAcrName so the Bicep wires up AcrPull and the registries block (otherwise the first image pull fails with 401 Unauthorized); see apps/README.md for the full az acr build + az deployment group create flow. Populates AppRequests with AppRoleName equal to the container app name (auto-injected by Container Apps via CONTAINER_APP_NAME) and the /error route returns HTTP 500 for Q5 validation. Once running, KQL pack Q5 surfaces app-handled 5xx telemetry (validates telemetry plumbing and exposes the /error route exception path) and capture C9 becomes capturable. The H0b verdict continues to rely on trigger.sh --client no-retry stdout totals (LoadEnd.fail / LoadEnd.total) — ingress-generated 503s during clustered churn never reach the Flask process and therefore produce no AppRequests row, so Q5 alone is not a KQL-backed H0b verdict. A true KQL-backed H0b would require ingesting ingress-side or client-side 5xx telemetry, which this lab defers to a follow-up.
• Azure Monitor workbook at labs/zone-redundancy-best-effort/workbook/ (3-panel: Q3 clustered churn table, Q4 recovery duration line chart, Q7 multi-app comparison bar chart). Deploy from the labs/zone-redundancy-best-effort/workbook/ directory via az deployment group create --resource-group $RG --template-file workbook-arm.json --parameters workbookSourceId=$LAW_ID (where $LAW_ID is the lab Log Analytics workspace resource ID; see workbook/README.md for the full deploy flow). Once deployed, the workbook becomes visible in the workspace gallery and capture C13 becomes capturable.

The infra/main.bicep template now provisions a workspace-based Application Insights resource (appi-zrlab-${suffix}, linked to the existing law Log Analytics workspace) and injects APPLICATIONINSIGHTS_CONNECTION_STRING as a secret-backed environment variable on each of the three subject Container Apps. The default appImage remains mcr.microsoft.com/azuredocs/containerapps-helloworld:latest so existing reproductions continue to work unchanged; the connection string is harmlessly ignored by the helloworld image.

Observed Evidence (Live Azure Reproduction)¶

Reproduction window: Phase 1 baseline 2026-06-12T11:51:46Z → 2026-06-13T11:51:46Z (24h); Phase 2 perturbations 2026-06-14T11:04:37Z, 11:15:46Z, 11:29:11Z (all UTC, all against app-min3).

Region / RG: koreacentral / rg-aca-zr-lab-260612114313.

Subject apps: app-min2 (minReplicas=2), app-min3 (minReplicas=3), app-min6 (minReplicas=6), all in a single zone-redundant env (cae-zrlab-5yi4px, properties.zoneRedundant=true).

Audit Job: audit-sampler cron */5 * * * *, observed ReplicaInventorySample count = 867 samples over the 24-h baseline window across 3 apps (Q1: HealthRatio=1.0, 867 / 864 expected = 100.3% completeness, 0 error samples).

Caveat 1 on H0b. The 0 / 1950 failure rate under this load profile (10 RPS, single-revision restart, 3 replicas, two 180-second windows) is a positive signal that minReplicas=3 plus Container Apps ingress drain behavior is sufficient to absorb single-revision churn for low-traffic apps. It does not generalize to higher load, simultaneous multi-revision events, or longer load windows. Re-run with trigger.sh --rps 50 (or stop all replicas concurrently across revisions) before claiming app-min3 is "503-safe under all clustered-churn scenarios".

Caveat 2 on Q6 baseline rows. Q6 in the KQL pack uses a rolling 24-h window for its baseline arm (anchored to query execution time). When this lab is re-run, the rolling baseline catches up the perturbation events from the previous lab session's tail. In this reproduction, Q6 shows Baseline: 3 events / 0.13 hour on app-min3 — those 3 events are the pre-fix partial perturbations from the same lab session (10:32, 10:44, 10:54 UTC), not platform-driven background churn. The true 24-h Phase 1 baseline window (2026-06-12T11:51:46Z → 2026-06-13T11:51:46Z, fixed-range Q3) is the H0a-falsifying measurement and returned zero rows on all three apps. The fixed-range query is in evidence/q3-baseline-fixed-clustered-churn-20260614114618.*.

Caveat 3 on run history. Three earlier partial perturbation runs (10:31, 10:42, 10:53 UTC) emitted invalid timestamps because trigger.sh used GNU-only date +%s%3N and date -u +"%Y-%m-%dT%H:%M:%S.%3NZ" — both produce literal %3N on macOS BSD date. The corresponding LoadEnd totals are uncomputable from those logs. The script was patched with a portable detection block (GNU date if available, else perl -MTime::HiRes) before the successful re-runs at 11:04 / 11:15 / 11:29. The three pre-fix logs were moved to evidence/.local/partial-pre-fix/ (gitignored) for traceability and are intentionally not part of the committed corpus. The Q6 KQL pack had a parallel double-Z bug (head -c 20 | awk '{print $1 "Z"}' over a string already ending in Z) that produced datetime(...ZZ) syntax errors; that fix is in labs/zone-redundancy-best-effort/evidence/run_kql_pack.sh.

Portal captures (2026-06-14 reproduction)¶

The Portal captures from this reproduction are committed under docs/assets/troubleshooting/zone-redundancy-best-effort/.

[Observed] C1 — Container Apps environment Overview. The environment cae-zrlab-5yi4px displays its zone-redundant status alongside the koreacentral region in the Essentials tile, confirming properties.zoneRedundant=true at the environment level.

[Observed] C2 — Container Apps environment Workload profiles tab. The Consumption workload profile is provisioned inside this workload-profile environment, which is the host context for app-min3. Zone redundancy is configured at the environment level (per C1) and not per profile.

[Observed] C3 — app-min3 Overview blade. The subject app app-min3 is in Running state, and the Configuration tile shows Min replicas = Max replicas = 3, which is the minReplicas value the H0a and H0b hypotheses are framed around.

[Observed] C4 — app-min3 Revisions and replicas tab (baseline). Before any perturbation, all 3 replicas are listed under a single active revision in the green Running state. This is the steady-state baseline that Q2 confirmed at SteadyStateOK=True for all 289 audit samples over the 24-h baseline window.

[Observed] C6 — Log Analytics Q1 ingestion check. The Q1 ingestion-check query is pasted into the Log Analytics Logs editor and returns HealthRatio near 1.0 (lab measurement: 1.0 over 867 samples), confirming the audit-sampler cron emitted ReplicaInventorySample events into ContainerAppConsoleLogs_CL reliably across the 24-h baseline window.

[Observed] C7 — Log Analytics Q3 clustered-churn result. The Q3 clustered-churn query returns the perturbation-induced row for app-min3 — the smoking-gun signal that az containerapp revision restart produced a cluster of ContainerTerminated events within a single 60-s bin, which is the MaxReplacementFraction=1.0 measurement summarized in the table above.

[Observed] C11 — app-min3 Metrics blade (Replica Count plus Restart Count). The Metrics blade chart shows the perturbation dip in Replica Count and the corresponding spike in Restart Count for app-min3, plus the recovery to steady-state within the 600-s deadline (Q4: RecoverySecs p50 approximately 83 s, p95 approximately 262 s, all 6 events WithinDeadline=True).

Mapping to Container Apps Non-Guarantee Claims¶

This lab tests three distinct claims about Container Apps zone redundancy. Each claim has a different evidence ceiling, and the ceilings are independent of how cleanly any single reproduction passes. Even with a perfect 24-h baseline, Claims 2 and 3 remain capped because the underlying signal is not exposed by the platform.

Why this matters for incident response. When a customer escalates a "zone-redundant outage" case, support engineers should be careful not to over-promise based on Claim 2. The permanent [Strongly Suggested] cap on Claim 2 means there is no internal Container Apps signal that can prove a customer's replicas were spread across zones at the moment of incident; the Container Apps management plane does not expose per-replica AZ identity, and that visibility limit is a known platform property rather than a defect. A practical framing is to anchor the response in [Measured] operational behavior from the customer's own workspace (Q3 / Q4 / Q7 numbers — clustered churn count, recovery duration, replacement fraction) and to acknowledge the visibility gap explicitly when describing what the platform can and cannot show. Customers whose RTO genuinely requires per-replica zone visibility have options beyond Container Apps — for example, AKS surfaces per-node zone identity via the standard Kubernetes topology labels — but that is an architectural choice driven by the RTO requirement, not the only available path.

12.1 Phase B 4-gate evidence pack¶

The committed Phase B overlay for this lab is a non-falsification with bounded coverage variant, not the standard H1-trigger / H2-fix repair loop used by canonical troubleshooting labs. It reuses the existing 24-hour reproduction corpus under labs/zone-redundancy-best-effort/evidence/ and adds four derived gate files that re-validate what the corpus can honestly support.

Gate semantics¶

Phase B 16-sub-gate result table¶

Bounded coverage disclosure¶

• Bounded subject scope. The perturbation evidence is limited to app-min3. app-min2 and app-min6 appear only in the negative-control baseline and in the zero-churn comparison rows from Q7.
• Bounded load scope. The only client-bearing measurements are the two successful 180 s runs at 10 RPS (Variant B no-retry, Variant C retry-backoff). The measured 0 / 1950 failures do not generalize beyond that envelope.
• Excluded Q6 artifacts. q6-baseline-vs-perturb-20260614114318.json is excluded because it is unparsable (datetime(...ZZ) syntax error). q6-baseline-vs-perturb-20260614114522.json is excluded from H0a because its rolling baseline bucket includes earlier partial perturbations and is therefore contaminated.
• Evidence ceilings. Claim 2 (zone distribution) stays at [Strongly Suggested] because Container Apps does not expose per-replica Availability Zone identity. Claim 3 (multi-replica platform events) also stays at [Strongly Suggested] because this corpus measures operator-triggered clustered churn, not platform-triggered clustered churn.

13. Solution¶

Apply the four-layer mitigation matrix from the companion playbook. Summary:

No single layer is sufficient. Pick the layers that match your RTO and document the assumption in your runbook.

14. Prevention¶

• Replace the "zone-redundant = zero 5xx" assumption wherever it appears (runbooks, service definitions, post-incident reviews). The pre-registered framing of H0a / H0b is the safer mental model.
• Set resources.requests and resources.limits on every container during the first deploy. Treat unset values as a deploy-time CI check failure.
• Run this lab once per quarter (or whenever the team onboards a new region) so the team retains hands-on familiarity with clustered-churn signatures.
• Wire the Q3 query into an Azure Monitor alert rule so the next clustered churn is detected platform-side, not from customer complaints.

15. Takeaway¶

Zone redundancy is a best-effort placement behavior, not a placement guarantee. The platform makes a strong attempt to spread replicas across zones, but per-replica zone placement is not exposed and not contractually guaranteed. The reliability you actually get is the product of the four mitigation layers, not the result of any single layer. Setting zoneRedundant=true alone changes the probability distribution of failure events; it does not change their existence.

16. Support Takeaway¶

When a customer escalates a "zone-redundant environment had a brief 5xx outage" case:

1. Run the Mass-Reschedule KQL pack Q1-Q3 against their workspace for the incident window. A non-zero Q3 row is expected and is not automatically a platform defect.
2. Inspect resource requests and probe configuration with the CLI commands in Section 12 before any escalation — vague resource shapes are the leading documented contributor.
3. If clustered-churn rate exceeds the customer's documented SLO after L1 + L2 are in place, that is the trigger for L3 / AKS escalation review — not the first clustered churn event.
4. Reference Microsoft Learn's exact wording on best-effort distribution in the case notes so the customer's mental model is reset before further investigation.

Clean Up¶

./cleanup.sh

Related Playbook¶

• Zone Redundancy Is Best-Effort

See Also¶

• Mass-Reschedule KQL Pack
• Multi-Region Failover Lab
• Multi-Region Failover Playbook
• Replica Load Imbalance Lab
• Replica Load Imbalance Playbook

Sources¶

• Reliability in Azure Container Apps
• Set up zone redundancy in Azure Container Apps
• Planned maintenance for Azure Container Apps
• Scale an app in Azure Container Apps
• Workload profiles in Azure Container Apps
• Containers in Azure Container Apps