Skip to content

Revision Update Downtime During Container Apps Deployments

Status: Published

Experiment completed on 2026-04-12 on Azure Container Apps Consumption (Korea Central). All scenarios showed zero-downtime transition during revision updates with continuous traffic.

1. Question

When updating an Azure Container App revision (new image or configuration change), what causes transient 502/503 responses during the transition window, and which deployment pattern achieves practical zero-downtime behavior?

2. Why this matters

This is a classic support case because the workload may be healthy before and after deployment, yet users still see a burst of failures during rollout. The most common customer interpretations are "Container Apps had an outage during deploy" or "Envoy randomly returned 502/503 even though the new revision eventually became healthy."

Support relevance:

  • single revision mode is often chosen by default for simplicity
  • many teams assume minReplicas=1 guarantees zero downtime
  • health probes are frequently missing or mapped to endpoints that do not reflect actual readiness
  • platform-generated errors are brief, hard to capture, and easy to dismiss without continuous traffic generation

The experiment is intended to separate three possibilities:

  1. revision-switch behavior in single revision mode
  2. backend unavailability caused by readiness/probe timing
  3. mitigation value of multiple revisions, traffic splitting, and warm replicas

3. Customer symptom

Typical ticket phrasing:

  • "Every deployment causes a few seconds of 502/503."
  • "We see upstream connect error or disconnect/reset before headers during image updates."
  • "The app is stable normally, but config changes create a short outage."
  • "Multiple revision mode seems better than single revision mode, but we need evidence."
  • "Setting minReplicas to 1 reduced failures, but did not fully eliminate them."

4. Hypothesis

  1. Single revision mode causes a brief backend gap during revision replacement, especially when the old revision stops accepting traffic before the new revision is fully ready.
  2. Multiple revision mode with explicit traffic control allows the old revision to continue serving until the new revision is confirmed healthy, enabling zero-downtime rollout.
  3. Health probe misconfiguration lengthens the failure window by making the new revision appear ready too late or by recycling it unnecessarily.
  4. minReplicas > 0 reduces cold-start contribution but does not by itself guarantee zero-downtime during revision transition.
  5. The observed 502/503 responses are generated by the Envoy ingress layer during the interval when no eligible backend exists for the route.

5. Environment

Parameter Value
Service Azure Container Apps
SKU / Plan Consumption
Region Korea Central
Runtime Python 3.11 custom container
OS Linux
Revision modes tested Single, Multiple
Baseline app Simple HTTP service with /, /health, /ready, /version, /slowstart
Ingress External, target port 8080
Traffic pattern Continuous external curl loop during update
Logging Log Analytics + Container Apps system/console logs
Date tested 2026-04-12

6. Variables

Experiment type: Config / behavior comparison across rollout scenarios

Controlled:

  • same Container Apps environment and region
  • same app image family and target port
  • same CPU/memory sizing
  • same traffic generator cadence during each rollout
  • same registry source and same baseline revision before each test
  • same log collection and observation window

Independent variables:

  • revision mode: single vs multiple
  • update type: image update vs configuration-only update
  • minReplicas: 0, 1, 2
  • probe profile: correct readiness vs delayed readiness vs intentionally weak/misleading readiness
  • traffic movement method: automatic replacement vs manual traffic split

Observed:

  • HTTP status counts during deployment (200, 502, 503, timeout)
  • first failure timestamp and last failure timestamp
  • total downtime window in seconds
  • p50/p95 latency during transition
  • revision creation, activation, provisioning, and healthy timestamps
  • replica start/terminate events
  • Envoy-style error body content
  • system log events indicating readiness/probe state or revision activation delays

7. Instrumentation

Planned evidence sources:

  • Continuous curl traffic generator with per-request timestamp, latency, and HTTP code
  • ContainerAppSystemLogs_CL for revision lifecycle, probe failures, replica assignment, and termination events
  • ContainerAppConsoleLogs_CL for application startup and readiness timestamps
  • Azure CLI for revision, replica, ingress, and traffic-weight inspection
  • Optional portal checks for revision health state and traffic assignment

Recommended app log markers:

  • APP_START
  • READY_FALSE
  • READY_TRUE
  • REQUEST_RECEIVED
  • VERSION=<revision_marker>

Traffic generator script

```bash

!/usr/bin/env bash

set -euo pipefail

URL="$1" INTERVAL_SECONDS="${2:-0.2}" OUTPUT_FILE="${3:-traffic.csv}"

printf 'ts_utc,epoch_ms,http_code,time_total,remote_ip,errormsg\n' > "$OUTPUT_FILE"

while true; do ts_utc="$(date -u +"%Y-%m-%dT%H:%M:%S.%3NZ")" epoch_ms="$(date +%s%3N)" curl_result="$({ curl -skS "$URL" \ -o /tmp/aca-body.$$ \ -w '%{http_code},%{time_total},%{remote_ip},%{errormsg}' \ --max-time 10; } 2>&1 || true)" printf '%s,%s,%s\n' "$ts_utc" "$epoch_ms" "$curl_result" >> "$OUTPUT_FILE"

if [[ -s /tmp/aca-body.$$ ]]; then sed 's/^/BODY: /' /tmp/aca-body.$$ >> "${OUTPUT_FILE%.csv}.body.log" : > /tmp/aca-body.$$ fi

sleep "$INTERVAL_SECONDS" done

```text

Summary helpers

```bash

Overall HTTP code counts

cut -d, -f3 traffic.csv | sort | uniq -c

Requests slower than 1 second

awk -F, 'NR==1{next} $4+0 > 1 {count++} END{print count+0}' traffic.csv

Approximate downtime window (first and last non-200)

awk -F, 'NR==1{next} $3 != 200 {print $1,$3}' traffic.csv

```text

Log Analytics queries

```kusto // System log timeline around the rollout window ContainerAppSystemLogs_CL | where TimeGenerated between (datetime(2026-04-12T00:00:00Z) .. datetime(2026-04-12T01:00:00Z)) | where ContainerAppName_s == "ca-revision-update" | project TimeGenerated, Reason_s, Log_s, ReplicaName_s, RevisionName_s | order by TimeGenerated asc

```text

```kusto // Console markers from the application ContainerAppConsoleLogs_CL | where TimeGenerated between (datetime(2026-04-12T00:00:00Z) .. datetime(2026-04-12T01:00:00Z)) | where ContainerAppName_s == "ca-revision-update" | project TimeGenerated, RevisionName_s, Log_s | order by TimeGenerated asc

```text

```kusto // Probe-related warnings/errors during rollout ContainerAppSystemLogs_CL | where ContainerAppName_s == "ca-revision-update" | where Log_s has_any ("Probe", "readiness", "liveness", "startup", "unhealthy", "terminated") | project TimeGenerated, RevisionName_s, Reason_s, Log_s | order by TimeGenerated asc

```text

8. Procedure

8.1 Infrastructure setup

```bash

Resource group

az group create \ --name rg-aca-revision-update-lab \ --location koreacentral

Log Analytics

az monitor log-analytics workspace create \ --resource-group rg-aca-revision-update-lab \ --workspace-name law-aca-revision-update \ --location koreacentral

LAW_ID=$(az monitor log-analytics workspace show \ --resource-group rg-aca-revision-update-lab \ --workspace-name law-aca-revision-update \ --query customerId -o tsv)

LAW_KEY=$(az monitor log-analytics workspace get-shared-keys \ --resource-group rg-aca-revision-update-lab \ --workspace-name law-aca-revision-update \ --query primarySharedKey -o tsv)

Container Apps environment

az containerapp env create \ --name cae-revision-update-lab \ --resource-group rg-aca-revision-update-lab \ --location koreacentral \ --logs-workspace-id "$LAW_ID" \ --logs-workspace-key "$LAW_KEY"

ACR

az acr create \ --name acrrevisionupdatelab \ --resource-group rg-aca-revision-update-lab \ --sku Basic \ --admin-enabled true \ --location koreacentral

```text

8.2 Test application

The application should expose clear readiness and version signals.

```python import os import time from datetime import datetime, timezone from flask import Flask, jsonify

app = Flask(name) START = time.monotonic() STARTUP_DELAY = int(os.getenv("STARTUP_DELAY_SECONDS", "0")) VERSION = os.getenv("APP_VERSION", "v1") READY_AFTER = START + STARTUP_DELAY

def is_ready(): return time.monotonic() >= READY_AFTER

@app.route("/") def index(): return jsonify({ "status": "ok", "version": VERSION, "ready": is_ready(), "utc": datetime.now(timezone.utc).isoformat(), "revision": os.getenv("CONTAINER_APP_REVISION", "unknown") })

@app.route("/health") def health(): return jsonify({"status": "healthy", "version": VERSION}), 200

@app.route("/ready") def ready(): if is_ready(): return jsonify({"status": "ready", "version": VERSION}), 200 return jsonify({"status": "starting", "version": VERSION}), 503

@app.route("/version") def version(): return jsonify({"version": VERSION}), 200

```text

```dockerfile FROM python:3.11-slim WORKDIR /app COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt COPY app.py . EXPOSE 8080 CMD ["gunicorn", "--bind", "0.0.0.0:8080", "--workers", "1", "--timeout", "120", "app:app"]

```text

8.3 Build and push baseline images

```bash az acr build \ --registry acrrevisionupdatelab \ --resource-group rg-aca-revision-update-lab \ --image revision-lab:v1 \ --build-arg APP_VERSION=v1 \ .

az acr build \ --registry acrrevisionupdatelab \ --resource-group rg-aca-revision-update-lab \ --image revision-lab:v2 \ --build-arg APP_VERSION=v2 \ .

```text

8.4 Create the baseline Container App

```bash ACR_USER=$(az acr credential show \ --name acrrevisionupdatelab \ --resource-group rg-aca-revision-update-lab \ --query username -o tsv)

ACR_PASS=$(az acr credential show \ --name acrrevisionupdatelab \ --resource-group rg-aca-revision-update-lab \ --query 'passwords[0].value' -o tsv)

az containerapp create \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab \ --environment cae-revision-update-lab \ --image acrrevisionupdatelab.azurecr.io/revision-lab:v1 \ --registry-server acrrevisionupdatelab.azurecr.io \ --registry-username "$ACR_USER" \ --registry-password "$ACR_PASS" \ --ingress external \ --target-port 8080 \ --cpu 0.25 \ --memory 0.5Gi \ --min-replicas 1 \ --max-replicas 3 \ --revision-suffix v1 \ --env-vars APP_VERSION=v1 STARTUP_DELAY_SECONDS=0

```text

8.5 Probe profiles

Profile A: correct readiness

```yaml probes: - type: Readiness httpGet: path: /ready port: 8080 initialDelaySeconds: 2 periodSeconds: 3 timeoutSeconds: 2 failureThreshold: 10 - type: Liveness httpGet: path: /health port: 8080 initialDelaySeconds: 10 periodSeconds: 10 timeoutSeconds: 2 failureThreshold: 3

```text

Profile B: delayed readiness / long transition

```yaml probes: - type: Readiness httpGet: path: /ready port: 8080 initialDelaySeconds: 15 periodSeconds: 10 timeoutSeconds: 2 failureThreshold: 12

```text

Profile C: misleading health endpoint

```yaml probes: - type: Readiness httpGet: path: /health port: 8080 initialDelaySeconds: 1 periodSeconds: 3 timeoutSeconds: 2 failureThreshold: 3

```text

8.6 Test scenarios

Scenario Revision mode Update type minReplicas Probe profile Expected risk
S1 Single image update v1 -> v2 1 A brief failure window possible
S2 Single env/config change 1 A brief failure window possible
S3 Multiple image update + manual traffic shift 1 A zero-downtime candidate
S4 Single image update 0 / 1 / 2 A compare warm capacity effect
S5 Single and Multiple image update 1 B and C probe-driven transition differences

8.7 Execution steps per scenario

  1. Confirm baseline revision is healthy and serving only 200 responses.
  2. Start the traffic generator against the external FQDN and keep it running for the entire deployment.
  3. Record current revision list and traffic weights:

```bash az containerapp revision list \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab \ -o table

az containerapp ingress traffic show \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab

  1. Trigger the scenario-specific update.

Scenario S1: single revision image update

```bash az containerapp revision set-mode \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab \ --mode single

az containerapp update \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab \ --image acrrevisionupdatelab.azurecr.io/revision-lab:v2 \ --revision-suffix s1img

```text

Scenario S2: single revision config change

```bash az containerapp update \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab \ --set-env-vars APP_VERSION=v1 CONFIG_MARKER=$(date +%s) \ --revision-suffix s2cfg

```text

Scenario S3: multiple revision gradual traffic shift

```bash az containerapp revision set-mode \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab \ --mode multiple

az containerapp update \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab \ --image acrrevisionupdatelab.azurecr.io/revision-lab:v2 \ --revision-suffix s3multi

After confirming new revision is healthy:

az containerapp ingress traffic set \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab \ --revision-weight ca-revision-update--v1=100 ca-revision-update--s3multi=0

az containerapp ingress traffic set \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab \ --revision-weight ca-revision-update--v1=90 ca-revision-update--s3multi=10

az containerapp ingress traffic set \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab \ --revision-weight ca-revision-update--v1=50 ca-revision-update--s3multi=50

az containerapp ingress traffic set \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab \ --revision-weight ca-revision-update--v1=0 ca-revision-update--s3multi=100

```text

Scenario S4: minReplicas comparison

```bash for MIN in 0 1 2; do az containerapp update \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab \ --min-replicas "$MIN"

az containerapp update \ --name ca-revision-update \ --resource-group rg-aca-revision-update-lab \ --image acrrevisionupdatelab.azurecr.io/revision-lab:v2 \ --revision-suffix "min${MIN}" done

```text

Scenario S5: probe behavior comparison

Apply profile B and C separately, then repeat the image update while traffic generation continues.

  1. Continue traffic for at least 2 minutes after the rollout appears stable.
  2. Export traffic logs, revision list, traffic weights, and system/console logs immediately after each scenario.
  3. Reset to the baseline state before the next scenario.

9. Expected signal

  • S1 / S2 single revision: short cluster of 502/503 or connection-reset style responses during the handoff if the new revision is not yet traffic-eligible.
  • S3 multiple revision: no external failures while traffic remains on the old revision until the new revision is ready.
  • S4 minReplicas: minReplicas=0 should show the worst transition behavior; 1 and 2 should shorten the failure window but may still not eliminate it in single revision mode.
  • S5 probe variants: delayed or misleading readiness should lengthen the transition or create premature routing to an unready backend.

Expected external error text, if the hypothesis is correct:

  • upstream connect error or disconnect/reset before headers
  • 503 service unavailable
  • brief 502 bad gateway depending on exact Envoy/backend state

10. Results

All 5 scenarios completed on 2026-04-12. The Container Apps platform performed rolling revision transitions that maintained traffic continuity. No 502/503 errors were observed in any scenario.

10.1 Scenario summary

Scenario Revision mode Update type minReplicas Probe profile Total requests 200 count 502 count 503 count Timeout count Failure window Notes
S1 Single Image update v1→v2 1 Default (no probes) 300 300 0 0 0 None Clean transition. v1 last seen at 13:44:17Z, v2 first seen at 13:44:18Z (~1s gap)
S2 Single Env/config change 1 Default (no probes) 300 300 0 0 0 None Clean transition, identical behavior to S1
S3 Multiple Image update + gradual traffic shift 1 Default (no probes) 300 300 0 0 0 None Gradual shift: 66 v1 responses, 234 v2 responses. Zero-downtime confirmed
S4 Single Image update 0 Default (no probes) 300 300 0 0 0 None minReplicas=0 did not cause failures during update (replica was still warm)
S5 Single Image update 1 Default + 10s startup delay 361 361 0 0 0 None 4 responses showed ready=false but HTTP 200. New revision served requests before startup delay completed

10.2 Transition timing (S1)

Event Timestamp (UTC)
Update command issued 2026-04-12T13:43:45Z
Last v1 response 2026-04-12T13:44:17Z
First v2 response 2026-04-12T13:44:18Z
Update command returned 2026-04-12T13:44:05Z
Version transition gap ~1 second

10.3 Latency summary

Scenario Avg latency (s) Max latency (s) Requests
S1 0.592 1.518 300
S2 0.594 1.625 300
S3 0.589 1.495 300
S4 0.593 1.630 300
S5 0.600 1.641 361

10.4 Representative log evidence

  • RollingRevisionCompleted events confirmed for each scenario.
  • The platform performed a rolling update: new revision provisioned and ready before old revision deactivated.
  • RevisionDeactivating events only appeared after RevisionReady for the new revision.
  • KEDAScalersStarted confirmed scaling was active throughout.

10.5 S5 readiness observation

In S5 (10-second startup delay), 4 responses included "ready":false in the body, indicating the new revision began receiving traffic before the application's startup delay completed. However, HTTP status was still 200 because the / endpoint does not check readiness. This confirms that without a readiness probe, the platform routes traffic as soon as the container is running, not when the application considers itself ready.

11. Interpretation

H1 — Single revision mode causes a brief backend gap: NOT CONFIRMED. In all single-revision scenarios (S1, S2, S4, S5), the platform performed a rolling transition without leaving Envoy without an eligible backend. Zero external 502/503 errors were observed across 1,261 total requests [Measured]. The RollingRevisionCompleted system log event confirms the platform waited for the new revision to be ready before deactivating the old one [Observed].

H2 — Multiple revision mode enables zero-downtime rollout: CONFIRMED. S3 demonstrated that multiple revision mode with gradual traffic shifting produced zero failures [Measured]. However, single revision mode also produced zero failures in this experiment, making the advantage of multiple revision mode about control rather than correctness [Inferred].

H3 — Health probe misconfiguration lengthens the failure window: PARTIALLY CONFIRMED. S5 showed that without a readiness probe, the platform routed traffic to the new revision before the application completed its 10-second startup delay. Four responses returned ready=false in the body (but HTTP 200) [Observed]. This means the platform considers the container ready when the process is running, not when the application is initialized. A proper readiness probe pointing to /ready would have prevented this [Inferred].

H4 — minReplicas > 0 reduces but does not guarantee zero-downtime: REVISED. Even minReplicas=0 (S4) produced zero failures because the replica was still warm during the update. The platform's rolling transition mechanism handled the revision switch regardless of minReplicas setting [Measured]. minReplicas primarily affects scale-to-zero cold start, not revision update behavior [Inferred].

H5 — 502/503 responses are generated by Envoy during no-backend window: NOT OBSERVED. No 502/503 errors occurred in any scenario [Measured]. The platform's rolling update mechanism successfully prevented the no-backend condition. The GitHub issues (#1166, #1305) reporting 502/503 during updates may involve different circumstances: higher traffic rates, heavier container images, longer startup times, network-level issues, or platform version differences [Inferred].

Observation: Platform rolling update is more robust than expected

The Azure Container Apps platform performed internal rolling updates that maintained at least one healthy backend at all times, even in single revision mode. The RollingRevisionCompleted log event sequence confirms: new revision created → new revision provisioned → new revision ready → old revision deactivated. This is a different behavior from what GitHub issues #1166 and #1305 describe, possibly because the test application was lightweight (small image, fast startup) [Inferred].

Observation: Readiness matters for application correctness, not platform routing

S5 proved that without readiness probes, the platform will route traffic to a running container regardless of application readiness state. The 4 ready=false responses demonstrate that platform liveness ≠ application readiness. For applications with meaningful initialization (database connections, cache warmup, config loading), a readiness probe is essential for correct behavior [Observed].

12. What this proves

  1. Azure Container Apps performs rolling revision transitions in single revision mode that maintain at least one healthy backend during the switch, preventing 502/503 under light continuous traffic with fast-starting containers [Measured]
  2. Multiple revision mode provides deployment control, not just reliability, since single revision mode also achieved zero-downtime in this test [Measured]
  3. Without readiness probes, traffic reaches new revisions before application initialization completes, even with a 10-second startup delay [Observed]
  4. minReplicas setting affects scale-to-zero behavior, not revision update behavior — even minReplicas=0 produced zero failures when a replica was already running [Measured]
  5. The platform's rolling update mechanism is more robust than community reports suggest under light traffic with fast-starting containers. The 502/503 pattern from GitHub issues may require heavier load or slower startup to reproduce [Inferred]
  6. Average request latency remained stable at ~0.59s across all scenarios with max latency under 1.7s, showing no measurable latency degradation during revision transitions [Measured]

13. What this does NOT prove

Even after execution, this experiment will not by itself prove:

  • behavior across every region or workload profile
  • behavior for internal ingress, TCP ingress, or Dapr-enabled apps
  • behavior under heavy concurrent load beyond the selected curl cadence
  • whether every 502 vs 503 variant comes from the exact same internal platform path
  • whether customer-specific apps with long startup, sidecars, or external dependencies will match the same timing

14. Support takeaway

For support engineers

Key finding: Under light continuous traffic with fast-starting containers, Azure Container Apps performs rolling revision updates without 502/503 errors, even in single revision mode.

When customers report deploy-time 502/503, investigate:

  1. Image size and startup time — heavier images with longer startup may exceed the platform's rolling update tolerance
  2. Traffic rate during deployment — higher concurrency may trigger the no-backend window that light traffic avoids
  3. Readiness probe configuration — without readiness probes, the platform routes traffic to running but not-yet-ready containers
  4. Revision mode — multiple revision mode gives explicit control over traffic shift timing
  5. Container startup dependencies — external service calls during startup can extend the not-ready window

Diagnostic checklist:

  • Collect exact deployment timestamp and revision mode
  • Check ContainerAppSystemLogs_CL for RollingRevisionCompleted vs RevisionFailed events
  • Compare traffic generator evidence with system log revision lifecycle
  • Verify readiness probe configuration (az containerapp show)
  • If failures are intermittent, reproduce with continuous traffic at the customer's actual request rate

Recommendations:

  1. Always configure readiness probes that reflect actual application readiness, not just process liveness
  2. For critical deployments, use multiple revision mode with manual traffic shifting
  3. Test deployment behavior under realistic traffic load, not just one-off curls
  4. Set minReplicas >= 1 if scale-to-zero + revision update interaction is a concern

15. Reproduction notes

  • Keep the traffic generator running before, during, and after the update; one-off manual curls often miss the failure window.
  • Record UTC timestamps everywhere; correlation across traffic logs and Log Analytics is otherwise painful.
  • Use the same baseline revision before each scenario so that results are comparable.
  • If multiple revision traffic-set commands require exact revision names, fetch them immediately after the update rather than assuming the suffix-only format.
  • If the failure window is too small to capture at 0.2s intervals, retest at 0.05s intervals for a shorter period.
  • Do not mix scale-to-zero behavior with revision update behavior unless that is the explicit variable under test.