Memory Leak OOMKilled Lab¶

Reproduce the canonical Azure Container Apps memory-pressure failure pattern (exit code 137 / OOMKilled), capture the evidence from CLI, KQL, and the Azure Portal Diagnose-and-Solve detectors, then verify that applying the fix creates a new healthy revision while preserving the failing revision for post-incident review.

Lab Metadata¶

1) Background¶

When a container's resident set size (RSS) crosses the cgroup memory ceiling configured by --memory (for example 0.5Gi), the Linux kernel OOM killer sends SIGKILL to the container's main PID. Azure Container Apps surfaces this as:

• Exit code 137 (128 + signal 9 = SIGKILL)
• Reason: ContainerTerminated, sub-reason ProcessExited
• HealthState: Unhealthy, ProvisioningState: Failed
• Restart loop driven by the platform's CrashLoopBackOff behavior

The exit code does not include an explicit "OOM" string in platform logs. Operators must correlate exit 137 with the MemoryPercentage / WorkingSetBytes metric curve to confirm OOM as the root cause versus other SIGKILL sources (admin kill, deployment cycle, manual restart).

Architecture¶

flowchart TD
    A["Azure Container Apps environment<br/>cae-memleak-* (koreacentral)"] --> B["Scenario A: ca-oom-hard"]
    A --> C["Scenario B: ca-oom-leak"]
    A --> D["Scenario C: ca-oom-healthy"]
    B --> B1["MODE=hard-oom<br/>allocate 600 MiB at startup<br/>memory 0.5Gi"]
    C --> C1["MODE=leak<br/>+30 MiB every 20s<br/>in background thread<br/>memory 0.5Gi"]
    D --> D1["MODE=healthy<br/>no allocations<br/>memory 1.0Gi"]
    B1 --> E["Observe:<br/>ContainerAppSystemLogs_CL<br/>WorkingSetBytes<br/>RestartCount<br/>D&S Exit Events detector"]
    C1 --> E
    D1 --> E
    E --> F["Apply fix to Scenario A:<br/>MODE=healthy + memory 1Gi<br/>creates new revision"]
    F --> G["Verify:<br/>HealthState=Healthy<br/>old revision retained for evidence"]

Comparing A and B against C proves the OOMs are caused by the workload's memory growth, not by the image, network, registry, or platform.

2) Hypothesis¶

IF three Container Apps share an environment but run workloads with different memory-allocation patterns, THEN:

• Scenario A (hard-oom): System logs show exit code '137' plus ProcessExited plus ContainerTerminated within the first 60 seconds after deployment. The revision's HealthState reports Unhealthy and ProvisioningState reports Failed. The container never serves HTTP traffic.
• Scenario B (leak): Console logs show progressive [leak] tick N: +30 MiB, total retained K MiB entries. MemoryPercentage climbs steadily over minutes. Around tick 12-15 (corresponding to ~360-450 MiB retained), the first OOM appears, followed by CrashLoopBackOff. The restart re-runs the leak, producing a periodic cliff pattern in the metric.
• Scenario C (healthy): WorkingSetBytes stays at ~30-50 MiB. RestartCount remains 0. HealthState reports Healthy. This is the control that proves the platform, the image, the network, and the registry are all fine.
• Fix: After trigger-fix.sh updates Scenario A to MODE=healthy with memory=1.0Gi, a new revision is created. The new revision reaches HealthState: Healthy within 90 seconds. MemoryPercentage drops to baseline. The Portal Diagnose-and-Solve Container App Memory Usage detector flips to green (no thresholds exceeded), and the failing revision is retained inactive in the Revisions blade for post-incident review.

3) Runbook¶

Deploy infrastructure¶

export RG="rg-aca-memleak-lab"
export LOCATION="koreacentral"
export BASE_NAME="memleak"

az group create --name "$RG" --location "$LOCATION"

az deployment group create \
    --resource-group "$RG" --name main \
    --template-file labs/memory-leak-oomkilled/infra/main.bicep \
    --parameters baseName="$BASE_NAME"

export ACR_NAME="$(az deployment group show --resource-group "$RG" --name main \
    --query properties.outputs.containerRegistryName.value --output tsv)"
export ENV_NAME="$(az deployment group show --resource-group "$RG" --name main \
    --query properties.outputs.environmentName.value --output tsv)"

Deploy the three scenarios¶

bash labs/memory-leak-oomkilled/trigger-scenario-a.sh
bash labs/memory-leak-oomkilled/trigger-scenario-b.sh
bash labs/memory-leak-oomkilled/trigger-scenario-c.sh

Wait, then verify each scenario¶

echo "Waiting 10 minutes for log ingestion and leak progression..."
sleep 600

APP_NAME=ca-oom-hard    bash labs/memory-leak-oomkilled/verify.sh
APP_NAME=ca-oom-leak    bash labs/memory-leak-oomkilled/verify.sh
APP_NAME=ca-oom-healthy bash labs/memory-leak-oomkilled/verify.sh

verify.sh collects every available signal per app: app configuration, active revisions, replica list, Log Analytics workspace ID, system-log query for exit-137 and OOM patterns, console-log query for [leak] tick entries, and RestartCount. Output goes to labs/memory-leak-oomkilled/evidence/<APP_NAME>/report-<TIMESTAMP>.txt. Note: MemoryPercentage and WorkingSetBytes time series are read from the Portal Metrics blade (captures 22-23, 27) rather than the CLI — the az monitor metrics list calls in verify.sh return empty payloads for short observation windows on newly created revisions, so the Portal Metrics chart is the authoritative source for memory-trace evidence in this lab.

Apply fix to Scenario A and verify recovery¶

bash labs/memory-leak-oomkilled/trigger-fix.sh
sleep 90
APP_NAME=ca-oom-hard bash labs/memory-leak-oomkilled/verify.sh

4) Experiment Log¶

Tested in Azure region Korea Central, 2026-06-20, az CLI 2.79.0, containerapp extension 1.3.0b4.

Scenario A: Hard OOM evidence [Observed]¶

# verify.sh output for ca-oom-hard (pre-fix, 2026-06-20T04:21:13Z)
Active revisions: ca-oom-hard--18xosgl
HealthState:      Unhealthy
ProvisioningState: Failed
RunningState:     Stopped

# System log entries (excerpt):
Container 'ca-oom-hard' was terminated with exit code '137' and reason 'ProcessExited'   ContainerTerminated
Container 'ca-oom-hard' was terminated with exit code '137'                              ProcessExited
Container 'ca-oom-hard' was terminated with exit code '137' and reason 'ProcessExited'   ContainerTerminated
... (76 such entries within ~17 minutes)

# Console log entries (excerpt): allocation progress prints only.
[hard-oom] allocating 600 MiB at startup (will exceed cgroup limit)
[hard-oom] allocated 50/600 MiB
[hard-oom] allocated 100/600 MiB
... (climbs to ~400-450/600 MiB)
# The line "[app] listening on :8000" is NEVER printed —
# the kernel killed the process during the allocation loop, before serve()
# was reached, so no HTTP traffic ever flowed.

Scenario B: Gradual leak evidence [Measured]¶

# verify.sh output for ca-oom-leak (2026-06-20T04:22:00Z)
Active revisions: ca-oom-leak--y1hn6jn
HealthState:      Healthy (at time of capture, before first OOM)
RestartCount:     0

# Console log entries (excerpt — progressive tick prints):
[leak] tick 1:  +30 MiB, total retained  30 MiB
[leak] tick 2:  +30 MiB, total retained  60 MiB
[leak] tick 3:  +30 MiB, total retained  90 MiB
...
[leak] tick 12: +30 MiB, total retained 360 MiB
[leak] tick 13: +30 MiB, total retained 390 MiB
[leak] tick 14: +30 MiB, total retained 420 MiB    ← OOM imminent

# Portal Metrics blade — WorkingSetBytes chart (capture 23, 60s bin):
# climbs linearly from ~50 MiB to ~450 MiB over the first 5 minutes,
# then drops to ~50 MiB at restart, then climbs again.

Scenario C: Healthy control evidence [Observed]¶

# verify.sh output for ca-oom-healthy (2026-06-20T04:22:46Z)
Active revisions: ca-oom-healthy--dfl8p2o
HealthState:      Healthy
ProvisioningState: Provisioned
RunningState:     RunningAtMaxScale
RestartCount:     0

# Console log entries: only the startup line (request logging is suppressed
# by `log_message = return` in the HTTP handler, so individual GET hits are
# not printed by design).
[app] listening on :8000
... (no errors, no terminations)

# Portal Metrics blade — WorkingSetBytes chart (capture 27): stable at
# ~30-50 MiB for the entire observation window.

Post-fix evidence [Observed]¶

# verify.sh output for ca-oom-hard (post-fix, 2026-06-20T06:02:57Z)
Active revisions: ca-oom-hard--0000001    ← NEW revision after fix
HealthState:      Healthy
ProvisioningState: Provisioned
RunningState:     RunningAtMaxScale
Replicas:         1
Memory:           1Gi                       ← raised from 0.5Gi
CPU:              0.5                       ← raised from 0.25

# System log entries during fix rollout:
Replica 'ca-oom-hard--0000001-67d777f859-v6brv' has been scheduled to run on a node.    AssigningReplica
Creating a new revision: ca-oom-hard--0000001                                            RevisionCreation
Rolling Transition: Successfully completed rolling over existing to latest revision...   RollingRevisionCompleted
Successfully updated containerApp: ca-oom-hard                                           ContainerAppReady
Deactivating old revisions for ContainerApp 'ca-oom-hard'                                RevisionDeactivating
Created container 'ca-oom-hard'                                                          ContainerCreated
Started container 'ca-oom-hard'                                                          ContainerStarted

# Failing revision (ca-oom-hard--18xosgl) status:
Active:           false
RunningState:     Stopped
(retained inactive for evidence; NOT deleted)

Operator takeaway from experiment¶

1. Exit code 137 alone does not prove OOM. SIGKILL also comes from admin actions, deployment cycles, and manual restarts. Confirm OOM with the MemoryPercentage curve — it should approach 100% just before the exit timestamp.
2. System logs do not explicitly say "OOMKilled". Look for exit code '137' + ProcessExited + ContainerTerminated. The kernel does not write an OOM message into the platform log surface — only the kill is visible.
3. Hard OOM versus gradual leak look different in metrics. Hard OOM: MemoryPercentage spikes to ~100% within seconds, container dies before HTTP traffic. Gradual leak: MemoryPercentage climbs over minutes, container serves traffic until the ceiling is hit, then restart history shows a periodic cliff pattern.
4. Always validate the fix. Raising memory alone does not fix a true leak — it only delays the OOM. The Scenario B pattern (re-leak after restart) makes this visible. The fix must change the workload (Scenario A's MODE=healthy switch) or include both axes (memory raise plus workload fix).
5. Preserve the failing revision. az containerapp update creates a new revision but does not delete the old one. The Revisions blade retains the failing revision's HealthState and RunningState for post-incident review. Do not delete it until evidence collection is complete.

4b) Phase B Falsification Gates¶

verify.sh is a pure file processor against the 35 historical-capture artifacts under evidence/ca-oom-{hard,leak,healthy}/ and emits four falsifiable gate JSONs under evidence/. Each gate is evaluated against a strict 2-path predicate per sub-gate — a Strong path that matches the exact lab specification (e.g. record-scoped 2-form OOM predicate matching the platform's two distinct exit-event row shapes, or §7 leak-tick regex parse) and a Fallback path that tolerates the same controlling behavior under minor numeric drift (e.g. the §6 5-minute bin histogram as a weaker proxy for the §5 record count). Eleven of the twelve sub-gates (4 gates × 3 sub-gates each) emitted on 2026-06-24 pass on both the Strong and Fallback paths against the canonical 2026-06-20 evidence. The twelfth — Gate 12 sub-gate (a) — passes on the Strong path only; its Fallback path is intentionally false by design (see the detailed disclosure two paragraphs below). All twelve *_pass flags evaluate to true because each sub-gate composes as Strong OR Fallback and the Strong path alone is sufficient.

The Gate 10 sub-gate (a) Strong path uses a record-scoped 2-form OOM predicate rather than a whole-file substring search. The Container Apps platform emits two distinct exit-event row shapes for the same OOM event: form A Reason_s == ContainerTerminated AND Log_s contains 'exit code \'137\'' AND Log_s contains 'ProcessExited' (11 records in hard) and form B Reason_s == ProcessExited AND Log_s contains 'exit code \'137\'' (5 records in hard). The gate matches both forms and sums them — a whole-file substring search would inflate the count by counting the same OOM event in both rows. This per-record discipline is why Gate 10's predicate_inputs lists the report path (parsed) rather than asserting against the raw text.

The Gate 12 sub-gate (a) Fallback path is intentionally false and documented as such in the gate JSON's section_6_bins_note: the §6 5-minute bin histogram uses a KQL has_any('memory') filter that produces false-positives on the healthy scenario (14 events in the 04:10 bin that are platform "memory" mentions unrelated to OOM). Only the Strong path (§5 record-scoped 2-form predicate, which returned 0) is authoritative for Gate 12. This is intentional defensive design — letting the fallback path silently mask a regression would defeat the purpose of the gate. The Gate 13 sub-gate (b) Strong path keys off §7 leak-tick count (not §6 bins) for the leak signature for the same reason. See labs/memory-leak-oomkilled/evidence/README.md "Honest disclosure" section for the full empirical platform behavior documented during the 2026-06-20 live run, including the three-timestamps-for-hard pattern (two PRE-fix at 04:19:00Z + 04:21:13Z plus one POST-fix at 06:02:57Z), the leak scenario's 1-OOM cross-cycle overlap, and the §10c PTY-failure-on-macOS that motivated using runningState: RunningAtMaxScale as the Gate 12 sub-gate (c) substitute for restart count.

The H1.a gate (10) proves hard-OOM has an immediate dense burst signature. The H1.b gate (11) proves leak has a delayed monotonic-runway signature distinct from hard. The H2 gate (12) proves healthy has zero OOM, falsifying any environmental-cause theory. The H3 cross-scenario gate (13) proves the three signatures are mutually distinguishable. Together these four gates yield the operator guidance in the Operator takeaway from experiment subsection above: exit code 137 alone does not classify the failure shape; the §6 bin density, §7 tick presence, and §5 record count together distinguish hard-OOM from gradual-leak from non-OOM-baseline.

5) Verification Queries¶

KQL: Cross-scenario exit-event summary¶

ContainerAppSystemLogs_CL
| where ContainerAppName_s in ("ca-oom-hard", "ca-oom-leak", "ca-oom-healthy")
| where Log_s has "exit code" or Reason_s in ("ContainerTerminated", "ProcessExited")
| summarize ExitEvents = count() by ContainerAppName_s, Reason_s
| order by ExitEvents desc

Expected: ca-oom-hard shows the highest count concentrated in ContainerTerminated and ProcessExited. ca-oom-leak shows a smaller but non-zero count. ca-oom-healthy shows zero.

KQL: Scenario B leak-tick console log¶

ContainerAppConsoleLogs_CL
| where ContainerAppName_s == "ca-oom-leak"
| where Log_s has "[leak] tick"
| project TimeGenerated, Log_s
| order by TimeGenerated asc

Expected: Progressive [leak] tick N: +30 MiB, total retained K MiB entries. The K value increases by 30 each tick until the container is killed and restarted.

KQL: Scenario A container-terminated detail¶

ContainerAppSystemLogs_CL
| where ContainerAppName_s == "ca-oom-hard"
| where Reason_s in ("ContainerTerminated", "ProcessExited")
| where Log_s has "137"
| project TimeGenerated, Reason_s, Log_s, RevisionName_s
| order by TimeGenerated asc

Expected: Repeated exit code '137' entries scoped to the failing revision (ca-oom-hard--18xosgl). Each entry is a separate SIGKILL event.

KQL: Memory growth timechart¶

ContainerAppConsoleLogs_CL
| where ContainerAppName_s == "ca-oom-leak"
| where Log_s has "[leak] tick"
| extend RetainedMiB = toint(extract(@"total retained (\d+) MiB", 1, Log_s))
| project TimeGenerated, RetainedMiB
| order by TimeGenerated asc
| render timechart

Expected: A staircase that climbs by 30 MiB every 20 seconds, with a sharp reset to baseline each time the container is OOMKilled and restarted.

KQL: ProbeFailed on Scenario B (after leak crosses ceiling)¶

ContainerAppSystemLogs_CL
| where ContainerAppName_s == "ca-oom-leak"
| where Reason_s == "ProbeFailed"
| project TimeGenerated, Log_s
| order by TimeGenerated asc

Expected: Probe-failure entries appearing once the leak exhausts the memory budget and the process slows or stops responding before SIGKILL.

KQL: ScaledObjectCheckFailed (KEDA correlation)¶

ContainerAppSystemLogs_CL
| where ContainerAppName_s == "ca-oom-hard"
| where Reason_s == "ScaledObjectCheckFailed"
| project TimeGenerated, Log_s
| order by TimeGenerated asc

Expected: KEDA scaler check failures while the revision is in CrashLoopBackOff (the scaler cannot read metrics from a container that is not Ready). This is a downstream symptom — not the root cause — and it disappears after the fix.

6) Portal Evidence¶

All captures live in docs/assets/troubleshooting/memory-leak-oomkilled/. Each capture was taken in a sanitized Azure Portal session with the standard PII helper applied before commit (see Portal capture rules for the capture policy and per-lab capture matrix).

Resource Group landing¶

Scenario A — ca-oom-hard (Hard OOM)¶

Scenario B — ca-oom-leak (Gradual leak)¶

Scenario C — ca-oom-healthy (Healthy control)¶

Environment and Log Analytics workspace¶

KQL verification queries (Logs blade)¶

Post-fix verification¶

Diagnose and Solve Problems (Portal-native OOM diagnosis)¶

The Azure Portal Diagnose-and-Solve Problems blade is the single best evidence surface for OOM diagnosis: it provides the Portal's own diagnosis text, the per-revision exit-event count, and a differential view that rules out alternative causes.

Clean Up¶

bash labs/memory-leak-oomkilled/cleanup.sh

Related Playbook¶

• Memory Leak OOMKilled

See Also¶

• CPU Throttling Lab
• CrashLoop OOM and Resource Pressure Playbook
• KEDA "No Metrics Returned" Lab

Sources¶

• Troubleshoot container start failures in Azure Container Apps
• Metrics in Azure Container Apps
• Containers in Azure Container Apps
• Application logging in Azure Container Apps
• Health probes in Azure Container Apps