Lab Guide (Level 3): SNAT Exhaustion on Azure App Service Linux¶

This lab is a full diagnostic reference for reproducing and proving outbound SNAT pressure on Azure App Service Linux using a Python/Flask workload. It expands the original scaffold into a complete investigation package with architecture background, falsifiable hypothesis, deterministic runbook, and artifact-backed experiment log.

Lab Metadata¶

1) Background¶

SNAT exhaustion on App Service is rarely a single-event failure. It is usually a cascade:

1. App generates high outbound connection churn.
2. Platform SNAT mapping inventory gets stressed.
3. Pending outbound calls wait longer for usable translated ports.
4. Upstream call latency grows.
5. Worker threads block and queue expands.
6. Gunicorn workers timeout and get killed/recycled.
7. Inbound availability degrades (499/503/000 symptoms).

1.1 Outbound flow on App Service Linux¶

The following logical flow explains where SNAT sits in the path:

flowchart TD
    A[Client Request] --> B[App Service Front End]
    B --> C[Linux Worker / Gunicorn]
    C --> D[Flask /outbound endpoint]
    D --> E[New outbound TCP connection]
    E --> F[Azure Load Balancer SNAT]
    F --> G[Internet target e.g. httpbin.org]
    G --> F
    F --> C

    style F fill:#f57c00,color:#fff
    style D fill:#1976d2,color:#fff
    style C fill:#455a64,color:#fff

Key point: SNAT mapping happens on platform egress. Your code does not directly manage SNAT tables, but your connection behavior determines churn pressure.

1.2 Why per-request TCP creation is dangerous¶

In this lab app:

• /outbound calls requests.get() with Connection: close for every call (no shared session). The April 2026 artifacts were captured with an earlier version that used urllib.request; both open a new connection per call.
• Every outbound call tends to create a fresh TCP socket.
• Under concurrency, sockets accumulate in active and post-close states.

When requests finish, connections do not disappear immediately due to TCP lifecycle behavior (for example, TIME_WAIT). That lag means transient churn can still consume port inventory for a while.

1.3 SNAT port inventory and App Service guidance¶

Microsoft guidance for App Service outbound troubleshooting describes finite SNAT inventory and recommends connection reuse/pooling to avoid intermittent failures.

Operationally relevant concepts:

• Finite SNAT mappings per instance.
• Port reuse delay due to TCP lifecycle.
• Connection pooling reduces churn and improves stability.
• Symptom signatures: outbound timeout, connection refused, intermittent spikes.

1.4 Causal mechanics with TCP states¶

sequenceDiagram
    participant W as Worker Thread
    participant A as App Code (/outbound)
    participant L as LB SNAT
    participant T as External Target

    W->>A: Handle inbound request
    A->>L: Open outbound TCP
    L->>T: Forward with translated source port
    T-->>L: Response
    L-->>A: Return response
    A-->>W: Complete call
    Note over L: Port mapping not always instantly reusable

Under low traffic this is fine. Under burst concurrency with no pooling, the per-call setup/teardown overhead becomes dominant and error-prone.

1.5 Why this lab can also show worker SIGKILL¶

The lab does not claim SNAT directly kills a worker process. The chain is indirect:

• Outbound calls stall.
• Request handlers exceed worker timeout thresholds.
• Worker recycling/kill events increase.
• Platform and app become unstable.

This is a classic cascading failure pattern, where initial network pressure manifests as process churn.

1.6 Lab code paths relevant to diagnosis¶

1.7 Why this is not only an outbound problem¶

Outbound instability can surface as inbound errors:

• 499 in HTTP logs (client closed or downstream timeout path).
• 503 when process/service is degraded.
• 000 in synthetic probes (curl transport failure).

1.8 Diagram: healthy vs exhausted behavior¶

flowchart TD
    A[Inbound request] --> B{Outbound call mode}
    B -->|Pooled| C[Reuse existing sockets]
    C --> D[Low churn]
    D --> E[Stable latency]
    E --> F[Healthy workers]

    B -->|No pooling| G[Create many new sockets]
    G --> H[High SNAT churn]
    H --> I[Outbound delay/timeout]
    I --> J[Worker timeout]
    J --> K[SIGKILL/recycle events]
    K --> L[499/503 increase]

    style C fill:#2e7d32,color:#fff
    style G fill:#ef6c00,color:#fff
    style K fill:#c62828,color:#fff

1.9 Baseline environment evidence (from artifacts)¶

Source files:

• baseline/diag-net.json
• baseline/diag-stats.json
• baseline/diag-env.json
• baseline/app-config.json
• baseline/health.json

Observed baseline values:

2) Hypothesis¶

2.1 Statement (falsifiable)¶

Hypothesis:

When a Python/Flask app creates a new outbound TCP connection per request without connection pooling, SNAT ports exhaust within minutes under concurrent load, causing timeouts and SIGKILL'd workers.

2.2 Causal chain under test¶

flowchart TD
    A[No pooling in /outbound] --> B[High outbound socket churn]
    B --> C[SNAT mapping pressure]
    C --> D[Outbound timeout growth]
    D --> E[Gunicorn worker timeout]
    E --> F[Worker SIGKILL / recycle]
    F --> G[HTTP 499/503 and curl 000]

    style C fill:#ef6c00,color:#fff
    style E fill:#d84315,color:#fff
    style F fill:#b71c1c,color:#fff

2.3 Proof criteria¶

All of the following must be observed in the same trigger window:

1. Transport failures appear under load
   • curl results include 000 responses and long (~60s) waits.
2. HTTP log degradation appears
   • Large share of 499/503 with elevated TimeTaken.
3. Application timeout signatures appear
   • Body samples include timeout text (for example, The read operation timed out).
4. Worker instability appears in console logs
   • WORKER TIMEOUT and SIGKILL events recorded.
5. Recovery indicator appears after pressure drops
   • Diagnostic endpoints become reachable again and counters restart/new PID appears.

2.4 Disproof criteria¶

Any one of the following disconfirms this specific chain:

• High concurrency produces no transport failures and no elevated HTTP time.
• No worker timeout/SIGKILL events during failure period.
• Failures occur equally in pooled and non-pooled paths with equivalent concurrency.
• Artifact evidence shows stable outbound behavior and no timeout signatures.

2.5 Scope boundaries¶

This lab tests application-driven outbound churn behavior, not every possible outbound failure root cause.

Not in scope:

• Upstream service outage as primary fault.
• DNS-wide outage.
• VNet routing misconfiguration.
• TLS certificate trust misconfiguration.

2.6 Expected measurable variables¶

2.7 Competing explanations considered¶

3) Runbook¶

This runbook is the repeatable execution path. Use long-form flags for Azure CLI commands.

3.1 Prerequisites¶

3.2 Variable setup¶

export RG="rg-lab-snat"
export LOCATION="koreacentral"
export TEMPLATE_FILE="labs/snat-exhaustion/main.bicep"

3.3 Deploy infrastructure¶

az group create --name "$RG" --location "$LOCATION"

az deployment group create \
  --resource-group "$RG" \
  --template-file "$TEMPLATE_FILE"

Capture app name:

export APP_NAME=$(az webapp list \
  --resource-group "$RG" \
  --query "[0].name" \
  --output tsv)

export APP_HOST=$(az webapp show \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --query "defaultHostName" \
  --output tsv)

export APP_URL="https://$APP_HOST"

3.4 Deploy lab app¶

az webapp deploy \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --src-path "labs/snat-exhaustion/app" \
  --type zip \
  --restart true

3.5 Baseline checks¶

curl --silent --show-error "$APP_URL/health"
curl --silent --show-error "$APP_URL/diag/env"
curl --silent --show-error "$APP_URL/diag/net"
curl --silent --show-error "$APP_URL/diag/stats"

Expected baseline shape:

• Health returns status=healthy.
• WEBSITES_PORT and/or PORT indicate container listener context.
• /diag/net returns low TCP pressure.

3.6 Trigger failure mode¶

bash "labs/snat-exhaustion/trigger.sh" "$APP_URL" outbound

Trigger behavior from script:

• Sends 200 /outbound?calls=40 requests, at most 20 in flight, each with a 180-second client timeout.
• Summarizes outer status codes (000 = no response) and the inner call results read from each JSON body, because the endpoint answers 200 even when inner calls fail.

3.7 Optional control check (pooled endpoint)¶

Wait about 5 minutes after the non-pooled phase, then run the same load shape against the pooled endpoint so the two phases are directly comparable (§4.14):

bash "labs/snat-exhaustion/trigger.sh" "$APP_URL" outbound-fixed

3.8 Collect platform diagnostics¶

Portal view: Networking blade (outbound IP context)¶

Diagnose and solve problems is the Portal first stop for SNAT investigations: the Networking card and the Network Troubleshooter link under Diagnostic Tools both pivot to detectors that surface SNAT port pressure and outbound IP saturation. The Web App Slow detector under Availability and Performance often lights up first as well, because the worker queue fills with TCP-blocked requests.

Purpose: Show the app's inbound and outbound network configuration on one page.

Look for: Public network access and Private endpoints under inbound traffic, and Virtual network integration, Outbound DNS, and the outbound address lists under outbound traffic.

Expected result: This capture shows the displayed configuration of a public app without VNet integration: the listed outbound addresses are the platform's potential outbound address pool (not a measured egress address for any connection), and the integration subnet settings show N/A.

The Networking blade is the Portal counterpart to the SNAT KQL queries below. The Outbound traffic configuration column confirms Virtual network integration is Not configured and shows the ~30 shared platform-pool Outbound IPv4 addresses (<ip-redacted>, <ip-redacted>, ...) that this app shares with other tenants. NAT gateway, Network security group, and User defined route all show N/A under Integration subnet configuration, which is consistent with the default platform SNAT allocation rather than a dedicated pool [Inferred from displayed configuration]. This capture does not measure the actual egress path or the available or consumed SNAT capacity; those need the separate evidence below. It does not by itself show that the ports were exhausted. After confirming this state, run the queries below to quantify the resulting 499/503 errors.

HTTP signal query¶

AppServiceHTTPLogs
| where TimeGenerated > ago(2h)
| where CsHost has "azurewebsites"
| where CsUriStem in ("/outbound", "/outbound-fixed", "/diag/net", "/health")
| project TimeGenerated, CsUriStem, ScStatus, TimeTaken, CsHost
| order by TimeGenerated desc

Console signal query¶

AppServiceConsoleLogs
| where TimeGenerated > ago(2h)
| where ResultDescription has_any (
    "WORKER TIMEOUT",
    "SIGKILL",
    "timed out",
    "connection refused",
    "Cannot assign requested address",
    "EADDRNOTAVAIL"
)
| project TimeGenerated, ResultDescription
| order by TimeGenerated desc

Platform signal query¶

AppServicePlatformLogs
| where TimeGenerated > ago(2h)
| where Message has_any (
    "warmup",
    "Container",
    "startup",
    "timeout"
)
| project TimeGenerated, Level, Message
| order by TimeGenerated desc

3.9 Azure CLI-based KQL execution (optional automation)¶

export WORKSPACE_ID="<log-analytics-workspace-id>"

az monitor log-analytics query \
  --workspace "$WORKSPACE_ID" \
  --analytics-query "AppServiceHTTPLogs | where TimeGenerated > ago(2h) | where CsUriStem in ('/outbound','/outbound-fixed','/diag/net') | project TimeGenerated, CsUriStem, ScStatus, TimeTaken | order by TimeGenerated desc" \
  --output json

3.10 Validate recovery state¶

curl --silent --show-error "$APP_URL/diag/net"
curl --silent --show-error "$APP_URL/diag/stats"
curl --silent --show-error "$APP_URL/health"

If reachable and stable after pressure window, recovery evidence is present.

4) Experiment Log¶

This section is derived from actual sanitized lab artifacts:

labs/snat-exhaustion/artifacts-sanitized/

4.1 Artifact inventory used¶

4.2 Baseline snapshot¶

4.2.1 /diag/stats baseline¶

Raw signal:

{"endpoint_counters":{"diag_stats":1},"outbound_call_counters":{"with-pooling":{"failures":0,"successes":0},"without-pooling":{"failures":0,"successes":0}},"pid":1907,"process_start_time":"2026-04-04T05:05:40.566103+00:00","request_count":1,"uptime_seconds":1642.522}

Interpretation:

• Process was long-lived before trigger.
• No outbound failures accumulated yet.

4.2.2 /diag/net baseline¶

Raw signal:

{"connection_count":10,"ip_local_port_range":{"end":"60999","start":"32768"},"sockstat":{"sockets":{"used":"10"},"tcp":{"alloc":"196","inuse":"5","mem":"18","orphan":"1","tw":"4"},"udp":{"inuse":"1","mem":"0"}}}

Interpretation:

• Low active pressure.
• Existing tw=4 confirms normal TCP post-close behavior.

4.3 Trigger output summary (targeted CSV)¶

Source: trigger/outbound-targeted-20260404T055433Z.csv

Derived summary:

Selected rows:

4.4 Error-body sample evidence¶

Sources:

• error-body-2_body-20260404T055433Z.txt
• error-body-3_body-20260404T055433Z.txt
• error-body-10_body-20260404T055433Z.txt
• error-body-11_body-20260404T055433Z.txt
• error-body-14_body-20260404T055433Z.txt

Aggregated findings:

Representative payload with failure:

{"calls":20,"elapsedMs":35774,"failures":1,"mode":"without-pooling","sampleErrors":["The read operation timed out"],"successes":19,"target":"https://httpbin.org/get"}

4.5 Diag endpoint reachability during pressure¶

This confirms transient unreachability during the failure window.

4.6 HTTP KQL analysis¶

Source: kql-http-20260404T060610Z.json

Dataset size:

• Total rows: 195

Status distribution:

Endpoint-focused findings:

Interpretation:

• HTTP telemetry matches timeout-dominated failure shape.
• The near-60s cluster strongly aligns with outbound wait/timeout behavior.

4.6.1 Log Analytics KQL query for /outbound status distribution (portal verification)¶

To cross-verify the /outbound status mix summarized in §4.6 (138 rows on /outbound from kql-http-20260404T060610Z.json, split 122 × 499 / 16 × 200) is a real property of the raw AppServiceHTTPLogs rows the Log Analytics service holds and not a JSON-parse artifact of the sanitized export, the Log Analytics workspace Logs blade was opened against the same workspace from a fresh live reproduction and a status-distribution query scoped to /outbound was executed interactively against the current live data:

Purpose: Provide an independent Portal-side execution of a /outbound-scoped status distribution query so a reviewer can confirm that the dominant-499 signature in §4.6 (122 × 499 vs 16 × 200 = ~7.6:1 fail-to-success ratio) is a real property of the raw AppServiceHTTPLogs rows the Log Analytics service holds — and can also see that the failure mode reproduces with the same shape (majority 499, minority 200) when the trigger is re-run against a freshly deployed lab instance, with a slightly more aggressive fail-to-success ratio (152:12 = ~12.7:1) that reflects a longer sustained degradation window than the original sanitized artifact captured.

Look for:

• Blade heading reads "log-labsnat-6kr7mkxulwioo | Logs" and the sub-heading reads "Log Analytics workspace" — this confirms the query ran against the same workspace attached to this lab's Web App via the Microsoft.Insights/diagnosticSettings resource declared in labs/snat-exhaustion/main.bicep, matching the workspace referenced by the sanitized JSON exports in §4.1.
• The KQL editor shows exactly the query AppServiceHTTPLogs | where TimeGenerated > ago(2h) | where CsUriStem == '/outbound' | summarize Count = count() by ScStatus | order by ScStatus asc — the CsUriStem == '/outbound' filter is an exact string match so rows for /health, /diag/stats, or /outbound-fixed are excluded even though they may share the same trigger window. The 2-hour lookback covers both the trigger execution and the log-ingestion delay (typically ~2-5 minutes for AppServiceHTTPLogs).
• The Results grid shows exactly 3 rows (pagination "1 - 3 of 3") — no other status class appeared on /outbound in this window. The absence of 202 is expected because this query is scoped to CsUriStem == '/outbound'. In the original sanitized export, the two 202 rows are SCM deployment traffic on the .scm host (/api/deployments/latest and /api/zipdeploy), not responses from /outbound-fixed in labs/snat-exhaustion/app/app.py.
• The 499 row shows Count 152 — this dominates the distribution at ~90% of all /outbound requests, matching the original §4.6's 122 × 499 / 138 total = ~88% ratio and showing the primary failure signature is a proxy-level abort rather than an application-side 5xx error. The April analysis attributed the aborts to SNAT-blocked outbound calls; the §4.14 rerun produced the same 499 signature with no failed outbound calls, so the abort alone does not identify the cause.
• The 200 row shows Count 12 — these are the small number of /outbound requests that completed successfully, which the April analysis attributed to the earliest requests in the burst running before SNAT ports were exhausted (128 preallocated ports per instance, up to 40 fresh connections per request). That attribution was not independently confirmed; see §4.14.
• The 502 row shows Count 4 — a low-volume tail absent from the sanitized §4.6 export. This capture proves only that a small minority of /outbound requests failed as platform gateway errors during the same saturation window; because AppServiceConsoleLogs were empty in this reproduction, do not attribute those four rows specifically to gunicorn's --timeout=120 worker recycle.
• Query duration in bottom left reads a small millisecond value (here 1s 531ms) — confirming the workspace is not throttled and the query hit indexed data, so the returned row count and status distribution are authoritative and not a partial-scan timeout.

Expected result: The 3 rows returned by the Portal show a dominant 499 count (>85% of /outbound requests) with minority 200 and a small 502 tail, matching §4.6's verdict for the April run, which attributed the pattern to SNAT-port exhaustion causing proxy-level connection aborts rather than application-side 5xx errors. The matched rerun in §4.14 produced the same 499 pattern with zero failed outbound calls, so the pattern alone does not prove SNAT exhaustion. If instead the Portal query returned a Count on 200 that exceeded 499 (i.e., majority success), the trigger did not reproduce the dominant degradation; check that its concurrency (200 requests with 20 in flight in labs/snat-exhaustion/trigger.sh) was not reduced. Neither outcome shows port-pool state on its own: that requires the SNAT Port Exhaustion detector on a production-tier plan.

Next step: If a future reproduction shows the 502 count growing above ~20% of the total, confirm the path with AppServiceConsoleLogs and inspect labs/snat-exhaustion/main.bicep for changes to appCommandLine (specifically --timeout=120); if the lab genuinely needs more outbound headroom, prefer connection reuse / pooling, scale-out, or NAT gateway rather than scaling up SKU. Conversely, if the 200 count grows above ~30% of the total, the SNAT port pool may have been enlarged (for example by enabling VNet integration with a NAT gateway giving 64K SNAT ports per outbound IP) — check for a Microsoft.Network/natGateways resource attached to the Web App's outbound subnet, because that would fundamentally change the reproducibility of this lab's SNAT-exhaustion hypothesis.

4.6.2 Log Analytics KQL query for top-10 slowest /outbound requests (portal verification)¶

To cross-verify the tail-latency evidence embedded in §4.6 (Rows with TimeTaken >= 59000ms: 123 and /outbound rows with 499 and TimeTaken >= 59000ms: 118 from the sanitized export) is not a JSON-parse rounding artifact and to characterize the extreme tail more precisely than the 59-second threshold that §4.6 uses, the same Log Analytics workspace Logs blade was reopened and a top-10 TimeTaken query scoped to /outbound was executed interactively against the current live data:

Purpose: Provide an independent Portal-side execution of a top-10 TimeTaken query on /outbound so a reviewer can identify the exact ceiling that clamps the tail rather than just the ~60-second lower bound recorded in §4.6. Where §4.6 tells you 118 /outbound requests exceeded 59 s, this query tells you the ceiling is extremely tight at ~240 s — a diagnostic signature of a hard proxy-level timeout, not variable-latency degradation.

Look for:

• Blade heading reads "log-labsnat-6kr7mkxulwioo | Logs" and the sub-heading reads "Log Analytics workspace" — confirms the query ran against the same workspace as §4.6.1 and matches the workspace referenced by the sanitized JSON exports in §4.1.
• The KQL editor shows exactly the query AppServiceHTTPLogs | where TimeGenerated > ago(2h) | where CsUriStem == '/outbound' | project TimeGenerated, CsUriStem, ScStatus, TimeTaken | order by TimeTaken desc | take 10 — project limits the returned columns to the four needed for latency analysis, keeping the payload small and the query fast.
• The Results grid shows exactly 10 rows (pagination "1 - 10 of 10") and every row's CsUriStem column reads /outbound and every row's ScStatus column reads 499 — no /outbound request in the top-10 latency slice succeeded with 200 or platform-errored with 502. This confirms the failure signature at the extreme tail is uniformly client-abort/proxy-abort (499), not a mix of statuses.
• TimeTaken values cluster extremely tightly at 240,015-240,036 ms (a 21 ms spread across all 10 rows) — this is the signature of a hard outer request-lifetime ceiling, not variable-latency degradation where you would expect a wider spread. The closest documented App Service timer is the ~230-second request timeout described in Why does my request time out after 230 seconds?; this capture is consistent with that class of upstream timeout, but the screenshot alone does not prove which exact timer owner produced the 240 s cutoff.
• The ~240 s ceiling is far above the app-side gunicorn --timeout=120 per-worker limit configured in labs/snat-exhaustion/main.bicep appCommandLine. That means the recorded HTTP lifetime is not explained by the 120 s worker timer alone. Because console logs were empty in this reproduction, avoid claiming a specific "worker killed at 120 s, then front-end waited another ~120 s" sequence; the defensible conclusion is only that an outer request timeout above the worker timeout dominates the tail.
• The 10 TimeGenerated values span a ~22 minute 42 second wall-clock window (6:04:20 to 6:27:02 UTC) — this is not a bounded burst. Requests initiated 22 minutes apart hit the same 240 s ceiling, so the degraded state persisted as long as the trigger kept issuing concurrent /outbound?calls=40 requests. The April analysis read this as sustained SNAT exhaustion; a persistent worker queue (§4.14) produces the same spread.
• Query duration in bottom left reads 0s 912ms — slightly faster than the §4.6.1 status-distribution query (1s 531ms) because take 10 after an order by TimeTaken desc is cheaper than a full summarize over all rows in the window; both queries are well under the 30-second workspace timeout, so both result sets are authoritative.

Expected result: The top-10 latency rows cluster at ~240 s with ScStatus 499 on /outbound, matching the April §4.11 verdict (qualified by the §4.14 rerun): requests stay pending long enough to hit a consistent outer request timeout. The April analysis attributed the wait to depleted SNAT ports; the §4.14 rerun shows the same ceiling can come from queued requests behind busy workers. If instead the top-10 clustered at ~120 s with ScStatus 502, the dominant cutoff would have moved closer to the gunicorn worker timer rather than the outer request timeout; that is still compatible with SNAT pressure, but it would need confirmation from AppServiceConsoleLogs before attributing the immediate cutoff to worker recycle.

Next step: If a future reproduction shows the top-10 TimeTaken cluster near ~120 s instead of ~240 s (matching gunicorn --timeout=120), verify that shift with AppServiceConsoleLogs before changing configuration, because the root SNAT mitigation remains outbound connection reuse / pooling or larger SNAT capacity rather than simply raising --timeout.

4.7 Console KQL analysis¶

Source: kql-console-20260404T060610Z.json

Dataset size and window:

Pattern counts:

Representative lines:

[2026-04-04 05:59:42 +0000] [1904] [CRITICAL] WORKER TIMEOUT (pid:1908)
[2026-04-04 05:59:43 +0000] [1904] [ERROR] Worker (pid:1908) was sent SIGKILL! Perhaps out of memory?
[2026-04-04 05:58:45 +0000] [1904] [CRITICAL] WORKER TIMEOUT (pid:1905)
[2026-04-04 05:58:46 +0000] [1904] [ERROR] Worker (pid:1905) was sent SIGKILL! Perhaps out of memory?

Interpretation:

• Strong process churn coincides with high outbound timeout phase.
• This supports cascading instability, not isolated request errors.

4.8 Platform KQL analysis¶

Source: kql-platform-20260404T060610Z.json

Rows sampled: 200

Dominant content:

• Container lifecycle messages.
• Startup/warmup informational traces.
• No contradictory signal that would independently explain the timeout burst.

4.9 PID rollover evidence¶

Compare baseline vs recovered diagnostics:

This indicates process turnover occurred during the trigger window.

4.10 Failure cascade timeline (reconstructed)¶

timeline
    title SNAT Lab Failure Cascade (artifact reconstruction)
    05:40:02 : Console window starts
    05:52:07 : New worker generation visible
    05:54-05:56 : Trigger pressure period
    05:56:00-05:57:26 : HTTP 499 near 59-60s dominates
    05:58-05:59 : Repeated WORKER TIMEOUT and SIGKILL events
    05:55+ : /diag endpoints eventually recover

4.11 Hypothesis verdict¶

Verdict (April 2026 artifacts): the timeout, 499, and worker-churn symptoms of the hypothesis were observed. SNAT exhaustion itself was not directly measured, so the hypothesis is not confirmed without port-pressure evidence.

4.12 Practical mitigation mapping¶

4.13 Recommended follow-up experiment¶

The matched run against /outbound-fixed is now recorded in 4.14. A remaining follow-up is a run on a production-tier plan (Standard or higher), where the SNAT Port Exhaustion detector reports port data, at a connection rate high enough to exhaust the preallocated ports.

4.14 Matched rerun against a lab-owned target (2026-10-02)¶

Setup differences from the April run: the outbound target is a storage account deployed by main.bicep (anonymous ?comp=list, answered with a fast 403) instead of httpbin.org; /outbound uses requests with a new connection per call so every call is recorded in AppDependencies; Application Insights is enabled. Plan: one Basic B1 instance in Korea Central, Gunicorn with 4 sync workers. Both phases used the same trigger shape (200 requests, calls=40, concurrency 20, 180-second client timeout), separated by a 5-minute recovery wait.

The rerun meets the 2.4 disproof criterion "No worker timeout/SIGKILL events during failure period" for the SNAT chain while still showing that pooling removes the inbound degradation.

Expected Evidence¶

This section defines what you SHOULD observe at each phase of the lab. Use it to validate your investigation is on track.

Before Trigger (Baseline)¶

During Incident¶

After Recovery¶

Evidence Timeline¶

graph TD
    A[Baseline Capture] --> B[Trigger Fault]
    B --> C[During: Collect Evidence]
    C --> D[After: Compare to Baseline]
    D --> E[Verdict: Confirmed/Falsified]

Evidence Chain: What Confirms or Falsifies the Hypothesis¶

Clean Up¶

az group delete --name "$RG" --yes --no-wait

Related Playbook¶

• SNAT or Application Issue?

See Also¶

• SNAT or Application Issue? (Playbook)
• First 10 Minutes: App Service Linux Troubleshooting

Sources¶

• Troubleshoot intermittent outbound connection errors in Azure App Service
• Azure Load Balancer outbound connections
• Configure a custom container for Azure App Service
• Enable diagnostic logging for apps in Azure App Service
• App Service diagnostics overview