Lab: Deployment Succeeded, Startup Failed (Wrong Module in Startup Command)¶

This lab reproduces a high-frequency Azure App Service Linux incident pattern: deployment appears successful, but runtime startup fails, so the site returns 503 or timeout errors.

The lab is intentionally designed to prove one specific distinction: deployment success and runtime success are separate control-plane and data-plane outcomes.

This guide helps you explain why az webapp deploy can succeed while the app still fails at runtime, trace the Linux startup lifecycle, detect wrong_module:app failures from logs, validate recovery after correcting the startup command to app:app, and produce an artifact-backed incident narrative.

Lab Metadata¶

1) Background¶

1.1 Why this lab exists¶

In App Service Linux, deployment and process startup are decoupled.

A deployment pipeline (ZIP deploy / Oryx build / Kudu APIs) can complete successfully, yet the process entrypoint may crash immediately once container startup begins.

This is exactly what happens when Gunicorn is pointed to a module that does not exist.

In this lab:

• The configured startup command initially uses wrong_module:app.
• The code package itself is valid.
• Deployment APIs return success signals.
• Runtime logs show ModuleNotFoundError.
• Health endpoint remains unavailable until startup command is corrected.

1.2 Deployment success vs runtime success¶

Use this mental model:

• Deployment success means the platform accepted and unpacked/build-prepared your package.
• Runtime success means worker process booted, app bound to expected port, and warmup probes succeeded.

Deployment success does not imply runtime success.

flowchart TD
    A[Client invokes az webapp deploy] --> B[Kudu ZIP Deploy API accepts package]
    B --> C[Build preparation / startup script generation]
    C --> D[Site container starts]
    D --> E{Gunicorn imports module?}
    E -->|Yes| F[Process listens on PORT]
    E -->|No| G[Process exits with startup exception]
    F --> H[Warmup and health probes pass]
    H --> I[HTTP 200 for app endpoints]
    G --> J[Container start failure recorded]
    J --> K[HTTP 503 or timeout]

    style A fill:#e3f2fd
    style B fill:#e3f2fd
    style C fill:#e3f2fd
    style D fill:#fff3e0
    style E fill:#fffde7
    style F fill:#e8f5e9
    style H fill:#e8f5e9
    style I fill:#e8f5e9
    style G fill:#ffebee
    style J fill:#ffebee
    style K fill:#ffebee

Portal view: Deployment Center (control-plane deployment status)¶

The Deployment Center blade is the Portal-visible surface of the same Kudu ZIP Deploy / Oryx build pipeline that the diagram above traces from box A to box C. When this lab's az webapp deploy --type zip command completes, this blade is where an operator first looks to confirm "the deployment worked" - the Settings tab reports the configured source, the Logs tab lists each deployment ID with a Success status, and the production-slot banner reflects the lab's single-slot setup. The critical insight is that this entire blade is a control-plane view: a green Success here only tells you the package was accepted, unpacked, and that the startup command was scheduled - it tells you nothing about whether the worker actually imported app:app, bound PORT, or passed the warmup probe (boxes D through K in the diagram). Treat this blade as "deployment delivered to the host" evidence only, and pair it with the runtime evidence from sections 3.6 through 3.10 before declaring an incident resolved.

1.3 Startup lifecycle details on App Service Linux¶

For Python on App Service Linux, the high-level startup chain is:

1. Site content is deployed.
2. Platform prepares startup invocation (often via Oryx-generated script).
3. Container starts.
4. Startup command launches Gunicorn.
5. Gunicorn resolves <module>:<application-object>.
6. Worker boots if module import succeeds.
7. Warmup probes verify app responsiveness.

If step 5 fails, Gunicorn exits, container cannot become healthy, and probes keep failing until timeout / restart.

sequenceDiagram
    participant Dev as Operator
    participant CLI as Azure CLI
    participant Kudu as Kudu/SCM
    participant Site as App Service Site Container
    participant Gunicorn as Gunicorn Master
    participant App as Flask Module
    participant Probe as Warmup Probe

    Dev->>CLI: az webapp deploy --src-path package.zip
    CLI->>Kudu: ZIP deploy request
    Kudu-->>CLI: 202 Accepted / deployment progressing
    Kudu->>Site: trigger restart with startup command
    Site->>Gunicorn: execute gunicorn ... wrong_module:app
    Gunicorn->>App: import wrong_module
    App-->>Gunicorn: ModuleNotFoundError
    Gunicorn-->>Site: worker failed to boot
    Probe->>Site: HTTP probe / warmup
    Site-->>Probe: 503 or timeout

1.4 Why wrong_module:app fails immediately¶

Gunicorn expects module:callable.

In this lab app, the Flask object is defined in app.py as app = Flask(__name__).

Therefore, valid module reference is:

app:app

Invalid reference:

wrong_module:app

The failure occurs before request handling, so the app never reaches a state where /health can respond.

1.5 Storage setting context for startup incidents¶

This lab focuses on module import failure, but startup incidents are often misattributed to storage behavior.

WEBSITES_ENABLE_APP_SERVICE_STORAGE is relevant mostly for custom containers and shared /home storage semantics.

Key point for this incident class:

• Startup command and module resolution failures happen even when storage is fine.
• Do not over-index on storage when logs clearly show import exceptions.

1.6 App Service startup timeout signal¶

Platform logs in this dataset include this diagnostic phrase:

Container did not start within expected time limit of 230s

Interpretation:

• Platform waited for startup and warmup completion.
• App process never reached healthy serving state in time.
• The 230s marker is a startup/warmup failure symptom, not a deployment pipeline failure.

1.7 Failure mode taxonomy for this lab¶

1.8 Diagram: control-plane success with data-plane failure¶

graph TD
    subgraph ControlPlane[Control Plane / Deployment]
        CP1[ZIP Package Upload]
        CP2[Kudu Deployment API]
        CP3[Deployment Status Success]
    end

    subgraph RuntimePlane[Runtime / Worker Execution]
        RP1[Container Start]
        RP2[Gunicorn Import wrong_module]
        RP3[Crash]
        RP4[Probe Failure 503]
    end

    CP1 --> CP2 --> CP3
    CP3 --> RP1 --> RP2 --> RP3 --> RP4

    style CP3 fill:#c8e6c9
    style RP3 fill:#ffcdd2
    style RP4 fill:#ffcdd2

1.9 Background takeaway¶

When an incident says:

“Deployment succeeded but app is down”

the first triage branch should include startup command validation, not only deployment rollback.

2) Hypothesis¶

2.1 Primary hypothesis (this lab)¶

When the startup command references a non-existent module (wrong_module:app instead of app:app), the container process crashes immediately, while deployment still reports success because package deployment and runtime boot are separate stages.

2.2 Causal chain¶

1. ZIP deploy uploads valid application code
      ↓
2. Kudu deployment APIs return success/accepted responses
      ↓
3. App restart runs startup command with wrong module
      ↓
4. Gunicorn import fails: ModuleNotFoundError
      ↓
5. Worker process exits before app can bind and serve
      ↓
6. Warmup/health probing fails (503/timeout)
      ↓
7. After startup command corrected to app:app, process boots and /health returns 200

2.3 Proof criteria for the checked-in procedure¶

All criteria below must be met:

1. The Bicep deployment provisions wrong_module:app before the application package is deployed.
2. az webapp deploy completes while that faulty startup command remains configured.
3. The trigger's first /health probe returns a non-200 result or times out.
4. The trigger changes the startup command to app:app without redeploying the package.
5. The trigger's post-repair /health probe returns 200, or a later manual probe confirms recovery after restart completes.
6. The verifier finds a matching ModuleNotFoundError, wrong_module, or No module named console-log row within its rolling two-hour query window. A zero count is inconclusive because ingestion may be delayed or the evidence may have aged out.

2.4 Disproof criteria¶

Any one condition disproves the hypothesis:

• Startup command is already correct during failure window.
• No module import errors exist in console logs.
• Runtime remains unhealthy even after command fixed to app:app.
• Failure is explained by unrelated root cause (for example platform outage, auth failures, or networking ACL blocks) with supporting evidence.

2.5 Alternate hypotheses considered¶

2.6 Expected observations if hypothesis is true¶

2.7 Risk to production if not detected¶

• False confidence from green deployment pipeline.
• Extended outage while teams chase irrelevant causes.
• Repeated restarts increasing noise in platform logs.
• Customer-facing availability incidents despite “successful release” status.

3) Runbook¶

The current procedure is fault-first: Bicep provisions the web app with wrong_module:app, the trigger deploys valid application code into that fault, records one failed probe, repairs the startup command to app:app, and records one post-repair probe. The verifier then searches a rolling two-hour console-log window. The archived artifacts below came from an earlier procedure and are not evidence that these checked-in scripts have run successfully.

Define the expected fault, repair, and rolling query window before execution. Preserve terminal output and any collected evidence outside the disposable resource group without overwriting earlier captures. A successful repair probe does not assert cleanup or independent reproduction, and a zero verifier count is inconclusive rather than a pass.

Use only a dedicated disposable resource group with permission to create App Service and Log Analytics, configure diagnostics, query logs, and delete the group. B1 compute and Log Analytics ingestion incur charges until cleaned up. Run one experiment at a time with no concurrent configuration changes. Read Clean Up before deployment; it also applies to failed or interrupted runs. Raw captures contain real resource IDs and configuration: keep them private outside Git, review and sanitize before publication, and retain public transformation records separately.

3.0 Structured runner¶

After deploying the checked-in Bicep template, the new runner can establish its own healthy baseline, inject wrong_module:app, collect bounded fault-window evidence, and restore app:app. This runner is additive: the checked-in trigger.sh and verify.sh still implement the fault-first procedure described throughout sections 3.1–3.11.

export APP_NAME="$(az webapp list --resource-group "$RG" --query '[0].name' --output tsv | tr -d '\r\n')"
python3 labs/deployment-succeeded-startup-failed/run.py execute "$RG" "$APP_NAME" --output /tmp/startup-lab-private
export RUN_DIR="/tmp/startup-lab-private/<printed-run-directory>"
python3 labs/deployment-succeeded-startup-failed/run.py evaluate "$RUN_DIR"

The runner contract is labs/deployment-succeeded-startup-failed/contract.json. Exit codes are 0 SUPPORTED, 1 CONTRADICTED, 2 INCONCLUSIVE, 3 NOT_TESTED, and 4 evidence-invalid. Evidence validity, hypothesis, cleanup, and independent reproduction remain separate. Do not edit a deficient run; preserve it and make a new run after correcting the procedure.

Run 20261016T075147Z-6d019de6e2f2 exercised this path in Korea Central on B1 Linux with Python 3.11. Its capture window was 2026-10-16T07:51:47.555754+00:00 through 2026-10-16T08:01:33.709704+00:00. The baseline returned 200, the fault reached 503, Log Analytics returned two exact fault-window ModuleNotFoundError: No module named 'wrong_module' rows, and recovery returned 200. The retained evidence includes every timestamped probe and the three phase windows. The cleanup record reached VERIFIED at 2026-10-16T08:02:54.778086+00:00, after which the separate group-existence check returned false. The replayable sanitized package is under evidence/runs/20261016T075147Z-6d019de6e2f2/.

3.1 Prerequisites¶

3.2 Environment variables¶

export RG="rg-lab-startup"
export LOCATION="koreacentral"

3.3 Deploy infrastructure¶

az group create \
  --name "$RG" \
  --location "$LOCATION"

az deployment group create \
  --resource-group "$RG" \
  --template-file "labs/deployment-succeeded-startup-failed/main.bicep" \
  --parameters "baseName=labstart"

Capture app name:

APP_NAME=$(az webapp list \
  --resource-group "$RG" \
  --query "[0].name" \
  --output tsv)

echo "$APP_NAME"

3.4 Trigger the incident¶

bash "labs/deployment-succeeded-startup-failed/trigger.sh" "$RG" "$APP_NAME"

The checked-in trigger packages and deploys the application while the Bicep-defined wrong_module:app startup command is active. After a 15-second wait it probes /health once, reports the non-200 result expected during the fault, changes the startup command to gunicorn --bind=0.0.0.0:8000 app:app, waits 20 seconds, and probes /health once more. A final 200 confirms recovery; any other status means the app may still be restarting and must be checked again manually.

bash "labs/deployment-succeeded-startup-failed/verify.sh" "$RG"

The checked-in verifier selects the first Log Analytics workspace in the resource group and queries AppServiceConsoleLogs from the previous two hours for ModuleNotFoundError, wrong_module, or No module named. It prints the matching count and up to 20 recent rows. A positive count is the expected startup-failure evidence; zero means evidence has not arrived or is outside the query window, not that the fault was absent.

Sections 3.5–3.11 provide optional manual diagnostics and recovery commands. Their illustrative rolling-window queries are separate from verify.sh, whose checked-in query searches the previous two hours. If interruption prevents automatic recovery, use 3.11 and then clean up; do not mark that interrupted run successful.

3.5 Verify startup command before and after¶

Before repair (Bicep-defined value for the checked-in procedure):

gunicorn --bind=0.0.0.0:8000 --timeout=120 wrong_module:app

After repair (value set by the checked-in trigger):

gunicorn --bind=0.0.0.0:8000 app:app

CLI verification command:

az webapp config show \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --query "appCommandLine" \
  --output tsv

3.6 Probe health endpoint during recovery¶

APP_HOST_NAME=$(az webapp show \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --query "defaultHostName" \
  --output tsv)

APP_URL="https://$APP_HOST_NAME"

for attempt in 1 2 3 4; do
  status_code=$(curl \
    --silent \
    --show-error \
    --output /dev/null \
    --write-out "%{http_code}" \
    --max-time 20 \
    "$APP_URL/health" || true)
  echo "$attempt,$status_code"
  sleep 8
done

Observed sequence from artifact recovery-probes-20260404T054537Z.csv:

1,000000,2026-04-04T05:46:52Z
2,000000,2026-04-04T05:47:05Z
3,000000,2026-04-04T05:47:24Z
4,200,2026-04-04T05:47:30Z

Interpretation:

• Attempts 1-3: startup not yet healthy.
• Attempt 4: service recovered after command correction and restart completion.

3.7 Collect diagnostics endpoints after recovery¶

curl --silent --show-error "$APP_URL/diag/stats"
curl --silent --show-error "$APP_URL/diag/env"
curl --silent --show-error "$APP_URL/health"

Observed post-fix health payload:

{"status":"healthy"}

Observed post-fix stats payload (artifact):

{"endpoint_counters":{"<unknown>":1,"diag_env":1,"diag_stats":1,"health":1},"pid":1896,"process_start_time":"2026-04-04T05:47:26.277049+00:00","request_count":5,"uptime_seconds":383.853}

3.8 Query KQL evidence (HTTP)¶

AppServiceHTTPLogs
| where TimeGenerated > ago(4h)
| where CsHost contains "azurewebsites"
| where CsUriStem in ("/", "/health", "/diag/stats", "/diag/env", "/api/zipdeploy")
| project TimeGenerated, CsUriStem, ScStatus, TimeTaken, CsHost
| order by TimeGenerated desc

Observed statuses in artifact kql-http-20260404T060610Z.json:

• 200: 12 rows
• 499: 3 rows
• 503: 5 rows
• 202: 2 rows

3.9 Query KQL evidence (console)¶

Log stream catches Python tracebacks the instant Gunicorn emits them, well ahead of the ingestion lag that affects the query below. Keep the Runtime radio selected and pin Instances to a single worker so application-side boot failures aren't interleaved across instances, then use the query below to confirm the same lines landed in AppServiceConsoleLogs for reporting.

AppServiceConsoleLogs
| where TimeGenerated > ago(4h)
| where ResultDescription has_any ("ModuleNotFoundError", "wrong_module", "Worker failed to boot")
| project TimeGenerated, ResultDescription
| order by TimeGenerated desc

Observed log lines:

2026-04-04T05:55:06.4678473Z [2026-04-04 05:55:06 +0000] [1895] [ERROR] Reason: Worker failed to boot.
2026-04-04T05:55:04.0857549Z No module named 'wrong_module'
2026-04-04T05:55:04.0838827Z ModuleNotFoundError: No module named 'wrong_module'
2026-04-04T05:54:49.4185745Z Site's appCommandLine: gunicorn --bind=0.0.0.0:8000 --timeout=120 wrong_module:app

3.10 Query KQL evidence (platform)¶

AppServicePlatformLogs
| where TimeGenerated > ago(4h)
| where Message has_any ("Container did not start within expected time limit of 230s", "Starting container", "stopped")
| project TimeGenerated, Level, Message
| order by TimeGenerated desc

Observed platform messages include:

State: Stopping, Action: StoppingSiteContainers, LastErrorDetails: Container did not start within expected time limit of 230s.
State: Stopping, Action: CancellingStartup, LastErrorDetails: Container did not start within expected time limit of 230s.
Starting container: 31a19ef7e66d_app-labstartup-... .
Site: app-labstartup-... stopped.

3.11 Remediation command (direct)¶

If reproducing manually and app remains unhealthy, force the known-good startup command:

az webapp config set \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --startup-file "gunicorn --bind=0.0.0.0:8000 app:app"

Then verify:

curl --silent --show-error --fail "$APP_URL/health"

4) Experiment Log¶

This section is a factual record from:

labs/deployment-succeeded-startup-failed/artifacts-sanitized/

4.1 Artifact inventory¶

4.2 Baseline observations¶

4.2.1 App is running as a resource, but not healthy as workload¶

File: baseline/app-state.json

{
  "kind": "app,linux",
  "state": "Running"
}

Interpretation:

• Control-plane resource state is Running.
• This does not guarantee app process health.

4.2.2 Startup command is misconfigured at baseline¶

File: baseline/startup-command.txt

gunicorn --bind=0.0.0.0:8000 --timeout=120 wrong_module:app

File: baseline/app-config.json (appCommandLine field)

gunicorn --bind=0.0.0.0:8000 --timeout=120 wrong_module:app

4.2.3 User-facing symptom at baseline¶

File: baseline/root-response.txt

Observed content includes:

:( Application Error
If you are the application administrator, you can access the diagnostic resources

This aligns with upstream 503 and startup failure patterns.

4.2.4 Control-plane view: Deployment Center reports success¶

While the runtime is failing, the App Service control plane still reports the deployment as successful. This is the central illusion of "deployment succeeded but startup failed."

Purpose: Show that Deployment Center — the primary UI operators consult after a az webapp deploy — reports Succeeded (Active) for the exact deployment that produces a broken runtime. Look for: The green "Succeeded" badge on the row with OneDeploy message and the timestamp of the deployment that immediately preceded the failure window. Expected result: Deployment ID visible, Status column shows Succeeded, no error indication anywhere on this blade. Next step: Cross-reference this timestamp against AppServicePlatformLogs runtime failure events (§4.4.4) — the two events are 90-120 seconds apart, which is the diagnostic window operators must learn to inspect.

4.3 Trigger and recovery observations¶

4.3.1 Command change is explicit and minimal¶

Before:

gunicorn --bind=0.0.0.0:8000 --timeout=120 wrong_module:app

After:

gunicorn --bind=0.0.0.0:8000 --timeout=120 app:app

Only the module reference changed.

4.3.2 Recovery probe timeline¶

File: trigger/recovery-probes-20260404T054537Z.csv

Interpretation:

• There is a short recovery window while restart stabilizes.
• Successful 200 appears once corrected startup process is serving.

4.3.3 Early post-fix process evidence¶

File: trigger/diag-stats-postfix-20260404T054537Z.json

{"endpoint_counters":{"<unknown>":1,"health":1},"pid":1896,"process_start_time":"2026-04-04T05:47:26.277049+00:00","request_count":3,"uptime_seconds":4.34}

Key detail:

• process_start_time is consistent with restart during fix.
• Process is now alive and counting requests.

4.4 Telemetry evidence from KQL artifacts¶

4.4.1 HTTP log evidence¶

Artifact: trigger/kql-http-20260404T060610Z.json

Status summary:

Selected rows:

4.4.2 Console log evidence (intended app-level cause, from prior artifact set)¶

Artifact: trigger/kql-console-20260404T060610Z.json

Critical rows:

This is direct proof of startup command misconfiguration when console output is captured. When it is not (as in the current run), the platform log evidence in §4.4.4 becomes the authoritative source for the failure lifecycle.

4.4.3 Platform log evidence¶

Artifact: trigger/kql-platform-20260404T060610Z.json

Representative messages:

This connects app-level exception to platform startup timeout behavior.

4.4.4 Live KQL evidence: startup lifecycle in AppServicePlatformLogs¶

The following KQL, executed against AppServicePlatformLogs in the current lab workspace, categorizes every startup lifecycle event into a human-readable Phase column. It is the authoritative evidence for this run because AppServiceConsoleLogs did not capture container STDOUT before the process exited (see §4.4.2 note).

AppServicePlatformLogs
| where TimeGenerated > ago(90m)
| extend Msg = tostring(coalesce(
    column_ifexists("Message", ""),
    column_ifexists("ResultDescription", ""),
    column_ifexists("Details", "")
))
| where Msg has_any (
    "PullingImage",
    "MountingVolumes",
    "exit code: 3",
    "ContainerTimeout",
    "terminated during site startup",
    "Failed to start site",
    "blocked due to multiple, consecutive cold start failures"
)
| extend Phase = case(
    Msg has "PullingImage", "Pulling image",
    Msg has "MountingVolumes", "Mounting volumes",
    Msg has "exit code: 3", "Process exited (code 3)",
    Msg has "ContainerTimeout", "Startup timeout (230s)",
    Msg has "terminated during site startup", "Startup cancelled by platform",
    Msg has "Failed to start site", "Site reverted/stopped",
    Msg has "blocked due to multiple, consecutive cold start failures", "Cold-start safety block",
    "Other"
)
| project TimeGenerated, Phase, Msg
| order by TimeGenerated asc

Purpose: Show the deterministic lifecycle that platform diagnostics record when a container fails to become healthy: image pull → volume mount → process exit (code 3) → platform cancels startup → startup timeout after 230s → site reverted/stopped. This lifecycle repeats on every restart attempt. Look for:

• The Phase column matches the six stages described above.
• The TimeGenerated column shows all six stages happen within one 40-60 second window per restart cycle.
• The Msg column contains the raw platform strings (Action: PullingImage, Container has finished running with exit code: 3, LastError: ContainerTimeout) that support each Phase label.

Expected result: 200+ rows (271 in this capture) over a 90-minute window, showing the container retry loop that runs until the platform's cold-start safety block engages. Next step: Use the timeline chart in §4.4.5 to see how these runtime events align against the control-plane deployment event.

4.4.5 Live KQL evidence: control-plane vs data-plane on the same timeline¶

The following KQL renders a column chart that overlays the successful deployment event (from Deployment Center metadata) against the platform-terminated startup events (from AppServicePlatformLogs) on a single minute-binned timeline. The single orange bar at the deploy timestamp against a sustained wall of red/blue/teal runtime-failure bars is the visual proof that "deployment succeeded" and "app running healthily" are two disjoint claims.

let deploy = datatable(TimeGenerated:datetime, Event:string) [
    datetime(2026-07-02T15:23:30Z), "Deploy succeeded (control-plane)"
];
let runtime =
    AppServicePlatformLogs
    | where TimeGenerated > ago(120m)
    | extend Msg = tostring(coalesce(
        column_ifexists("Message", ""),
        column_ifexists("ResultDescription", ""),
        column_ifexists("Details", "")
    ))
    | where Msg has_any ("exit code: 3", "ContainerTimeout", "terminated during site startup")
    | extend Event = case(
        Msg has "exit code: 3", "Container exited (code 3)",
        Msg has "ContainerTimeout", "Startup timeout",
        "Platform cancelled startup"
    )
    | project TimeGenerated, Event;
union deploy, runtime
| summarize Count = count() by bin(TimeGenerated, 1m), Event
| render columnchart

The deploy datatable row is synthesized from the Deployment Center capture in §4.2.4 because AppServiceHTTPLogs only captures traffic to the front-door *.azurewebsites.net hostname, not to the SCM/Kudu (*.scm.azurewebsites.net) endpoint that receives ZIP deploys. This is by design in App Service diagnostic settings.

Purpose: Visually collapse the entire lab hypothesis into a single chart. The orange bar and the teal/blue bars are literally on the same time axis, in the same query result, in the same LAW workspace — and yet they represent completely disjoint outcomes. Look for:

• The single, isolated orange bar at ~3:24 PM: this is the control-plane's report of success.
• The ~2 minute quiet gap immediately after: the platform is pulling the image and starting the container.
• The first sustained wave of colored bars from 3:25 PM to ~3:35 PM: the container repeatedly fails to boot and platform terminates each attempt.
• The quiescence from 3:36 PM to 3:48 PM: the platform's cold-start safety block engages and stops retrying.
• The second wave at 3:50-3:52 PM: external HTTP probes to the front-door woke the container up, and it failed again identically.

Expected result: One tiny orange bar (count = 1) against 35+ runtime failure bars. If your capture shows any other pattern (for example, no runtime bars at all, or the orange bar appearing after the runtime bars), the hypothesis is falsified — treat the container as healthy or investigate a different failure mode. Next step: In your own investigations, save this query into a Log Analytics workbook so any incident responder can render the same control-plane vs data-plane split for any deployment ID in under 60 seconds.

4.5 Postfix observations¶

4.5.1 Final health¶

File: postfix/health-20260404T055349Z.json

{"status":"healthy"}

4.5.2 Final process stats¶

File: postfix/diag-stats-20260404T055349Z.json

{"endpoint_counters":{"<unknown>":1,"diag_env":1,"diag_stats":1,"health":1},"pid":1896,"process_start_time":"2026-04-04T05:47:26.277049+00:00","request_count":5,"uptime_seconds":383.853}

4.5.3 Final environment snapshot¶

File: postfix/diag-env-20260404T055349Z.json

{"APP_STARTUP_COMMAND":"<unset>","PORT":"8000","SCM_DO_BUILD_DURING_DEPLOYMENT":"true","STARTUP_COMMAND":"<unset>","WEBSITES_PORT":"<unset>","WEBSITE_HOSTNAME":"app-labstartup-pt7yrjbt2zmv2.<azurewebsites-domain-redacted>","WEBSITE_INSTANCE_ID":"f7914080a1e04baaab7e548dc057c38527ceb352eed71d9074f19e190f413990","WEBSITE_SLOT_NAME":"<unset>"}

4.6 Timeline reconstruction¶

4.7 Hypothesis verdict¶

Verdict: INCONCLUSIVE for a single ordered causal run.

[Observed] The stored recovery probes reach 200 around 05:47 UTC, while the retained wrong-module console rows are timestamped 05:54–05:55 UTC. The set also lacks a pre-fault HTTP 200 capture and verified cleanup record. These records preserve useful examples of startup failure and health responses, but do not establish that the shown fault preceded the shown recovery in one isolated run.

[Not Proven] The historical collection does not prove that only the entrypoint changed or exclude concurrent restarts, networking failures, or mixed collection windows. Reevaluate the original timestamps without relabeling them as a new Azure run. The checked-in fault-first procedure requires a new deployment/fault/repair sequence, evidence from the verifier's rolling two-hour window, cleanup verification, and independent review before promotion.

4.8 Operational guidance distilled from this experiment¶

1. Always verify appCommandLine during startup incidents.
2. Separate deployment telemetry from runtime telemetry in dashboards.
3. Query console logs for import exceptions before scaling or redeploying.
4. Capture platform timeout messages (230s) for incident chronology.
5. Promote a pre-release startup command lint rule for Python apps.

4.9 Reusable KQL snippets¶

Console startup failures¶

AppServiceConsoleLogs
| where TimeGenerated > ago(24h)
| where ResultDescription has_any (
    "ModuleNotFoundError",
    "No module named",
    "Worker failed to boot",
    "appCommandLine"
)
| project TimeGenerated, ResultDescription
| order by TimeGenerated desc

HTTP availability around deployment¶

AppServiceHTTPLogs
| where TimeGenerated > ago(24h)
| where CsUriStem in ("/", "/health", "/api/zipdeploy", "/api/deployments/latest")
| summarize count() by CsUriStem, ScStatus
| order by CsUriStem asc, ScStatus asc

Platform startup timeout tracing¶

AppServicePlatformLogs
| where TimeGenerated > ago(24h)
| where Message has_any (
    "Container did not start within expected time limit of 230s",
    "Starting container",
    "CancellingStartup",
    "stopped"
)
| project TimeGenerated, Level, Message
| order by TimeGenerated desc

Expected Evidence¶

This section defines what you SHOULD observe at each phase of the lab. Use it to validate your investigation is on track.

After Provisioning, Before the Trigger¶

During Incident¶

After Recovery¶

Evidence Timeline¶

graph TD
    A[Provision App with wrong_module] --> B[Deploy Package into Fault]
    B --> C[Probe Failure and Repair Command]
    C --> D[Probe Recovery]
    D --> E[Query Prior Two Hours and Evaluate Limits]

Evidence interpretation and falsification¶

For the checked-in fault-first procedure, deployment success while wrong_module:app is configured, a failed first probe, same-resource import-failure evidence within the verifier's rolling two-hour window, and recovery after the trigger sets app:app support the scoped hypothesis. A healthy app under the wrong-module configuration contradicts the predicted fault. A missing log match, a transport-only error, or a post-repair non-200 leaves the cause unresolved and requires additional collection rather than a pass. Platform restart, dependency problems, and concurrent changes remain competing explanations; a 503 or screenshot alone does not establish the module as the cause.

Clean Up¶

Preserve terminal output and evidence outside the resource group, then clean up the dedicated group directly:

az group delete --name "$RG" --yes
az group exists --name "$RG"

Save both command results with UTC timestamps. An accepted deletion request alone is not cleanup verification. No deletion or residual-resource check has been executed for the revised procedure.

Related Playbook¶

• Deployment Succeeded but Startup Failed

See Also¶

• Playbook: Deployment Succeeded but Startup Failed
• Playbook: Container Didn't Respond to HTTP Pings
• Playbook: Warmup vs Health Check
• Lab: Failed to Forward Request

Sources¶

• Configure a Linux Python app for Azure App Service
• Configure a custom startup file for Linux apps in Azure App Service
• Configure a custom container for Azure App Service
• Deploy files to Azure App Service
• Enable diagnostics logging for Azure App Service
• Monitor App Service with Azure Monitor