Lab: No Space Left on Device (Persistent /home Exhaustion vs Ephemeral /tmp Pressure)¶

This lab reproduces App Service Linux filesystem pressure and makes one critical diagnostic distinction:

• /home is persistent quota-backed storage.
• /tmp is ephemeral worker-local storage.

In the captured experiment, /home reaches 100% utilization, write operations start failing with ENOSPC, but health probes still return HTTP 200.

That combination is the core troubleshooting lesson.

This guide helps you explain App Service Linux storage surfaces and quota behavior, differentiate persistent-storage exhaustion from runtime health failures, prove ENOSPC with application and telemetry evidence, validate that /health can remain 200 while business writes fail, and build a disk-pressure runbook.

Lab Metadata¶

1) Background¶

1.1 Why this failure mode is often misdiagnosed¶

Teams often map “disk full” to “site down”.

But disk failures are frequently partial failures:

• read-only and lightweight endpoints still succeed,
• health probes remain green,
• write-heavy paths fail with No space left on device.

This can create dangerous false confidence if monitoring relies only on /health.

1.2 App Service Linux storage model (practical view)¶

In a typical Linux app on App Service, you will observe:

• Application root filesystem overlay mounted at /.
• Persistent storage mounted under /home.
• Temporary scratch space under /tmp.

/home/site/wwwroot is commonly used for app content and app-writable folders, depending on deployment style and runtime behavior.

1.3 Persistent vs ephemeral semantics¶

1.4 Quota context for this lab¶

The user scenario asks to reason with Free vs Paid tier intuition:

• Free tier often has much smaller quota (commonly around 1 GB).
• Paid tiers provide larger persistent quota.

In this experiment’s artifacts, /home total bytes are exactly:

10,737,418,240 bytes (10 GB decimal-class quota representation at binary byte level).

Portal view: App Service Plan Overview blade (pricing tier and shared-storage context)¶

The App Service Plan Overview blade confirms three platform-side facts that anchor this lab's assumptions - the Pricing plan value (here P0v3) confirms the SKU tier that controls compute resources (CPU, memory, scale-up options), the Operating System: Linux field confirms which storage model from section 1.2 applies, and the App(s) / Slots: 1 / 1 field confirms that this plan currently hosts 1 app and 1 deployment slot. The App(s) / Slots field matters because /home storage is shared across every app and deployment slot bound to the same plan: a second app on the same plan would consume from the same pool described in section 1.3's persistent-vs-ephemeral semantics, which is the most common real-world source of "I didn't change anything and now I'm out of space". The 10,737,418,240 byte quota cited above comes from this lab's runtime artifacts (section 1.4) and from the official App Service limits documentation - this Overview blade does not surface that quota value as a field, so do not infer the quota directly from anything visible on this capture. Use this blade as your first stop when investigating "no space left on device" errors in production: confirm the SKU and OS first, click the Pricing plan link to reach the Scale up blade if compute starvation is in scope, or expand the left navigation to locate the plan's Apps blade and enumerate other apps that may be competing for the shared /home quota.

1.5 Why writes fail while health still succeeds¶

The lab app exposes:

• /health (simple JSON response, no large disk write dependency),
• /fill-home (writes large files into /home/site/wwwroot/temp),
• /fill-tmp (writes large files into /tmp/no-space-lab).

If /home is exhausted:

• /fill-home fails with OSError: [Errno 28] No space left on device.
• /health can still return 200.

This is exactly what the artifact set shows.

1.6 Storage architecture diagram¶

flowchart TD
    A[Incoming HTTP Request] --> B{Endpoint type}
    B -->|/health| C[In-memory response path]
    B -->|/fill-home| D[Write payload to /home/site/wwwroot/temp]
    B -->|/fill-tmp| E[Write payload to /tmp/no-space-lab]

    D --> F{Persistent /home has free space?}
    F -->|Yes| G[Write succeeds 200]
    F -->|No| H[ENOSPC -> app returns 507]

    E --> I{Ephemeral /tmp has free space?}
    I -->|Yes| J[Write succeeds 200]
    I -->|No| K[ENOSPC -> app returns 507]

    C --> L[Health remains 200]

    style L fill:#c8e6c9
    style H fill:#ffcdd2
    style K fill:#ffcdd2

1.7 Lifecycle of disk pressure in this lab¶

sequenceDiagram
    participant Operator
    participant App as Flask App
    participant Home as /home storage
    participant Tmp as /tmp storage
    participant Logs as Log Analytics

    Operator->>App: GET /disk-status (baseline)
    App-->>Operator: /home ~0.53% used
    Operator->>App: Repeat GET /fill-home?size_mb=100
    App->>Home: create diskfill files
    Home-->>App: writes succeed
    Operator->>App: Repeat GET /fill-tmp?size_mb=100
    App->>Tmp: create diskfill files
    Tmp-->>App: writes succeed
    Operator->>App: Aggressive GET /fill-home?size_mb=500
    App->>Home: write until quota edge
    Home-->>App: ENOSPC
    App-->>Operator: HTTP 507
    App->>Logs: emit OSError Errno 28
    Operator->>App: GET /health
    App-->>Operator: HTTP 200

1.8 Isolation misconception to avoid¶

A common misconception:

“If /home is full, the whole app must crash immediately.”

Reality:

• Failures are path-dependent.
• Endpoints that allocate large files fail first.
• Lightweight endpoints may continue serving.

1.9 Diagram: symptom matrix by endpoint behavior¶

graph TD
    A[/health/] --> A1[No large write]
    A1 --> A2[200 OK]

    B[/fill-home/] --> B1[Write to /home]
    B1 --> B2{/home free bytes > 0?}
    B2 -->|Yes| B3[200 written]
    B2 -->|No| B4[507 ENOSPC]

    C[/fill-tmp/] --> C1[Write to /tmp]
    C1 --> C2{/tmp free bytes > 0?}
    C2 -->|Yes| C3[200 written]
    C2 -->|No| C4[507 ENOSPC]

    style A2 fill:#c8e6c9
    style B4 fill:#ffcdd2
    style C4 fill:#ffcdd2

1.10 Filesystem metrics used in this guide¶

1.11 Diagnostic endpoints in the lab app¶

1.12 Background takeaway¶

Disk-pressure incidents require endpoint-level behavioral validation, not only app-level up/down checks.

2) Hypothesis¶

2.1 Primary hypothesis (this lab)¶

When application code writes large files to /home until the persistent quota is exhausted, subsequent write operations fail with ENOSPC, while health probes can continue returning 200 because they do not require disk writes.

2.2 Causal chain¶

1. App exposes write endpoints and health endpoint
      ↓
2. Repeated /fill-home writes consume /home persistent quota
      ↓
3. /home free bytes approach zero
      ↓
4. Kernel returns OSError Errno 28 (No space left on device)
      ↓
5. /fill-home endpoint returns HTTP 507 with ENOSPC payload
      ↓
6. /health endpoint still returns HTTP 200
      ↓
7. Incident appears as partial degradation, not full outage

2.3 Proof criteria¶

All criteria below must be satisfied:

1. Baseline /home has high free space and low used percent.
2. Repeated /fill-home calls increase /home usage significantly.
3. At high utilization, /fill-home returns status error with Errno 28.
4. HTTP telemetry records at least one 507 for /fill-home.
5. Console logs include No space left on device.
6. /health remains 200 during or after ENOSPC event.

2.4 Disproof criteria¶

Any one condition disproves the hypothesis:

• /fill-home never fails despite verified /home exhaustion.
• ENOSPC does not appear in either app payloads or console logs.
• /health fails for unrelated reasons before storage quota is reached.
• Failures occur only in /tmp while /home remains largely free.

2.5 Controlled and observed variables¶

2.6 Expected observation matrix¶

3) Runbook¶

This runbook maps directly to artifacts in:

labs/no-space-left-on-device/artifacts-sanitized/

3.1 Prerequisites¶

3.2 Environment variables¶

Convention reminder:

• Documentation uses $RG, $APP_NAME naming style.

Operational shell form:

export RG="rg-lab-nospace"
export LOCATION="koreacentral"

3.3 Deploy infrastructure¶

az group create \
  --name "$RG" \
  --location "$LOCATION"

az deployment group create \
  --resource-group "$RG" \
  --template-file "labs/no-space-left-on-device/main.bicep" \
  --parameters "baseName=labnospace"

Resolve app name from deployment outputs or app list:

APP_NAME=$(az webapp list \
  --resource-group "$RG" \
  --query "[0].name" \
  --output tsv)

3.4 Deploy lab app code¶

az webapp deploy \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --src-path "labs/no-space-left-on-device/app" \
  --type startup

If your CLI version requires explicit ZIP packaging for directory deploy, package and deploy with ZIP:

python3 -m zipfile -c "/tmp/no-space-lab.zip" "labs/no-space-left-on-device/app"

az webapp deploy \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --src-path "/tmp/no-space-lab.zip" \
  --type zip

3.5 Resolve app URL¶

APP_HOST_NAME=$(az webapp show \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --query "defaultHostName" \
  --output tsv)

APP_URL="https://$APP_HOST_NAME"

3.6 Baseline checks¶

curl --silent --show-error "$APP_URL/health"
curl --silent --show-error "$APP_URL/disk-status"
curl --silent --show-error "$APP_URL/diag/disk"
curl --silent --show-error "$APP_URL/diag/stats"

Artifact-backed baseline /disk-status value:

{"home":{"free_bytes":10680426496,"path":"/home/site/wwwroot/temp","total_bytes":10737418240,"used_bytes":56991744,"used_percent":0.53},"status":"ok","tmp":{"free_bytes":15628582912,"path":"/tmp/no-space-lab","total_bytes":36670308352,"used_bytes":21024948224,"used_percent":57.34}}

3.7 Trigger moderate fill (scripted)¶

bash "labs/no-space-left-on-device/trigger.sh" "$APP_URL"

The script performs:

1. Baseline /disk-status.
2. 8 x /fill-home?size_mb=100.
3. 5 x /fill-tmp?size_mb=100.
4. Post-fill /disk-status.
5. /health probe.

3.8 Trigger aggressive /home fill (manual)¶

To reproduce the 500 MB chunk behavior seen in artifacts:

for chunk_index in $(seq 1 11); do
  curl \
    --silent \
    --show-error \
    "$APP_URL/fill-home?size_mb=500"
  printf "\n"
done

Expected sequence from artifacts:

• Chunks 1-10: status=written
• Chunk 11: status=error, Errno 28

3.9 Verify final disk state¶

curl --silent --show-error "$APP_URL/disk-status"
curl --silent --show-error "$APP_URL/diag/disk"

Artifact-backed final /disk-status:

{"home":{"free_bytes":0,"path":"/home/site/wwwroot/temp","total_bytes":10737418240,"used_bytes":10737418240,"used_percent":100.0},"status":"ok","tmp":{"free_bytes":15103889408,"path":"/tmp/no-space-lab","total_bytes":36670308352,"used_bytes":21549641728,"used_percent":58.77}}

3.10 Confirm health behavior under disk exhaustion¶

curl --silent --show-error --output /dev/null --write-out "%{http_code}\n" "$APP_URL/health"

Expected from artifacts:

200

3.11 KQL queries for evidence capture¶

Log stream is the live counterpart to the queries below: it carries the same Gunicorn stdout/stderr that lands in AppServiceConsoleLogs minutes later, with no ingestion delay. Keep the Runtime radio selected and pin Instances to a single worker so write-side errors aren't interleaved, then watch for No space left on device lines the moment the trigger runs. Use the Stop and Clear toolbar actions to capture a clean tail before moving to the queries for 507 correlation.

HTTP outcomes¶

AppServiceHTTPLogs
| where TimeGenerated > ago(6h)
| where CsUriStem in ("/fill-home", "/fill-tmp", "/health", "/disk-status", "/diag/disk")
| project TimeGenerated, CsUriStem, ScStatus, TimeTaken, CsHost
| order by TimeGenerated desc

Console ENOSPC signatures¶

AppServiceConsoleLogs
| where TimeGenerated > ago(6h)
| where ResultDescription has_any ("No space left on device", "ENOSPC", "Errno 28")
| project TimeGenerated, ResultDescription
| order by TimeGenerated desc

Platform activity context¶

AppServicePlatformLogs
| where TimeGenerated > ago(6h)
| where Message has_any ("Starting container", "stopped", "Container did not start")
| project TimeGenerated, Level, Message
| order by TimeGenerated desc

3.12 Recovery actions¶

For rapid remediation after reproduction:

curl --silent --show-error "$APP_URL/cleanup"
curl --silent --show-error "$APP_URL/disk-status"

Operational options in real incidents:

1. Delete stale files under /home.
2. Move temp write patterns from /home to /tmp where appropriate.
3. Stream data to external durable storage (Blob Storage) instead of local disk.
4. Add app-level write guards and quota telemetry alerts.

4) Experiment Log¶

All findings in this section come from:

labs/no-space-left-on-device/artifacts-sanitized/

4.1 Artifact inventory summary¶

This is one of the richest labs in the repository, with many phase snapshots and per-request payload captures.

4.2 Baseline state¶

4.2.1 Baseline runtime counters¶

File: baseline/diag-stats.json

{"endpoint_counters":{"<unknown>":1,"index":2},"pid":1898,"process_start_time":"2026-04-04T05:30:21.367012+00:00","request_count":4,"total_bytes_written":0,"uptime_seconds":169.694}

Initial interpretation:

• Process healthy and serving.
• No write payload generated yet.

4.2.2 Baseline disk usage snapshot¶

File: baseline/disk-status.json

Important baseline fact:

• /home starts almost empty.

4.2.3 Baseline extended diagnostics¶

File: baseline/diag-disk.json

Key values:

4.3 Moderate fill phase (8 x 100 MB home + 5 x 100 MB tmp)¶

4.3.1 Before and after endpoint snapshots¶

Before (disk-status-before-20260404T053504Z.json):

After (disk-status-after-20260404T053504Z.json):

Observation:

• Moderate fill increases /home usage, but still far from quota limit.

4.3.2 Midfill diagnostic snapshot¶

File: diag-disk-midfill-20260404T053504Z.json

4.4 Aggressive fill phase details¶

The aggressive phase combines multiple series:

1. 20 writes of 200 MB (fill-home-big-*) = 4,000 MB
2. 11 writes of 500 MB (fill-home-500mb-*) where chunk 11 fails

Additionally, earlier 100 MB sequences are present.

4.4.1 Aggregate write statistics from artifacts¶

Computed from sanitized trigger files:

Critical error file:

• fill-home-500mb-11-20260404T055739Z.json

Payload:

{"error":"[Errno 28] No space left on device","status":"error","target":"home"}

4.4.2 500 MB chunk series timeline¶

This is direct proof of quota edge behavior.

4.5 Disk state progression¶

4.5.1 Key snapshots table¶

Key insight:

• /tmp remained with substantial free bytes, while /home exhausted to zero.

4.5.2 Kudu SSH df -h output (portal verification)¶

To cross-verify the final /home used_percent = 100.00 row in the §4.5.1 snapshots table against live filesystem state, the Kudu SCM Debug Console was opened against the same Web App from a fresh live reproduction and df -h was executed with the Filesystem column intentionally omitted for PII safety:

Purpose: Cross-verify visually that the platform-reported /home mount is at 100% Use% at the same point in time when the app's own /disk-status endpoint reports used_percent = 100.00 and free_bytes = 0 (row 6 of the §4.5.1 snapshots table). This complements the app-layer JSON evidence by showing the OS-layer df view directly, so a reviewer can rule out the possibility that the app's shutil.disk_usage() reading disagrees with what the kernel reports.

Look for:

• Top navigation bar shows the Kudu SCM header ("Azure App Service") with the Bash tab active — this confirms the capture is from the Kudu Debug Console, not the runtime container's own shell.
• The row 10G 10G 0 100% /home — Size, Used, Avail, Use%, Mounted on. Used equals Size, Available is 0, Use% is 100.
• Every other mount is well below 100% (max is 59% on the overlay root at 35G/20G) — proving the exhaustion is scoped to the /home SMB share and is not a broader disk failure.
• The Filesystem column is intentionally missing (only 5 columns instead of the default 6) because --output=size,used,avail,pcent,target was used to prevent the SMB mount path from being rendered in the capture.
• Bottom green status bar reads "SSH CONNECTION ESTABLISHED", indicating a live SSH channel into the container.

Expected result: The /home Use% shown in this Kudu capture (100%) equals the used_percent value in the final row of the §4.5.1 snapshots table (100.00) — the OS-layer df output and the app-layer /disk-status endpoint agree that the mount is fully exhausted. If instead the Kudu output showed /home at some intermediate percent (say 87%) while the app kept reporting 100%, that disagreement would suggest a stale disk-usage cache inside the Python process or a bug in shutil.disk_usage() on this platform, and the ENOSPC classification would need to be reconsidered.

Next step: If a future reproduction shows the same /home 100% state in Kudu but the app's /health endpoint still returns 200 (as documented in §4.10), this is the expected split-behavior signature — the app process is alive and answers health probes, but any code path that writes to /home will raise OSError: [Errno 28]. Do not restart the app to "clear" the state; first delete or move files out of /home (for example via the app's cleanup path or manual file removal) so the same evidence pattern can be captured again on the next cycle.

4.6 HTTP telemetry evidence¶

Artifact: trigger/kql-http-20260404T060610Z.json

Summary status counts:

Representative rows:

Interpretation:

• Write endpoint failure appears as 507 at quota edge.
• Health endpoint still returns 200.

4.7 Console telemetry evidence¶

Artifact: trigger/kql-console-20260404T060610Z.json

Rows containing ENOSPC indicators:

This confirms application-level exception path, not only HTTP status symptom.

4.7.1 Log Analytics KQL query (portal verification)¶

To cross-verify the two ENOSPC console rows exported to trigger/kql-console-20260404T060610Z.json above, the Log Analytics workspace Logs blade was opened against the same workspace from a fresh live reproduction and the identical KQL query was executed interactively:

Purpose: Provide an independent Portal-side execution of the same KQL query whose exported results are stored in trigger/kql-console-20260404T060610Z.json, so a reviewer can confirm that the two ENOSPC rows shown in the §4.7 table are not an artifact of local export tooling (az monitor log-analytics query) but are the same records the Log Analytics service returns to an interactive Portal query against the AppServiceConsoleLogs table.

Look for:

• Blade heading reads "log-labdisk-uz2rvn6exc56q | Logs" and the sub-heading reads "Log Analytics workspace" — this confirms the query ran against the same workspace used for this lab's exported KQL evidence.
• The KQL editor shows exactly the query AppServiceConsoleLogs | where ResultDescription has "No space left on device" | project TimeGenerated, ResultDescription | order by TimeGenerated desc | take 10 — the has operator matches whole tokens, so the filter is anchored on the specific phrase "No space left on device" and cannot silently match a substring inside an unrelated log line.
• The Results grid shows exactly 2 rows (pagination "1 - 2 of 2") — the same row count as the §4.7 table.
• Row 1 ResultDescription contains OSError: [Errno 28] No space left on device — Python's standard exception message for ENOSPC.
• Row 2 ResultDescription contains [Errno 28] No space left on device inside the app-level fill-home failed with disk error string — the app's own exception handler surfaced the same errno.
• Query duration in bottom left reads a small millisecond value (here 0s 459ms) — confirming the workspace is not throttled and the query hit indexed data.

Expected result: The row count in the Portal (2) matches the row count in the exported JSON referenced by §4.7 (also 2). The two TimeGenerated values in the Portal fall within seconds of each other and are ordered by descending time, consistent with a single write attempt raising the OS-level OSError (row 1) and the app's exception handler logging the wrapped error message (row 2, logged milliseconds earlier). If instead the Portal returned zero rows against the same 24-hour window, suspect a Diagnostic Settings misconfiguration on the Web App (the AppServiceConsoleLogs category may not be enabled) and re-check az monitor diagnostic-settings list --resource $APP before concluding the ENOSPC log path is broken.

Next step: If the KQL row count in a future reproduction is much higher than 2 (say 20+), the app is retrying the failing /fill-home endpoint on its own — check the client-side reproduction script (labs/no-space-left-on-device/trigger.sh) for an inadvertent retry loop, because the intended lab shape is a single failed write followed by manual investigation, not a retry storm.

4.8 Platform telemetry context¶

Artifact: trigger/kql-platform-20260404T060610Z.json

Representative rows include startup and stop events, plus timeout history strings.

Important for this lab:

• Platform logs provide lifecycle context.
• Root-cause storage evidence still comes from app and console layers.

4.9 App config and startup context¶

Artifact: baseline/app-config.json

Key field:

appCommandLine = gunicorn --bind=0.0.0.0:8000 --timeout=120 --workers=2 app:app

Meaning:

• Startup command is valid.
• This incident is not module-resolution failure.
• Failure is induced by storage pressure from write endpoints.

4.10 Health vs write behavior proof table¶

4.11 Timeline reconstruction¶

4.12 Hypothesis verdict¶

Verdict: Supported.

Evidence chain:

1. /home increased from ~0.53% to 100% used.
2. /fill-home failed with Errno 28 at high utilization.
3. KQL HTTP captured a 507 for /fill-home.
4. KQL console captured explicit No space left on device.
5. /health remained 200 during the failure window.

4.13 Operational recommendations from this experiment¶

1. Add dedicated synthetic probes for write paths, not only /health.

2. Alert on /home free space thresholds (for example <15%, <5%).

3. Store large generated files outside local persistent quota when possible.
4. Implement app-level graceful fallback on OSError for disk writes.
5. Add periodic cleanup jobs for temporary artifacts in /home.

4.14 Suggested SLO instrumentation¶

4.15 Reusable KQL snippets¶

4.15.1 ENOSPC in console¶

AppServiceConsoleLogs
| where TimeGenerated > ago(24h)
| where ResultDescription has_any (
    "No space left on device",
    "ENOSPC",
    "Errno 28"
)
| project TimeGenerated, ResultDescription
| order by TimeGenerated desc

4.15.2 HTTP write failure profile¶

AppServiceHTTPLogs
| where TimeGenerated > ago(24h)
| where CsUriStem in ("/fill-home", "/fill-tmp", "/health")
| summarize count() by CsUriStem, ScStatus
| order by CsUriStem asc, ScStatus asc

4.15.3 Disk pressure and endpoint latency¶

AppServiceHTTPLogs
| where TimeGenerated > ago(24h)
| where CsUriStem in ("/fill-home", "/health")
| summarize
    requests=count(),
    avg_time_ms=avg(TimeTaken),
    p95_time_ms=percentile(TimeTaken, 95)
  by CsUriStem, ScStatus
| order by CsUriStem asc, ScStatus asc

4.16 Known limitations of this experiment¶

1. Synthetic workload writes deterministic binary payloads, not production data patterns.

2. Timing and status distributions are specific to this run and SKU.

3. /tmp was not driven to exhaustion in this dataset, so /tmp ENOSPC behavior is inferred from architecture and app logic, not observed as terminal failure in this run.

4. Platform logs include startup lifecycle noise; root-cause proof relied on app and console evidence.

4.17 Experiment closure statement¶

This experiment demonstrates a production-relevant anti-pattern:

Green health checks can coexist with failed business writes under persistent storage exhaustion.

Therefore, incident playbooks must include storage-aware functional probes and log correlation, not only endpoint-level liveness.

Expected Evidence¶

This section defines what you SHOULD observe at each phase of the lab. Use it to validate your investigation is on track.

Before Trigger (Baseline)¶

During Incident¶

After Recovery¶

Evidence Timeline¶

graph TD
    A[Baseline Capture] --> B[Trigger Fault]
    B --> C[During: Collect Evidence]
    C --> D[After: Compare to Baseline]
    D --> E[Verdict: Confirmed/Falsified]

Evidence Chain: Why This Proves the Hypothesis¶

Clean Up¶

az group delete --name "$RG" --yes --no-wait

Related Playbook¶

• No Space Left on Device / Ephemeral Storage Pressure

See Also¶

• Playbook: No Space Left on Device / Ephemeral Storage Pressure
• Playbook: Memory Pressure and Worker Degradation
• Lab: Memory Pressure
• Lab: Slow Start / Cold Start

Sources¶

• Operating system functionality on Azure App Service
• Azure App Service plan overview
• Best practices for Azure App Service diagnostics
• Monitor App Service with Azure Monitor
• Configure Linux Python app on App Service
• Azure App Service reliability