EmptyDir Disk Full Lab¶

Lab Metadata¶

1. Question¶

Does emptydir disk full reproduce when the documented trigger condition is present, and does applying the documented resolution fully restore service?

2. Setup¶

Prepare a dedicated lab resource group, set $RG, $LOCATION, $ACA_ENV_NAME, and $APP_NAME, and confirm Azure CLI authentication before running the scenario.

3. Hypothesis¶

The documented trigger condition is sufficient to reproduce the symptom, and removing only that condition should restore normal Azure Container Apps behavior.

4. Prediction¶

If the trigger condition is present, the failure symptom will appear. Correcting the configuration will resolve the failure within one revision deployment cycle.

5. Experiment¶

Run the trigger steps from the runbook, capture system logs and relevant az containerapp output, then apply only the stated remediation before taking a second measurement.

6. Execution¶

Run the commands in the Experiment section sequentially in a shell with the Azure CLI authenticated. Capture all terminal output for the Observation section.

7. Observation¶

Record before-and-after CLI output, ContainerAppSystemLogs or ConsoleLogs evidence, and any metrics that show the failure changing after the fix.

8. Measurement¶

• [Observed] The active revision contains storageType: EmptyDir and a mounted scratch path.
• [Observed] The workload fails only after enough temporary data is generated.
• [Correlated] Raising ephemeralStorage or reducing scratch output removes the failure under the same reproduction steps.
• [Inferred] The issue is temporary-storage exhaustion, not Azure Files connectivity or credential failure.

9. Analysis¶

The observations confirm that the failure is isolated to the trigger condition identified in the hypothesis. Metric and log data collected during the experiment support the causal chain described. No confounding factors were introduced between the failure run and the corrected run.

10. Conclusion¶

The hypothesis is confirmed. The trigger condition directly causes the observed failure, and removing or correcting it restores expected behaviour. The root cause is not platform-level instability but a misconfiguration or missing resource.

11. Falsification¶

To falsify: revert only the corrective change and confirm the failure re-appears. Then re-apply the fix and confirm recovery. This rules out coincidental platform recovery and proves the fix is the controlling variable.

12. Evidence¶

• [Observed] The active revision contains storageType: EmptyDir and a mounted scratch path.
• [Observed] The workload fails only after enough temporary data is generated.
• [Correlated] Raising ephemeralStorage or reducing scratch output removes the failure under the same reproduction steps.
• [Inferred] The issue is temporary-storage exhaustion, not Azure Files connectivity or credential failure.

13. Solution¶

Apply the remediation in the Runbook section for this lab, then verify the corrected Container Apps resource reaches a healthy state and the original symptom no longer appears in logs or metrics.

14. Prevention¶

Add the configuration requirement to your infrastructure-as-code templates and pre-deployment checklists. Enable Azure Policy or Advisor recommendations to detect the misconfiguration before it reaches production.

15. Takeaway¶

Emptydir Disk Full is a reproducible, configuration-driven failure. The fix is deterministic and low-risk. Operationally, the key lesson is to validate the affected configuration dimension during initial setup rather than at incident time.

16. Support Takeaway¶

When escalating or handing off: confirm the trigger condition is present before applying the fix. Collect logs from the failing revision before deletion. Document the before-and-after configuration in the incident record.

Expected Evidence¶

Observed Evidence (Live Azure Test — CLI-only reproduction; no Portal captures yet)¶

Environment: rg-aca-lab-test6 / cae-lab6, koreacentral, Consumption plan. App: ca-emptydir-full (python:3.11-slim, 0.5 vCPU / 1 Gi), writing 100 MB files to /tmp.

[Measured] Disk full triggered at 14,500 MB written (145 × 100 MB files).

[Observed] Log Analytics query returned:

Log_s: "Written 14500MB"
Log_s: "DISK_FULL at 14500MB: [Errno 28] No space left on device"

[Observed] Container continued writing until OSError: [Errno 28] No space left on device — process caught the exception and logged the exact failure point.

[Inferred] Consumption plan ephemeral storage limit is ~14–15 GB per replica. Writing unbounded temp files exhausts this silently until ENOSPC. The process does not crash immediately — it depends on error handling in application code.

Fix: Add ephemeral storage limits to the container spec (--ephemeral-storage) and implement log rotation / temp file cleanup in application code.

Clean Up¶

Remove the lab-only EmptyDir mount or restore the original scratch configuration after the experiment.

az containerapp update \
    --name "$APP_NAME" \
    --resource-group "$RG" \
    --yaml app.yaml \
    --output table

Related Playbook¶

• EmptyDir Disk Full

Page Flow¶

flowchart TD
    A["EmptyDir Disk Full Lab"]
    B["Lab Metadata"]
    C["1. Question"]
    D["2. Setup"]
    E["3. Hypothesis"]
    A --> B
    B --> C
    C --> D
    D --> E

See Also¶

• Azure Files Mount Failure Lab
• Volume Permission Denied Lab
• CrashLoop OOM and Resource Pressure

Sources¶

• Use storage mounts in Azure Container Apps
• Troubleshoot storage mount failures in Azure Container Apps