Skip to content

Fault Lab 01: Image Pull Failure

Use this falsification lab to prove that an ImagePullBackOff symptom is caused by a bad image reference, not by probe, ingress, or runtime logic.

Lab Metadata

Field Value
Difficulty Intermediate
Estimated duration 20-30 minutes
Lab tier AKS workload-level falsification lab
Failure class Pod start failure / registry image lookup
Namespace workload
Companion assets labs/image-pull-failure/
Paired playbook Image Pull Failure

1) Background

The Python sample in apps/python/ normally starts a FastAPI container from an image you build and push to a registry. This lab intentionally deploys the same workload name with a broken image reference so the kubelet cannot pull it.

flowchart TD
    A[Render broken manifest] --> B[Apply workload in namespace workload]
    B --> C[Observe ErrImagePull or ImagePullBackOff]
    C --> D[Capture pod events and image reference evidence]
    D --> E[Apply fixed manifest with valid image]
    E --> F[Re-check pod readiness and falsify hypothesis]

2) Hypothesis

If the pod references an image tag that does not exist, then AKS will keep the pod in ErrImagePull or ImagePullBackOff, and kubectl describe pod will show registry pull errors tied to the image name or tag.

3) Runbook

  1. Build and push the sample app image from apps/python/, then export IMAGE_REPOSITORY to that real image reference.
  2. Review the shared cluster baseline in infra/README.md if you need a lab cluster first.

  3. Run the broken scenario:

    ./labs/image-pull-failure/trigger-scenario.sh

  4. Collect evidence before changing anything:

    ./labs/image-pull-failure/verify.sh

  5. Apply the fix:

    ./labs/image-pull-failure/trigger-fix.sh

  6. Re-run verification and compare the before/after evidence pack.

4) Experiment Log

Record a real run here after execution. This section is intentionally a template and must not be filled with invented command output.

Timestamp (UTC) Action Expected observation Actual observation
fill after real run Apply broken manifest Pod enters ErrImagePull or ImagePullBackOff fill after real run
fill after real run Capture evidence Events mention manifest lookup or auth failure fill after real run
fill after real run Apply fixed manifest New pod reaches Running and Ready fill after real run

Expected Evidence

  • Before remediation, kubectl describe pod shows ErrImagePull or ImagePullBackOff and references the broken image value.
  • kubectl get events --namespace workload --sort-by=.lastTimestamp shows pull failures before any fix is applied.
  • After the fix, the replacement pod becomes Running and Ready.
  • Falsification-after-fix: if the pod still fails after restoring a valid image reference, the original hypothesis is false or incomplete, and you should pivot to identity, network, or registry authorization checks in the paired playbook.

Clean Up

./labs/image-pull-failure/cleanup.sh

Set DELETE_RESOURCE_GROUP=true and RG=<resource-group-name> only if the whole cluster resource group was created just for this lab.

See Also

Sources