Skip to content

Fault Lab 05: Node Pressure and Scheduling Boundaries

Use this falsification lab to prove that a pod is unschedulable because of node-pool placement rules and taints, not because the node is NotReady or the image is broken.

Lab Metadata

Field Value
Difficulty Intermediate
Estimated duration 20-30 minutes
Lab tier AKS node and scheduling falsification lab
Failure class Scheduler constraint / taint mismatch
Namespace workload
Companion assets labs/node-pressure-scheduling/
Paired playbook Node Not Ready

1) Background

AKS system pools are commonly tainted to reserve them for critical add-ons. This lab intentionally forces the sample app toward a system-pool scheduling boundary so you can distinguish node-health evidence from scheduler-constraint evidence.

flowchart TD
    A[Apply manifest with restrictive nodeSelector] --> B[Scheduler checks taints and labels]
    B --> C[Pod remains Pending]
    C --> D[Capture scheduler messages and node conditions]
    D --> E[Apply fixed manifest without bad placement rule]
    E --> F[Confirm scheduling recovery and falsify hypothesis]

2) Hypothesis

If the workload is forced onto nodes that do not tolerate or satisfy the target pool constraints, then the pod will remain Pending, and scheduler events will cite node selector or taint mismatch rather than node NotReady conditions.

3) Runbook

  1. Build and push the Python sample image, then export IMAGE_REPOSITORY.

  2. Trigger the broken scheduling scenario:

    ./labs/node-pressure-scheduling/trigger-scenario.sh

  3. Capture the pending state before remediation:

    ./labs/node-pressure-scheduling/verify.sh

  4. Apply the fixed manifest:

    ./labs/node-pressure-scheduling/trigger-fix.sh

  5. Re-run verification and compare scheduler messages with post-fix pod placement.

4) Experiment Log

This log stays blank until you execute the lab for real.

Timestamp (UTC) Action Expected observation Actual observation
fill after real run Apply broken manifest Pod stays Pending with scheduler warnings fill after real run
fill after real run Capture evidence Nodes remain Ready, disproving node failure fill after real run
fill after real run Apply fixed manifest Pod schedules successfully fill after real run

Expected Evidence

  • kubectl describe pod shows taint or selector mismatch messages instead of image or crash-loop errors.
  • kubectl get nodes and kubectl describe node show the nodes are still Ready, helping disprove a node-health incident.
  • After the fixed manifest removes the bad placement rule, the same image schedules successfully.
  • Falsification-after-fix: if the pod still does not schedule after removing the placement mismatch, the original hypothesis is false or incomplete, and you should investigate capacity pressure, autoscaler limits, or real node degradation.

Clean Up

./labs/node-pressure-scheduling/cleanup.sh

See Also

Sources