Skip to content

Evidence Packs

Use this index to standardize what you capture before remediation. Every row pairs a falsification lab with the matching playbook so operators preserve the same evidence set that supports the troubleshooting hypothesis.

Evidence Pack Matrix

Failure class Preserve before remediation CLI commands KQL queries Healthy-signal baseline PII masking guidance Lab Playbook
Image pull failure Pod events, image reference, namespace secrets, node egress clues kubectl get pods --namespace workload --output wide
kubectl describe pod <pod-name> --namespace workload
kubectl get events --namespace workload --sort-by=.lastTimestamp		 KubeEvents | where TimeGenerated > ago(30m) | where Namespace == "workload" and Reason in ("Failed", "BackOff") | project TimeGenerated, Name, Reason, Message
KubePodInventory | where TimeGenerated > ago(30m) | where Namespace == "workload" | project TimeGenerated, Name, ContainerStatusReason, ContainerID, Image		 Fixed image reference starts pods and clears pull-backoff reason Mask real registry names only when they reveal tenant or customer identity; never publish tokens, pull secrets, subscription IDs, tenant IDs, or email addresses Fault Lab 01 Image Pull Failure
CrashLoopBackOff Previous container logs, exit code, restart count, probe config kubectl get pods --namespace workload
kubectl logs <pod-name> --namespace workload --previous
kubectl describe pod <pod-name> --namespace workload		 KubePodInventory | where TimeGenerated > ago(30m) | where Namespace == "workload" | project TimeGenerated, Name, PodStatus, ContainerRestartCount, ContainerStatusReason
KubeEvents | where TimeGenerated > ago(30m) | where Namespace == "workload" and Reason in ("BackOff", "Killing", "Unhealthy") | project TimeGenerated, Name, Reason, Message		 Pod restart count stays flat and readiness remains true after the fix Mask stack traces only when they expose secrets, internal hostnames, or customer data; keep exit codes and reasons intact Fault Lab 02 CrashLoop
Pending pod from oversized requests Scheduler events, requested resources, node allocatable, autoscaler state kubectl describe pod <pod-name> --namespace workload
kubectl get nodes --output wide
kubectl describe node <node-name>		 KubeEvents | where TimeGenerated > ago(30m) | where Namespace == "workload" and Reason == "FailedScheduling" | project TimeGenerated, Name, Message
Perf | where TimeGenerated > ago(30m) | where ObjectName == "K8SNode" and CounterName in ("cpuCapacityNanoCores", "memoryCapacityBytes") | project TimeGenerated, Computer, CounterName, CounterValue		 Scheduler places the pod without FailedScheduling once requests are right-sized Mask node names if they embed customer naming conventions; keep resource values and scheduler text Fault Lab 03 Pending Pods
Ingress misconfiguration Ingress spec, service, endpoints, controller events, hostname mapping kubectl get ingress --namespace workload --output wide
kubectl describe ingress <ingress-name> --namespace workload
kubectl get service,endpoints --namespace workload		 KubeEvents | where TimeGenerated > ago(30m) | where Namespace == "workload" and ObjectKind == "Ingress" | project TimeGenerated, Name, Reason, Message
KubePodInventory | where TimeGenerated > ago(30m) | where Namespace == "workload" | project TimeGenerated, Name, PodStatus, PodIP		 Pods and endpoints stay healthy while ingress routes successfully after the fix Mask public IPs, DNS names, TLS secret names, and any hostname tied to a real environment; keep port numbers and path rules Fault Lab 04 Ingress Failure
Node pressure and scheduling boundary Scheduler events, node readiness, labels, taints, allocatable snapshot kubectl describe pod <pod-name> --namespace workload
kubectl get nodes --show-labels
kubectl describe node <node-name>		 KubeEvents | where TimeGenerated > ago(30m) | where Namespace == "workload" and Reason == "FailedScheduling" | project TimeGenerated, Name, Message
Perf | where TimeGenerated > ago(30m) | where ObjectName == "K8SNode" and CounterName in ("memoryRssBytes", "cpuUsageNanoCores") | project TimeGenerated, Computer, CounterName, CounterValue		 Nodes remain Ready and the pod schedules after removing the bad placement rule Mask VMSS instance IDs, node resource IDs, subscription IDs, and tenant-linked labels; keep taint keys and scheduler messages Fault Lab 05 Node Not Ready

Usage Notes

  • Preserve evidence before scaling, restarting, or reapplying manifests.
  • Prefer kubectl describe, kubectl get events, and read-only KQL queries before remediation.
  • If you export JSON or screenshots, sanitize subscription IDs as <subscription-id>, tenant IDs as <tenant-id>, object IDs as <object-id>, and emails as user@example.com.
  • Do not attach fabricated output to the repo. Store real run artifacts in the matching labs/<slug>/evidence/ directory only after execution.

See Also

Sources