VM Boot Failures¶

Symptoms¶

• Administrative or workload impact is visible to users or operators.
• The VM is deployed, but one part of the expected control plane or data path is failing.
• You need a fast way to narrow the problem before making a risky change.

flowchart TD
    A[VM Boot Failures] --> B[Confirm current symptom and blast radius]
    B --> C[Collect platform evidence first]
    C --> D[Collect guest or workload evidence]
    D --> E[Map findings to the most likely hypothesis]
    E --> F[Apply the smallest safe fix]
    F --> G[Validate recovery and prevention actions]

1. Summary¶

Use this playbook when a VM never reaches a usable OS state, cycles through boot diagnostics errors, or shows a black screen, GRUB issue, BitLocker prompt, or kernel panic in serial console output.

Boot diagnostics, serial console, extension state, OS disk health, and recovery actions.

2. Common Misreadings¶

3. Competing Hypotheses¶

4. What to Check First¶

1. Review VM instance view

   az vm get-instance-view             --resource-group $RG             --name $VM_NAME             --output json
   

1. Review boot diagnostics settings

   az vm boot-diagnostics get-boot-log             --resource-group $RG             --name $VM_NAME
   

1. Review NIC effective security rules

   az network nic list-effective-nsg             --resource-group $RG             --name $NIC_NAME             --output json
   

1. Review recent platform changes

   az monitor activity-log list             --resource-group $RG             --offset 24h             --output table
   

5. Evidence to Collect¶

5.1 KQL Queries¶

// Activity and maintenance events
AzureActivity
| where TimeGenerated > ago(24h)
| where ResourceProviderValue =~ "MICROSOFT.COMPUTE"
| where ResourceId has "/virtualMachines/"
| project TimeGenerated, OperationNameValue, ActivityStatusValue, Caller, ResourceId
| order by TimeGenerated desc

// Heartbeat and guest gap
Heartbeat
| where TimeGenerated > ago(24h)
| summarize LastHeartbeat=max(TimeGenerated) by Computer, _ResourceId
| order by LastHeartbeat asc

5.2 CLI Investigation¶

az vm show     --resource-group $RG     --name $VM_NAME     --query "{powerState:powerState,vmSize:hardwareProfile.vmSize,priority:priority,provisioningState:provisioningState}"     --output json

az network nic show     --resource-group $RG     --name $NIC_NAME     --query "{ipConfigs:ipConfigurations[].privateIPAddress,acceleratedNetworking:enableAcceleratedNetworking,networkSecurityGroup:networkSecurityGroup.id}"     --output json

Interpretation:

• If the VM is not in a usable power state, fix power and boot issues before guest remediation.
• If the NIC or NSG binding is wrong, repair the control plane before changing guest settings.
• If accelerated networking or disk attachment changed after a resize, validate that the new size still supports the intended feature set.

6. Validation and Disproof by Hypothesis¶

Hypothesis 1: Configuration drift or recent change¶

Proves if: The Activity Log records a change (resize, disk swap, NSG/route edit, or extension update) whose timestamp precedes the first failed signal.

Disproves if: No control-plane change is recorded in the incident window and the resource configuration matches the last-known-good baseline.

Recommended validation steps:

1. Filter the Activity Log to this resource for the 24 hours before symptom onset.
2. Diff the current configuration against the approved landing-zone template.
3. Correlate the earliest recorded change with the first failed signal.
4. Roll back the single suspected change and re-test the symptom.

Hypothesis 2: Guest OS or service failure¶

Proves if: Boot diagnostics or serial console show a guest-level fault (kernel panic, failed fsck, stopped service, or full OS disk) while the platform reports the VM as running.

Disproves if: Boot diagnostics show a clean boot and the guest-agent heartbeat is healthy.

Recommended validation steps:

1. Open the boot-diagnostics screenshot and serial log.
2. Check the guest agent (heartbeat / waagent) status.
3. Inspect the guest system and service logs for the failing unit.
4. Repair in-guest (or from a rescue VM) and re-test the boot or login.

Hypothesis 3: Capacity limit or SKU mismatch¶

Proves if: A start or resize returns an allocation or quota error, or metrics show the VM pinned at its SKU's CPU, disk, or network cap.

Disproves if: Allocation succeeds and utilization sits below the SKU's documented limits.

Recommended validation steps:

1. Read the allocation or quota error from the start or resize attempt.
2. Compare the observed draw against the SKU's core, IOPS, and bandwidth caps.
3. Check regional and zonal capacity for the target size.
4. Resize to an available SKU or request quota, then re-test.

Hypothesis 4: Security control blocking the expected path¶

Proves if: An NSG rule, expired JIT grant, firewall, or route change denies the management or data path exactly where the symptom appears.

Disproves if: Effective NSG rules, routes, and JIT grants all permit the expected path and the block occurs elsewhere.

Recommended validation steps:

1. Compute the effective security rules and routes for the NIC.
2. Confirm JIT or Bastion access is currently granted.
3. Trace the denied hop with IP Flow Verify or Connection Troubleshoot.
4. Restore the least-privilege allow rule and re-test.

7. Likely Root Cause Patterns¶

8. Immediate Mitigations¶

1. Capture a fresh screenshot and serial log before restarting so you do not lose the failure signature.
2. Validate whether the issue started after patching, kernel update, disk encryption change, or availability host maintenance.
3. Repair boot loader, filesystem, or driver issues using the Azure serial console or VM repair workflow.

Step-by-step resolution:

1. Stabilize the VM or admin path without erasing forensic evidence.
2. Correct the control-plane configuration first when Azure intent is clearly wrong.
3. Apply guest-side repair only after confirming the platform path is healthy.
4. Re-run the original command, probe, or sign-in flow to verify recovery.
5. Record the exact evidence that proved the fix, not just that the symptom disappeared.

CLI commands commonly used during fixes:

az vm run-command invoke     --resource-group $RG     --name $VM_NAME     --command-id RunShellScript     --scripts "sudo systemctl status walinuxagent"

az vm restart     --resource-group $RG     --name $VM_NAME

9. Prevention¶

Prevention checklist¶

• [ ] Keep a documented healthy baseline for VM size, NIC, disk, and admin-path settings
• [ ] Alert on drift in critical VM security and connectivity controls
• [ ] Test boot diagnostics, Bastion, serial console, and backup restore before production go-live
• [ ] Review SKU feature compatibility before resize operations
• [ ] Capture post-incident evidence and turn it into a reusable guardrail

See Also¶

• Troubleshooting Index
• First 10 Minutes
• Decision Tree

Sources¶

• Troubleshoot Azure virtual machines
• Boot diagnostics for Azure virtual machines
• Connect to a VM with Azure Bastion
• Azure Managed Disks performance
• Accelerated networking overview