RDP and SSH Connection Failures¶
Symptoms¶
- Administrative or workload impact is visible to users or operators.
- The VM is deployed, but one part of the expected control plane or data path is failing.
- You need a fast way to narrow the problem before making a risky change.
flowchart TD
A[RDP and SSH Connection Failures] --> B[Confirm current symptom and blast radius]
B --> C[Collect platform evidence first]
C --> D[Collect guest or workload evidence]
D --> E[Map findings to the most likely hypothesis]
E --> F[Apply the smallest safe fix]
F --> G[Validate recovery and prevention actions] 1. Summary¶
Use this playbook when administrators cannot sign in to a VM over RDP or SSH, receive timeouts, authentication prompts that always fail, or see the service listening but unreachable.
Public or private admin path, Bastion, JIT access, guest firewall, credential reset, and service health.
2. Common Misreadings¶
| Observation | Often misread as | Actually means |
|---|---|---|
| One failed probe or one stale metric | Total VM outage | The issue may be scoped to one path or one recovery dependency. |
| A successful extension deployment | Guest health is good | Extensions can succeed while the underlying guest service still fails. |
| A recent change record | Guaranteed root cause | Recent changes are strong leads, but they still need proof from evidence. |
| A restart fixes the issue | Permanent resolution | Recovery after restart may only hide the real structural cause. |
3. Competing Hypotheses¶
| Hypothesis | Likelihood | Key discriminator |
|---|---|---|
| Control-plane or configuration drift | High | Azure resource state no longer matches the intended pattern. |
| Guest OS or agent issue | High | Guest or serial evidence shows service, boot, or firewall failure. |
| Capacity or platform dependency bottleneck | Medium | Metrics or SKU limits explain the symptom better than configuration drift. |
| Security control blocked expected behavior | Medium | NSG, ASG, JIT, or policy state changed before the incident. |
| External dependency issue | Low | VM appears healthy, but a downstream service path is broken. |
4. What to Check First¶
-
Review VM instance view
| Command | Purpose |
|---|---|
az vm get-instance-view | Retrieve the runtime instance view of the virtual machine. |
--resource-group | Resource group that contains the virtual machine. |
--name | Name of the virtual machine to inspect. |
--output | Output format for the response (JSON here). |
-
Review boot diagnostics settings
| Command | Purpose |
|---|---|
az vm boot-diagnostics get-boot-log | Retrieve the serial boot log for the virtual machine. |
--resource-group | Resource group that contains the virtual machine. |
--name | Name of the virtual machine to inspect. |
-
Review NIC effective security rules
| Command | Purpose |
|---|---|
az network nic list-effective-nsg | List the effective NSG rules applied to a network interface. |
--resource-group | Resource group that contains the network interface. |
--name | Name of the network interface to inspect. |
--output | Output format for the response (JSON here). |
-
Review recent platform changes
| Command | Purpose |
|---|---|
az monitor activity-log list | List recent control-plane activity-log events. |
--resource-group | Resource group to scope activity-log events to. |
--offset | Look-back window for events (24h here). |
--output | Output format for the response (table here). |
5. Evidence to Collect¶
5.1 KQL Queries¶
// Administrative sign-in and activity correlation
AzureActivity
| where TimeGenerated > ago(24h)
| where ResourceProviderValue =~ "MICROSOFT.COMPUTE"
| project TimeGenerated, OperationNameValue, Caller, ActivityStatusValue, ResourceId
| order by TimeGenerated desc
| Field | Interpretation |
|---|---|
TimeGenerated | Incident sequence and correlation window. |
| Resource identifier | Confirms the signal belongs to the affected VM. |
| Operation or metric value | Explains whether the failure is change-driven, capacity-driven, or guest-driven. |
How to read this
Compare these results with the last known healthy window. A change in shape matters more than a single absolute value.
// Heartbeat plus security posture context
Heartbeat
| where TimeGenerated > ago(24h)
| summarize LastHeartbeat=max(TimeGenerated) by Computer, _ResourceId
| order by LastHeartbeat asc
| Field | Interpretation |
|---|---|
TimeGenerated | Incident sequence and correlation window. |
| Resource identifier | Confirms the signal belongs to the affected VM. |
| Operation or metric value | Explains whether the failure is change-driven, capacity-driven, or guest-driven. |
How to read this
Compare these results with the last known healthy window. A change in shape matters more than a single absolute value.
5.2 CLI Investigation¶
az vm show --resource-group $RG --name $VM_NAME --query "{powerState:powerState,vmSize:hardwareProfile.vmSize,priority:priority,provisioningState:provisioningState}" --output json
az network nic show --resource-group $RG --name $NIC_NAME --query "{ipConfigs:ipConfigurations[].privateIPAddress,acceleratedNetworking:enableAcceleratedNetworking,networkSecurityGroup:networkSecurityGroup.id}" --output json
| Command | Purpose |
|---|---|
az vm show | Retrieve the current configuration of the virtual machine. |
--resource-group | Resource group that contains the virtual machine. |
--name | Name of the virtual machine to inspect. |
--query | JMESPath expression selecting power state, size, priority, and provisioning state. |
--output | Output format for the response (JSON here). |
az network nic show | Retrieve the configuration of a network interface. |
--resource-group | Resource group that contains the network interface. |
--name | Name of the network interface to inspect. |
--query | JMESPath expression selecting private IPs, accelerated networking, and NSG association. |
--output | Output format for the response (JSON here). |
Interpretation:
- If the VM is not in a usable power state, fix power and boot issues before guest remediation.
- If the NIC or NSG binding is wrong, repair the control plane before changing guest settings.
- If accelerated networking or disk attachment changed after a resize, validate that the new size still supports the intended feature set.
6. Validation and Disproof by Hypothesis¶
Hypothesis 1: Configuration drift or recent change¶
Proves if: The Activity Log records a change (resize, disk swap, NSG/route edit, or extension update) whose timestamp precedes the first failed signal.
Disproves if: No control-plane change is recorded in the incident window and the resource configuration matches the last-known-good baseline.
Recommended validation steps:
- Filter the Activity Log to this resource for the 24 hours before symptom onset.
- Diff the current configuration against the approved landing-zone template.
- Correlate the earliest recorded change with the first failed signal.
- Roll back the single suspected change and re-test the symptom.
Hypothesis 2: Guest OS or service failure¶
Proves if: Boot diagnostics or serial console show a guest-level fault (kernel panic, failed fsck, stopped service, or full OS disk) while the platform reports the VM as running.
Disproves if: Boot diagnostics show a clean boot and the guest-agent heartbeat is healthy.
Recommended validation steps:
- Open the boot-diagnostics screenshot and serial log.
- Check the guest agent (heartbeat / waagent) status.
- Inspect the guest system and service logs for the failing unit.
- Repair in-guest (or from a rescue VM) and re-test the boot or login.
Hypothesis 3: Capacity limit or SKU mismatch¶
Proves if: A start or resize returns an allocation or quota error, or metrics show the VM pinned at its SKU's CPU, disk, or network cap.
Disproves if: Allocation succeeds and utilization sits below the SKU's documented limits.
Recommended validation steps:
- Read the allocation or quota error from the start or resize attempt.
- Compare the observed draw against the SKU's core, IOPS, and bandwidth caps.
- Check regional and zonal capacity for the target size.
- Resize to an available SKU or request quota, then re-test.
Hypothesis 4: Security control blocking the expected path¶
Proves if: An NSG rule, expired JIT grant, firewall, or route change denies the management or data path exactly where the symptom appears.
Disproves if: Effective NSG rules, routes, and JIT grants all permit the expected path and the block occurs elsewhere.
Recommended validation steps:
- Compute the effective security rules and routes for the NIC.
- Confirm JIT or Bastion access is currently granted.
- Trace the denied hop with IP Flow Verify or Connection Troubleshoot.
- Restore the least-privilege allow rule and re-test.
7. Likely Root Cause Patterns¶
| Pattern | Evidence | Resolution |
|---|---|---|
| Unsupported feature after resize or redeploy | Settings drift, feature not enabled, or SKU capability mismatch | Move back to a supported size or re-enable the feature with validation. |
| NSG or route or JIT drift | Effective rules do not match the intended admin or workload path | Repair the policy and document the expected flow. |
| Guest service stopped or corrupted | Serial, extension, or guest evidence points to OS-level failure | Repair the guest service, driver, boot loader, or firewall configuration. |
| Performance bottleneck blamed as an outage | CPU, memory, or disk metrics saturate before the user-visible failure | Resize, retier, or redistribute workload pressure. |
8. Immediate Mitigations¶
- Confirm whether the issue is path-related, policy-related, or credential-related before resetting accounts.
- Prefer Azure Bastion and JIT over permanent public RDP or SSH exposure.
- Use Run Command or VMAccess extensions only when the platform path works and you need guest-side repair.
Step-by-step resolution:
- Stabilize the VM or admin path without erasing forensic evidence.
- Correct the control-plane configuration first when Azure intent is clearly wrong.
- Apply guest-side repair only after confirming the platform path is healthy.
- Re-run the original command, probe, or sign-in flow to verify recovery.
- Record the exact evidence that proved the fix, not just that the symptom disappeared.
CLI commands commonly used during fixes:
az vm run-command invoke --resource-group $RG --name $VM_NAME --command-id RunShellScript --scripts "sudo systemctl status walinuxagent"
az vm restart --resource-group $RG --name $VM_NAME
| Command | Purpose |
|---|---|
az vm run-command invoke | Run a shell script inside the guest OS of the VM. |
--resource-group | Resource group that contains the virtual machine. |
--name | Name of the virtual machine to target. |
--command-id | Built-in run-command identifier (RunShellScript). |
--scripts | Inline script to execute in the guest OS. |
az vm restart | Restart the virtual machine. |
--resource-group | Resource group that contains the virtual machine. |
--name | Name of the virtual machine to restart. |
9. Prevention¶
Prevention checklist¶
- [ ] Keep a documented healthy baseline for VM size, NIC, disk, and admin-path settings
- [ ] Alert on drift in critical VM security and connectivity controls
- [ ] Test boot diagnostics, Bastion, serial console, and backup restore before production go-live
- [ ] Review SKU feature compatibility before resize operations
- [ ] Capture post-incident evidence and turn it into a reusable guardrail