Lab Guide: Slot Swap Config Drift (Sticky vs Non-Sticky Settings)¶

This Level 3 lab guide reproduces a slot swap on Azure App Service Linux and proves, using real artifacts, how configuration behaves across swap boundaries. The lab shows why teams often perceive "drift" after a healthy swap.

Lab Metadata¶

1) Background¶

Slot swap incidents are often not platform failures. They are frequently interpretation failures.

To troubleshoot accurately, you must separate:

1. Slot routing behavior.
2. Sticky/non-sticky config behavior.
3. Process restart and warm-up lifecycle behavior.

1.1 Deployment slots are separate runtime surfaces¶

Each slot is a separate runtime endpoint with its own hostname and runtime lifecycle.

flowchart TD
    subgraph AppService[Single App Service App]
        P[Production Slot\nHostname: app.azurewebsites.net]
        S[Staging Slot\nHostname: staging.azurewebsites.net]
    end
    U[Users] --> P
    O[Operators/Testers] --> S

Even when slots share plan resources, they maintain independent runtime state and per-slot app setting context.

1.2 Swap behavior model¶

Swap can be thought of as role exchange with warm-up safeguards, not just a file copy.

sequenceDiagram
    participant Ops as Operator
    participant Platform as App Service Platform
    participant Stage as Staging Slot
    participant Prod as Production Slot

    Ops->>Platform: Start swap (staging -> production)
    Platform->>Stage: Validate and warm source runtime
    Stage-->>Platform: Warm-up probe succeeded
    Platform->>Prod: Shift production role
    Platform->>Stage: Shift staging role
    Platform-->>Ops: Swap completed

1.3 Sticky and non-sticky settings¶

The critical rule set:

This lab intentionally mixes one of each to produce a clear differential.

1.4 Why this creates perceived "drift"¶

After swap, operators may expect all values to follow code.

If that mental model is wrong:

• Non-sticky setting movement appears surprising.
• Sticky setting stability appears inconsistent.

In reality, both are correct outcomes.

1.5 Template-level implementation in this lab¶

From main.bicep:

• Production and staging are both configured with Python 3.11 runtime.
• FEATURE_FLAG differs by slot (v1 vs v2) and is non-sticky.
• DB_CONNECTION_STRING differs by slot and is marked sticky via slotConfigNames.

slotConfigNames excerpt:

resource slotConfigNames 'Microsoft.Web/sites/config@2023-12-01' = {
  name: '${webApp.name}/slotConfigNames'
  properties: {
    appSettingNames: [
      'DB_CONNECTION_STRING'
    ]
    connectionStringNames: []
  }
}

1.6 Expected before/after state map¶

flowchart TB
    subgraph BeforeSwap[Before Swap]
        P1[Production\nFEATURE_FLAG=v1\nDB_CONNECTION_STRING=prod-server]
        S1[Staging\nFEATURE_FLAG=v2\nDB_CONNECTION_STRING=staging-server]
    end

    subgraph AfterSwap[After Swap]
        P2[Production\nFEATURE_FLAG=v2\nDB_CONNECTION_STRING=prod-server]
        S2[Staging\nFEATURE_FLAG=v1\nDB_CONNECTION_STRING=staging-server]
    end

    P1 --> S2
    S1 --> P2

1.7 Warm-up lifecycle context¶

Swap operations are tightly coupled with startup/warm-up behavior:

• Container and app startup events occur around swap windows.
• Probe success transitions appear in AppServicePlatformLogs.
• Gunicorn startup appears in AppServiceConsoleLogs.

This guide includes both config-state proof and lifecycle proof.

1.8 Why this lab matters¶

For production release safety, teams must be able to answer:

1. Did the swap perform correctly?
2. Did sticky settings stay where intended?
3. Did non-sticky settings move as expected?
4. Were restarts/warm-up transitions healthy?

This lab provides an evidence-driven method to answer all four.

2) Hypothesis¶

2.1 Formal hypothesis statement¶

When performing a slot swap, non-sticky app settings move with the swapped app context while slot settings (deployment slot settings) remain with the destination slot, creating apparent config drift only for operators who expect all settings to swap uniformly.

2.2 Causal chain¶

flowchart TD
    A[Pre-swap config divergence by slot] --> B[Swap staging to production]
    B --> C[Non-sticky values exchange]
    B --> D[Sticky values remain slot-bound]
    C --> E[Production FEATURE_FLAG changes]
    D --> F[Production DB setting remains]
    E --> G[Perceived drift if sticky model unknown]
    F --> G

2.3 Proof criteria¶

The hypothesis is supported only if all conditions are met:

1. Pre-swap production and staging values are different for both targeted settings.
2. Post-swap FEATURE_FLAG values are exchanged across slots.
3. Post-swap DB_CONNECTION_STRING values are unchanged per physical slot.
4. App settings metadata confirms DB_CONNECTION_STRING is sticky (slotSetting=true).
5. Lifecycle logs show restart/warm-up progression around swap time.

2.4 Disproof criteria¶

Any of these disproves the hypothesis:

• FEATURE_FLAG does not exchange across slots after confirmed swap.
• DB_CONNECTION_STRING exchanges despite sticky metadata.
• Sticky metadata absent or inconsistent in pre-swap app settings.
• No lifecycle evidence near swap operation window.

2.5 Controlled and dependent variables¶

2.6 Anticipated data pattern¶

3) Runbook¶

This runbook is designed for deterministic replay. Commands use long flags only.

3.1 Prerequisites¶

3.2 Environment setup¶

RG="rg-lab-slotswap"
LOCATION="koreacentral"
BASE_NAME="labswap"

Convention note:

• In command usage, this guide references $RG, $APP_NAME, etc.
• In declarations, variable names are written without $ (shell syntax).

3.3 Deploy lab infrastructure¶

az group create \
  --name "$RG" \
  --location "$LOCATION"

az deployment group create \
  --resource-group "$RG" \
  --template-file "labs/slot-swap-config-drift/main.bicep" \
  --parameters "baseName=$BASE_NAME"

3.4 Resolve app and slot URLs¶

APP_NAME=$(az webapp list \
  --resource-group "$RG" \
  --query "[0].name" \
  --output tsv)

PRODUCTION_HOST=$(az webapp show \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --query "defaultHostName" \
  --output tsv)

STAGING_HOST="$APP_NAME-staging.azurewebsites.net"
PRODUCTION_URL="https://$PRODUCTION_HOST"
STAGING_URL="https://$STAGING_HOST"

3.5 Validate slot settings metadata¶

Inspect production settings:

az webapp config appsettings list \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --query "[?name=='SCM_DO_BUILD_DURING_DEPLOYMENT' || name=='DB_CONNECTION_STRING' || name=='FEATURE_FLAG'].{name:name,value:value,slotSetting:slotSetting}"

Inspect staging settings:

az webapp config appsettings list \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --slot "staging" \
  --query "[?name=='SCM_DO_BUILD_DURING_DEPLOYMENT' || name=='DB_CONNECTION_STRING' || name=='FEATURE_FLAG'].{name:name,value:value,slotSetting:slotSetting}"

Expected pre-swap model:

Portal view: Configuration blade (slot-aware settings entry point)¶

The Configuration blade is the Portal control plane for the same settings the CLI commands above expose. The Application settings sub-tab (not shown in this capture, which focuses on General settings) is where the per-row Deployment slot setting checkbox appears - that checkbox is the visual representation of the slotSetting:true flag in the expected pre-swap table above (DB_CONNECTION_STRING row). General settings (active here) governs platform behavior like Always on and Session affinity and follows different swap rules than App Settings; consult section 1.3 before changing anything here mid-incident. Use this blade to spot-check the sticky/non-sticky shape of your settings without parsing CLI JSON, then cross-reference the post-swap behavior with the KQL evidence in section 3.10.

3.6 Capture pre-swap runtime evidence¶

curl --silent --show-error "$PRODUCTION_URL/config"
curl --silent --show-error "$STAGING_URL/config"

curl --silent --show-error "$PRODUCTION_URL/diag/slots"
curl --silent --show-error "$STAGING_URL/diag/slots"

curl --silent --show-error "$PRODUCTION_URL/diag/stats"
curl --silent --show-error "$PRODUCTION_URL/diag/env"

Representative pre-swap output:

{"DB_CONNECTION_STRING":"prod-server.database.windows.net","FEATURE_FLAG":"v1","PROCESS_START_UTC":"2026-04-04T05:13:59.196136+00:00","WEBSITE_SLOT_NAME":"Production"}

{"DB_CONNECTION_STRING":"staging-server.database.windows.net","FEATURE_FLAG":"v2","PROCESS_START_UTC":"2026-04-04T05:34:02.102318+00:00","WEBSITE_SLOT_NAME":"Production"}

3.7 Execute swap¶

az webapp deployment slot swap \
  --resource-group "$RG" \
  --name "$APP_NAME" \
  --slot "staging" \
  --target-slot "production"

Wait for immediate post-swap stabilization:

sleep 20

3.8 Capture post-swap runtime evidence¶

curl --silent --show-error "$PRODUCTION_URL/config"
curl --silent --show-error "$STAGING_URL/config"

curl --silent --show-error "$PRODUCTION_URL/diag/slots"
curl --silent --show-error "$PRODUCTION_URL/diag/stats"

Representative post-swap output:

{"DB_CONNECTION_STRING":"prod-server.database.windows.net","FEATURE_FLAG":"v2","PROCESS_START_UTC":"2026-04-04T05:46:56.764695+00:00","WEBSITE_SLOT_NAME":"Production"}

{"DB_CONNECTION_STRING":"staging-server.database.windows.net","FEATURE_FLAG":"v1","PROCESS_START_UTC":"2026-04-04T05:49:05.541050+00:00","WEBSITE_SLOT_NAME":"Production"}

3.9 Run full trigger script (optional)¶

bash "labs/slot-swap-config-drift/trigger.sh" "$RG" "$APP_NAME"

The script does:

1. Package deployment to both slots.
2. Pre-swap config capture.
3. Swap execution.
4. Post-swap capture.
5. Behavior summary booleans:
   • feature swapped
   • sticky DB remained
   • process restart observed

3.10 KQL runbook queries¶

The Activity log blade is the Portal anchor for swap-drift investigations, giving you the audit trail of every ARM operation against the app with timestamps, initiator, and status. Scope the Resource chip to the app and set Timespan to the swap window, then read Event initiated by to see which principal triggered each operation when correlating against the sticky-versus-non-sticky settings table.

3.10.1 HTTP request evidence¶

AppServiceHTTPLogs
| where TimeGenerated > ago(2h)
| where CsHost has "app-labswap"
| project TimeGenerated, CsUriStem, ScStatus, TimeTaken, CsHost
| order by TimeGenerated desc

3.10.2 Console startup evidence¶

AppServiceConsoleLogs
| where TimeGenerated > ago(2h)
| where ResultDescription has_any ("Starting gunicorn", "Booting worker", "App path is set", "Oryx", "Listening at")
| project TimeGenerated, ResultDescription
| order by TimeGenerated desc

3.10.3 Platform lifecycle evidence¶

AppServicePlatformLogs
| where TimeGenerated > ago(2h)
| where Message has_any ("WarmUpProbeSucceeded", "Site startup probe succeeded", "Site started", "StoppingSite", "CreatingContainer", "PullingImage")
| project TimeGenerated, Level, Message
| order by TimeGenerated desc

3.11 Real output snippets from this lab¶

Platform warm-up progression:

2026-04-04T05:49:05.8804667Z  State: Starting, Action: WarmUpProbeSucceeded ... Site startup probe succeeded after 55.2013831 seconds.
2026-04-04T05:49:06.6769441Z  Site started.

Console startup progression:

[2026-04-04 05:49:04 +0000] [1895] [INFO] Starting gunicorn 25.3.0
[2026-04-04 05:49:04 +0000] [1896] [INFO] Booting worker with pid: 1896
[2026-04-04 05:49:04 +0000] [1897] [INFO] Booting worker with pid: 1897

HTTP verification calls:

2026-04-04T05:51:21.610129Z  /config      200   47
2026-04-04T05:51:22.477189Z  /config      200   14
2026-04-04T05:51:24.143418Z  /diag/slots  200   11

3.12 Operator checklist¶

3.13 Decision tree for incident responders¶

flowchart TD
    A[Post-swap config surprise] --> B{Is setting sticky?}
    B -->|Yes| C[Expected to remain with slot]
    B -->|No| D[Expected to move with swapped context]
    C --> E{Observed unchanged?}
    D --> F{Observed exchanged?}
    E -->|Yes| G[Behavior correct]
    E -->|No| H[Investigate setting metadata]
    F -->|Yes| G
    F -->|No| H
    G --> I[Check warm-up and HTTP health]
    H --> J[Review slotConfigNames and deployment pipeline]

4) Experiment Log¶

All data below is sourced from:

labs/slot-swap-config-drift/artifacts-sanitized/

4.1 Artifact inventory¶

4.2 Baseline evidence¶

4.2.1 Baseline config snapshots¶

Production baseline:

{"DB_CONNECTION_STRING":"prod-server.database.windows.net","FEATURE_FLAG":"v1","PROCESS_START_UTC":"2026-04-04T05:13:59.196136+00:00","WEBSITE_SLOT_NAME":"Production"}

Staging baseline:

{"DB_CONNECTION_STRING":"staging-server.database.windows.net","FEATURE_FLAG":"v2","PROCESS_START_UTC":"2026-04-04T05:34:01.911025+00:00","WEBSITE_SLOT_NAME":"Production"}

4.2.2 Baseline app setting metadata¶

Production (baseline/app-settings-prod.json):

Staging (baseline/app-settings-staging.json):

4.2.3 Baseline diagnostics¶

baseline/diag-stats-prod.json:

{"endpoint_counters":{"<unknown>":1,"diag_env":1,"diag_slots":1,"diag_stats":1,"index":1},"last_config_snapshot":null,"pid":1898,"process_start_time":"2026-04-04T05:13:59.135627+00:00","request_count":6,"uptime_seconds":1212.651}

baseline/diag-env-prod.json highlights:

4.3 Trigger pre-swap captures¶

From trigger artifacts:

4.4 Trigger post-swap captures¶

From trigger artifacts:

4.5 Assertion table: expected vs observed¶

4.6 Programmatic summary checks (artifact-derived)¶

These checks produce a deterministic proof of sticky/non-sticky mechanics.

4.7 Process restart evidence¶

Using timestamp differences from before/after JSON:

This indicates new process generations became active during swap lifecycle.

4.7.1 Sticky/non-sticky configuration (portal verification)¶

To confirm which app settings are eligible to move with the swap (and therefore which can drive the process regeneration observed in the timestamp deltas above), the Environment variables blade was captured against the production slot from a fresh live reproduction:

Purpose: Cross-verify visually that the Portal control-plane marks DB_CONNECTION_STRING as slot-sticky and FEATURE_FLAG and SCM_DO_BUILD_DURING_DEPLOYMENT as non-sticky. This complements the runtime timestamp deltas in the table above by showing the platform-level precondition consistent with the swap-driven restart interpretation used here: only non-sticky app settings move to the destination slot on swap, and only settings whose value actually changes on swap trigger a process regeneration.

Look for:

• Blade heading reads "app-labslot-x6ybh6bakgipy | Environment variables" with the App settings tab active.
• DB_CONNECTION_STRING row: Deployment slot setting checkbox CHECKED.
• FEATURE_FLAG row: Deployment slot setting checkbox UNCHECKED.
• SCM_DO_BUILD_DURING_DEPLOYMENT row: Deployment slot setting checkbox UNCHECKED.
• Apply and Discard buttons at the bottom are both disabled (no pending edits, so the displayed state equals the persisted state).

Expected result: The sticky checkbox on DB_CONNECTION_STRING matches the slotConfigNames.appSettingNames array in main.bicep (only DB_CONNECTION_STRING is registered as sticky). If the Portal instead showed FEATURE_FLAG as sticky or DB_CONNECTION_STRING as non-sticky, the runtime behavior in section 4.5 (FEATURE_FLAG swapped, DB stayed) would not reproduce, and the process-restart deltas above would need to be attributed to an unrelated cause such as a worker recycle.

Next step: Cross-check this Portal state against section 4.6's programmatic assertions (feature_swapped = True, db_sticky = True) — the two evidence layers should agree. If they disagree (Portal shows one contract, runtime observes another), suspect a race between Bicep re-deployment and the swap trigger, and re-verify by running az webapp config appsettings list --slot production against the parent site resource.

4.8 /diag/slots snapshots and config hash evolution¶

Pre-swap production (baseline/diag-slots-prod.json):

{"config_hash":"027d574d88e7b2722dc7da98bc5d1bb337a75f1331bb795d2a83aba6ce291fd7","current_config":{"DB_CONNECTION_STRING":"prod-server.database.windows.net","FEATURE_FLAG":"v1","PROCESS_START_UTC":"2026-04-04T05:13:59.196136+00:00","WEBSITE_SLOT_NAME":"Production"}}

Pre-swap staging (baseline/diag-slots-staging.json):

{"config_hash":"7f5ed03ca7b0fdd301f27ed036155631ca62ffce8a18b4fe5bce0235f02216bc","current_config":{"DB_CONNECTION_STRING":"staging-server.database.windows.net","FEATURE_FLAG":"v2","PROCESS_START_UTC":"2026-04-04T05:34:02.102318+00:00","WEBSITE_SLOT_NAME":"Production"}}

Post-swap sample (trigger/diag-slots-postswap-20260404T055128Z.json):

{"config_hash":"4069415d1d95ca3d38f723e476f8956726634a0b7fddf338499168a7d7198b3d","current_config":{"DB_CONNECTION_STRING":"prod-server.database.windows.net","FEATURE_FLAG":"v2","PROCESS_START_UTC":"2026-04-04T05:46:57.025552+00:00","WEBSITE_SLOT_NAME":"Production"}}

4.8.1 Swap operation activity log (portal verification)¶

To confirm that the config hash transition captured in the JSON snapshots above is correlated with a distinct ARM control-plane action, the Activity log blade was captured and the "Swap Web App Slots" row was expanded from a fresh live reproduction:

Purpose: Provide an independent ARM control-plane record that the swap operation whose runtime effects are captured in the JSON snapshots above actually completed as a discrete action, correlated in time with the config hash transition from 027d… (pre-swap) to 4069… (post-swap). The Activity log and the /diag/slots runtime endpoint measure different layers (ARM operation acknowledgement vs. data-plane environment observation), so this capture cross-verifies for existence and time-window alignment — not for exact timestamp equality with PROCESS_START_UTC.

Look for:

• Blade heading reads "app-labslot-x6ybh6bakgipy | Activity log".
• The grid contains a row with Operation name "Swap Web App Slots".
• The selected "Swap Web App Slots" row shows Status "Succeeded".
• Detail panel Resource path ends in /slots/staging — the swap operation is scoped to the source slot per ARM semantics.
• Detail panel Time stamp falls in the same window as the PROCESS_START_UTC regeneration observed in section 4.7's timestamp deltas.
• Detail panel Event initiated by shows a real identity (sanitized in this capture to user@example.com).

Expected result: Exactly one "Swap Web App Slots" row per swap trigger, with a Succeeded status. The detail panel time stamp is consistent with (not necessarily identical to) the process-restart deltas in section 4.7 — the ARM operation completes when the platform accepts the swap, while the runtime PROCESS_START_UTC advances when the destination worker observes the new environment and re-initializes.

Next step: If the Activity log shows the swap as Failed or Cancelled while section 4.6's feature_swapped = True still holds, treat it as a control-plane vs. data-plane inconsistency — capture the JSON tab of the failed operation record and open a support case referencing both the Activity log JSON and the /diag/slots config hash transition so the platform team can reconcile the two evidence layers.

4.9 KQL export volume summary¶

4.10 HTTP evidence details¶

Representative rows from HTTP export:

Observation: both slots remained reachable and healthy during/after swap evidence window.

4.11 Console evidence details¶

Representative console lines:

Observation: process lifecycle aligns with swap-time startup sequence.

4.12 Platform lifecycle evidence details¶

Representative platform lines:

These lines validate that the swap involved observable start/stop/warm-up transitions.

4.13 Hypothesis evaluation matrix¶

Final hypothesis verdict: Supported.

4.14 Key finding (required conclusion)¶

4.15 Practical lessons for release engineering¶

1. Treat sticky/non-sticky classification as release-critical metadata.
2. Include pre/post slot snapshots in every production swap runbook.
3. Align incident response language with slot semantics ("expected movement" vs "drift").
4. Use KQL lifecycle evidence to distinguish swap mechanics from app defects.

4.16 Recommended production controls¶

4.17 Pre-swap checklist template¶

4.18 Post-swap checklist template¶

4.19 Reproducibility and data integrity notes¶

• Every numeric and string value in this section is sourced from checked-in sanitized artifacts.
• Hostnames and subscription identifiers are redacted where required.
• No placeholders were used for experimental result tables.

4.20 Limitations and scope¶

This lab validates swap/config semantics for this specific setup:

• Linux App Service, Python runtime.
• S1 plan, single app with one staging slot.
• Two controlled app settings.

It does not attempt to benchmark multi-slot, multi-instance, or multi-region swap complexity.

Expected Evidence¶

This section defines what you SHOULD observe at each phase of the lab. Use it to validate your investigation is on track.

Before Trigger (Baseline)¶

During Incident¶

After Recovery¶

Evidence Timeline¶

graph TD
    A[Baseline Capture] --> B[Trigger Fault]
    B --> C[During: Collect Evidence]
    C --> D[After: Compare to Baseline]
    D --> E[Verdict: Confirmed/Falsified]

Evidence Chain: Why This Proves the Hypothesis¶

Clean Up¶

az group delete --name "$RG" --yes --no-wait

Related Playbook¶

• Slot Swap Restart / Config Drift / Warm-up Race

See Also¶

• Playbook: Slot Swap Restart / Config Drift / Warm-up Race
• Playbook: Slot Swap Failed During Warm-up
• Playbook: Warm-up vs Health Check
• KQL: Restart Timing Correlation
• Troubleshooting Method

Sources¶

• Set up staging environments in Azure App Service
• Configure an App Service app in Azure App Service
• Enable diagnostic logging for apps in Azure App Service
• Monitor Azure App Service
• App Service plan overview