Production ARM Configuration Update Returns HTTP 400¶

1. Summary¶

Use this playbook when an App Service configuration update aimed at production returns HTTP 400, while a seemingly comparable update to a staging slot succeeds. It belongs at docs/troubleshooting/playbooks/arm-configuration-http-400.md because the symptom is a management-plane configuration failure with several competing hypotheses, not a deployment procedure or an application-response problem.

The successful slot request is a comparison point, not proof that the production payload, target, authorization context, or current resource state is valid. Do not classify the difference as an App Service platform defect until the exact requests and resource state have been normalized and every supported discriminator below is exhausted.

Verification scope (2026-10-14): Azure CLI command syntax was checked with Azure CLI 2.89.0 (az ... --help), using the Windows-hosted CLI from Ubuntu 26.04 on WSL2. The JSON normalizer was executed with Python 3.12.14 on Ubuntu 26.04/WSL2 against synthetic files, including a nested name and type difference. No request was sent to an Azure resource, no configuration was changed, and reproduction against a failing resource is NOT_RUN.

flowchart TD
    A[HTTP 400 reported] --> B{Request host is ARM endpoint?}
    B -->|No, app hostname| X[Application HTTP 400<br/>use application logs and contract]
    B -->|Yes, management endpoint| C[Capture method, API version,<br/>target, payload, headers, and body]
    C --> D{Production and slot requests<br/>are structurally equivalent?}
    D -->|No| E[Test target, payload,<br/>identity, and API-version differences]
    D -->|Yes| F[GET production and staging state]
    E --> F
    F --> G[Normalize and diff properties]
    G --> H[Test each hypothesis separately]
    H --> I{Evidence identifies a cause?}
    I -->|Yes| J[Apply approved minimum fix<br/>with rollback plan]
    I -->|No| K[INCONCLUSIVE<br/>preserve evidence and escalate]

2. Common Misreadings¶

3. Competing Hypotheses¶

4. What to Check First¶

4.1 Separate ARM from application traffic¶

Record the full destination host before interpreting the status:

• ARM/control plane: the Azure environment's Resource Manager endpoint. In Azure global, this is management.azure.com, and the path contains /subscriptions/.../providers/Microsoft.Web/....
• Application traffic: the app's default/custom hostname, API gateway, Front Door endpoint, or another data-plane route.

An ARM error normally has a top-level error object with fields such as code, message, target, and details. Do not rely on shape alone: proxies can rewrite responses, so host, method, and URI are the stronger boundary.

Decision:

• If the captured request went to an app hostname, stop this playbook and investigate the application's route, validation, and logs.
• If it went to the Resource Manager endpoint, continue with the exact operation.
• [Unknown] If the host or request trace is unavailable, label the origin INCONCLUSIVE; do not call it ARM validation.

4.2 Build the immutable request record¶

Capture the failing production request and the successful slot request independently. Do not reconstruct one from memory.

The App Service 2024-11-01 REST reference documents these site-configuration targets:

PATCH https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/rg-example/providers/Microsoft.Web/sites/app-example/config/web?api-version=2024-11-01

PATCH https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/rg-example/providers/Microsoft.Web/sites/app-example/slots/staging/config/web?api-version=2024-11-01

The first path targets production. The second targets staging. Keep the method and API version with the path; path-only comparison is incomplete.

4.3 Capture the error and management timeline¶

Preserve error.code, error.message, error.target, every error.details[] entry, and innererror if returned. Treat the following as a synthetic shape, not an observed response:

{
  "error": {
    "code": "<arm-error-code>",
    "message": "<sanitized-message>",
    "target": "<property-if-returned>",
    "details": [
      {
        "code": "<detail-code>",
        "message": "<sanitized-detail>",
        "target": "<property-if-returned>"
      }
    ]
  }
}

Query management activity around the failure. Substitute a narrow UTC interval; do not publish the returned caller identity or raw resource ID.

az monitor activity-log list \
    --resource-group $RG \
    --start-time 2026-10-14T08:00:00Z \
    --end-time 2026-10-14T08:15:00Z \
    --query "[?contains(resourceId, '/providers/Microsoft.Web/sites/')].{time:eventTimestamp,operation:operationName.value,status:status.value,correlationId:correlationId,resourceId:resourceId}" \
    --output json

If the failed response supplied a correlation ID, use the Activity Log correlation-ID filter to narrow the same time window. A matching event strengthens the request-to-operation link; no match is not proof that the request bypassed ARM.

4.4 Record concurrency context¶

Record whether the failing request supplied If-Match, what ETag or revision the client read before writing, and whether another deployment, portal save, IaC run, policy remediation, or automation job overlapped. Fetch current state again before any approved retry. Do not invent an ETag requirement when the specific App Service operation or client did not use one.

5. Evidence to Collect¶

5.1 Read production and staging configuration¶

These reads intentionally keep the two targets explicit:

az webapp config show \
    --resource-group $RG \
    --name $APP_NAME \
    --output json > /tmp/production-config.json

az webapp config show \
    --resource-group $RG \
    --name $APP_NAME \
    --slot staging \
    --output json > /tmp/staging-config.json

App settings are a secret-bearing surface. The commands below project setting names inside Azure CLI before writing output; they never print or persist setting values. Compare this name inventory first. If an investigation requires value hashes, use an approved secret-handling tool and a mode-600 output inside a mode-700 temporary directory with automatic cleanup. Never print values or write raw app-setting output to an ordinary /tmp file.

az webapp config appsettings list \
    --resource-group $RG \
    --name $APP_NAME \
    --query "[].name" \
    --output json > /tmp/production-appsetting-names.json

az webapp config appsettings list \
    --resource-group $RG \
    --name $APP_NAME \
    --slot staging \
    --query "[].name" \
    --output json > /tmp/staging-appsetting-names.json

Use REST GETs when the failing writer used a raw REST target, so comparison uses the same API version and resource shape:

az rest \
    --method get \
    --url "https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/rg-example/providers/Microsoft.Web/sites/app-example/config/web?api-version=2024-11-01" \
    --output json > /tmp/production-rest.json

az rest \
    --method get \
    --url "https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/rg-example/providers/Microsoft.Web/sites/app-example/slots/staging/config/web?api-version=2024-11-01" \
    --output json > /tmp/staging-rest.json

5.2 Normalize property-level differences¶

The following illustrative, standard-library-only Python 3.11+ script recursively sorts object keys and removes resource identity or commonly volatile metadata only at the response root. Nested name and type fields remain available for comparison. It does not know which arrays are semantically unordered and does not redact secrets. Use it only on protected site-configuration JSON, review the root-level ignored fields for the API shape, and never feed raw app-setting values into a shared artifact.

import json
import sys
from pathlib import Path
from typing import TypeAlias, assert_never

JSON: TypeAlias = None | bool | int | float | str | list["JSON"] | dict[str, "JSON"]

ROOT_IGNORED_KEYS = frozenset(
    {
        "etag",
        "id",
        "lastModifiedTimeUtc",
        "name",
        "resourceGroup",
        "systemData",
        "type",
    }
)


def normalize(value: JSON, path: tuple[str | int, ...] = ()) -> JSON:
    match value:
        case dict() as mapping:
            return {
                key: normalize(item, (*path, key))
                for key, item in sorted(mapping.items())
                if path or key not in ROOT_IGNORED_KEYS
            }
        case list() as items:
            return [normalize(item, (*path, index)) for index, item in enumerate(items)]
        case str() | int() | float() | bool() | None:
            return value
        case unreachable:
            assert_never(unreachable)


for filename in sys.argv[1:]:
    document: JSON = json.loads(Path(filename).read_text(encoding="utf-8"))
    output = Path(filename).with_suffix(".normalized.json")
    output.write_text(
        json.dumps(normalize(document), indent=2, sort_keys=True) + "\n",
        encoding="utf-8",
    )

Run it locally, then use a normal text diff:

python3 /tmp/normalize-appservice-config.py \
    /tmp/production-config.json \
    /tmp/staging-config.json

diff --unified \
    /tmp/production-config.normalized.json \
    /tmp/staging-config.normalized.json

Interpret differences one property at a time. A difference is correlated evidence, not proof of rejection. Compare app-setting names, sticky-setting declarations, and redacted value hashes separately because ordinary configuration GET responses can omit secret-bearing properties.

5.3 Synthetic request cases¶

These payloads are documentation fixtures. They were not sent to Azure, and no response is claimed.

Synthetic valid-shape request: a documented property with the production resource target.

PATCH https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/rg-example/providers/Microsoft.Web/sites/app-example/config/web?api-version=2024-11-01
Content-Type: application/json

{
  "properties": {
    "alwaysOn": true
  }
}

Whether the property is available depends on the target's plan and platform context. “Valid-shape” does not promise the request will succeed on every app.

Synthetic invalid-property request: a deliberately undocumented property used only to illustrate provider validation.

PATCH https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/rg-example/providers/Microsoft.Web/sites/app-example/config/web?api-version=2024-11-01
Content-Type: application/json

{
  "properties": {
    "notARealSiteConfigProperty": true
  }
}

Do not execute this request. Its exact response is intentionally unspecified.

Synthetic wrong-target request: the intent is staging, but the path omits /slots/staging and therefore addresses production.

PATCH https://management.azure.com/subscriptions/<subscription-id>/resourceGroups/rg-example/providers/Microsoft.Web/sites/app-example/config/web?api-version=2024-11-01
Content-Type: application/json

{
  "properties": {
    "alwaysOn": true
  }
}

The corresponding staging target would be /sites/app-example/slots/staging/config/web. A syntactically valid wrong-target request might succeed, fail, or change the wrong resource; URI review must happen before any controlled mutation.

6. Validation and Disproof by Hypothesis¶

H1: Application response mistaken for ARM validation¶

Supports: request host is the app/custom hostname, or the trace shows the request reached application middleware or a route.

Weakens: host is the Resource Manager endpoint, target is a Microsoft.Web resource path, and the response carries an ARM error envelope plus management correlation identifiers.

Inconclusive: only a screenshot saying “400” survives.

H2: Invalid, immutable, or unsupported property¶

1. Compare the failing serialized payload against the request-body schema for the exact API version.
2. Remove response-only metadata from comparison, but do not silently remove a field from a proposed retry.
3. Map error.details[].target to the normalized property diff.
4. If one property is suspected, seek a documented validation mechanism. Do not probe production by repeatedly deleting fields from live writes.

Supports: provider detail names the field, type, allowed value, or read-only behavior.

Weakens: the exact field/value succeeds against the same target and current state in an authorized, approved test while all other request dimensions are held constant.

H3: Site-versus-slot targeting¶

Compare the complete paths:

• Production: /providers/Microsoft.Web/sites/{name}/config/web
• Slot: /providers/Microsoft.Web/sites/{name}/slots/{slot}/config/web

Also compare the client command's --slot argument, SDK resource ID, and pipeline variable expansion. A staging success does not validate a production path, and a production path without /slots/{slot} is not an alias for staging.

Supports: intent and resource path disagree.

Disproves: both recorded paths match their intended targets, and the failure remains after correcting no target variable.

H4: App-setting and sticky-setting treatment¶

Site configuration, app settings, and slot stickiness are related but distinct surfaces. Check:

1. Whether the client intended /config/web or an app-settings operation.
2. Production and staging app-setting names, with values redacted or compared by protected hash.
3. Which names are deployment-slot settings and therefore remain pinned during swaps.
4. Whether a full-object update accidentally included app settings or connection strings that normal configuration GETs do not return.

Supports: the request used the wrong subresource or the only meaningful difference is a sticky-setting declaration directly involved in the operation.

Inconclusive: secrets were omitted from both reads and the original serialized request is unavailable.

H5: RBAC scope or identity¶

Record the principal used by each request. Compare effective permissions at the exact production and slot scopes, including group membership, deny assignments, and conditions. See Security Operations for access-management safeguards.

Supports: the actual error or Activity Log shows authorization denial for the production action and the principal lacks that action at production scope.

Weakens: the same principal has the required action at the exact target and no deny applies. A plain unexplained 400 alone does not support RBAC.

H6: Locks or policy¶

Inspect locks inherited from subscription/resource-group scope and policy evidence without removing either control. A ReadOnly lock prevents updates even for otherwise authorized users. Resource Manager-mode Azure Policy deny normally returns 403, so preserve the actual status rather than relabeling it.

Supports: the response or Activity Log names a lock/policy, or read-only inventory shows an applicable inherited control.

Weakens: no applicable control exists and the provider error identifies payload validation instead.

H7: Feature, plan, OS, or runtime compatibility¶

Use the normalized GETs to compare only properties relevant to the failing field: Linux versus Windows runtime fields, plan-dependent options, runtime version, or feature enablement. Then verify that combination against the exact App Service REST/API or feature documentation.

Supports: provider detail states the target combination is unsupported, and the production/staging state difference matches that constraint.

Inconclusive: the environments differ but no documentation or provider detail connects the difference to rejection.

H8: Concurrent updates or stale ETag context¶

Build a timeline from client logs and Activity Log. Identify overlapping portal saves, deployments, IaC runs, policy remediation, and automation. Compare any If-Match value sent by the failed request with the ETag from the read that preceded it and with a fresh GET.

Supports: a conditional-write failure or operation timeline proves the request used stale state or overlapped another writer.

Weakens: no conditional header was used, no other write overlaps, and the ARM error names a deterministic field validation problem.

7. Likely Root Cause Patterns¶

If none of these patterns has direct evidence, the result remains INCONCLUSIVE. Preserve both request records, normalized diffs, Activity Log export, and exact UTC interval for escalation. Production-versus-staging difference alone is not evidence of a platform defect.

8. Immediate Mitigations¶

1. Stop automatic retries and overlapping configuration writers.
2. Keep production unchanged while collecting read-only GETs and Activity Log evidence.
3. If production is healthy, leave the rejected value unapplied until the cause is identified.
4. If an earlier approved change caused impact, use the team's known-good declarative configuration or documented rollback process; do not copy staging wholesale into production.
5. Escalate with the sanitized request pair, ARM error envelope, UTC timestamps, correlation identifiers, Activity Log evidence, normalized diff, and an explicit list of hypotheses that remain inconclusive.

Safeguards for any controlled mutation¶

A mutation is outside the read-only diagnosis and requires all of the following before execution:

• Named authorization from the production owner and an approved change window.
• Exact target URI, method, API version, minimal payload, calling identity, and expected impact peer-reviewed.
• Fresh pre-change GET stored securely, including concurrency context supported by the client.
• Rollback payload or last-known-good declarative source prepared and independently reviewed.
• Health, restart, and representative business-request checks defined before the write.
• One writer only; automation and portal edits paused or coordinated.
• Stop conditions defined. An unexpected response means capture evidence and stop, not field-by-field retries.

Never use the synthetic invalid-property or wrong-target examples as mutation tests.

9. Prevention¶

• [ ] Log method, sanitized ARM path, API version, principal, payload hash, client request ID, response status, ARM error fields, and returned correlation identifiers for every configuration write.
• [ ] Generate production and slot resource IDs from typed deployment inputs; test that slot intent inserts /slots/{slot}.
• [ ] Send minimal documented PATCH bodies instead of serializing a prior GET response.
• [ ] Separate site configuration, app settings, connection strings, and slot-stickiness workflows.
• [ ] Serialize portal, IaC, pipeline, and policy-remediation writers or adopt supported conditional-update behavior.
• [ ] Add a pre-production check that reads and normalizes both targets without treating their difference as a failure by itself.
• [ ] Keep authorization, lock, and policy checks in the release preflight without weakening those controls.
• [ ] Retain a redacted diagnostic bundle template that preserves INCONCLUSIVE as a valid result.

See Also¶

• Deployment Failures
• Security Operations
• Deployment Slots Operations
• Playbooks

Sources¶

• Control plane and data plane operations (Microsoft Learn)
• Web Apps - Update Configuration, API 2024-11-01 (Microsoft Learn)
• Web Apps - Update Configuration Slot, API 2024-11-01 (Microsoft Learn)
• Deployments - Validate REST example with ARM correlation and request headers (Microsoft Learn)
• Activity Log in Azure Monitor (Microsoft Learn)
• Lock Azure resources (Microsoft Learn)
• Azure Policy deny effect (Microsoft Learn)
• Azure role-based access control overview (Microsoft Learn)
• Set up staging environments in Azure App Service (Microsoft Learn)