Observability and Distributed Tracing Lab¶

Troubleshoot Application Insights connectivity issues by simulating a misconfigured telemetry connection string.

Lab Metadata¶

1) Background¶

This lab starts with working observability: the Container Apps environment sends logs to Log Analytics, the app has APPLICATIONINSIGHTS_CONNECTION_STRING configured through a secret reference, and Application Insights receives telemetry. The trigger replaces that working configuration with an invalid literal connection string, causing telemetry export to fail.

The main troubleshooting pattern is to compare the app's environment variable configuration with the expected secret reference, then validate telemetry absence in Application Insights and Log Analytics.

Architecture¶

flowchart TD
    A[Container App] --> B[APPLICATIONINSIGHTS_CONNECTION_STRING]
    B -->|Valid secretRef| C[Application Insights]
    A --> D[Container Apps Environment]
    D --> E[Log Analytics Workspace]
    D --> F[daprAIConnectionString]
    B -->|Invalid literal value| G[Telemetry dropped]

Telemetry Flow in Container Apps¶

2) Hypothesis¶

IF APPLICATIONINSIGHTS_CONNECTION_STRING is replaced with an invalid literal value instead of the working secret reference, THEN the Container App's env var source flips from "Reference a secret" to "Manual entry" with an invalid literal value (the directly-falsifiable half of the hypothesis), and any application that emits Application Insights telemetry through this env var will stop reaching the real Application Insights resource until the valid secret-backed configuration is restored.

3) Runbook¶

Deploy baseline infrastructure¶

Prerequisites:

• Azure CLI with the Container Apps extension
• Basic understanding of Application Insights concepts

az extension add --name containerapp --upgrade
az login

export RG="rg-aca-lab-observability"
export LOCATION="koreacentral"

az group create --name "$RG" --location "$LOCATION"

az deployment group create \
    --name "lab-obs" \
    --resource-group "$RG" \
    --template-file "./labs/observability-tracing/infra/main.bicep" \
    --parameters baseName="labobs"

Expected output:

• Resource group creation succeeds.
• Deployment creates a Container App, Container Apps environment, Application Insights component, and Log Analytics workspace.

Capture deployment outputs¶

export APP_NAME="$(az deployment group show \
    --resource-group "$RG" \
    --name "lab-obs" \
    --query "properties.outputs.containerAppName.value" \
    --output tsv)"

export ACA_ENV_NAME="$(az deployment group show \
    --resource-group "$RG" \
    --name "lab-obs" \
    --query "properties.outputs.containerAppsEnvironmentName.value" \
    --output tsv)"

export APPINSIGHTS_NAME="$(az deployment group show \
    --resource-group "$RG" \
    --name "lab-obs" \
    --query "properties.outputs.appInsightsName.value" \
    --output tsv)"

export LOG_ANALYTICS_WORKSPACE_NAME="$(az deployment group show \
    --resource-group "$RG" \
    --name "lab-obs" \
    --query "properties.outputs.logAnalyticsWorkspaceName.value" \
    --output tsv)"

Expected output:

• Commands return no console output.
• Variables resolve to the app, environment, Application Insights, and workspace names.

Verify baseline observability¶

az containerapp show \
    --name "$APP_NAME" \
    --resource-group "$RG" \
    --query "properties.template.containers[0].env[?name=='APPLICATIONINSIGHTS_CONNECTION_STRING']" \
    --output table

Expected output:

• The app shows secretRef: appinsights-connection-string for APPLICATIONINSIGHTS_CONNECTION_STRING.

Trigger the failure¶

./labs/observability-tracing/trigger.sh

The trigger applies this misconfiguration:

az containerapp update \
    --resource-group "$RG" \
    --name "$APP_NAME" \
    --set-env-vars "APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://invalid/"

Expected output:

• The script prints that telemetry settings were misconfigured.
• The Container App now uses an invalid literal connection string.

Observe the broken state¶

az containerapp show \
    --name "$APP_NAME" \
    --resource-group "$RG" \
    --query "properties.template.containers[0].env[?name=='APPLICATIONINSIGHTS_CONNECTION_STRING']" \
    --output json

APPINSIGHTS_ID="$(az monitor app-insights component show \
    --app "$APPINSIGHTS_NAME" \
    --resource-group "$RG" \
    --query "appId" \
    --output tsv)"

az monitor app-insights query \
    --app "$APPINSIGHTS_ID" \
    --analytics-query "requests | where timestamp > ago(5m) | count"

Expected output:

• The env var now shows a literal invalid value instead of a secret reference.
• Recent Application Insights queries are empty or stale when the application image is SDK-instrumented. With the baseline azuredocs/containerapps-helloworld:latest image, App Insights queries will also be empty in the baseline state (see the [Not Proven] caveat in the Hypothesis).

Diagnose with additional evidence and restore the valid configuration¶

Useful debugging commands:

az containerapp env show --name "$ACA_ENV_NAME" --resource-group "$RG" --query "properties.daprAIConnectionString"
az containerapp logs show --name "$APP_NAME" --resource-group "$RG" --type console --tail 50

WORKSPACE_ID=$(az monitor log-analytics workspace show \
    --resource-group "$RG" \
    --workspace-name "$LOG_ANALYTICS_WORKSPACE_NAME" \
    --query customerId \
    --output tsv)

az monitor log-analytics query \
    --workspace "$WORKSPACE_ID" \
    --analytics-query "union isfuzzy=true AppTraces, traces | where TimeGenerated > ago(15m) | summarize count()"

Restore the valid connection string using the original secret reference:

APPINSIGHTS_CONNECTION_STRING="$(az monitor app-insights component show \
    --app "$APPINSIGHTS_NAME" \
    --resource-group "$RG" \
    --query "connectionString" \
    --output tsv)"

az containerapp update \
    --name "$APP_NAME" \
    --resource-group "$RG" \
    --set-env-vars "APPLICATIONINSIGHTS_CONNECTION_STRING=secretref:appinsights-connection-string"

Expected output:

• The environment-level daprAIConnectionString remains configured.
• The app env var returns to the secret reference.
• Telemetry resumes after the new revision is applied when the application image is SDK-instrumented (see the [Not Proven] caveat in the Hypothesis).

Verify recovery¶

./labs/observability-tracing/verify.sh

Expected output (with the baseline azuredocs/containerapps-helloworld:latest image):

• PASS: Application Insights connection string is configured on $APP_NAME. — this row is directly falsifiable from the Container App template and corresponds to the env-var half of the hypothesis.
• The trace-record check is conditional: PASS: Found <count> trace record(s) in Log Analytics. is expected only when the application image is SDK-instrumented. With the baseline helloworld image (no Application Insights SDK), the trace count is zero in both the baseline and the misconfigured state, so this row may report zero records and the script may not reach PASS on it — see the [Not Proven] caveat in the Hypothesis.
• Verification complete.

4) Experiment Log¶

Expected Evidence¶

Before trigger¶

During incident¶

After fix¶

Observed Evidence (Live Azure Reproduction — 2026-06-03)¶

Reproduced end-to-end in koreacentral against resource group rg-aca-lab-observability using Azure CLI 2.71.0. The Container App revisions advanced as follows:

CLI evidence (with PII redacted to placeholder GUIDs and lab-owned resource names):

# Baseline — secret-backed configuration
az containerapp show --name ca-labobs-622oal --resource-group rg-aca-lab-observability \
  --query "properties.template.containers[0].env[?name=='APPLICATIONINSIGHTS_CONNECTION_STRING']"
→ [{ "name": "APPLICATIONINSIGHTS_CONNECTION_STRING",
     "secretRef": "appinsights-connection-string" }]

# After ./labs/observability-tracing/trigger.sh — literal invalid string applied
az containerapp show --name ca-labobs-622oal --resource-group rg-aca-lab-observability \
  --query "properties.template.containers[0].env[?name=='APPLICATIONINSIGHTS_CONNECTION_STRING']"
→ [{ "name": "APPLICATIONINSIGHTS_CONNECTION_STRING",
     "value": "InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://invalid/" }]

# After restore — secretRef back in place on revision 0000002
az containerapp update --name ca-labobs-622oal --resource-group rg-aca-lab-observability \
  --set-env-vars "APPLICATIONINSIGHTS_CONNECTION_STRING=secretref:appinsights-connection-string"
→ [{ "name": "APPLICATIONINSIGHTS_CONNECTION_STRING",
     "secretRef": "appinsights-connection-string" }]

Each Portal capture below documents one observable fact. [Observed] paragraphs cite only what is visible in the screenshot at capture time. [Inferred] paragraphs connect the captures across revisions to support or falsify the hypothesis.

Capture 1 — Container App Overview¶

[Observed] The Overview blade for ca-labobs-622oal shows the Essentials panel with Status: Running, Location: Korea Central, Environment type: Workload profiles, an Application Url link pointing to https://ca-labobs-622oal.jollytree-c58df5c9.koreacentral.azurecontaine..., and Tags lab: observability-tracing and cloud: AzureCloud at capture time.

[Inferred] A Status: Running overview at this moment is consistent with the app being deployed and provisioned. It does not on its own prove the app is currently serving requests, nor that telemetry is or is not being emitted — those facts must come from the env var blade (Captures 2, 3, 6) and the App Insights blades (Captures 4, 5).

Capture 2 — Baseline environment variables (secret-backed)¶

[Observed] The Containers → Environment variables tab, with the "Based on revision" selector at the top of the blade reading ca-labobs-622oal--0m6ek7p, shows a row for APPLICATIONINSIGHTS_CONNECTION_STRING with Source = "Reference a secret" and Value = "appinsights-connection-stri..." (truncated by the table cell).

[Inferred] This matches the documented control state: the connection string is injected from the secret store, so the runtime resolves it to the real Application Insights connection string at container start.

Capture 3 — Misconfigured environment variables (literal invalid value)¶

[Observed] The same Containers → Environment variables tab, with the "Based on revision" selector now reading ca-labobs-622oal--0000001, shows the same APPLICATIONINSIGHTS_CONNECTION_STRING row with Source = "Manual entry" and the Value textarea showing the literal string InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint (the rest of the literal value is clipped by the textarea at capture time and is not legible in the screenshot). A second row, OTEL_SERVICE_NAME, also shows Source = "Manual entry" with Value = "ca-labobs-622oal".

[Inferred] Comparing this capture to Capture 2 falsifies any claim that the secret store was modified: only the env var's Source changed from "Reference a secret" to "Manual entry", on a different revision ID. The misconfiguration is at the Container App template layer, not the Container Apps environment secret layer.

Capture 4 — Application Insights Transaction search blade (no executed result)¶

[Observed] The Transaction search blade for the lab's Application Insights resource (appi-labobs-622oal) shows the filter chips Local Time: Last 24 hours (Automatic), View as: Traces, and Event types = All selected; an empty search input placeholder reading "Place search terms here, e.g. a trace ID or log text (optional)"; and a blue prompt button labeled "See all data in the last 24 hours" with the text "or" / "Visit our syntax query help page to get started" below it. No results table has been rendered at capture time.

[Inferred] Because no search has been executed, this capture documents only the configured filters (last 24 hours, Traces view) on the right Application Insights resource. It is consistent with — but does not by itself prove — telemetry absence; the executed traces | count measurement in Capture 5 is the stronger evidence, and the [Not Proven] caveat below still applies.

Capture 5 — Logs blade KQL traces | count returns 0¶

[Observed] On the Logs blade for appi-labobs-622oal, the query editor shows two lines (traces on line 1 and | count on line 2), the toolbar shows Time range: Last hour and Show: 500000 results, the Results tab shows a single column with header Count and one row containing the value 0, the footer reads 0s 881ms execution time and 1 - 1 of 1.

[Inferred] The zero count is consistent with telemetry export failing, but as noted in the Hypothesis caveat and Capture 4, the helloworld baseline image's lack of an Application Insights SDK means a zero count would also be observed in the baseline. This capture documents the measurable state during the incident; it does not by itself prove the connection-string change is responsible.

Capture 6 — Restored environment variables (secret-backed)¶

[Observed] The Containers → Environment variables tab, with the "Based on revision" selector now reading ca-labobs-622oal--0000002, shows the APPLICATIONINSIGHTS_CONNECTION_STRING row with Source = "Reference a secret" and Value = "appinsights-connection-stri...". A secondary OTEL_SERVICE_NAME row is also visible with Source = "Manual entry" and Value = "ca-labobs-622oal".

[Inferred] Captures 2 and 6 together (both showing the same Source / Value for the same env var name, on different revision IDs) falsify any claim that the misconfiguration is still in place. The misconfiguration was bounded to revision 0000001 and is no longer present on the active revision 0000002.

Falsification summary¶

• [Inferred] Three distinct revision IDs are visible across the env-var captures: 0m6ek7p in Capture 2, 0000001 in Capture 3, 0000002 in Capture 6. Each capture's [Observed] paragraph cites the revision selector value plus the Source / Value for APPLICATIONINSIGHTS_CONNECTION_STRING at capture time.
• [Inferred] The Reference a secret ↔ Manual entry flip on the same env var name across those three captures, bounded by trigger.sh and the restore command, is sufficient to confirm the configuration-change half of the hypothesis: the trigger does replace the working secret reference with an invalid literal, and the restore returns it to the baseline.
• [Not Proven] The telemetry-blocking half of the hypothesis (that the invalid connection string would actually drop traces that the SDK is emitting) cannot be empirically falsified by this reproduction, because the baseline azuredocs/containerapps-helloworld:latest image used by infra/main.bicep does not ship an Application Insights SDK. Captures 4 and 5 are therefore consistent with the hypothesis but not a smoking gun. To upgrade this to [Measured], swap the lab image for one of the reference apps under apps/ (e.g. apps/python/) instrumented with the OpenTelemetry Distro for Azure Monitor.

Environment: koreacentral, rg-aca-lab-observability, cae-labobs-622oal, appi-labobs-622oal, log-labobs-622oal.

Observed Evidence (Live Azure Test — 2026-06-24)¶

A second end-to-end live Azure run executed on 2026-06-24 with az CLI version 2.79.0 (containerapp extension 1.3.0b4; see labs/observability-tracing/evidence/21-cli-versions.json and 22-cli-containerapp-ext.json) against resource group rg-aca-lab-observability2 (subscription redacted to placeholder 00000000-0000-0000-0000-000000000000, Korea Central). The run produced 24 numbered evidence prefixes (01-* through 24-*, totaling 26 physical files when counting .stderr companions to 06-* and 15-*; prefixes 07-* and 16-* use a .log extension) under labs/observability-tracing/evidence/, covering H1 (telemetry-misconfiguration env-var Source flip) and H2 (env-var restoration with revision progression). All 11 sub-gates across both hypotheses passed.

H1 reproduction — evidence/12-h1-gate.json:

Gate classification: telemetry_misconfiguration_env_var_source_flipped_to_literal.

H2 restoration — evidence/24-h2-gate.json:

Gate classification: telemetry_configuration_restored_to_secretref_app_intact. The fix command was az containerapp update --set-env-vars APPLICATIONINSIGHTS_CONNECTION_STRING=secretref:appinsights-connection-string — a Container App template edit that mints a new revision.

Honest disclosure of platform behavior¶

Three platform-behavior observations from this run worth flagging to future maintainers:

1. Telemetry-blocking is [Not Proven] with the helloworld baseline image. The H1 and H2 gates intentionally restrict their falsifiable claims to the env-var Source/Value flip + revision progression + secret-store integrity + data-plane HTTP 200, all observable from the Container App template and revision list. The upstream claim "the misconfigured connection string actually drops SDK-emitted traces" remains [Not Proven] in this evidence pack because mcr.microsoft.com/azuredocs/containerapps-helloworld:latest ships no Application Insights SDK — App Insights / Log Analytics will report zero traces in ALL THREE states (baseline / post-trigger / post-fix) and a zero count is therefore consistent with both the hypothesis and the null hypothesis. A future evidence pack would need an SDK-instrumented baseline image (e.g. apps/python/ with azure-monitor-opentelemetry, or a .NET app with Microsoft.ApplicationInsights.AspNetCore) to lift this claim to [Observed]. The original 2026-06-03 reproduction recorded the same [Not Proven] caveat under the Hypothesis section above; this 2026-06-24 run preserves the same caveat.

2. Container Apps single-revision-mode revision history pruning. The post-fix revisions snapshot (evidence/18-post-fix-revisions.json) lists only ca-labobs-ewwmkr--0000003 and --0000004; the baseline --0000002 has been pruned from the revision list by the time the third revision is created. The H2 sub-gate f_revision_progression_documented therefore looks up each revision's createdTime from the snapshot taken when that revision was the latest — baseline from evidence/02-baseline-revisions.json, post-trigger from evidence/09-post-trigger-revisions.json, post-fix from evidence/18-post-fix-revisions.json — rather than relying on a single post-fix snapshot. This is why the lab requires running trigger.sh and verify.sh in sequence without an intervening manual revision update; a future maintainer changing the cadence must preserve this snapshot-lookup-per-revision pattern, otherwise verify.sh lookup_created_time will return None for the pruned baseline and the sub-gate will fail (createdTime_strictly_increasing=false) rather than passing with stale data.

3. Secret-store PII safety. The managed-environment secret store entry appinsights-connection-string contains a real Application Insights connection string with a real instrumentation key. Logging the resolved value to an evidence file would be a P0 PII leak. The 05-*, 10-*, and 19-* snapshots redact the secret value to REDACTED_NEVER_LOG_REAL_CONNECTION_STRINGS and emit only name, value_present (boolean), keyVaultUrl (None for this lab — secrets are inline, not Key Vault references), and identity (None for this lab — no managed identity reference). The raw secret-store output is captured to a -raw.json temp file and immediately deleted by trigger.sh Phase 5 / Phase 10 and verify.sh Phase 19. The H1 sub-gate d_secret_store_unchanged and H2 sub-gate d_secret_store_value_unchanged_after_fix compare the redacted snapshots' name + keyVaultUrl + identity + value_present fields — they explicitly do NOT compare the resolved value.

Evidence provenance note¶

All 26 physical files (24 numbered prefixes) come from a single coherent 2026-06-24 01:12–01:14 UTC run against rg-aca-lab-observability2 (Korea Central). The full provenance map, capture timeline, PII-safety policy, and reproducibility instructions are in labs/observability-tracing/evidence/README.md.

Operator caveat¶

The H1 sub-gate c_env_var_now_literal_with_expected_value hard-codes the expected literal value InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://invalid/ (set by trigger.sh as EXPECTED_LITERAL_VALUE). The H1 sub-gate a_baseline_env_was_secretref, the H2 sub-gate c_env_var_back_to_secretref, and the secret-store integrity sub-gates hard-code the expected secret name appinsights-connection-string (set by trigger.sh as EXPECTED_SECRET_NAME, and emitted by infra/main.bicep as the secret name in the Container App resource). If a future maintainer changes any of these constants in trigger.sh or infra/main.bicep, the corresponding gate string matches must be updated in lockstep across both trigger.sh and verify.sh.

Clean Up¶

az group delete --name "$RG" --yes --no-wait

Related Playbook¶

• Secret and Key Vault Reference Failure

See Also¶

• Monitoring Operations
• KQL Query Catalog

Sources¶

• Application Insights for Azure Container Apps
• Observability in Azure Container Apps
• Enable Azure Monitor OpenTelemetry