High Ingestion Cost¶

1. Summary¶

Your Log Analytics bill is growing faster than expected, or the workspace usage chart shows a sudden step up in daily volume. This playbook applies when the cost problem is primarily ingestion, not query charges or long-term archive retention, and you need to identify which tables, resources, solutions, or categories are driving billable growth.

Azure Monitor cost spikes are usually explainable. Microsoft Learn emphasizes starting with Usage and _Usage, because most incidents come from noisy diagnostic settings, a new AMA or DCR stream, verbose application logging, or a solution that enabled a wider set of tables than the team expected. Treat this as a data-shape problem before treating it as a billing mystery.

Typical incident window: 4-24 hours because ingestion cost anomalies usually surface after an hourly or daily usage step-up. Time to resolution: 1-2 business cycles for full cost normalization, with first mitigations often applied within 30-90 minutes.

Use it when:

• Daily GB jumped after a deployment, policy assignment, or monitoring rollout.
• A workspace is approaching budget even though retention did not change.
• One or two tables suddenly dominate usage charts.
• Teams need evidence before disabling logs or moving tables to Basic plan.

flowchart TD
    A[High ingestion cost] --> B{Which table grew first?}
    B -->|One or two tables| C[Check source resources and categories]
    B -->|Many tables| D[Check solution rollout or workspace scope]
    C --> E{Platform logs or AMA data?}
    E -->|Platform logs| F[Review diagnostic settings]
    E -->|AMA data| G[Review DCR streams and transformations]
    D --> H[Check enabled solutions and policy changes]
    F --> I[Disable noisy categories]
    G --> J[Filter before ingestion]
    H --> K[Reduce scope or move tables to lower-cost plan]

2. Common Misreadings¶

3. Competing Hypotheses¶

4. What to Check First¶

1. Capture workspace identifiers before running usage queries

   az monitor log-analytics workspace show \
       --resource-group $RG \
       --workspace-name $WORKSPACE_NAME \
       --query "{id:id,customerId:customerId}"
   

   Command	Purpose
   az monitor log-analytics workspace show	Show a Log Analytics workspace.
   --resource-group	Resource group that contains the resource.
   --workspace-name	Name of the Log Analytics workspace.
   --query	JMESPath projection of the fields to return.

2. Replay top billable tables directly from the workspace

   az monitor log-analytics query \
       --workspace $WORKSPACE_ID \
       --analytics-query "Usage | where TimeGenerated > ago(24h) | where IsBillable == true | summarize TotalGB=round(sum(Quantity)/1024.0,2) by DataType | order by TotalGB desc | take 10" \
       --timespan "P1D"
   

   Command	Purpose
   az monitor log-analytics query	Run a KQL query against a Log Analytics workspace.
   --workspace	Log Analytics workspace ID for the query.
   --analytics-query	Kusto (KQL) query to execute.
   --timespan	Time range for the query.

3. Review workspace pricing, retention, and cap settings

   az monitor log-analytics workspace show \
       --resource-group $RG \
       --workspace-name $WORKSPACE_NAME \
       --query "{sku:sku.name,retentionInDays:retentionInDays,workspaceCapping:workspaceCapping.dailyQuotaGb}"
   

   Command	Purpose
   az monitor log-analytics workspace show	Show a Log Analytics workspace.
   --resource-group	Resource group that contains the resource.
   --workspace-name	Name of the Log Analytics workspace.
   --query	JMESPath projection of the fields to return.

4. Verify whether the spike lines up with a DCR rollout

   az monitor data-collection rule list \
       --resource-group $RG \
       --output table
   

   Command	Purpose
   az monitor data-collection rule list	List data collection rules.
   --resource-group	Resource group that contains the resource.
   --output	Output format for the result.

5. If platform logs are implicated, inspect diagnostic settings on a top resource

   az monitor diagnostic-settings list \
       --resource $RESOURCE_ID \
       --output json
   

   Command	Purpose
   az monitor diagnostic-settings list	List diagnostic settings for a resource.
   --resource	Target resource ID or name for the operation.
   --output	Output format for the result.

6. If Application Insights tables grew, check the component and workspace connection

   az monitor app-insights component show \
       --app $APP_INSIGHTS_NAME \
       --resource-group $RG \
       --query "{workspaceResourceId:workspaceResourceId,applicationType:applicationType,samplingPercentage:samplingPercentage}"
   

   Command	Purpose
   az monitor app-insights component show	Show an Application Insights component.
   --app	Application Insights component name.
   --resource-group	Resource group that contains the resource.
   --query	JMESPath projection of the fields to return.

5. Evidence to Collect¶

5.1 KQL Queries¶

// Billable usage by table for the last 7 days
Usage
| where TimeGenerated > ago(7d)
| where IsBillable == true
| summarize TotalGB = round(sum(Quantity) / 1024, 2) by DataType
| order by TotalGB desc

// Billable usage by resource for the last 24 hours
_Usage
| where TimeGenerated > ago(24h)
| summarize BillableGB = round(sum(Quantity) / 1024, 2) by _ResourceId, DataType
| order by BillableGB desc
| take 20

// Daily trend by top billable tables
Usage
| where TimeGenerated > ago(14d)
| where IsBillable == true
| summarize TotalGB = round(sum(Quantity) / 1024, 2) by bin(TimeGenerated, 1d), DataType
| order by TimeGenerated asc

// Application telemetry mix when App Insights cost appears high
union requests, dependencies, traces, exceptions
| where timestamp > ago(24h)
| summarize Items = sum(itemCount) by itemType, cloud_RoleName
| order by Items desc

5.2 CLI Investigation¶

# Review workspace commercial settings
az monitor log-analytics workspace show \
    --resource-group $RG \
    --workspace-name $WORKSPACE_NAME \
    --output json

Sample output:

{
  "id": "/subscriptions/<subscription-id>/resourceGroups/rg-monitor/providers/Microsoft.OperationalInsights/workspaces/law-prod",
  "name": "law-prod",
  "provisioningState": "Succeeded",
  "retentionInDays": 30,
  "sku": {
    "name": "PerGB2018"
  },
  "workspaceCapping": {
    "dailyQuotaGb": -1
  }
}

Interpretation:

• PerGB2018 means ingestion growth maps directly to cost growth.
• retentionInDays matters, but it does not explain a sudden same-day ingestion jump.
• dailyQuotaGb = -1 means there is no hard cap to stop runaway spend automatically.

# Inspect a DCR when AMA-collected data is suspected
az monitor data-collection rule show \
    --resource-group $RG \
    --name $DCR_NAME \
    --output json

Sample output:

{
  "name": "dcr-vm-baseline",
  "dataFlows": [
    {
      "destinations": [
        "la-workspace"
      ],
      "streams": [
        "Microsoft-Perf",
        "Microsoft-InsightsMetrics",
        "Microsoft-Event"
      ]
    }
  ],
  "destinations": {
    "logAnalytics": [
      {
        "name": "la-workspace",
        "workspaceResourceId": "/subscriptions/<subscription-id>/resourceGroups/rg-monitor/providers/Microsoft.OperationalInsights/workspaces/law-prod"
      }
    ]
  }
}

Interpretation:

• New or unnecessary streams can explain cost growth immediately.
• If many machines share this DCR, a small change can multiply workspace volume quickly.
• Absence of transformation rules means all matching source data lands unfiltered.

# Inspect diagnostic settings on a top contributing resource
az monitor diagnostic-settings list \
    --resource $RESOURCE_ID \
    --output json

Sample output:

[
  {
    "logs": [
      {
        "categoryGroup": "allLogs",
        "enabled": true
      }
    ],
    "metrics": [
      {
        "category": "AllMetrics",
        "enabled": true
      }
    ],
    "workspaceId": "/subscriptions/<subscription-id>/resourceGroups/rg-monitor/providers/Microsoft.OperationalInsights/workspaces/law-prod"
  }
]

Interpretation:

• allLogs is convenient but often expensive on noisy services.
• Category groups hide which specific log types are enabled, so compare against service-level guidance before leaving it broad.
• If the resource is the top _Usage contributor, this output often identifies the fastest reduction opportunity.

# Review Application Insights component configuration when app tables are expensive
az monitor app-insights component show \
    --app $APP_INSIGHTS_NAME \
    --resource-group $RG \
    --output json

Sample output:

{
  "applicationType": "web",
  "connectionString": "<connection-string>",
  "name": "appi-prod",
  "samplingPercentage": null,
  "workspaceResourceId": "/subscriptions/<subscription-id>/resourceGroups/rg-monitor/providers/Microsoft.OperationalInsights/workspaces/law-prod"
}

Interpretation:

• If samplingPercentage is null or effectively disabled, high-volume apps may send far more telemetry than needed.
• A workspace-based component means Application Insights cost appears through the shared Log Analytics workspace.
• Use this output with the application telemetry KQL to decide whether to tune SDK sampling or log levels.

6. Validation and Disproof by Hypothesis¶

Hypothesis 1: A small set of tables is generating most billable GB¶

Proves if: Section 5.1 Query 1 shows one or two tables dominating recent usage.

Disproves if: Volume is broadly distributed with no obvious top table.

Test with: Section 5.1 Queries 1 and 3.

Hypothesis 2: A specific resource or diagnostic setting is the main contributor¶

Proves if: Section 5.1 Query 2 points to a single _ResourceId and Section 5.2 CLI command 3 shows broad or noisy categories.

Disproves if: No top resource stands out or the top resource is already tightly scoped.

Test with: Section 5.1 Query 2 plus Section 5.2 CLI command 3.

Hypothesis 3: A DCR or solution rollout expanded collection scope¶

Proves if: Cost rose immediately after a DCR update or after enabling VM Insights, Container Insights, or another solution.

Disproves if: No rollout aligned with the usage change and DCR streams are unchanged.

Test with: Section 5.2 CLI command 2 and the 14-day trend in Section 5.1 Query 3.

Hypothesis 4: Application telemetry increased because of verbose logs, retries, or exceptions¶

Proves if: traces, dependencies, or exceptions surged in Section 5.1 Query 4 and the Application Insights component is writing to the same workspace.

Disproves if: App tables remain flat while another table explains the spend.

Test with: Section 5.1 Query 4 plus Section 5.2 CLI command 4.

Hypothesis 5: Plan choice and retention are amplifying a bad ingestion pattern¶

Proves if: High-volume operational tables remain in Analytics plan even though Basic plan or shorter retention would satisfy the use case.

Disproves if: Table plan and retention are already optimized and ingestion volume itself is still the dominant issue.

Test with: Section 5.2 CLI command 1, then compare high-volume table purpose against workspace table plan and retention requirements.

Hypothesis 6: Daily cap is hiding runaway ingestion instead of solving it¶

Proves if: Usage drops only because the cap is reached, not because the source volume was reduced.

Disproves if: There is no cap or usage remains below the configured cap.

Test with: Section 5.2 CLI command 1 and the workspace Operation table if cap-related warnings are suspected.

7. Likely Root Cause Patterns¶

Normal vs Abnormal Comparison¶

8. Immediate Mitigations¶

1. Disable or narrow noisy diagnostic categories on the top source resource.

   az monitor diagnostic-settings create \
       --name send-to-law \
       --resource $RESOURCE_ID \
       --workspace $WORKSPACE_ID \
       --logs '[{"category":"AppServiceHTTPLogs","enabled":true},{"category":"AppServiceConsoleLogs","enabled":true}]' \
       --metrics '[{"category":"AllMetrics","enabled":true}]'
   

   Command	Purpose
   az monitor diagnostic-settings create	Create a diagnostic setting.
   --name	Name of the resource.
   --resource	Target resource ID or name for the operation.
   --workspace	Log Analytics workspace name or resource ID that receives the logs.
   --logs	Log categories or settings to collect.
   --metrics	Metric categories to collect.

2. Remove unnecessary AMA streams or destinations from the DCR.

   az monitor data-collection rule show \
       --resource-group $RG \
       --name $DCR_NAME \
       --query "dataFlows"
   

   Command	Purpose
   az monitor data-collection rule show	Show a data collection rule.
   --resource-group	Resource group that contains the resource.
   --name	Name of the resource.
   --query	JMESPath projection of the fields to return.

3. Move a suitable high-volume table to Basic plan.

   az monitor log-analytics workspace table update \
       --resource-group $RG \
       --workspace-name $WORKSPACE_NAME \
       --name AppServiceHTTPLogs \
       --plan Basic
   

   Command	Purpose
   az monitor log-analytics workspace table update	Update a table in a Log Analytics workspace.
   --resource-group	Resource group that contains the resource.
   --workspace-name	Name of the Log Analytics workspace.
   --name	Name of the resource.
   --plan	Table plan (Analytics or Basic) for the Log Analytics table.

4. Reintroduce or increase telemetry sampling for a high-volume application.

   az monitor app-insights component show \
       --app $APP_INSIGHTS_NAME \
       --resource-group $RG \
       --query "{name:name,workspaceResourceId:workspaceResourceId}"
   

   Command	Purpose
   az monitor app-insights component show	Show an Application Insights component.
   --app	Application Insights component name.
   --resource-group	Resource group that contains the resource.
   --query	JMESPath projection of the fields to return.

5. Use a temporary daily cap only as a guardrail while fixes are being applied.

   az monitor log-analytics workspace update \
       --resource-group $RG \
       --workspace-name $WORKSPACE_NAME \
       --quota 20
   

   Command	Purpose
   az monitor log-analytics workspace update	Update a Log Analytics workspace.
   --resource-group	Resource group that contains the resource.
   --workspace-name	Name of the Log Analytics workspace.
   --quota	Daily ingestion quota for the workspace.

9. Prevention¶

Preventing ingestion cost incidents is mainly about design discipline. Microsoft Learn recommends understanding table usage, choosing the right table plan, and filtering or scoping data before it hits the workspace.

Review top tables and top resources on a schedule, not only during billing incidents.

az monitor log-analytics workspace show \
    --resource-group $RG \
    --workspace-name $WORKSPACE_NAME \
    --query "{id:id,customerId:customerId,sku:sku.name}"

Make DCR streams explicit and minimal, especially when onboarding many VMs or Arc servers.

az monitor data-collection rule list \
    --resource-group $RG \
    --output table

Replace blanket diagnostic settings with category choices that map to real operational use cases.

az monitor diagnostic-settings categories list \
    --resource $RESOURCE_ID \
    --output table

Align table plans with query frequency and retention needs.

az monitor log-analytics workspace table show \
    --resource-group $RG \
    --workspace-name $WORKSPACE_NAME \
    --name AppServiceHTTPLogs

For application telemetry, keep logging levels and sampling part of release review. A bug that floods traces or exceptions is both a reliability issue and a cost issue.

Finally, do not treat daily cap as your primary budget strategy. It prevents unlimited spend, but it also shuts off observability exactly when noisy incidents are happening.

See Also¶

• No Data in Workspace
• Slow Query Performance
• Operations: Cost Control
• Operations: Diagnostic Settings
• KQL: Ingestion Volume

Sources¶

• Microsoft Learn: Manage cost and usage in Log Analytics workspaces
• Microsoft Learn: Data collection rules in Azure Monitor
• Microsoft Learn: Create diagnostic settings in Azure Monitor
• Microsoft Learn: Application Insights pricing and data volume reduction guidance
• Microsoft Learn: Table plans in Azure Monitor Logs
• Microsoft Learn: Usage table reference