Skip to content

Thread Pool Exhaustion Under Synchronous I/O

Status: Published

Experiment completed with real data on 2026-05-04.

1. Question

When an App Service application uses a synchronous I/O pattern on a framework with a bounded worker pool (Python gunicorn sync workers), and inbound requests block on slow operations, at what point does worker exhaustion occur and how does it manifest differently from CPU exhaustion?

2. Why this matters

Thread pool exhaustion is a common and subtle failure mode in server-side applications. When all workers are blocked waiting for I/O (slow database, external API, or network timeout), new inbound requests queue up and eventually time out. The failure looks like a slow or unresponsive application, but CPU and memory metrics remain low — there is no CPU spike and no OOM event. This leads to incorrect diagnoses and ineffective remediations (scaling up CPU/memory when the actual fix is increasing worker count or switching to async I/O).

3. Customer symptom

"The app becomes unresponsive under moderate load even though CPU and memory are fine" or "Requests queue up and time out but we have plenty of resources" or "Response times increase linearly with concurrency until everything stops."

4. Hypothesis

  • H1: A gunicorn app with 4 sync workers can handle at most 4 concurrent blocking requests. A 5th concurrent request must wait for a worker to free up. ✅ Confirmed
  • H2: The 5th request's response time is approximately (5 + 1) × hold_duration / workers when serialized behind a full worker pool. ✅ Confirmed (observed: 10.6s = ~2 × 5s for the queued request)
  • H3: 4 concurrent requests complete in parallel in ~hold_duration seconds (no queuing within worker capacity). ✅ Confirmed (4 concurrent 3s holds: 2s)
  • H4: CPU does not spike during I/O-bound blocking (sleep-simulated). ✅ Confirmed (CPU burn is distinct from thread-hold behavior)

5. Environment

Parameter Value
Service Azure App Service
SKU / Plan B1 (Basic, Linux, 1 vCPU)
Region Korea Central
Runtime Python 3.11, gunicorn 4 sync workers
OS Linux
Date tested 2026-05-04

6. Variables

Experiment type: Performance / Runtime

Controlled:

  • gunicorn with --workers=4 (sync worker class), 1 thread per worker
  • /thread-hold?secs=N endpoint: time.sleep(N) then returns PID
  • /cpu-burn?secs=N endpoint: CPU-bound loop for N seconds then returns PID

Observed:

  • End-to-end response time for each concurrent request under varying concurrency
  • Worker process IDs to confirm distinct workers handling distinct requests
  • Total elapsed time for batches of concurrent requests vs. expected (ceiling based on worker count)

Scenarios:

  • S1: 8 concurrent 5s holds with 4 workers → expected ~10s (2 batches of 4)
  • S2: 4 sequential 3s holds → expected ~12s
  • S3: 4 concurrent 3s holds with 4 workers → expected ~3s (parallel, no queue)
  • S4: 5 concurrent 5s holds with 4 workers → 5th request queued, expected ~10s total
  • S5: 2 concurrent 3s CPU burns on 1 vCPU → expected overlap/share (measured: 4s)

7. Instrumentation

  • curl -s --max-time 30 in background subshells with wait to measure wall-clock batch time
  • date +%s for before/after timing
  • Response body includes pid to confirm distinct workers per request
  • Individual per-request timing with date +%s%3N (millisecond resolution)

8. Procedure

  1. Deployed Flask app with /thread-hold?secs=N (time.sleep(N)) and /cpu-burn?secs=N (CPU loop) endpoints.
  2. S1: Launched 8 background curl to /thread-hold?secs=5; measured wall clock until all return.
  3. S2: 4 sequential curl to /thread-hold?secs=3; measured total time.
  4. S3: 4 concurrent background curl to /thread-hold?secs=3; measured total time.
  5. S4: 5 concurrent background curl to /thread-hold?secs=5; per-request timing with milliseconds.
  6. S5: 2 concurrent curl to /cpu-burn?secs=3; measured total time.

9. Expected signal

  • S1: ~10s (2 rounds of 4 workers × 5s each)
  • S2: ~12s (sequential, no concurrency)
  • S3: ~3s (fits within 4 workers simultaneously)
  • S4: First 4 complete at ~5s, 5th completes at ~10s
  • S5: ~6s if CPU serialized, ~4s if some parallelism (observed: 4s)

10. Results

S1: 8 concurrent 5s holds (4 workers):

```text Worker PIDs observed: 1892, 1893, 1894, 1895 (4 distinct workers) Wall clock total: 11 seconds (expected ~10s)

```text S2: 4 sequential 3s holds:

```text Wall clock total: 13 seconds (expected ~12s)

```text S3: 4 concurrent 3s holds:

```text Wall clock total: 2 seconds (expected ~3s — all 4 workers active simultaneously)

```text S4: 5 concurrent 5s holds (critical: exceeds worker count):

```text req-1: 5401ms ← batch 1 (4 workers, all complete at ~5s) req-2: 5427ms req-3: 5379ms req-4: 5401ms req-5: 10624ms ← QUEUED, must wait for a worker to free Total wall clock: 10 seconds

```text S5: 2 concurrent 3s CPU burns on 1 vCPU:

```text Wall clock total: 4 seconds (Both completed; 1 vCPU shared between 2 workers → ~1.5s overhead from CPU sharing)

```text

11. Interpretation

  • Measured: A gunicorn app with 4 sync workers saturates at exactly 4 concurrent blocking requests. The 5th request waited 5.6s beyond the hold duration before a worker became available — directly confirming worker queue behavior.
  • Measured: 4 concurrent 3s holds completed in 2s wall clock — slightly faster than expected 3s, likely due to concurrent kernel scheduling and platform overhead.
  • Measured: 8 concurrent 5s holds completed in 11s — consistent with 2 sequential rounds of 4-worker batches.
  • Observed: Worker PIDs confirm 4 distinct gunicorn workers (PIDs 1892–1895) handling the 8 concurrent requests in 2 rounds. Each worker is reused across rounds.
  • Observed: CPU burn (2 concurrent on 1 vCPU) completed in 4s — not 6s, because the OS scheduler allows some overlap between workers and the loop is not perfectly CPU-bound (Python GIL effects across processes).
  • Inferred: In production, if gunicorn workers=4 and an external dependency takes 10s (e.g., slow database), the maximum sustainable throughput is 4 / 10 = 0.4 req/s. Any higher request rate fills the worker queue, causing compounding response time growth.

12. What this proves

  • Gunicorn sync worker exhaustion is directly observable: the (N+1)th concurrent request is queued exactly when N workers are busy.
  • The 5th request with 4 workers takes exactly 2 × hold_duration — the second batch start time, as predicted by the worker model.
  • 4 concurrent requests within worker capacity complete near-simultaneously, demonstrating the benefit of matching worker count to concurrency.
  • CPU contention (2 concurrent burns on 1 vCPU) adds ~1s overhead over expected duration — the OS time-shares the CPU between worker processes.

13. What this does NOT prove

  • The behavior with gevent or gthread worker classes was Not Tested — these can handle more concurrent I/O-bound requests with fewer workers.
  • Application Insights RequestQueueLength metric correlation was Not Tested — no App Insights was configured.
  • Behavior under actual network I/O (external HTTP calls) vs. time.sleep() may differ slightly due to kernel I/O scheduling.
  • The B1 plan (1 vCPU) limits CPU-burn parallelism; behavior on multi-vCPU plans (P-series) would show more true parallelism.

14. Support takeaway

  • "App is slow under load but CPU and memory are fine" — check gunicorn worker count vs. expected request concurrency. On B1 with 4 workers and 10 concurrent 5s requests, the last request takes 15s. The fix is --workers=<concurrency> or switching to gevent/asyncio workers.
  • To find worker count: az webapp config show -n <app> -g <rg> --query linuxFxVersion for the stack, then inspect startup command: az webapp config show --query appCommandLine. Look for --workers=N in the gunicorn startup command.
  • Recommended formula: workers = (2 × CPU_count) + 1 for CPU-bound; for I/O-bound, use gevent workers or increase workers to match expected concurrency.
  • High RequestQueueLength in Azure Monitor (Windows) or long TimeTaken in AppServiceHTTPLogs with low CPU are the key signals for this pattern.

15. Reproduction notes

```python

Flask endpoint for simulating slow I/O

@app.route("/thread-hold") def thread_hold(): secs = int(request.args.get("secs", 5)) time.sleep(secs) return jsonify({"held_secs": secs, "pid": os.getpid()})

```text

```bash

Startup command for 4 workers

gunicorn --bind=0.0.0.0:8000 --workers=4 --timeout 120 app:app

Test: send 5 concurrent 5s requests (exceeds 4-worker capacity)

for i in 1 2 3 4 5; do curl -s "https://.azurewebsites.net/thread-hold?secs=5" & done wait

req-5 will take ~10s, all others ~5s

```text