Concurrency, utilization, saturation, and bottlenecks

Orientation

Definition

Concurrency is the number of requests in progress at one moment. Utilization is the fraction of time a resource is busy. Saturation is the amount of work waiting because that resource is fully occupied. A bottleneck is the resource that limits a system's throughput, which is the one that saturates first as load grows.

Why it matters

These four terms are how you diagnose nearly every performance problem. Each one reveals something the others can't: a server at 40% average CPU may have one core pinned at 100%, and an idle database may be stalled on a single lock. Utilization tells you how close to the limit you are, and saturation tells you that you're past it. Knowing which resource limits you prevents the classic waste of scaling something that was never the constraint, and you can often compute the limit in advance from simple measured numbers.

Prerequisites

Throughput

Study time

About 1.5 hours, including the terminal lab and challenges. This is an estimate.

Related topics

Little’s Law and practical queueing theory (not yet written), Golden Signals, RED, and USE methods (not yet written), Load shedding and admission control (not yet written), Throughput, CPU, heap, lock-contention, and I/O profiling (not yet written)

Learning objectives

• Distinguish concurrency, utilization, saturation, and errors, and measure each for a resource.
• Apply the utilization law and bottleneck bound to predict a system's maximum throughput.
• Use Little's Law to convert between rate, latency, and concurrency.
• Identify non-hardware bottlenecks such as locks, pools, and single-threaded loops.
• Protect a bottleneck with a concurrency limit and bounded queue.

Production scenario

An order service at 450 requests per second starts timing out. Average CPU is 45% on web servers, 70% on application servers, and 34% on the database. Disk is at 90%, which draws attention, and the team orders faster disks. Nothing improves. Then someone measures a resource nobody charted: every order updates one row of a global counter, holding its lock for 2.5 ms. Since one lock serves one holder at a time, the maximum is 1 ÷ 0.0025 s = 400 updates per second. At 450 requests per second, that lock's utilization is 112.5%, so its queue grows forever.

Disk was close to its limit too, but the lock had already hit its own. The fix replaces the single counter with four sub-counters, which are summed when read. Lock capacity rises to 1,600 updates per second, and the bottleneck moves to disk at 500 per second, which is now worth upgrading. The operational analysis example in this lesson reproduces these numbers from measured demand alone, without a load test.

Mental model

Analogy

Picture a supermarket. Concurrency is how many customers are inside. Utilization is the fraction of time a checkout clerk is scanning. Saturation is the line at the checkouts. The bottleneck is whichever stage has the longest wait relative to its capacity. Opening more aisles doesn't help if there's one clerk. An empty shop with one clerk away on a break has low "CPU" and terrible service, just like a server waiting on a lock.

Where the analogy breaks

You can see a line in a supermarket, but software queues are hidden in kernels, thread pools, locks, and network buffers, so many go unmeasured. Clerks also don't slow one another down as you add more, whereas extra software workers can contend and lower total output. Finally, a shopper in a long line usually waits, whereas a software client times out and retries, adding even more work to the line.

Entities, state, flow, and invariants

• Entities: requests, resources, servers per resource, queues.
• State: busy servers, queue length, throughput.
• Flow: arrive, wait if all busy, get served, leave, possibly visiting several resources.
• Invariant 1: U = X × D. Utilization is throughput times service demand.
• Invariant 2: X ≤ 1 ÷ D_max. Throughput can't exceed what the largest demand allows.
• Invariant 3: L = X × R. Concurrency is throughput times response time.

Foundations

Concepts and terminology

First-principles explanation

Measure a system for a period, and three relationships must hold regardless of the arrival pattern, as they are bookkeeping identities. If a resource completes X requests per second, each taking D seconds of its time, then it is busy X × D of each second, which is the utilization law. Utilization can't exceed 100%, so X can't exceed 1 ÷ D. Therefore the resource with the biggest demand per server is the bottleneck, and it sets the cap on the whole system.

This is powerful, since you can find the bottleneck from measurements at low load. CPU per request, disk time per request, and lock hold time per request, along with server counts, predict the limit before you ever reach it. Behavior near the limit is the exception, as contention and cache effects can raise demand. So treat the result as an upper bound, and confirm it with a test. See Load, stress, spike, and soak testing (not yet written).

Below the bottleneck's limit, response time stays near the sum of demands. Above it, queues grow. The transition happens near a "knee" of concurrency, N* = (ΣD + Z) ÷ D_max, where Z is client think time. Past this knee, adding concurrency only adds waiting.

Average utilization misleads for three reasons. It averages across time, hiding bursts. It averages across servers or cores, hiding one busy instance or a single-threaded loop: one pinned core of 16 reads as 6.25%. And it hides logical resources entirely, such as locks and pools, which have no OS-level utilization metric. Saturation signals are usually more direct: any non-trivial queue on a resource means it's limiting something right now.

Mathematics

Utilization law: U = X × D ÷ c, for c servers. At 300 requests per second with an app-tier demand of 25 ms over 16 cores: 300 × 0.025 ÷ 16 = 46.9%.

Bottleneck bound: X_max = min over resources of (c ÷ D).

Response floor: R ≥ ΣD = 8 + 25 + 6 + 4 + 2.5 = 45.5 ms.

Knee: N* = 45.5 ÷ 2.5 = 18.2 concurrent requests with zero think time. Beyond that, extra requests wait.

Little's Law: L = X × R. At 300 requests per second and 50 ms: 15 in flight. Check it: the knee is 18.2, which means that 300 requests per second sits below it. See Little’s Law and practical queueing theory (not yet written).

Complexity

Operational analysis is O(r) for r resources. A concurrency limiter's acquire and release are O(1) amortized, although the example's Array.shift is O(q) for a queue of length q, which is fine for small bounded queues. Measurement overhead also matters: per-request timing adds little, but heavy lock profiling can distort what it measures.

Mechanism

Normal path

1. A request arrives, and concurrency rises by one.
2. If the resource has a free server, service starts.
3. Otherwise, it waits in a queue, which is saturation.
4. It uses each resource in turn: web CPU, app CPU, database, disk, and locks.
5. When done, it leaves, and concurrency falls by one.
6. As arrivals approach the bottleneck's rate, its queue grows quickly and response time climbs.
7. A limit in front of the bottleneck makes excess work wait outside it, or get rejected.

State transitions

stateDiagram-v2
    [*] --> Idle
    Idle --> PartlyBusy: arrivals below capacity
    PartlyBusy --> NearSaturation: utilization above about 80 percent
    NearSaturation --> PartlyBusy: load falls
    NearSaturation --> Saturated: arrivals exceed capacity
    Saturated --> Rejecting: bounded queue is full
    Saturated --> TimingOut: unbounded queue grows
    Rejecting --> NearSaturation: load falls
    TimingOut --> NearSaturation: load removed and queue drained

Saturated resources go one of two ways. Rejecting is a controlled outcome, and recovers once load falls. TimingOut wastes capacity on requests that will never be useful. See Throughput. The 80% figure is an illustrative threshold, which varies with variability and server count.

Concurrency behavior

Concurrency limits are often the right control, more so than rate limits, because concurrency reflects true load: a slow dependency raises concurrency for the same request rate. This is Little's Law in action: when response time doubles at a constant rate, concurrency doubles. Concurrency limiting therefore reacts automatically when a dependency slows down, with no need for a new rate setting. See Bulkheads and isolation pools (not yet written).

Ordering and consistency

Queues impose order, and it matters who waits. First-in-first-out is fair, but under overload it serves the oldest requests, which are the most likely to have expired. Last-in-first-out and priority queues can preserve goodput for fresh or important work. Locks affect consistency too: reducing lock scope is the classic throughput gain, but it has to preserve the invariant that the lock protected. See Deadlocks (not yet written).

Ownership of data and control

Each shared resource needs an owner who sets its limits. Without one, several teams can each stay "within their rate" while together overloading the same database. The owner of a limit also has to decide who gets rejected, which is a business question as well as a technical one. See Load shedding and admission control (not yet written).

Architecture

flowchart LR
    C["Clients"] -->|"HTTPS"| LB["Load balancer: accept queue"]
    LB --> W["Web tier: 8 cores"]
    W --> AP["App tier: 16 cores, thread pool queue"]
    AP --> LM["Concurrency limiter: bounded queue"]
    LM -->|"TCP, connection pool"| DB["Database, owns data: 8 cores"]
    DB --> DK["Disk: 2 volumes"]
    DB --> RL["BOTTLENECK: one row lock, 400 per second"]

Each arrow has a queue that may be invisible on standard dashboards. The bottleneck here is a logical resource that won't appear on an infrastructure graph. The limiter sits in front of the database, so excess work waits or gets rejected in a place where this is cheap, and the bottleneck's own queue stays short.

flowchart LR
    R["HTTP GET /query"] --> L["ConcurrencyLimiter.run"]
    L --> SL["Slots: limit 10"]
    L --> WQ["Waiters: limit 20"]
    WQ -->|"queue full"| E["HTTP 503 with Retry-After"]
    SL --> WK["Simulated database work"]
    L --> ST["GET /stats: in-flight, utilization, queued, rejected"]
    IN["Measured demands and server counts"] --> OA["Python: analyze"]
    OA --> RP["Utilization per resource, bottleneck, X max, knee"]

The Python tool predicts limits, and the TypeScript service enforces one and reports on it. Together they form the working cycle: compute the limit from demands, set the limiter just below it, and verify the prediction using its metrics.

sequenceDiagram
    participant C as Clients
    participant L as Limiter, 2 slots, queue 1
    participant D as Database
    C->>L: Request 1
    L->>D: Start, 1 of 2 slots
    C->>L: Request 2
    L->>D: Start, 2 of 2 slots
    C->>L: Request 3
    Note over L: Waits in queue, saturation is 1
    C->>L: Request 4
    L-->>C: 503 Retry-After, queue is full
    D-->>L: Request 1 finished
    L->>D: Slot passes to request 3
    D-->>L: Request 2 finished

The rejection takes almost no resources, which is what makes it useful during overload. The database never sees more than its limit, so it keeps running at full throughput throughout. Clients need to treat a 503 with Retry-After as "try later", or retries will undo the protection.

flowchart TD
    A["Timeouts at 450 rps"] --> B["Dashboards: disk at 90 percent is the highest"]
    B --> C["Upgrade disk"]
    C --> D["No improvement"]
    D --> E["Measure lock wait, previously not charted"]
    E --> F["Row lock at 112.5 percent"]
    F --> G["Split counter into 4 sub-counters"]
    G --> H["Lock limit 1,600 per second"]
    H --> I["Bottleneck moves to disk at 500 per second"]

Highest-charted is not the same as highest. The team could only compare what they had measured, so unmeasured resources are the first thing to suspect when a fix doesn't help. The final step shows that fixing a bottleneck always reveals the next, and that's the cue to repeat the analysis.

Trade-offs

Advantages

• A small set of signals diagnoses most problems.
• Demand-based bounds give predictions without expensive tests.
• Concurrency limits adapt to slow dependencies automatically.

Costs

• Measuring logical resources requires custom instrumentation.
• Running at low utilization to keep queues short costs money.
• Limits reject real users, so choosing them is a business decision.
• Fixed limits go stale as hardware and code change.

Alternatives

• Rate limiting: better for fairness and quotas, but slower to respond to dependency slowdowns. See Rate limiting (not yet written).
• Adaptive limits: adjust to latency automatically, at the cost of stability tuning.
• Autoscaling: adds capacity where the tier can scale, and is too slow for bursts. See Autoscaling (not yet written).
• Load testing alone: finds limits empirically, at greater cost. Use it to confirm predictions.

When to use it

For every performance investigation and capacity plan. Add concurrency limits around any shared resource whose overload would hurt others.

When not to use it

Don't over-instrument a small internal service with no contention. Don't set a tight limit without measuring, as that creates errors that weren't there before. Don't rely on these identities outside steady state, for example in the seconds after a burst or a restart.

Reconsider when

• A fix to the "bottleneck" has no effect.
• Demand per request changes after a code or schema change.
• CPU or disk looks healthy, but latency and queues are climbing.
• Hardware changes make old limits wrong.

Capacity planning

Inputs and assumptions

Measured demands from the table above. Target: 1,000 requests per second at 70% utilization or below. Assume demand stays constant up to that load, which you should verify.

Formula and units

• Servers = rate (requests per second) × demand (seconds) ÷ target utilization.
• Max rate = servers ÷ demand (seconds).
• In flight = rate × response time (seconds).

Intermediate calculations

• Web CPU: 1,000 × 0.008 ÷ 0.7 = 11.4, so 12 cores.
• App CPU: 1,000 × 0.025 ÷ 0.7 = 35.7, so 36 cores.
• Database CPU: 1,000 × 0.006 ÷ 0.7 = 8.6, so 9 cores.
• Disk: 1,000 × 0.004 ÷ 0.7 = 5.7, so 6 volumes.
• Row lock: 1,000 × 0.0025 ÷ 0.7 = 3.6, so 4 sub-counters.
• In flight at a 60 ms response: 1,000 × 0.06 = 60, which sets the connection pool's size.

Headroom

The 70% target is the headroom. A lower target means shorter queues and higher cost. Single-server resources, such as one lock or a single-threaded loop, need more margin than pooled resources, because a pool absorbs variation much better. So the 70% target may be too high for the lock.

Failure capacity

With three zones, losing one raises per-zone load by 50%. So 36 app cores across three zones become 24, and utilization goes from 70% to 105%. For failure tolerance at 70%: 36 × 3 ÷ 2 = 54 cores. Alternatively, plan to run near 90% during a failure: 1,000 × 0.025 ÷ 0.9 = 27.8 cores surviving, or 42 total. State which you've chosen. The row lock's capacity doesn't change with zones, since it isn't horizontally spread.

Failure modes

Security and privacy

Threats

Attackers find your bottleneck too. Sending requests that take a contended lock, or that have the highest demand, gets maximum damage from minimum traffic. Slow clients can hold connections to exhaust a concurrency limit, and metrics endpoints reveal your limits and current load, which helps attackers time their traffic.

Trust boundaries

Protect shared resources with limits placed inside your trust boundary, so clients can't bypass them. Don't expose /stats publicly. Set read and write timeouts on untrusted connections, to prevent slow-client attacks from occupying slots.

Authentication and authorization

Authentication can itself be the bottleneck. Password hashing is deliberately CPU-heavy, so a login flood can saturate the tier. Run it in a separate limited pool, and apply limits before expensive checks.

Tenant isolation

A single global limit lets one tenant use all slots. Use per-tenant concurrency limits or fair queues, so that one tenant's slow queries reject only that tenant's traffic. See Tenant isolation and noisy-neighbor controls (not yet written).

Sensitive data

Load and queue metrics rarely contain personal data, but labels might. Do not label saturation metrics with user identifiers. Rejection logs should use hashed or tiered tenant labels where appropriate.

Abuse resistance

Make rejection cheaper than service, and reject before doing expensive parsing. Limit concurrency per client as well as globally. Monitor for clients that retry immediately after a 503 and ignore Retry-After.

Observability

Metrics

For every resource, including logical ones, use the USE method: utilization, saturation, and errors. Examples: CPU per core; run queue length; Linux pressure stall information; pool active count and wait time; lock wait time; queue depth and age; rejected counts. For every service, add request rate, errors, and duration. See Golden Signals, RED, and USE methods (not yet written).

Logs

Log rejections as a sampled counter, with the reason and limiter name. Log when limits change. Avoid per-rejection logging during overload, as it could add to the problem.

Traces

Show waits as spans: pool.wait, lock.wait, and queue.wait. If these aren't instrumented, they appear as unexplained gaps. Comparing wait to work in a span shows whether a slow request was doing something or merely waiting.

Dashboards

Build a resource table: utilization, saturation, and errors per resource, sorted by saturation. Add a load-versus-latency scatter plot, in which the knee appears directly. Add per-instance views, so that averages don't hide a single bad node.

Alerts

Page on user-visible SLO burn. Warn on queue age, sustained saturation, and rising rejections. Avoid CPU threshold pages, since high CPU with short queues is efficient and low CPU with long queues is a real problem.

SLO examples

• Queue wait on db-limiter stays below 20 ms for 99% of minutes.
• Under 0.1% of requests are rejected by concurrency limits over 28 days.
• Bottleneck utilization stays at or below 70% at daily peak, reviewed monthly.

Runbook actions

Look for saturation first, sorted across all resources. If a logical resource is saturated, reduce pressure before scaling. If CPU saturates, check whether the tier can scale without moving load to the database. After any fix, re-measure, since the bottleneck will have moved.

Implementation

Architecture of the examples

TypeScript provides an Express service with a ConcurrencyLimiter around simulated database work, plus a /stats route. Python applies operational laws to demand data to predict limits. The Express layer is justified here, since real concurrent HTTP traffic is the thing being controlled.

Data structures

The limiter has a slot count and a waiters array of resolver functions. Utilization is measured as slot-time: on each change, it adds inFlight × elapsed to a running total, then divides by elapsed × limit. Python's Resource computes effective demand as demand ÷ servers.

Interfaces

limiter.run(work) runs work under the limit, or throws QueueFullError. limiter.stats() returns a snapshot. GET /query returns 200 or 503 with Retry-After. Python exposes analyze(resources, rps) and required_servers(resources, rps, target).

Error handling

release runs in a finally block, so a failed task can't leak its slot, and a test confirms it. A full queue produces a typed error mapped to 503, not a 500, since overload is an expected condition. Invalid limits are rejected at construction.

Concurrency considerations

Node's single thread makes the counters safe with no atomics, but the await points still interleave. On release, the slot passes directly to the next waiter, with no decrement and increment, so a new arrival can't jump the queue in that gap. In Go or Rust, this logic would require a semaphore or channel, and there is some subtlety around cancelling a queued waiter. This example does not support cancelling while queued.

Educational limitations

There's no queue timeout, so a waiter can wait as long as the queue takes to drain. The limit is fixed. Work is a sleep, so CPU contention never appears. Operational analysis assumes constant demand, so treat the result as a bound.

Production hardening

Add queue timeouts tied to deadlines, cancellation, per-tenant fairness, adaptive limits, histograms for wait time, and priorities. Measure real lock and pool waits with your database and runtime's built-in instrumentation.

Example source

// Express service protecting a simulated database with a concurrency limit. Excess requests queue
// briefly and are then rejected with 503, and /stats reports concurrency, utilization, and saturation.
import { fileURLToPath } from 'node:url';
import express from 'express';
import type { Express, NextFunction, Request, Response } from 'express';
import { ConcurrencyLimiter, QueueFullError } from './limiter.ts';

const log = (level: string, msg: string, fields: Record<string, unknown> = {}): void => {
  console.log(JSON.stringify({ ts: new Date().toISOString(), level, msg, ...fields }));
};
const sleep = (ms: number): Promise<void> => new Promise((resolve) => { setTimeout(resolve, ms); });

export interface AppOptions { limit: number; queueLimit: number; workMs: number }

export function createApp(opts: AppOptions): Express {
  const app = express();
  const limiter = new ConcurrencyLimiter(opts.limit, opts.queueLimit);
  app.get('/healthz', (_req: Request, res: Response) => { res.json({ status: 'ok' }); });
  app.get('/stats', (_req: Request, res: Response) => { res.json(limiter.stats()); });
  app.get('/query', async (_req: Request, res: Response, next: NextFunction) => {
    try {
      await limiter.run(() => sleep(opts.workMs));
      res.json({ ok: true });
    } catch (err) {
      if (err instanceof QueueFullError) {
        res.status(503).set('Retry-After', '1').json({ error: 'Overloaded, retry later.' });
        return;
      }
      next(err);
    }
  });
  app.use((err: unknown, _req: Request, res: Response, _next: NextFunction) => {
    log('error', 'request.failed', { error: String(err) });
    res.status(500).json({ error: 'Internal error.' });
  });
  return app;
}

if (process.argv[1] === fileURLToPath(import.meta.url)) {
  const port = Number(process.env['PORT'] ?? 3000);
  const server = createApp({ limit: 10, queueLimit: 20, workMs: 100 }).listen(port, () => { log('info', 'server.started', { port }); });
  const stop = (): void => { server.close(() => { process.exit(0); }); };
  process.on('SIGTERM', stop);
  process.on('SIGINT', stop);
}

"""Operational analysis: find the bottleneck from measured service demands, before any load test.

Service demand D is the CPU, disk, or other resource time one request needs in total. These laws hold
regardless of the arrival distribution:

    Utilization law:   U = X * D
    Bottleneck limit:  X <= 1 / D_max
    Response floor:    R >= sum(D)
    Saturation knee:   N* = (sum(D) + Z) / D_max
"""

from __future__ import annotations

import json

from pydantic import BaseModel, Field


class Resource(BaseModel):
    name: str
    demand_ms: float = Field(gt=0, description="total service time per request")
    servers: int = Field(default=1, gt=0)

    @property
    def effective_demand_ms(self) -> float:
        return self.demand_ms / self.servers


class Row(BaseModel):
    name: str
    utilization_pct: float
    status: str
    max_rps: float


class Report(BaseModel):
    bottleneck: str
    max_throughput_rps: float
    min_response_ms: float
    knee_concurrency: float
    rows: list[Row]


def classify(u: float) -> str:
    if u >= 1:
        return "saturated"
    if u >= 0.8:
        return "near saturation"
    return "ok"


def analyze(resources: list[Resource], rps: float, think_ms: float = 0.0) -> Report:
    if not resources:
        raise ValueError("at least one resource is required")
    if rps < 0:
        raise ValueError("rps must be non-negative")
    worst = max(resources, key=lambda r: r.effective_demand_ms)
    total = sum(r.demand_ms for r in resources)
    rows = [
        Row(
            name=r.name,
            utilization_pct=round(rps * r.effective_demand_ms / 10, 1),  # rps x ms / 1000 x 100
            status=classify(rps * r.effective_demand_ms / 1_000),
            max_rps=round(1_000 / r.effective_demand_ms, 1),
        )
        for r in resources
    ]
    return Report(
        bottleneck=worst.name,
        max_throughput_rps=round(1_000 / worst.effective_demand_ms, 1),
        min_response_ms=round(total, 1),
        knee_concurrency=round((total + think_ms) / worst.effective_demand_ms, 1),
        rows=rows,
    )


def required_servers(resources: list[Resource], rps: float, target_utilization: float) -> dict[str, int]:
    """Servers per resource so that each one stays at or below the target utilization."""
    if not 0 < target_utilization <= 1:
        raise ValueError("target_utilization must be in (0, 1]")
    out = {}
    for r in resources:
        needed = rps * r.demand_ms / 1_000 / target_utilization
        out[r.name] = max(1, int(needed) + (needed > int(needed)))
    return out


SAMPLE = [
    Resource(name="web_cpu", demand_ms=8, servers=8),
    Resource(name="app_cpu", demand_ms=25, servers=16),
    Resource(name="db_cpu", demand_ms=6, servers=8),
    Resource(name="db_disk", demand_ms=4, servers=2),
    Resource(name="db_row_lock", demand_ms=2.5, servers=1),
]


def main() -> int:
    print(json.dumps({
        "at_300_rps": analyze(SAMPLE, 300).model_dump(),
        "at_450_rps": analyze(SAMPLE, 450).model_dump(),
        "servers_for_1000_rps_at_70pct": required_servers(SAMPLE, 1_000, 0.7),
    }, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Terminal lab

Setup

cd topics/concurrency-utilization-saturation-and-bottlenecks/examples/typescript
pnpm install --ignore-workspace

Run

Start the service, which has a limit of 10 slots, a queue of 20, and 100 ms of work:

PORT=3000 pnpm start

In another terminal, send 60 requests at once, then read the stats:

seq 1 60 | xargs -P 60 -I{} curl -s -o /dev/null -w '%{http_code}\n' localhost:3000/query | sort | uniq -c

curl -s localhost:3000/stats | jq -c .

Now predict a real system's limit:

cd ../python && uv run python -m app.main | jq -c '.at_300_rps | {bottleneck,max_throughput_rps,min_response_ms,knee_concurrency}'

uv run python -m app.main | jq -c '.at_450_rps.rows[] | {name,utilization_pct,status}'

On Linux, look at real saturation signals:

vmstat 1 5

cat /proc/pressure/cpu

Test

cd ../typescript && pnpm test

cd ../python && uv run pytest -q

Expected output

Measured on the review date. With 60 requests at once, 10 run and 20 queue, and the other 30 are rejected:

  30 200
  30 503

After two such rounds, the stats read:

{"inFlight":0,"limit":10,"queued":0,"queueLimit":20,"utilization":0.17,"completed":60,"rejected":60,"meanWaitMs":90.4}

Utilization of 0.17 shows that the service was idle between bursts, so an average over time hid a total overload. That's the lesson's point in one number. The mean wait is near 100 ms, since the three groups of ten waited 0, 100, and 200 ms. Your split may vary slightly if your machine launches curl slowly. The analysis prints:

{"bottleneck":"db_row_lock","max_throughput_rps":400.0,"min_response_ms":45.5,"knee_concurrency":18.2}
{"name":"web_cpu","utilization_pct":45.0,"status":"ok"}
{"name":"app_cpu","utilization_pct":70.3,"status":"ok"}
{"name":"db_cpu","utilization_pct":33.8,"status":"ok"}
{"name":"db_disk","utilization_pct":90.0,"status":"near saturation"}
{"name":"db_row_lock","utilization_pct":112.5,"status":"saturated"}

Cleanup

Stop the service with Ctrl-C, then:

rm -rf ../typescript/node_modules .venv

Troubleshooting

• /proc/pressure exists only on Linux kernels with pressure stall information enabled. On macOS, use vm_stat and iostat.
• Fewer 503 responses than expected: curl is starting too slowly to create overlap, so raise the count to 200.
• EADDRINUSE: change PORT, or find the holder using lsof -i :3000.

Practical challenges

Industry standards and trends

Standards and specifications

RFC 9110, June 2022, defines 503 and Retry-After, which signal overload. Linux pressure stall information, in kernel documentation, standardizes saturation signals on Linux. These operational laws come from research, in particular The Operational Analysis of Queueing Network Models, Denning and Buzen, 1978.

Established practice

The USE Method, by Brendan Gregg, gives a checklist for each resource. The Four Golden Signals, in Site Reliability Engineering, 2016, includes saturation. These are widely used conventions, not formal standards.

Managed-service options

Managed databases expose connection counts, and often wait-event data. Connection proxies from cloud vendors limit concurrency toward a database. Serverless platforms have per-function concurrency settings that behave as limits. Check what happens when you reach each one, since it might queue or reject.

Emerging approaches

Adaptive concurrency limits, such as Netflix's open-source concurrency-limits library, adjust limits from measured latency. eBPF tools measure off-CPU time and lock waits with low overhead. GPU workloads are now common bottlenecks whose utilization metric can mislead, since a busy indicator doesn't reveal how fully the device is used. Evaluate each against your own workload.

Recent changes

Pressure stall information is widely available in current Linux distributions and is used by container tools. Check your kernel version and configuration. No standards change was identified at the review date.

Adoption constraints

Rejecting traffic needs clients that back off, and many don't. Adaptive limiters need careful evaluation for stability. Logical resources require instrumentation in application code, which takes ongoing effort.

Links in this section were checked on 2026-11-03 (access date). Items labelled as emerging or inferred are not settled facts and should be re-checked before you rely on them.

Interview preparation

Review

Key takeaways

• Concurrency is how many are in progress, utilization is how busy, and saturation is how much is waiting.
• U = X × D, and X ≤ c ÷ D, so demand measurements predict limits.
• L = X × R links rate, latency, and concurrency.
• Averages hide bursts, single-threaded limits, and logical resources.
• Limit concurrency in front of a bottleneck, and reject the excess cheaply.
• Fixing a bottleneck reveals the next one, so measure again.

Decision checklist

• Which resources are on the path, including locks and pools?
• What is each resource's demand and server count?
• Which one has the lowest maximum rate?
• Is any queue growing right now?
• What's the next bottleneck after this fix?

Production-readiness checklist

• Every bounded resource exports utilization and saturation.
• Slow traces separate waiting from working.
• Limits and queues are bounded, and they reject cheaply.
• Autoscaling maximums respect downstream limits.
• Failure capacity has been computed per resource.

Glossary

• Service demand: total time a request needs from a resource.
• Knee: the concurrency beyond which waiting grows.
• USE method: a check of utilization, saturation, and errors for every resource.
• Off-CPU time: time a thread spends blocked or waiting.
• Bulkhead: isolating resources so that one failure can't drain all capacity.
• Pressure stall information: a Linux metric reporting time lost waiting on a resource.

Quiz

Further study

Go on to Little’s Law and practical queueing theory (not yet written) for how waits grow near saturation, CPU, heap, lock-contention, and I/O profiling (not yet written) for measuring these resources with profilers, and Golden Signals, RED, and USE methods (not yet written) for monitoring methods. Load shedding and admission control (not yet written) covers what to do when saturation arrives anyway.

Progress

Progress is stored only in this browser. Export it from the curriculum page.