Throughput

Orientation

Definition

Throughput is the rate at which a system completes work, measured in units per time: requests per second, messages per second, or bytes per second. Goodput is the part of that work that was actually useful: completed successfully, in time, and not duplicated.

Why it matters

Throughput sets how much a system can do and therefore how much it costs per unit of work. It has a hard limit at the most constrained resource, called the bottleneck, and improving anything else accomplishes nothing. The gap between throughput and goodput is where many major outages live. A server can run flat out at its maximum rate while serving nobody, because every response arrives after the client has left. Dashboards show a busy, productive system during what users experience as a total outage.

Prerequisites

Latency

Study time

About 1.3 hours, including the terminal lab and challenges. This is an estimate.

Related topics

Concurrency, utilization, saturation, and bottlenecks, Little’s Law and practical queueing theory (not yet written), Back-of-the-envelope capacity estimation

Learning objectives

• Differentiate throughput, goodput, capacity, and bandwidth, each with its correct unit.
• Find the bottleneck in a pipeline, and explain why other improvements do nothing.
• Explain why adding workers eventually stops helping, using Amdahl's Law and the Universal Scalability Law.
• Show why an unbounded queue destroys goodput during overload, and how bounding it helps.
• Describe when batching improves throughput and what it costs in latency.

Production scenario

A payments company processes settlement files and handles 900 transactions per second at month end. Leadership asks for 3,000. The team doubles application servers from 40 to 80 and gets just 940 transactions per second. They double them again, and throughput falls to 880. Each transaction needs a row lock on a merchant balance, and a few large merchants receive most of the volume. Application servers were never the bottleneck: lock wait on those rows was, and more servers meant more contention.

The bottleneck analysis takes one afternoon. Transactions wait on locks for 70% of their duration. The fix keeps the servers at 40, batches balance updates per merchant, applying 200 ledger entries in one transaction, and partitions balances into sub-accounts for the largest merchants. Throughput reaches 3,400 per second. Doubling hardware cost a quarter's budget and made things worse, and a design change tripled the throughput at no extra hardware cost.

Mental model

Analogy

Picture a highway leading to a two-lane bridge. Traffic flow is limited by the bridge, however many lanes lead up to it. Widening the highway elsewhere only lengthens the queue before the bridge. Past a certain density, cars get in each other's way and flow actually falls, which is why jammed highways move fewer cars than busy ones moving freely.

Where the analogy breaks

Cars physically occupy space, whereas software requests occupy abstract resources such as locks and connections, so a bottleneck may be invisible on any infrastructure graph. Drivers also don't abandon a trip halfway and leave a ghost car still using the bridge. Software requests do: a server may keep processing work whose client timed out. This wasted work has no road equivalent, and it is the main source of goodput collapse.

Entities, state, flow, and invariants

• Entities: arrivals, queues, workers, shared resources such as locks and disks.
• State: queue length, in-flight count, and resource utilization.
• Flow: arrive, wait, get served, complete, with possible timeout and retry.
• Invariant 1: in steady state, throughput cannot exceed the capacity of the bottleneck.
• Invariant 2: in steady state, completions equal arrivals, otherwise a queue is growing or the system is rejecting work.
• Invariant 3: concurrency = throughput × response time, which is Little's Law. See Little’s Law and practical queueing theory (not yet written).

Foundations

Concepts and terminology

Bits and bytes cause frequent errors: 1 Gbit/s is 125 MB/s at most, and less after protocol overhead.

First-principles explanation

For a pipeline of stages, the sustainable rate equals the smallest stage capacity. If an API can do 5,000 rps, a cache 50,000, and a database 1,200, the system does at most 1,200 for requests requiring the database. Doubling API servers changes nothing. Hence the first question in any throughput problem is "which resource is saturated?" See Concurrency, utilization, saturation, and bottlenecks.

Adding workers scales linearly only if they share nothing. Amdahl's Law says that a serial fraction σ limits speedup to 1 ÷ σ, however many workers you add. With 5% serialized, the limit is 20 times. The Universal Scalability Law adds a coherency cost that grows with pairs of workers, since each must stay consistent with others. This term makes throughput actually fall after a peak, matching the scenario above. Locks, cache-line invalidation, cluster gossip, and distributed coordination can all cause it.

Batching works because many operations have a large fixed cost and a small per-item cost: a disk flush, a network round trip, a transaction commit. Combining items pays the fixed cost once. If a commit costs 5 ms and each row costs 0.05 ms, one row per commit gets about 198 rows per second, while 100 rows per commit gets 10,000 rows per second. The cost is that each item waits for the batch to fill. See Batching, pipelining, and parallelism (not yet written).

Mathematics

Amdahl: speedup(N) = 1 ÷ (σ + (1 − σ) ÷ N).

USL: X(N) = λN ÷ (1 + σ(N − 1) + κN(N − 1)), peaking at N = √((1 − σ) ÷ κ).

Example: λ = 1,000 rps, σ = 0.05, κ = 0.0005. The peak is at √(0.95 ÷ 0.0005) = √1,900 ≈ 44 workers. X(44) = 44,000 ÷ (1 + 2.15 + 0.0005 × 44 × 43) = 44,000 ÷ (3.15 + 0.946) = 44,000 ÷ 4.096 ≈ 10,742 rps. Past that, adding workers reduces throughput.

Little's Law: L = λW. At 2,000 rps and 50 ms: L = 2,000 × 0.05 = 100 requests in flight. So 80 connections cap throughput at 80 ÷ 0.05 = 1,600 rps, no matter how fast everything else is.

TCP limit: throughput ≤ window size ÷ RTT. A 64 KB window at 100 ms RTT gives 640 KB/s, regardless of bandwidth. See Congestion control and head-of-line blocking (not yet written).

Complexity

Throughput per worker is the inverse of service time, so reducing an operation from O(n) to O(log n) can change capacity by orders of magnitude where n is large. Lock contention behaves differently: it does not show up in big-O analysis, since it depends on concurrency rather than input size. Sharding removes the shared term by giving each partition its own lock, which is why it often beats optimizing code.

Mechanism

Normal path

1. Work arrives at some offered rate.
2. If a worker is free, service begins. Otherwise the work is queued.
3. As long as offered load stays below capacity, throughput matches offered load, and latency stays near service time.
4. Close to capacity, throughput still rises, but queues and latency grow quickly.
5. At capacity, throughput flattens, and the excess must either queue or be rejected.
6. Beyond it, one of two things happens. With a bounded queue, goodput stays near capacity. With an unbounded queue, wait time exceeds deadlines, and goodput collapses.

State transitions

stateDiagram-v2
    [*] --> Underloaded: throughput equals offered load
    Underloaded --> NearCapacity: load nears bottleneck
    NearCapacity --> Underloaded: load falls
    NearCapacity --> Shedding: overload with bounded queue
    NearCapacity --> Collapse: overload with unbounded queue
    Shedding --> NearCapacity: load falls
    Collapse --> Collapse: retries sustain the queue
    Collapse --> NearCapacity: load removed and queue drained

The notable difference between Shedding and Collapse is the exit. Shedding recovers as soon as load falls. Collapse can sustain itself through retries after the original cause has ended, and it often requires deliberate removal of load.

Concurrency behavior

Throughput increases with concurrency until a shared resource saturates. Past that, more concurrency only lengthens queues, and it might degrade throughput through contention and context switching. Concurrency limits, connection pools and semaphores are therefore useful even when CPU is free, as they keep work outside the contended region. See Connection pooling (not yet written) and Bulkheads and isolation pools (not yet written).

Ordering and consistency

Strict ordering reduces throughput, because ordered items can't be processed in parallel. Partition keys are the usual compromise: ordering within a key and parallelism across keys. A single very hot key caps throughput at what one consumer can do. See Ordering and partition keys (not yet written). Likewise, a stronger durability guarantee requires a flush or a remote acknowledgment for each commit, and batching commits is how databases keep throughput while preserving it.

Ownership of data and control

Whoever owns the shared resource owns its capacity. When several services share one database, none can guarantee its own throughput without quotas or a designated capacity owner. Flow control is another ownership question: the consumer must be able to signal the producer. If nothing does, the queue in between absorbs the mismatch until it fails. See Backpressure (not yet written).

Architecture

flowchart LR
    C["Clients: 3,000 rps offered"] -->|"HTTPS"| LB["Load balancer: 50,000 rps"]
    LB --> A["API servers: 5,000 rps"]
    A -->|"reads"| CA["Cache: 50,000 rps, holds copies"]
    A -->|"TCP, 80 connections"| P["Connection pool"]
    P --> D["Database, owns data: 1,200 writes per second"]
    D --> L["BOTTLENECK: row lock on hot balance"]

Every stage has its own limit, and the system's write rate equals the lowest figure on the path, here the lock. Doubling API servers leaves that number unchanged. Sometimes the limit is logical, such as a lock, and doesn't appear on CPU or disk graphs. All rates shown are illustrative.

flowchart LR
    subgraph py["Python: goodput simulation"]
      AR["Seeded arrivals"] --> Q["FIFO queue: optional limit"]
      Q --> W["Worker heap: 10 workers"]
      W --> R["Throughput, goodput, rejected"]
      Q -->|"queue is full"| REJ["Rejected early"]
    end
    subgraph ts["TypeScript: scaling model"]
      S["Load test samples: workers, rps"] --> FIT["Least-squares USL fit"]
      FIT --> PK["Peak concurrency and predictions"]
    end

The two examples answer different questions. The USL fit asks how far adding workers will carry you, and the simulation asks what happens when load exceeds the result. Together they describe capacity and overload behavior.

sequenceDiagram
    participant C as Client
    participant Q as Queue
    participant W as Worker
    C->>Q: Request A
    Note over Q: Queue wait exceeds 200 ms
    C--xQ: Deadline passes, client gives up
    C->>Q: Retry A2 joins the back of the queue
    Q->>W: Start original A
    W-->>Q: A done, but nobody is waiting
    Note over W: Counted in throughput, not goodput
    Q->>W: Start A2, which is also late

This is how throughput stays high while goodput reaches zero. The fix has two parts: bound the queue, so that waiting stays below the deadline, and propagate deadlines, so that workers can drop expired work rather than do it. See Deadlines and cancellation propagation (not yet written).

flowchart TD
    A["Throughput below target"] --> B["Add application servers"]
    B --> C["More concurrent transactions"]
    C --> D["Lock contention on hot row grows"]
    D --> E["Throughput falls while cost rises"]
    E --> B
    D -->|"profile shows 70 percent lock wait"| F["Batch updates and split hot accounts"]
    F --> G["Bottleneck moves. Measure again"]

The loop back to B represents what happens if nobody identifies the bottleneck. Notice that the good path ends with "bottleneck moves". Removing a bottleneck always reveals the next one, so capacity work is iterative and each step needs another measurement.

Trade-offs

Advantages

High throughput per unit of hardware reduces cost, leaves room for growth, and shortens backlogs. Batching, partitioning, and asynchronous processing often raise throughput dramatically compared with adding hardware.

Costs

• Batching and queuing raise latency for each item.
• Partitioning adds operational complexity and loses cross-partition transactions.
• Running near capacity leaves little headroom, and it raises queueing delay.
• Bounded queues reject work on purpose, which requires clients to handle the rejection.

Alternatives

• Vertical scaling: simple, often enough, and removes coordination costs up to a limit. See Vertical scaling (not yet written).
• Horizontal scaling: effective if work has low contention. See Horizontal scaling (not yet written).
• Reducing work: caching, removing unneeded calls, and doing less per request.
• Asynchronous processing: decouples the rate of acceptance from the rate of processing.

When to use it

Focus on throughput for batch pipelines, ingestion, month-end processes, and any system whose cost scales with volume. Use goodput as the headline measure for services with deadlines.

When not to use it

Do not maximize throughput on interactive paths at latency's expense: a large batch window can ruin user experience. Do not tune below your actual load: an unused capacity gain is merely a cost.

Reconsider when

• A load test shows that adding workers no longer increases throughput.
• Goodput and throughput diverge.
• The bottleneck has moved since your last analysis.
• The cost per unit of work is rising along with volume, which suggests contention.

Capacity planning

Inputs and assumptions

Target: 3,000 transactions per second at peak. A single-row commit costs 5 ms of a shard's serialized time. Batching can apply 200 entries in one commit, costing 15 ms in total. Each batch waits at most 50 ms to fill. Each application server handles 500 transactions per second (assumption from a load test).

Formula and units

• Unbatched per-partition rate = 1 ÷ commit time, in transactions per second.
• Batched per-partition rate = batch size ÷ batch commit time.
• Partitions = target ÷ per-partition rate.
• Connections = throughput in commits per second × commit duration in seconds, from Little's Law.

Intermediate calculations

• Unbatched: 1 ÷ 0.005 s = 200 per second for each hot row. 3,000 ÷ 200 = 15 partitions if work spreads evenly.
• Batched: 200 ÷ 0.015 s = 13,333 per second per partition, which is more than the target on one.
• Batches needed: 3,000 ÷ 200 = 15 batches per second at full size, with a 67 ms fill time at that rate. So the 50 ms limit triggers first, with 150 entries per batch and 20 batches per second.
• Utilization: 20 × 15 ms = 300 ms of commit time each second, or 30% of one partition's serialized time.
• App servers: 3,000 ÷ 500 = 6.

Headroom

Run at 70% of capacity for latency and forecast error: 3,000 ÷ 0.7 = 4,286 per second to provision, or 9 application servers (8.57 rounded up). Since batching adds up to 50 ms, confirm it still meets the latency budget.

Failure capacity

Losing one of three zones: 9 × 3 ÷ 2 = 13.5, so 14 servers. If a ledger primary fails, writes pause during failover, so the backlog at 3,000 per second for 30 s reaches 90,000 entries. Draining this within 5 minutes requires 300 extra per second, an easy load at 13,333 per partition, so the batched design recovers quickly. Without batching, a 200-per-second hot row could never catch up.

Failure modes

Security and privacy

Threats

Throughput limits are a target. A denial-of-service attacker aims at the most expensive operation, which is effectively your real capacity limit. A small number of slow or costly requests, such as complex searches, can do as much damage as a flood of cheap ones. Compression bombs and oversized payloads turn small requests into large amounts of work.

Trust boundaries

Enforce rate and size limits at the first point that you control, before expensive work starts. Internal services need limits as well, since a buggy internal caller can do more damage than an external attacker.

Authentication and authorization

Expensive cryptographic checks reduce throughput, and they can be abused by sending invalid credentials that must be verified. Cache verified results, rate-limit failures, and place cheap checks before expensive ones. When a limit depends on identity, authenticate first using an inexpensive method.

Tenant isolation

Without isolation, one tenant's traffic consumes throughput meant for everyone. Use per-tenant quotas measured in resource cost, not just request count. Fair queueing prevents any one tenant from dominating a shared queue. See Tenant isolation and noisy-neighbor controls (not yet written).

Sensitive data

Batching can mix different tenants' data in one payload or log line, so keep tenant identifiers per item and avoid logging batch contents. Encryption costs CPU, but don't remove it to gain throughput. Measure it first, since it's often far smaller than assumed.

Abuse resistance

Limit by cost, reject oversized and over-complex requests early, and treat rejection as cheap. A rejected request should cost a tiny fraction of a served one. If it doesn't, rejecting offers no protection.

Observability

Metrics

Report throughput by route and outcome, goodput as successful requests within deadline, rejected rate, and utilization and saturation for each resource. Also report in-flight requests, pool wait time, queue depth and age, and batch size distribution. See Golden Signals, RED, and USE methods (not yet written).

Logs

Log rejected and expired work in sampled form with a reason, as high-volume logging during overload can itself cause harm. Batches should log size and outcome.

Traces

Traces reveal lock waits and pool waits as gaps or spans. Use span links to represent batches, since one batch trace may relate to many request traces.

Dashboards

Plot offered load, throughput, and goodput together. A separation between them is the clearest sign of overload. Add a chart of throughput against concurrency to spot retrograde scaling.

Alerts

Page on goodput SLO burn, and on queue age over the deadline. Raise a ticket when peak utilization of a bottleneck stays above its planned level or when adding instances no longer increases throughput.

SLO examples

• 99.9% of accepted requests complete successfully within their deadline over 28 days.
• Settlement backlog stays under 5 minutes at peak in 99% of month-end runs.
• Queue age remains under 100 ms for 99% of minutes.

Runbook actions

If goodput drops: compare offered load with historical capacity, and find the saturated resource. Shed non-critical traffic. If retries amplify load, limit them. Confirm expired work is being dropped. Only add instances if the bottleneck really is the instance tier.

Implementation

Architecture of the examples

Python is a seeded discrete-event simulator of a 10-worker pool with a capacity of 1,000 requests per second. TypeScript fits the USL to load-test results. Both are offline tools. Simulated time makes overload behavior repeatable, and it lets a 30-second run complete in well under a second.

Data structures

The simulator uses a deque for FIFO waiting and a min-heap of finish times for workers, which gives O(log w) per event with w workers. Result separates throughput, goodput, and rejection, since merging those would hide the lesson. The USL fit uses closed-form normal equations for two parameters.

Interfaces

simulate(offered_rps, Config) returns Result. Run python -m app.main [--queue-limit N] to print a load sweep as JSON lines. TypeScript exports uslThroughput, uslPeak, and fitUsl, and pnpm start [samples.json] runs a fit.

Error handling

Invalid rates and worker counts raise explicit errors. The fit requires a baseline at n = 1 and two more samples, because a fit from too little data would look authoritative while being meaningless.

Concurrency considerations

Neither tool uses real concurrency. The simulator advances a virtual clock, so results depend only on the seed, and tests never flake. That's a deliberate choice: testing overload behavior with real timers tends to be slow and unreliable. See Deterministic distributed-system simulation (not yet written).

Educational limitations

The simulation has no retries, no dropped expired work, and a single queue, so it may understate real collapse. The USL is an empirical model: it describes data, and it does not explain causes. The TypeScript sample data is invented for teaching and isn't from any real system.

Production hardening

Measure real systems at several concurrency levels. Add deadline propagation and expired-work dropping, per-tenant fairness, and adaptive concurrency limits. Validate the fitted model against a new measurement before basing a purchase on it.

Example source

// Universal Scalability Law: why adding workers can stop helping, and then start to hurt.
// X(N) = lambda * N / (1 + sigma * (N - 1) + kappa * N * (N - 1))
// sigma models contention (serialized work), and kappa models coherency (cross-talk between workers).
import { readFileSync } from 'node:fs';
import { fileURLToPath } from 'node:url';

export interface UslParams { lambda: number; sigma: number; kappa: number }
export interface Sample { n: number; throughput: number }

export class InputError extends Error {}

export function uslThroughput(n: number, p: UslParams): number {
  if (!Number.isInteger(n) || n < 1) throw new InputError(`n must be a positive integer, got ${n}.`);
  return (p.lambda * n) / (1 + p.sigma * (n - 1) + p.kappa * n * (n - 1));
}

// Peak concurrency, from setting dX/dN = 0. With no coherency cost, throughput never peaks.
export function uslPeak(p: UslParams): number | null {
  if (p.kappa <= 0) return null;
  return Math.sqrt((1 - p.sigma) / p.kappa);
}

// The USL rearranges to a quadratic: N / (X(N) / X(1)) - 1 = kappa * x^2 + (sigma + kappa) * x,
// where x = N - 1. This fits it by least squares with no intercept.
export function fitUsl(samples: Sample[]): UslParams {
  const base = samples.find((s) => s.n === 1);
  if (!base || base.throughput <= 0) throw new InputError('Samples must include n = 1 with positive throughput.');
  const points = samples.filter((s) => s.n > 1);
  if (points.length < 2) throw new InputError('Need at least two samples with n > 1.');
  let x2 = 0, x3 = 0, x4 = 0, xy = 0, x2y = 0;
  for (const s of points) {
    const x = s.n - 1;
    const y = s.n / (s.throughput / base.throughput) - 1;
    x2 += x * x; x3 += x ** 3; x4 += x ** 4; xy += x * y; x2y += x * x * y;
  }
  const det = x4 * x2 - x3 * x3;
  if (det === 0) throw new InputError('Samples are degenerate; use more distinct values of n.');
  const a = (x2y * x2 - xy * x3) / det; // kappa
  const b = (xy * x4 - x2y * x3) / det; // sigma + kappa
  return { lambda: base.throughput, sigma: Math.max(0, b - a), kappa: Math.max(0, a) };
}

const SAMPLE: Sample[] = [
  { n: 1, throughput: 955 }, { n: 4, throughput: 3_400 }, { n: 8, throughput: 5_900 },
  { n: 16, throughput: 8_700 }, { n: 32, throughput: 10_100 }, { n: 64, throughput: 8_400 },
];

if (process.argv[1] === fileURLToPath(import.meta.url)) {
  try {
    const path = process.argv[2];
    const samples = path ? (JSON.parse(readFileSync(path, 'utf8')) as Sample[]) : SAMPLE;
    const params = fitUsl(samples);
    const peak = uslPeak(params);
    console.log(JSON.stringify({ level: 'info', msg: 'usl.fit', sigma: +params.sigma.toFixed(4), kappa: +params.kappa.toFixed(5), peakWorkers: peak === null ? null : Math.round(peak) }));
    for (const n of [1, 8, 16, 32, 64, 128]) {
      console.log(JSON.stringify({ n, predictedRps: Math.round(uslThroughput(n, params)), linearRps: Math.round(params.lambda * n) }));
    }
  } catch (err) {
    console.error(JSON.stringify({ level: 'error', msg: 'usl.failed', error: err instanceof Error ? err.message : String(err) }));
    process.exitCode = 2;
  }
}

"""Discrete-event simulation of a server pool, separating throughput from goodput.

Throughput counts everything completed. Goodput counts only what finished before the client's
deadline. Under overload with an unbounded queue, throughput stays high while goodput collapses,
because servers spend their time on requests whose clients have already given up. A bounded queue
rejects the excess early and keeps goodput near capacity. The simulation is seeded and repeatable.
"""

from __future__ import annotations

import argparse
import heapq
import json
import random
from collections import deque

from pydantic import BaseModel, Field


class Config(BaseModel):
    workers: int = Field(default=10, gt=0)
    service_ms: float = Field(default=10, gt=0)
    deadline_ms: float = Field(default=200, gt=0)
    duration_s: float = Field(default=30, gt=0)
    queue_limit: int | None = Field(default=None, ge=0)
    seed: int = 11


class Result(BaseModel):
    offered_rps: float
    throughput_rps: float
    goodput_rps: float
    rejected_rps: float
    mean_response_ms: float


def simulate(offered_rps: float, cfg: Config) -> Result:
    if offered_rps <= 0:
        raise ValueError("offered_rps must be positive")
    rng = random.Random(cfg.seed)
    end_ms = cfg.duration_s * 1_000
    queue: deque[float] = deque()
    finish: list[float] = []  # min-heap of worker completion times
    t = 0.0
    completed = good = rejected = 0
    total_response = 0.0

    def start_ready(now: float) -> None:
        nonlocal completed, good, total_response
        while queue and len(finish) < cfg.workers:
            arrived = queue.popleft()
            done = now + rng.expovariate(1 / cfg.service_ms)
            heapq.heappush(finish, done)
            if done <= end_ms:
                completed += 1
                total_response += done - arrived
                if done - arrived <= cfg.deadline_ms:
                    good += 1

    while t < end_ms:
        t += rng.expovariate(offered_rps / 1_000)
        while finish and finish[0] <= t:
            now = heapq.heappop(finish)
            start_ready(now)
        if cfg.queue_limit is not None and len(queue) >= cfg.queue_limit and len(finish) >= cfg.workers:
            rejected += 1
            continue
        queue.append(t)
        start_ready(t)

    seconds = cfg.duration_s
    return Result(
        offered_rps=offered_rps,
        throughput_rps=round(completed / seconds, 1),
        goodput_rps=round(good / seconds, 1),
        rejected_rps=round(rejected / seconds, 1),
        mean_response_ms=round(total_response / completed, 1) if completed else 0.0,
    )


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--queue-limit", type=int, default=None)
    args = parser.parse_args(argv)
    cfg = Config(queue_limit=args.queue_limit)
    # 10 workers at 10 ms each: capacity is 1,000 requests per second.
    for load in (250, 500, 800, 950, 1_100, 1_500, 3_000):
        print(simulate(load, cfg).model_dump_json())
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Terminal lab

Setup

cd topics/throughput/examples/python
uv sync --quiet

Run

Sweep the offered load with an unbounded queue:

uv run python -m app.main | jq -c '{offered_rps, throughput_rps, goodput_rps, mean_response_ms}'

Now repeat with a queue bounded to 50 requests:

uv run python -m app.main --queue-limit 50 | jq -c '{offered_rps, throughput_rps, goodput_rps, rejected_rps, mean_response_ms}'

Fit the scalability model:

cd ../typescript && pnpm install --ignore-workspace --silent && pnpm --silent start

To check a connection-limited throughput on a real system, compare in-flight requests with the Little's Law value. This counts established connections on a port:

lsof -nP -iTCP:3000 -sTCP:ESTABLISHED | wc -l

Test

cd ../python && uv run pytest -q

cd ../typescript && pnpm test

Expected output

Measured on the review date, with capacity at 1,000 requests per second. Without a queue limit:

{"offered_rps":950.0,"throughput_rps":949.6,"goodput_rps":949.6,"mean_response_ms":22.3}
{"offered_rps":1100.0,"throughput_rps":998.8,"goodput_rps":60.0,"mean_response_ms":1547.7}
{"offered_rps":1500.0,"throughput_rps":999.5,"goodput_rps":21.0,"mean_response_ms":5140.0}
{"offered_rps":3000.0,"throughput_rps":1003.0,"goodput_rps":8.6,"mean_response_ms":9943.4}

With --queue-limit 50:

{"offered_rps":950.0,"throughput_rps":951.2,"goodput_rps":951.2,"rejected_rps":2.7,"mean_response_ms":23.8}
{"offered_rps":1100.0,"throughput_rps":997.6,"goodput_rps":997.6,"rejected_rps":106.2,"mean_response_ms":50.1}
{"offered_rps":1500.0,"throughput_rps":999.5,"goodput_rps":999.5,"rejected_rps":505.7,"mean_response_ms":57.9}
{"offered_rps":3000.0,"throughput_rps":1003.0,"goodput_rps":1003.0,"rejected_rps":2001.6,"mean_response_ms":59.2}

Throughput is the same in both runs. A 10% overload takes goodput from about 950 down to 60 without a limit, and rejecting just 106 requests per second keeps it near 1,000. The USL fit prints sigma 0.0324, kappa 0.00105, and a peak near 30 workers. Predicted throughput peaks around 10,035 rps at 32 workers and falls to 5,512 at 128. The input data is invented for this lesson.

Cleanup

rm -rf ../python/.venv node_modules

Troubleshooting

• command not found: uv: install uv or use another Python 3.12+ environment.
• lsof prints only a header when nothing is listening on the port. This is expected if no server is running.
• To fit your own data, include a measurement at n = 1, or else the fit is rejected.

Practical challenges

Industry standards and trends

Standards and specifications

• RFC 2544, March 1999, describes a benchmarking method for network devices, including throughput with zero loss.
• RFC 6349, August 2011, describes a framework for TCP throughput testing.
• RFC 9293, August 2022, is the current TCP specification.

These cover network measurement. There is no general standard for application throughput, so state your method.

Established practice

The USL is described in Neil Gunther's USL work. Goodput and overload handling are discussed in Handling Overload and Addressing Cascading Failures, chapters of Site Reliability Engineering, 2016. These are influential published practices, not standards.

Managed-service options

Managed queues and streams, serverless databases, and managed load balancers all have quotas that define your limit. Treat each published quota as a bottleneck candidate and confirm whether burst and sustained limits differ. Provisioned capacity generally trades cost for predictable throughput.

Emerging approaches

Adaptive concurrency limits, which adjust limits automatically from observed latency, are used in several service proxies and libraries. io_uring improves I/O throughput on Linux by reducing system-call overhead, though its security history has led some environments to restrict it. For GPU inference, batching and token throughput matter more than request rate. These observations rest on public projects and should be checked against your own workload.

Recent changes

No major standards change was identified in this review. In practice, processor core counts continue to rise, so contention and memory-locality limits increasingly matter more than raw CPU speed. See CPU caches, locality, false sharing, and NUMA (not yet written).

Adoption constraints

Rejecting traffic requires clients that handle rejection well, and many do not. Batching demands idempotent retries and per-item error handling. Measuring goodput depends on knowing the client's deadline, which needs propagation across services.

Links in this section were checked on 2026-11-03 (access date). Items labelled as emerging or inferred are not settled facts and should be re-checked before you rely on them.

Interview preparation

Review

Key takeaways

• Throughput is limited by the bottleneck, and everything else is irrelevant until it moves.
• Goodput, not throughput, measures useful work.
• With unbounded queues, overload can leave throughput high and goodput near zero.
• Bounded queues reject work on purpose, and this raises useful output.
• Contention caps speedup, and coherency can make throughput fall as workers increase.
• Batching spreads fixed costs at the price of added latency.

Decision checklist

• Have I named and measured the bottleneck?
• Am I using goodput for services with deadlines?
• Have I converted every limit to the same unit?
• Have I checked in-flight limits using Little's Law?
• Does a scaling test show that more workers really help?

Production-readiness checklist

• All queues are bounded, and waiting is less than the deadline.
• Every bounded resource exports a saturation metric.
• Expired work gets dropped.
• Retries are limited by a budget.
• Autoscaling maximums account for downstream limits.

Glossary

• Bottleneck: the resource that limits total throughput.
• Goodput: useful work completed per unit of time.
• Offered load: the rate at which work arrives.
• Contention: waiting on a shared resource.
• Coherency cost: the cost of keeping workers' state in agreement.
• Retrograde scaling: throughput falling as concurrency rises.

Quiz

Further study

Go on to P99 latency and Concurrency, utilization, saturation, and bottlenecks for how to read saturation, then Little’s Law and practical queueing theory (not yet written) for queue math. Batching, pipelining, and parallelism (not yet written) covers batching in depth, and Load shedding and admission control (not yet written) covers staying productive under overload.

Progress

Progress is stored only in this browser. Export it from the curriculum page.