Latency

Orientation

Definition

Latency is the time between a cause and its effect: for a request, the time from when it is sent to when a response arrives. It always needs two qualifiers: where it is measured, and which part of the distribution you mean. "50 ms" means nothing until you say "p99, measured at the load balancer, including time to last byte."

Why it matters

Latency is what users feel, and it is a capacity signal: it rises well before failure. It also composes unforgivingly. Each hop adds to it, and retries multiply it. Physics sets a floor no budget can buy back, since a round trip between New York and London takes at least roughly 56 ms in fiber on a direct path, and more on a real route. That floor decides whether cross-region synchronous writes are feasible at all. Distinguishing the delay you can remove from the delay you cannot is one of the most useful reflexes in design.

Prerequisites

None

Study time

About 1.3 hours, including the terminal lab and challenges. This is an estimate.

Related topics

P99 latency, Tail latency, Latency budgets and critical-path analysis

Learning objectives

• Decompose a request's latency into propagation, transmission, processing, and queueing delay.
• Compute physical floors and round-trip costs for connection setup.
• Explain why latency rises sharply as utilization approaches saturation.
• Choose the correct measurement point and metric for a latency claim.
• Use sequential and parallel composition to find where time can actually be saved.

Production scenario

A travel booking site has customers in Sydney and servers in Virginia. Search takes 3.2 seconds to display, and the team plans a database upgrade. A breakdown tells a different story. The distance is about 15,700 km, so with real routes the round trip is about 220 ms. A new connection costs three round trips before the first response byte: TCP, TLS 1.3, and the request itself, for 660 ms. The page makes 8 sequential API calls, each a round trip, for 1,760 ms. Server work totals only 300 ms, of which the database is 90 ms.

A faster database would save perhaps 50 ms of 3,200 ms. Instead, the team terminates TLS at a Sydney edge node and reuses connections to the origin, batches the eight calls into one, and caches search facets at the edge. Time to display falls under 900 ms, with no database change. The decomposition cost an afternoon, and the upgrade would have cost a quarter.

Mental model

Analogy

Sending a package has four sources of delay. The truck's travel time depends on distance (propagation). Loading many boxes takes time, depending on box count and door width (transmission). Sorting at the depot takes some time per box (processing). Waiting in line at the depot depends on how busy it is (queueing). A faster truck does not help if the depot line is the real delay, and more depots don't shorten the road.

Where the analogy breaks

Trucks carry a whole load, while networks split data into packets, which may be lost and retransmitted, and congestion control limits how fast a sender may go. A "shipment" also frequently requires several trips, such as handshakes, before any goods move. Finally, in software, waiting clients often send duplicate requests when delayed, so a slow system gets busier, which no postal customer would cause.

Entities, state, flow, and invariants

• Entities: client, network hops, proxies, service instances, queues, dependencies.
• State: connections (new or reused), queue depth, cache contents, garbage-collection state.
• Flow: DNS, TCP, TLS, request transmission, queueing, processing, dependency calls, response.
• Invariant 1: total latency is at least the sum of the physically required hops on the critical path.
• Invariant 2: sequential steps add, while parallel steps cost at least the slowest member.
• Invariant 3: a latency number has a location, percentile, and window, or it is not meaningful.

Foundations

Concepts and terminology

Note the different terms for bandwidth and latency. Bandwidth is capacity per unit time, and latency is delay. A truck full of disks has huge bandwidth and terrible latency. See Throughput.

First-principles explanation

Light in fiber travels at about 200,000 km/s, or two-thirds of c. That's 200 km per millisecond, or 5 ms per 1,000 km one way. No software changes this. For small messages over long distances, propagation dominates. For large messages over slow links, transmission does. Within a data center, propagation is negligible, around 500 µs round trip or less, so queueing and processing dominate.

Queueing comes from variation. If requests arrive at random, then some wait even if the server is idle on average. As utilization ρ grows, the mean wait grows in proportion to ρ ÷ (1 − ρ). So going from 50% to 90% multiplies wait ninefold, and 90% to 99% multiplies it another elevenfold. This nonlinearity is the reason systems look fine right up until they don't, and why a small traffic increase can cause a large latency increase. See Little’s Law and practical queueing theory (not yet written).

Protocols add round trips. New TCP needs one RTT before sending a request. TLS 1.3 adds one, and TLS 1.2 adds two. QUIC combines transport and TLS in one RTT, and resumption can achieve zero. Connection reuse eliminates all of this, which is often the greatest single saving for long-distance traffic. See Connection establishment, keep-alive, and draining (not yet written) and HTTP/2 and HTTP/3 (not yet written).

Mathematics

• Propagation (ms) = distance (km) × route factor ÷ 200 km per ms.
• Transmission (ms) = size (bytes) × 8 ÷ (bandwidth (Mbit/s) × 1,000).
• Queueing, M/D/1 mean wait = ρ ÷ (2(1 − ρ)) × service time.
• Queueing, M/M/1 mean wait = ρ ÷ (1 − ρ) × service time. Random service time doubles the M/D/1 wait.
• Time to first byte, new connection with TLS 1.3 = 3 × RTT, plus server time.

Example: 5,600 km × 1.4 ÷ 200 = 39.2 ms one way, or 78.4 ms RTT. New connection: 235 ms. Reused: 78 ms. M/D/1 with a 10 ms service time at 90% utilization: 0.9 ÷ 0.2 × 10 = 45 ms wait.

Complexity

Latency is commonly a function of algorithmic complexity plus system behavior. An O(n) scan over 10,000 items at 10 ns each adds 100 µs, which is negligible next to one disk seek or round trip. The useful habit is comparing magnitudes:

These are approximate teaching values. Measure your own hardware, since they vary substantially.

Mechanism

Normal path

1. DNS resolution, if not cached: 0 to tens of milliseconds, sometimes more. See DNS (not yet written).
2. TCP handshake: 1 RTT, unless reused.
3. TLS handshake: 1 RTT with TLS 1.3, unless resumed or reused.
4. Client transmits the request.
5. Propagation to the server, through any load balancers and proxies.
6. The request waits in queues: socket accept queue, thread pool, event loop.
7. Processing, and calls to dependencies, each repeating some of these steps.
8. Response transmission, limited by bandwidth and TCP congestion window. Large responses need several RTTs early in a connection.

State transitions

stateDiagram-v2
    [*] --> Connecting: new connection, 1 to 3 RTT
    [*] --> Sending: reused connection
    Connecting --> Sending
    Sending --> Queued: arrives at server
    Queued --> Processing: worker becomes free
    Processing --> AwaitingDependency: downstream call
    AwaitingDependency --> Processing: reply received
    Processing --> Responding
    Responding --> [*]: last byte received
    Queued --> TimedOut: deadline passes
    AwaitingDependency --> TimedOut: deadline passes
    TimedOut --> [*]

Each state has its own remedy. Connecting time falls with reuse and edge termination. Queued time falls with capacity or load shedding. Processing falls through profiling, and waiting on dependencies falls through parallelism or caching. A breakdown tells you which remedy is even relevant.

Concurrency behavior

Parallel calls cost as much as the slowest, not the sum, but they raise the probability of meeting a slow outlier. Concurrent requests contend for shared resources: connection pools, locks, CPU. Under contention, what looks like service time turns into queueing delay. On a single-threaded event loop, one slow synchronous task delays all others behind it. See Blocking versus non-blocking I/O (not yet written).

Ordering and consistency

Latency trades off against consistency. A strongly consistent write across regions must wait for a remote acknowledgment, so it pays at least one inter-region RTT. Reading from a nearby replica is fast but may be stale. This relationship is what PACELC (not yet written) formalizes: even without a partition, you're choosing between latency and consistency. Physical ordering matters too: in a sequential chain, time saved at any step counts only if the step lies on the critical path. See Latency budgets and critical-path analysis.

Ownership of data and control

Client teams own connection reuse, payload size, and the number of calls. Edge and platform teams own CDNs, TLS termination, and routing. Service teams own processing and dependency calls. Data location is an ownership decision too: placing data far from its readers imposes a fixed cost that no later optimization can remove. Each hop needs an owner for its share of the budget.

Architecture

flowchart LR
    U["User device"] -->|"last mile: variable, 1 to 3 RTT setup"| E["Edge node: TLS terminates here"]
    subgraph private["Private network: operator-owned"]
      E -->|"backbone: reused connections"| LB["Origin load balancer"]
      LB -->|"HTTP, under 1 ms"| S["Service: queueing, CPU, GC"]
      S -->|"under 1 ms"| C["Cache: owns copies"]
      S -->|"1 to 10 ms"| D["Database: owns data, disk, locks"]
    end

Moving TLS termination close to the user shortens the handshake's round trips to the nearby edge, while the long haul uses already-open connections. The trust boundary sits at the edge, so traffic behind it needs its own protection. Time values shown are illustrative only and vary widely.

flowchart LR
    C["curl or browser"] -->|"HTTP GET /feed"| R["Express route"]
    R --> T["PhaseTimer.measure"]
    T --> A["auth phase"]
    T --> D["db phase"]
    T --> P["ranking and ads: sequential or parallel"]
    T -->|"Server-Timing header"| C
    T -->|"JSON log line"| L["Structured log"]
    I["Distance, bandwidth, size"] --> Q["Python calculator"]
    Q --> O["Propagation, transmission, queueing, handshake report"]

The TypeScript side measures time inside a running service, while the Python side predicts the time physics requires. Use both in practice. Prediction tells you the floor, and measurement tells you how far you are above it.

sequenceDiagram
    participant C as Client
    participant S as Server
    Note over C,S: New connection with TLS 1.3, 3 RTT to first byte
    C->>S: TCP SYN
    S-->>C: SYN-ACK (1 RTT)
    C->>S: TLS ClientHello with ACK
    S-->>C: TLS ServerHello and Finished (2 RTT)
    C->>S: HTTP request with TLS Finished
    S-->>C: First response byte (3 RTT)
    Note over C,S: Reused connection, 1 RTT to first byte
    C->>S: HTTP request
    S-->>C: First response byte (1 RTT)

At 200 ms RTT, the difference between these is 400 ms on each new connection. Since cold-start costs land on first impressions and on clients that reconnect often, connection reuse usually beats server-side tuning for distant clients. It also explains why a short idle timeout on a load balancer can raise user latency.

flowchart TD
    A["Traffic reaches 90 percent utilization"] --> B["Queue wait grows non-linearly"]
    B --> C["Latency exceeds client timeout"]
    C --> D["Clients retry, adding load"]
    D --> E["Utilization reaches 100 percent"]
    E --> F["Servers work on requests whose clients already gave up"]
    F --> C
    E -->|"detected: queue age and saturation"| G["Shed load and cap retries"]
    G --> H["Queue drains and latency recovers"]

The loop from F back to C is the dangerous part, as the system can stay broken even after the original traffic rise has ended. That is a metastable failure. Scaling alone may not end it, so you need to shed load and cap retries. See Load shedding and admission control (not yet written) and Retry budgets (not yet written).

Trade-offs

Advantages

Lower latency improves user experience and leaves more of the budget for retries and work. Faster requests also hold resources for less time, which improves capacity. Moving work to the edge can eliminate entire round trips.

Costs

• Edge nodes, replicas, and caches add money, complexity, and consistency issues.
• Lower utilization to reduce queueing means paying for idle capacity.
• Parallel calls raise total load, and hedging can double it.
• Pursuing the last few milliseconds has quickly diminishing returns.

Alternatives

Reducing latency is not always the right goal. Perceived latency can improve via progressive rendering and optimistic UI. Asynchronous workflows remove the need for a fast response altogether. See Long-running operation APIs (not yet written). Better tail control may matter more than a better median. See Tail latency.

When to use it

Prioritize latency for interactive journeys, for time-critical systems such as trading and bidding, and at deep fan-out, where each millisecond multiplies. Use measured user or business impact where possible.

When not to use it

Do not optimize batch work for latency when throughput matters instead, and don't add a cache or new region where the gain is imperceptible. Optimizing processing time is pointless if network round trips make up 90% of the total.

Reconsider when

• Your user base moves to a new geography.
• A breakdown shows a different dominant component than last time.
• Utilization targets change, since less headroom means more queueing.
• Protocol or edge offerings change what round trips are necessary.

Capacity planning

Inputs and assumptions

Service time 10 ms with little variation, peak arrival rate 3,000 requests per second, a target mean wait of at most 5 ms. Assume random arrivals and an even split between servers, and treat each server as a single queue with one worker. These are simplifying assumptions, and real services have more workers.

Formula and units

• Utilization ρ = arrival rate per server × service time in seconds.
• Mean wait (ms) = ρ ÷ (2(1 − ρ)) × service time (ms).
• Solve for the maximum ρ, then servers = total arrival rate × service time ÷ ρ.

Intermediate calculations

• Required wait of 5 ms at 10 ms service: ρ ÷ (2(1 − ρ)) = 0.5, so ρ = 0.5.
• Each server takes 0.5 ÷ 0.010 s = 50 requests per second at that utilization.
• Servers: 3,000 ÷ 50 = 60.
• If 15 ms of wait were acceptable: ρ ÷ (2(1 − ρ)) = 1.5, so ρ = 0.75. Then each server handles 75 per second, for 40 servers.
• So the 10 ms latency saving costs 20 extra servers, or 50% more.

Headroom

Latency targets already imply headroom, because they fix the safe utilization. Add a further margin for forecast error, such as 20%: 60 ÷ 0.8 = 75 servers. Illustrative figures should be replaced with measured values.

Failure capacity

Losing one of three zones sends all load to 2 ÷ 3 of the servers. At 60 servers total, the remaining 40 each take 75 requests per second: ρ = 0.75, and the wait rises to 15 ms, which may still be tolerable, since latency degrades and nothing fails. To hold the 5 ms target during failure, you'd need 60 × 3 ÷ 2 = 90 servers. So you should state whether the latency target applies during failure, since that costs 50% more.

Failure modes

Security and privacy

Threats

Timing itself can leak information: differences in response time can reveal whether a username exists or whether a secret comparison failed early. The Server-Timing header can expose internal structure, so it should be limited in public responses. Attackers can also target slow routes for cheap denial of service, or hold connections open with slow request techniques.

Trust boundaries

Clients control how fast they send and read. Protect servers from slow clients using timeouts at the edge. Internal timing details should not cross to the public internet without review.

Authentication and authorization

Authentication adds latency, particularly when it needs remote calls on every request. Cache verification results carefully, within revocation requirements. Compare secrets in constant time, even though this is marginally slower, since speed here would be a vulnerability. Decide whether authorization fails closed if it can't respond within its budget.

Tenant isolation

One tenant's heavy query can raise latency for others through shared queues. Isolate through per-tenant concurrency limits and report latency per tenant to detect it. See Tenant isolation and noisy-neighbor controls (not yet written).

Sensitive data

Latency logs can include URLs with user identifiers or tokens. Log route templates instead. Real-user monitoring collects timing from personal devices, so it needs consent and minimization under applicable privacy law.

Abuse resistance

Enforce header and body read timeouts, and limit concurrent connections per client. Find routes where the cost per request is highest, since an attacker will find them too, and add limits there.

Observability

Metrics

Use a latency histogram by route and status, measured at more than one location. Track queue depth and age, active connections, new connection rate, and DNS and TLS times where available. Separately track successful and failed latency: fast failures make averages look better. See Golden Signals, RED, and USE methods (not yet written).

Logs

Log total duration and phase durations in structured form, as the example does, with route template, status, and trace ID. Log slow requests in more detail, using a threshold or sampling to bound the volume.

Traces

Traces show the waterfall: which spans are sequential, which overlap, and where gaps between spans indicate queueing or uninstrumented work. A gap inside a parent span is often the most informative thing on the page. See Distributed tracing (not yet written).

Dashboards

Show p50, p95, and p99 together at the load balancer, with a breakdown by phase and instance. Show utilization on the same time axis, as seeing latency rise along with utilization separates queueing from a code regression.

Alerts

Alert on user-visible SLO burn instead of raw thresholds. Create a warning when queue age climbs, because it comes before user impact. See Multi-window burn-rate alerts (not yet written).

SLO examples

• 99% of GET /search requests finish within 400 ms at the load balancer over 28 days.
• 75% of real-user page loads reach largest contentful paint within 2.5 s, per region.
• 99.9% of internal auth calls finish within 20 ms.

Runbook actions

On a latency alert: check whether it affects all instances or one, compare with recent deployments, and look at utilization and queue age. If it is one instance, remove it. If it is load, shed or scale. If it is a dependency, look at its traces and consider its fallback.

Implementation

Architecture of the examples

The TypeScript example is an Express service that wraps phases in PhaseTimer and reports them with Server-Timing. A mode=parallel query shows the effect of composition. The Python example is a calculator that models what physics and queueing require, with no HTTP layer since it would add nothing.

Data structures

PhaseTimer holds an ordered list of Phase records. It takes an injectable clock, so tests are deterministic and need no real delays. Python uses a Pydantic Hop for inputs, with utilization restricted to below 1 because the formula diverges there, and Breakdown for outputs.

Interfaces

GET /feed?mode=sequential|parallel returns total time and phase names, plus the header. GET /healthz reports liveness. Python functions propagation_ms, transmission_ms, md1_queueing_ms, and rtts_to_first_byte can be used individually.

Error handling

measure records a phase in finally, so failed phases are timed too. Slow failures are important signals, and dropping them would bias results. Route errors go to a logging error handler with no internal details in the response.

Concurrency considerations

Promise.all runs ranking and ads concurrently. Since Node runs these timers on a single event loop, a CPU-heavy synchronous task would delay every in-flight phase. The timer is per-request, so concurrent requests do not share measurement state.

Educational limitations

Delays are simulated by timers, so nothing here shows real contention. The measurement is inside the process, so it misses the accept queue and the network. The M/D/1 formula is a simple model, and a multi-worker service queues differently.

Production hardening

Use OpenTelemetry spans rather than manual timers, record histograms, cover client-side measurement, remove detailed Server-Timing from public responses, and add deadlines so that requests stop when their budget is spent.

Example source

// Express service that breaks a request's latency into named phases and reports them with the
// W3C Server-Timing header, so a browser or curl can see where the time went.
import { fileURLToPath } from 'node:url';
import express from 'express';
import type { Express, NextFunction, Request, Response } from 'express';

export interface Phase { name: string; durationMs: number }

export class PhaseTimer {
  readonly phases: Phase[] = [];
  private readonly now: () => number;
  constructor(now: () => number = () => performance.now()) { this.now = now; }

  async measure<T>(name: string, work: () => Promise<T>): Promise<T> {
    const start = this.now();
    try {
      return await work();
    } finally {
      this.phases.push({ name, durationMs: this.now() - start });
    }
  }

  header(): string {
    return this.phases.map((p) => `${p.name};dur=${p.durationMs.toFixed(1)}`).join(', ');
  }

  total(): number { return this.phases.reduce((sum, p) => sum + p.durationMs, 0); }
}

const sleep = (ms: number): Promise<void> => new Promise((resolve) => { setTimeout(resolve, ms); });
const log = (level: string, msg: string, fields: Record<string, unknown>): void => {
  console.log(JSON.stringify({ ts: new Date().toISOString(), level, msg, ...fields }));
};

// Simulated dependency delays in milliseconds. Sequential phases add, and parallel phases cost
// only as much as the slowest member.
export const DELAYS = { auth: 5, db: 20, ranking: 30, ads: 45 } as const;

export function createApp(): Express {
  const app = express();
  app.get('/healthz', (_req: Request, res: Response) => { res.json({ status: 'ok' }); });

  app.get('/feed', async (req: Request, res: Response, next: NextFunction) => {
    try {
      const mode = req.query['mode'] === 'parallel' ? 'parallel' : 'sequential';
      const timer = new PhaseTimer();
      await timer.measure('auth', () => sleep(DELAYS.auth));
      await timer.measure('db', () => sleep(DELAYS.db));
      if (mode === 'parallel') {
        await timer.measure('rank_and_ads', () => Promise.all([sleep(DELAYS.ranking), sleep(DELAYS.ads)]));
      } else {
        await timer.measure('ranking', () => sleep(DELAYS.ranking));
        await timer.measure('ads', () => sleep(DELAYS.ads));
      }
      res.setHeader('Server-Timing', timer.header());
      log('info', 'feed.served', { mode, totalMs: Math.round(timer.total()), phases: timer.phases });
      res.json({ mode, totalMs: Math.round(timer.total()), phases: timer.phases.map((p) => p.name) });
    } catch (err) { next(err); }
  });

  app.use((err: unknown, _req: Request, res: Response, _next: NextFunction) => {
    log('error', 'request.failed', { error: String(err) });
    res.status(500).json({ error: 'Internal error.' });
  });
  return app;
}

if (process.argv[1] === fileURLToPath(import.meta.url)) {
  const port = Number(process.env['PORT'] ?? 3000);
  const server = createApp().listen(port, () => { log('info', 'server.started', { port }); });
  process.on('SIGTERM', () => { server.close(() => { process.exit(0); }); });
  process.on('SIGINT', () => { server.close(() => { process.exit(0); }); });
}

"""Calculate one-way network latency from physics, to show what no design can buy back.

Latency = propagation + transmission + processing + queueing. The speed of light in fiber sets a
hard floor on propagation, and that floor is often the deciding number in a multi-region design.
"""

from __future__ import annotations

import argparse
import json

from pydantic import BaseModel, Field

FIBER_KM_PER_MS = 200.0  # about two-thirds of c in vacuum
ROUTE_FACTOR = 1.4  # illustrative: real fiber paths are longer than the great-circle distance


class Hop(BaseModel):
    distance_km: float = Field(ge=0)
    bandwidth_mbps: float = Field(gt=0)
    payload_bytes: int = Field(ge=0)
    processing_ms: float = Field(default=0.0, ge=0)
    utilization: float = Field(default=0.0, ge=0, lt=1)
    service_ms: float = Field(default=0.0, ge=0)


class Breakdown(BaseModel):
    propagation_ms: float
    transmission_ms: float
    processing_ms: float
    queueing_ms: float
    total_ms: float


def propagation_ms(distance_km: float, route_factor: float = ROUTE_FACTOR) -> float:
    return distance_km * route_factor / FIBER_KM_PER_MS


def transmission_ms(payload_bytes: int, bandwidth_mbps: float) -> float:
    return payload_bytes * 8 / (bandwidth_mbps * 1_000)


def md1_queueing_ms(utilization: float, service_ms: float) -> float:
    """Mean M/D/1 wait: rho / (2 * (1 - rho)) * service time. Grows sharply as rho nears 1."""
    return utilization / (2 * (1 - utilization)) * service_ms


def breakdown(hop: Hop) -> Breakdown:
    prop = propagation_ms(hop.distance_km)
    tx = transmission_ms(hop.payload_bytes, hop.bandwidth_mbps)
    queue = md1_queueing_ms(hop.utilization, hop.service_ms)
    total = prop + tx + hop.processing_ms + queue
    return Breakdown(
        propagation_ms=round(prop, 3), transmission_ms=round(tx, 3),
        processing_ms=hop.processing_ms, queueing_ms=round(queue, 3), total_ms=round(total, 3),
    )


def rtts_to_first_byte(rtt_ms: float, *, tls13: bool = True, reused: bool = False) -> float:
    """Time to first response byte, in milliseconds: TCP handshake, TLS, then request and response."""
    if reused:
        return rtt_ms
    return rtt_ms * (1 + (1 if tls13 else 2) + 1)


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--km", type=float, default=5_600, help="distance; default is roughly New York to London")
    parser.add_argument("--mbps", type=float, default=100)
    parser.add_argument("--bytes", type=int, default=1_000_000)
    args = parser.parse_args(argv)
    hop = Hop(distance_km=args.km, bandwidth_mbps=args.mbps, payload_bytes=args.bytes)
    rtt = 2 * propagation_ms(args.km)
    print(json.dumps({
        "one_way": breakdown(hop).model_dump(),
        "rtt_floor_ms": round(rtt, 1),
        "first_byte_new_tls13_ms": round(rtts_to_first_byte(rtt), 1),
        "first_byte_reused_ms": round(rtts_to_first_byte(rtt, reused=True), 1),
        "queueing_ms_at_utilization": {
            str(u): round(md1_queueing_ms(u, 10), 1) for u in (0.5, 0.8, 0.9, 0.95, 0.99)
        },
    }, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Terminal lab

Setup

cd topics/latency/examples/typescript
pnpm install --ignore-workspace

Run

Start the service:

PORT=3000 pnpm start

In another terminal, compare the two modes:

curl -s localhost:3000/feed | jq -c '{mode, totalMs}'

curl -s 'localhost:3000/feed?mode=parallel' | jq -c '{mode, totalMs}'

Use curl's timing variables to split up real connection time for a site you may test:

curl -so /dev/null -w 'dns %{time_namelookup}  tcp %{time_connect}  tls %{time_appconnect}  ttfb %{time_starttransfer}  total %{time_total}\n' https://www.rfc-editor.org/

Then calculate the physical floor:

cd ../python && uv run python -m app.main | jq -c '{rtt_floor_ms, first_byte_new_tls13_ms, first_byte_reused_ms, queueing_ms_at_utilization}'

Test

cd ../typescript && pnpm test

cd ../python && uv run pytest -q

Expected output

Sequential mode reports a totalMs of around 100 and parallel about 70, because ads and ranking overlap, so the parallel step costs 45 ms and not 75 ms. Timer resolution will add a few milliseconds. The Python tool prints:

{"rtt_floor_ms":78.4,"first_byte_new_tls13_ms":235.2,"first_byte_reused_ms":78.4,"queueing_ms_at_utilization":{"0.5":5.0,"0.8":20.0,"0.9":45.0,"0.95":95.0,"0.99":495.0}}

Going from 90% to 99% utilization raises the wait 11 times, for a 10% increase in traffic.

Cleanup

Stop the server with Ctrl-C, then:

rm -rf ../typescript/node_modules .venv

Troubleshooting

• The curl timing values are cumulative from the start of the request. Subtract adjacent values to get each phase.
• EADDRINUSE: another process is holding the port. Find it with lsof -i :3000.
• time_appconnect is zero for plain HTTP, because TLS is not used.

Practical challenges

Industry standards and trends

Standards and specifications

• RFC 7679, January 2016, and RFC 2681, September 1999, define one-way and round-trip delay metrics for IP networks.
• RFC 8446, August 2018, TLS 1.3, which reduced the full handshake to one RTT.
• RFC 9000, May 2021, QUIC, which combines transport and cryptographic setup.
• W3C Server Timing, a W3C working draft, defines the header used in this lesson.
• W3C Navigation Timing Level 2 defines browser timing measurements.

Established practice

The Tail at Scale, Dean and Barroso, 2013, is the widely cited paper on latency variability in large fan-out systems. Web Vitals, Google guidance, updated periodically, is a common set of field metrics for page experience. It's vendor guidance and not a formal standard, though it's influential.

Managed-service options

CDNs and global accelerators terminate connections near users. Real-user monitoring products report field latency, and cloud load balancers publish latency metrics. Compare where each one starts and stops its clock.

Emerging approaches

HTTP/3 deployment continues to grow, particularly for mobile clients on lossy networks. Edge compute moves logic close to users, though the benefit disappears if every request must still return to a distant database. LLM streaming has changed the common measure to time to first token and tokens per second. These are observations from public products, and they are not settled standards.

Recent changes

Interaction to Next Paint replaced First Input Delay as a Core Web Vital in March 2024. Check current thresholds in the guidance above, since they may change.

Adoption constraints

Corporate networks or middleboxes may block UDP, which requires a fallback from QUIC to TCP. Edge deployment has data-residency implications, and client-side measurement needs privacy review.

Links in this section were checked on 2026-11-03 (access date). Items labelled as emerging or inferred are not settled facts and should be re-checked before you rely on them.

Interview preparation

Review

Key takeaways

• Latency = propagation + transmission + processing + queueing.
• Physics sets a floor of about 5 ms per 1,000 km one way in fiber.
• A new TLS 1.3 connection takes 3 RTTs to first byte, and a reused one takes 1.
• Queueing grows as ρ ÷ (1 − ρ), so saturation hits suddenly.
• Decompose before you optimize. The dominant part is often not your code.
• Always state location, percentile, and window.

Decision checklist

• Do I have a breakdown of the time, not just a total?
• Have I calculated the physical floor for this path?
• Does the optimization target the critical path?
• What is the target utilization, given the target wait?
• Does the latency target apply during failure?

Production-readiness checklist

• Latency histograms exist per route, at the load balancer.
• Real-user timing is collected by region.
• Connection reuse is monitored.
• Queue age and saturation are monitored.
• Deadlines and bounded queues prevent runaway waiting.

Glossary

• RTT: round-trip time.
• TTFB: time to first byte.
• Service time: the time spent actively working on a request.
• Response time: service time plus waiting.
• Utilization (ρ): the fraction of time a resource is busy.
• Critical path: the sequence of steps that determines total time.
• Metastable failure: an overload that sustains itself after the trigger is gone.

Quiz

Further study

Go on to Throughput for throughput, P99 latency and Tail latency for percentiles and the tail, and Latency budgets and critical-path analysis to allocate time across a critical path. Little’s Law and practical queueing theory (not yet written) goes deeper into queueing, and Connection establishment, keep-alive, and draining (not yet written) goes deeper into connections.

Progress

Progress is stored only in this browser. Export it from the curriculum page.