P99 latency

Orientation

Definition

The p99 latency is the value that 99% of requests finish within, for a stated population and time window. Put differently, 1 request in 100 is slower than it. It describes one point on the latency distribution and says nothing about how much slower the remaining 1% are.

Why it matters

Averages describe a request almost nobody receives. A service where 98% of requests take 5 ms and 2% take 2 seconds has a mean near 45 ms, which matches no real experience. The p99 exposes the slow group. It also matters more than "1 in 100" suggests, since a user session or a fanned-out page makes many requests: with 100 requests, the chance that at least one exceeds p99 is 1 − 0.99^100 = 63%. A p99 is also easy to compute wrongly. Averaging across servers, using coarse histogram buckets, sampling too little, and choosing a long window can each give you a number that looks exact and is badly wrong.

Prerequisites

Latency

Study time

About 1.5 hours, including the terminal lab and challenges. This is an estimate.

Related topics

Tail latency, Metric cardinality and histogram design (not yet written), Benchmark design and coordinated omission (not yet written), Latency

Learning objectives

• Compute a percentile correctly and state its population and window.
• Explain why percentiles cannot be averaged, while histograms can be merged.
• Estimate how many samples a stable p99 requires.
• Choose histogram buckets so that a reported p99 is trustworthy near your target.
• Decide when p99 is the right objective and when another measure is better.

Production scenario

An API has 10 servers. The dashboard shows "p99: 180 ms", computed as the average of each server's p99, and the target is 300 ms, so everything looks fine. Support is nevertheless receiving complaints. One server has a failing disk, with a p99 of about 1,200 ms, while the other nine are near 65 ms. The true p99 for all requests, computed from combined data, is 649 ms: over twice the target and over three times what the dashboard said. These figures come from the Python example in this lesson.

The same team has a second problem. Their histogram's buckets jump from 100 ms to 1,000 ms, and interpolation inside such a wide bucket produces smooth, believable numbers that bear little relation to the data. After they switch to merged histograms with buckets concentrated around 300 ms, the bad server is obvious within a minute, and the alert fires as it should. Nothing about the system changed, only how it was measured.

Mental model

Analogy

Line up 100 runners by finish time. The p50 is the time of runner 50, and the p99 is the time of runner 99. The average depends on everyone, but the p99 only asks where a given position falls. If you have the finish lists of ten separate races, you can't find the overall runner-99 time by averaging each race's runner-99 time. You have to merge the lists, because one race might be much slower than the rest.

Where the analogy breaks

Runners all start together and finish once, whereas requests overlap and interfere with each other, so a slow one can cause others to be slow. Also, nobody stores the complete finish list in production. Systems store a compressed summary, so the answer is an approximation whose error depends on the summary. Finally, a slow request that never finishes because it timed out may vanish from the list altogether, which makes the numbers look better.

Entities, state, flow, and invariants

• Entities: samples, histograms, servers, time windows, percentile queries.
• State: a count for each bucket, per instance and interval.
• Flow: record a duration, increment a bucket, export the counts, merge across instances and time, then query.
• Invariant 1: a percentile has a stated population, window, and measurement point.
• Invariant 2: aggregate raw counts first and derive percentiles afterward, never the reverse.
• Invariant 3: failed and timed-out requests are counted, or their exclusion is stated.

Foundations

Concepts and terminology

First-principles explanation

A percentile is a statement about rank, and rank depends on all the data together. If server A has a p99 of 65 ms and server B has 1,200 ms, the combined p99 could be anywhere from the lower to the higher value, depending on request counts and shapes. No formula recovers it from the two numbers, because the information has gone. Histograms keep the needed information: the counts add, so you can merge them across servers and over time, then compute the percentile.

Estimating a tail from a sample needs many samples. With n samples, only about n ÷ 100 of them lie beyond p99, and those few determine the estimate. With 100 samples, a single outlier determines the result. Also, what you see is a summary of a time window: a percentile over one hour can miss a five-minute failure that hits less than 1% of that hour's requests. In this lesson's example, an incident with a median of 2,012 ms leaves the hourly p99 at 41.7 ms, since it covers just 0.28% of requests.

Mathematics

Nearest rank: p = sorted[⌈q × n⌉ − 1], using zero-based indexing.

Requests exceeding p99 at least once, across k independent requests: 1 − 0.99^k. For k = 10: 9.6%. For k = 100: 63.4%. For k = 500: 99.3%.

Rank uncertainty: the number of samples below the true p99 is binomial, with a standard deviation of √(n × q × (1 − q)). For n = 10,000: √99 ≈ 10 ranks, so a roughly 95% interval spans ranks 9,880 to 9,920. Translate those ranks into milliseconds using the local shape of the distribution.

Measured stability, from the example with a true p99 near 64 ms, showing the middle 90% of repeated estimates:

Relative error of log-linear buckets: with s sub-buckets per power of two, width ÷ value ≤ 1 ÷ s. For s = 16, that is 6.25%.

Complexity

Here n is the sample count and b the bucket count. A wide value range needs few log-linear buckets: the test shows under 350 for values spanning 1 to 1,000,000.

Mechanism

Normal path

1. Measure duration at a defined point, such as the load balancer or a server middleware.
2. Find the bucket and increment its counter, including for failed requests.
3. Export cumulative counts per instance at a regular interval.
4. At query time, sum per-bucket counts across instances for the chosen window.
5. Find the bucket containing rank q × total, then report its bound or interpolate inside it.
6. Compare with the objective, and alert using burn rates rather than one noisy point. See Multi-window burn-rate alerts (not yet written).

State transitions

stateDiagram-v2
    [*] --> Measured: request finishes
    [*] --> Lost: process killed or client disconnected unrecorded
    Measured --> Bucketed: counter incremented
    Bucketed --> Exported: scrape or push
    Exported --> Merged: counts summed by bucket
    Merged --> Reported: quantile computed at query time
    Bucketed --> Lost: instance dies before export
    Reported --> [*]
    Lost --> [*]

The Lost state biases results in the flattering direction. Instances that crash are usually the slow ones, and their unexported counts disappear. Load-balancer measurement partly fixes this, as it records outcomes even when a backend dies.

Concurrency behavior

Several threads record at once, so counters need atomic increments or per-thread shards that merge on read. Reading while writers run gives a slightly inconsistent snapshot, which is acceptable for monitoring. Node's single-threaded event loop avoids this issue in the example. Under high load, the recording path must stay cheap: a lock in the metrics library can itself add latency. See Thread safety (not yet written).

Ordering and consistency

Instances scrape at slightly different times, so a merged window is approximate, and for rates this is usually harmless. Counter resets on restarts must be handled when computing increases. Be careful with percentiles of percentiles: a dashboard taking the average or maximum of minute-level p99s over an hour is not the hour's p99. Roll-ups must keep the histogram, or the result is meaningless.

Ownership of data and control

The service team owns instrumentation and bucket layout, since they know the target. The platform team owns the metrics store and its cardinality limits. SLO owners agree on the measurement point and the definition of the population, including whether errors and health checks count. See SLAs, SLOs, and SLIs (not yet written).

Architecture

flowchart LR
    subgraph client["Client devices: outside operator control"]
      U["Browser or app: real-user timing"]
    end
    subgraph op["Operator network"]
      LB["Load balancer: sees all outcomes"]
      A1["Server 1: owns its histogram"]
      A2["Server N: owns its histogram"]
      M["Metrics store: merges counts by bucket"]
      D["Dashboards and SLO alerts"]
    end
    U -->|"HTTPS"| LB
    LB --> A1
    LB --> A2
    A1 -->|"scrape: bucket counts"| M
    A2 -->|"scrape: bucket counts"| M
    LB -->|"export: bucket counts"| M
    U -->|"beacon: sampled timings"| M
    M --> D

Each measurement point answers a different question. Servers can break latency down by route but miss what happens before their handler runs. The load balancer sees failures and upstream queues. Real-user data includes the last mile, though it needs privacy review. The arrows carry bucket counts, not percentiles, as only counts can be merged.

flowchart LR
    R["HTTP GET /work"] --> MW["Express timing middleware"]
    MW -->|"record duration"| H["Log-linear histogram: Map of bucket to count"]
    H --> Q["quantile function"]
    Q -->|"JSON"| MR["GET /metrics/latency"]
    H2["Another instance's histogram"] -->|"merge: add counts"| H
    S["Seeded lognormal data"] --> P["Python analysis"]
    P --> O["Exact vs averaged vs bucketed vs windowed"]

The merge arrow is the most important one here. Because a histogram merges by adding counts, its percentile matches what a single histogram of all requests would report, and the tests confirm it.

sequenceDiagram
    participant D as Dashboard
    participant M as Metrics store
    participant S1 as Series, server 1
    participant S2 as Series, server 2
    D->>M: p99 for route over 5 minutes
    M->>S1: Bucket count increases
    S1-->>M: le10=900, le100=95, le1000=5
    M->>S2: Bucket count increases
    S2-->>M: le10=100, le100=300, le1000=600
    M->>M: Sum: le10=1000, le100=395, le1000=605
    M->>M: Rank 0.99 x 2000 = 1980 falls in le1000
    M-->>D: Between 100 and 1000 ms

Server 1 alone would report a p99 inside its last bucket, and server 2 is much worse. Summing counts weights each server by its traffic. Note the answer's honest form: "between 100 and 1,000 ms". A precise-looking interpolated number from such a bucket would be false precision.

flowchart TD
    A["One of ten servers degrades"] --> B["Its p99 reaches 1,227 ms"]
    B --> C{"How is the fleet p99 computed?"}
    C -->|"average of per-server p99s"| D["Dashboard: 180 ms, below target"]
    D --> E["No alert, while users complain"]
    C -->|"merged histograms"| F["True p99: 649 ms, above target"]
    F --> G["Alert fires. Per-instance view isolates the server"]

The same telemetry leads to opposite outcomes, depending on one aggregation decision. This failure is silent, which is what makes it serious: a broken measurement looks exactly like a healthy system. The numbers are from this lesson's seeded example.

Trade-offs

Advantages

• Describes what the slow users actually experience, unlike a mean.
• Robust to a few extreme outliers, compared with a maximum.
• Widely understood, and works well as an SLO target.

Costs

• Histograms take more metric series than a single average.
• Needs adequate traffic to be stable, so it is noisy on low-volume routes.
• Ignores everything past the 99th position: the worst 1% could be 500 ms or 30 s.
• Optimizing one percentile can move problems just beyond it.

Alternatives

• Threshold SLI: "99% of requests under 300 ms" counts good requests. It's simple, it aggregates correctly, and only needs one bucket boundary at the target.
• p99.9 or maximum: for high-volume systems or pause detection. See Tail latency.
• Latency heat map: shows multiple modes, which percentiles conceal.
• Sketches: t-digest or DDSketch give accuracy across a wide range, at some implementation cost.

When to use it

For user-facing and internal services with enough volume. As a rule of thumb, you want at least 1,000 samples in the evaluation window, more if the decision is important.

When not to use it

Avoid p99 on routes with few requests, where a threshold ratio over a longer window is better. For batch jobs, track completion time per job. For bimodal traffic, split by route or cache status before computing percentiles, as a single number will describe neither group.

Reconsider when

• The p99 and p50 move in different directions, indicating a mix change.
• Your target moves, so bucket boundaries must follow it.
• Fan-out increases, so a higher percentile now determines what users experience.

Capacity planning

The capacity concern here is the telemetry cost of the percentile data itself.

Inputs and assumptions

200 routes, 5 status classes, 12 histogram buckets, and 300 instances, scraped every 15 s. Assumed storage of about 2 bytes per sample after compression. This depends on your system, so measure it.

Formula and units

• Series = routes × status classes × buckets × instances.
• Samples per second = series ÷ scrape interval in seconds.
• Storage per day (bytes) = samples per second × 86,400 s × bytes per sample.

Intermediate calculations

• Per instance: 200 × 5 × 12 = 12,000 series.
• Fleet: 12,000 × 300 = 3,600,000 series.
• Samples: 3,600,000 ÷ 15 = 240,000 per second.
• Storage: 240,000 × 86,400 × 2 B ≈ 41.5 GB per day.
• After aggregating away the instance label for long-term storage: 12,000 series, 800 samples per second, and about 138 MB per day.

Headroom

Allow 30% for growth in routes and instances, or about 54 GB per day of raw data. Keep per-instance data for a short period, such as 7 days, for debugging, and fleet aggregates for longer. Retention figures are illustrative.

Failure capacity

During a zone loss or an autoscaling event, instance churn creates new series, because each new instance has a new label value. Replacing all 300 instances doubles active series temporarily, to 7.2 million. Size the metrics store for this, as monitoring needs to work best during failures. Also, metrics should not depend on the failing zone. See Metric cardinality and histogram design (not yet written).

Failure modes

Security and privacy

Threats

Metric labels can leak or explode. If a label takes values from user input, an attacker can create unlimited series and take down your metrics store, thus blinding your monitoring during an attack. Inspection endpoints like /metrics/latency reveal traffic volumes and internal route names.

Trust boundaries

Keep metrics endpoints on an internal network or behind authentication, and do not serve them through the public load balancer. Treat real-user beacons as untrusted input, since anyone can send false timings and distort your numbers.

Authentication and authorization

Scrape targets should authenticate the scraper when they carry anything sensitive. Restrict who can read per-tenant latency, and who can edit dashboards and alert rules. A silently edited query can hide an outage.

Tenant isolation

A global p99 can hide a single tenant's bad experience. Measure latency per tenant class, and for your largest tenants individually. Avoid one label value per tenant when you have many: use a bounded set, or keep per-tenant data in logs instead.

Sensitive data

Never put user IDs, emails, or raw URLs in metric labels, where they can't be easily deleted. Use route templates. Real-user monitoring may collect personal data, depending on jurisdiction, and needs review. See Privacy-by-design (not yet written).

Abuse resistance

Normalize route labels server-side, and map unknown paths to one label so that scanners can't create new series. Set per-target series limits in the metrics system. Filter or tag known bot traffic, so attackers can't distort the percentile that drives your alerts.

Observability

Metrics

Use a latency histogram with labels for route template, method, and status class. Add request count beside it. Add metrics about the metrics pipeline: scrape failures, series count, and dropped samples.

Logs

Log individual slow requests above a threshold with route, duration, and trace ID, so that a bad percentile can be traced to examples. Sample the logs to bound their volume.

Traces

Sample slow requests preferentially, as head sampling at 1% keeps few from the tail. Exemplars link a histogram bucket to a trace ID, moving you from "p99 is high" straight to an actual slow request. See Trace sampling and context propagation (not yet written).

Dashboards

Show p50, p90, p99, and p99.9 on one graph, plus count. Add a heat map to reveal multiple modes, and a per-instance view to spot single bad servers. Display the target as a line.

Alerts

Alert on the ratio of requests slower than the target, using multi-window burn rates, instead of on an instantaneous p99. A ratio is cheap to compute, aggregates correctly, and handles low traffic better.

SLO examples

• 99% of GET /search requests complete within 300 ms at the load balancer, over 28 days.
• 99.9% of POST /orders finish within 1 s over 28 days, including failures at their full duration.
• A ratio form is preferred: count of requests at or below 300 ms, divided by all requests.

Runbook actions

When p99 rises: check whether p50 rose too (a general slowdown) or only the tail (a subset). Split by instance, route, and status. If it is one instance, drain it. If it is one route, inspect slow traces. Check request count first, since low volume can make a number meaningless.

Implementation

Architecture of the examples

The TypeScript example is an Express service with a timing middleware and a log-linear Histogram class, using an inspection route. The Python example is a seeded analysis comparing exact percentiles with the common wrong methods. The TypeScript side shows how to measure correctly, and the Python side shows how measurement goes wrong.

Data structures

Histogram maps a bucket index to a count. The index combines a value's power of two with one of 16 linear sub-buckets, bounding relative error at 6.25%. A Map keeps only non-empty buckets, so memory depends on the range seen and not on the number of samples.

Interfaces

record(ms), merge(other), and quantile(q), plus GET /work, GET /healthz, and GET /metrics/latency. In Python: percentile, average_of_p99s, p99_spread, bucket_p99, and window_effect.

Error handling

record rejects negative and non-finite values, since a bad clock must not silently corrupt a histogram. merge rejects mismatched layouts. quantile throws on an empty histogram, because a reported 0 ms would look like excellent performance. The route handles the empty case by reporting just a count of zero.

Concurrency considerations

Node runs record on one thread, so no synchronization is needed. In a multi-threaded runtime you would use striped atomic counters. The middleware records on finish, which misses clients that disconnect early. A production version would also listen for close.

Educational limitations

There are no time windows: the histogram accumulates forever. It reports bucket upper bounds and does no interpolation. curl in a loop is closed-loop, so it understates queueing. Lognormal data is a simplification.

Production hardening

Use a maintained library such as an OpenTelemetry exponential histogram or HdrHistogram, add rotating windows, export to your metrics system, capture aborted requests, and attach exemplars.

Example source

// Express service recording request latency in a histogram and exposing percentiles for inspection.
import { fileURLToPath } from 'node:url';
import express from 'express';
import type { Express, NextFunction, Request, Response } from 'express';
import { Histogram } from './histogram.ts';

const log = (level: string, msg: string, fields: Record<string, unknown> = {}): void => {
  console.log(JSON.stringify({ ts: new Date().toISOString(), level, msg, ...fields }));
};
const sleep = (ms: number): Promise<void> => new Promise((resolve) => { setTimeout(resolve, ms); });

// 2% of requests are slow, so the median stays low and p99 reveals them.
export const pickDelay = (r: number): number => (r < 0.02 ? 200 : 5);

export function createApp(random: () => number = Math.random): Express {
  const app = express();
  const latency = new Histogram();

  app.use((req: Request, res: Response, next: NextFunction) => {
    const start = performance.now();
    res.on('finish', () => { if (req.path === '/work') latency.record(performance.now() - start); });
    next();
  });
  app.get('/healthz', (_req: Request, res: Response) => { res.json({ status: 'ok' }); });
  app.get('/work', async (_req: Request, res: Response) => {
    await sleep(pickDelay(random()));
    res.json({ done: true });
  });
  app.get('/metrics/latency', (_req: Request, res: Response) => {
    if (latency.count === 0) { res.json({ count: 0 }); return; }
    const pct = (q: number): number => Math.round(latency.quantile(q) * 10) / 10;
    res.json({ count: latency.count, p50: pct(0.5), p90: pct(0.9), p99: pct(0.99), p999: pct(0.999), buckets: latency.bucketCount });
  });
  app.use((err: unknown, _req: Request, res: Response, _next: NextFunction) => {
    log('error', 'request.failed', { error: String(err) });
    res.status(500).json({ error: 'Internal error.' });
  });
  return app;
}

if (process.argv[1] === fileURLToPath(import.meta.url)) {
  const port = Number(process.env['PORT'] ?? 3000);
  const server = createApp().listen(port, () => { log('info', 'server.started', { port }); });
  const stop = (): void => { server.close(() => { process.exit(0); }); };
  process.on('SIGTERM', stop);
  process.on('SIGINT', stop);
}

"""Show the common ways percentile numbers go wrong, using seeded data.

1. Averaging per-server p99s does not give the fleet p99.
2. Few samples give an unstable p99.
3. Coarse histogram buckets decide the answer you see.
4. Percentiles over a window hide short bad periods.
"""

from __future__ import annotations

import json
import math
import random
from bisect import bisect_left
from statistics import mean

from pydantic import BaseModel, Field


class ServerProfile(BaseModel):
    name: str
    median_ms: float = Field(gt=0)
    sigma: float = Field(gt=0)
    requests: int = Field(gt=0)


def percentile(values: list[float], q: float) -> float:
    """Nearest-rank percentile. Sorts a copy, so it is O(n log n)."""
    if not values:
        raise ValueError("no values")
    if not 0 < q <= 1:
        raise ValueError("q must be in (0, 1]")
    ordered = sorted(values)
    return ordered[math.ceil(q * len(ordered)) - 1]


def sample(rng: random.Random, p: ServerProfile) -> list[float]:
    return [rng.lognormvariate(math.log(p.median_ms), p.sigma) for _ in range(p.requests)]


def average_of_p99s(seed: int = 3) -> dict[str, float]:
    rng = random.Random(seed)
    fleet = [
        ServerProfile(name=f"healthy-{i}", median_ms=20, sigma=0.5, requests=50_000) for i in range(9)
    ] + [ServerProfile(name="degraded", median_ms=300, sigma=0.6, requests=50_000)]
    per_server = [sample(rng, p) for p in fleet]
    p99s = [percentile(v, 0.99) for v in per_server]
    pooled = [x for v in per_server for x in v]
    return {
        "mean_of_p99s_ms": round(mean(p99s), 1),
        "max_of_p99s_ms": round(max(p99s), 1),
        "true_pooled_p99_ms": round(percentile(pooled, 0.99), 1),
        "true_pooled_p50_ms": round(percentile(pooled, 0.50), 1),
    }


def p99_spread(samples: int, trials: int = 300, seed: int = 5) -> dict[str, float]:
    """Repeat a measurement and report how far the estimated p99 moves."""
    rng = random.Random(seed)
    estimates = sorted(
        percentile([rng.lognormvariate(math.log(20), 0.5) for _ in range(samples)], 0.99)
        for _ in range(trials)
    )
    return {
        "samples": samples,
        "low_5pct_ms": round(estimates[int(trials * 0.05)], 1),
        "high_95pct_ms": round(estimates[int(trials * 0.95)], 1),
    }


def bucket_p99(values: list[float], bounds: list[float]) -> float:
    """What a histogram with these upper bounds reports: linear interpolation inside a bucket."""
    ordered = sorted(values)
    rank = 0.99 * len(ordered)
    below = 0
    lower = 0.0
    for upper in bounds:
        count = bisect_left(ordered, upper) - below
        if below + count >= rank:
            return lower + (upper - lower) * (rank - below) / max(count, 1)
        below += count
        lower = upper
    return bounds[-1]


def bucket_effect(seed: int = 9) -> dict[str, float]:
    rng = random.Random(seed)
    values = [rng.lognormvariate(math.log(20), 0.5) for _ in range(100_000)]
    return {
        "exact_p99_ms": round(percentile(values, 0.99), 1),
        "coarse_buckets_p99_ms": round(bucket_p99(values, [10, 100, 1_000]), 1),
        "default_like_buckets_p99_ms": round(bucket_p99(values, [5, 10, 25, 50, 100, 250, 500, 1_000]), 1),
        "fine_buckets_p99_ms": round(bucket_p99(values, [20, 30, 40, 50, 55, 60, 65, 70, 80, 100, 1_000]), 1),
    }


def window_effect(seed: int = 13) -> dict[str, float]:
    """A 5-minute incident hides inside a 1-hour p99 if it covers under 1% of requests."""
    rng = random.Random(seed)
    normal = [rng.lognormvariate(math.log(20), 0.3) for _ in range(359_000)]
    incident = [rng.lognormvariate(math.log(2_000), 0.2) for _ in range(1_000)]
    return {
        "hour_p99_ms": round(percentile(normal + incident, 0.99), 1),
        "incident_window_p50_ms": round(percentile(incident, 0.5), 1),
        "incident_share_pct": round(100 * len(incident) / (len(normal) + len(incident)), 2),
    }


def main() -> int:
    print(json.dumps({
        "average_of_p99s": average_of_p99s(),
        "stability": [p99_spread(n) for n in (100, 1_000, 10_000)],
        "bucket_effect": bucket_effect(),
        "window_effect": window_effect(),
    }, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Terminal lab

Setup

cd topics/p99-latency/examples/typescript
pnpm install --ignore-workspace

Run

Start the service:

PORT=3000 pnpm start

In another terminal, send 500 requests and read the percentiles:

for i in $(seq 1 500); do curl -s localhost:3000/work >/dev/null; done
curl -s localhost:3000/metrics/latency | jq -c .

Then run the analysis:

cd ../python && uv run python -m app.main | jq -c '.average_of_p99s, .stability[], .bucket_effect, .window_effect'

Test

cd ../typescript && pnpm test

cd ../python && uv run pytest -q

Expected output

In the service, about 2% of requests take the 200 ms path, so p50 stays near 6 ms, and p99 lands just above 200 ms. Exact values vary because the slow path is random. The analysis output is seeded, and measured on the review date as:

{"mean_of_p99s_ms":180.3,"max_of_p99s_ms":1227.2,"true_pooled_p99_ms":649.1,"true_pooled_p50_ms":21.4}
{"samples":100,"low_5pct_ms":45.2,"high_95pct_ms":77.2}
{"samples":1000,"low_5pct_ms":57.2,"high_95pct_ms":70.2}
{"samples":10000,"low_5pct_ms":62.1,"high_95pct_ms":65.9}
{"exact_p99_ms":64.3,"coarse_buckets_p99_ms":99.1,"default_like_buckets_p99_ms":85.7,"fine_buckets_p99_ms":64.4}
{"hour_p99_ms":41.7,"incident_window_p50_ms":2011.8,"incident_share_pct":0.28}

Read those lines as four separate warnings. The averaged p99 is 3.6 times too low, 100 samples leave a wide range, buckets move the answer from 64 ms to 99 ms, and a 2-second incident is invisible in the hourly p99.

Cleanup

Stop the service with Ctrl-C, then:

rm -rf ../typescript/node_modules .venv

Troubleshooting

• {"count":0} from the metrics route: send traffic to /work first.
• Percentile values look too high: an event loop busy with other work adds to all measurements.
• The analysis takes a few seconds, since it sorts over a million values.

Practical challenges

Industry standards and trends

Standards and specifications

The OpenTelemetry metrics data model, a living specification, defines explicit-bucket and exponential histograms. The exponential form gives bounded relative error without manual bucket selection. There is no single standard for percentile calculation: tools differ in interpolation, so note the method. RFC 7679, January 2016, covers delay percentiles in network measurement.

Established practice

Prometheus's Histograms and summaries documentation explains why summaries generally can't be aggregated and how bucket choice affects error. HdrHistogram is a widely used log-linear implementation. Service Level Objectives, from Site Reliability Engineering, 2016, recommends percentiles rather than averages. These are project documentation and published practice, not formal standards.

Managed-service options

Cloud monitoring and observability vendors offer histogram and sketch types. Read how each computes percentiles, whether they merge correctly across dimensions, and what their sampling does to tails.

Emerging approaches

Native or exponential histograms reduce series counts and bucket-tuning work, and adoption is growing in Prometheus and OpenTelemetry. Mergeable sketches such as DDSketch, Masson et al., 2019, offer relative-error guarantees. Check support in your full metrics pipeline before depending on either.

Recent changes

Prometheus 3.0, released in 2024, continued developing native histograms. Verify their current status in your version, because the details have been changing.

Adoption constraints

Switching histogram types can break existing dashboards and historical comparison. Exemplars and exponential histograms require support from every component between application and dashboard.

Links in this section were checked on 2026-11-03 (access date). Items labelled as emerging or inferred are not settled facts and should be re-checked before you rely on them.

Interview preparation

Review

Key takeaways

• p99 means position 99 of 100, for a stated population, window, and measurement point.
• Never average percentiles. Merge histograms, then compute.
• Buckets determine accuracy, so place them around the target.
• Few samples give unstable tails, and long windows hide short incidents.
• Include failures and timeouts, or say clearly that they're excluded.
• A threshold ratio is often a better SLI than a percentile.

Decision checklist

• Are the population, window, and measurement point stated?
• Does the query merge histograms before computing?
• Is a bucket boundary near the target?
• Are there enough samples in the window?
• Are failures and timeouts counted?

Production-readiness checklist

• Histograms exist per route at the load balancer or server.
• Route labels are normalized, to protect cardinality.
• Alerts use ratios and burn rates.
• Slow requests link to traces.
• The pipeline has been tested using injected delay.

Glossary

• Nearest rank: a percentile method that picks an actual sample at a computed position.
• Mergeable: summaries can be combined without loss beyond their stated error.
• Exemplar: a sample trace ID attached to a metric bucket.
• Heat map: a time plot of latency density across buckets.
• Coordinated omission: a measurement error in which stalls suppress samples.

Quiz

Further study

Next, read Tail latency for what happens beyond p99, Metric cardinality and histogram design (not yet written) for histogram design at scale, and Multi-window burn-rate alerts (not yet written) for alerting on these numbers. Benchmark design and coordinated omission (not yet written) explains how load tests can corrupt percentile measurements.

Progress

Progress is stored only in this browser. Export it from the curriculum page.