What is a confidence interval in benchmarking?

A confidence interval in benchmarking is a statistical range around a measured metric — recall, latency, QPS, and so on — that quantifies how uncertain that estimate is given finite samples, usually stated with a confidence level such as 95%.

Why is a single point estimate not enough?

One run that reports “12.4 ms mean latency” or “97.1% recall@10” is a sample from a noisy process. Query sets differ, insertion order and seeds change graph structure slightly, OS scheduling jitters timings, and cold versus warm cache shifts tails. If you re-run the same config tomorrow you will not get the identical number. A confidence interval answers: given this experiment design, which values for the true long-run mean (or other parameter) remain plausible? Without that band, a 0.3% recall gap between two HNSW settings can look like a win when it is indistinguishable from noise — the classic path to cherry-picked “victories.”

How you build the interval depends on what you measured.

How are confidence intervals typically formed for search benchmarks?

For metrics that average many i.i.d.-ish observations — per-query latency samples, per-query recall indicators — a common approach is to compute the sample mean and a standard error, then form a normal- or t-based interval when n is large and the distribution is not too wild. Latency is often skewed, so practitioners use medians with bootstrap intervals, or report percentile estimates with uncertainty from many repeated trials. For QPS under a fixed load harness, the natural unit is often a timed trial (or a block of seconds): repeat the whole load test R times and build an interval over those R throughput numbers rather than pretending every individual query is an independent QPS sample. Bootstrap resampling of the query set is popular when you want interval estimates for recall@k without strong parametric assumptions: resample queries with replacement, recompute the metric, and read percentiles of that bootstrap distribution.

Sample size and dependence decide whether the interval is honest.

What makes ANN confidence intervals too narrow or too wide?

Too few queries inflate uncertainty — a recall estimate on 50 queries swings wildly. Very large query files shrink intervals, which is why serious suites use thousands of requests. Treating autocorrelated samples as independent (for example consecutive latencies under overload) understates error; blocking or trial-level aggregation fixes that. Changing two variables at once (new ef and new machine) attributes noise to the wrong cause. Mixing cold-start trials into a “warm” mean widens and biases the interval unless you stratify. Comparing two systems by eye without checking whether their intervals overlap is how 1% differences become marketing slides.

Weaviate-style ANN reporting implicitly fights uncertainty with volume, but you still need explicit intervals for close calls.

How should you apply confidence intervals when reading or running Weaviate-class benchmarks?

Public tables that run on the order of 10,000 requests per configuration drive sampling error on mean latency and recall down — a good start — yet a one-shot multi-threaded QPS number still deserves repeated trials if you will pick hardware on a small gap. When you tune ef, efConstruction, or quantization, require non-overlapping intervals (or a formal test) before declaring a winner at fixed recall. For p99 especially, uncertainty is large unless you have huge n or many repetitions; quote intervals or at least the trial-to-trial spread. Hold seeds, insertion order, dataset slice, and cache policy fixed so the interval reflects measurement noise, not uncontrolled confounds. If two Weaviate configs’ recall intervals overlap but latency intervals do not, you have a real speed story without a quality story — or vice versa.

Use intervals as a decision gate, not decoration.

How do you report them clearly?

State the metric, the estimator (mean, median, p99), the confidence level, the method (t-interval, bootstrap, trial repeats), and n (queries and/or trials). Plot error bars on recall–QPS curves so Pareto fronts are not drawn sharper than the data support. Precommit the comparison rule: e.g. prefer A only if A’s latency CI lies entirely below B’s at equal recall within tolerance. Revisit intervals after changing concurrency or batching — those change variance, not only the center.

A confidence interval turns a brittle headline number into an honest range so HNSW tuning decisions survive re-runs. Next, read “What is a reproducibility manifest?” to lock down the seeds and environment that keep those intervals meaningful, then “What is ground-truth computation?” because recall intervals are only as trustworthy as the exact neighbors you score against.