Benchmarking and evaluation concepts

Benchmarking and evaluation concepts define honest measurement: QPS, latency percentiles, distance computations, build cost, energy, cache warmth, reproducibility, and Pareto trade-offs between recall and speed.
Created: Updated: 2 min read

These are the topics in this section, each on its own page with a stable path you can bookmark or share.

What topics are covered in this section?

Benchmarking and evaluation concepts define honest measurement: QPS, latency percentiles, distance computations, build cost, energy, cache warmth, reproducibility, and Pareto trade-offs.

What separates a trustworthy ANN benchmark from a misleading one?

Report recall with latency (p50/p95/p99) and QPS at stated thread counts, plus distance computations per query when comparing algorithms. Build time, peak build RAM, and update throughput matter for lifecycle cost. Cold versus warm caches and singleton versus batch queries change results dramatically. Confidence intervals, reproducibility manifests, ground-truth computation, seed and insertion-order reporting, and hardware-environment disclosure keep comparisons fair. Cherry-picked operating points and missing Pareto frontiers are common failure modes of marketing benches.

Part VIII exists to institutionalize these habits.

How should you benchmark Weaviate?

Use Weaviate’s ANN benchmark guidance and your own golden queries: fix dataset, metric, concurrency, and cache policy; sweep ef and compression; record percentiles and recall together. Energy per query belongs when power or carbon matters. Do not compare a warm in-process microbench to a cold end-to-end HTTP path without labeling both.

Evaluation vocabulary also supports tuning and capacity planning.

Open QPS, latency percentiles, Pareto frontier, reproducibility manifest, and cold-versus-warm cache pages, then Part VIII. For datasets, continue into the benchmark-suites hub.

This section is how we keep score. Next, read Part VIII’s methodology chapter or “What is a Pareto frontier for recall versus latency?”, then apply the same discipline to your Weaviate configs.