What should a complete HNSW benchmark record to be trustworthy?

A complete HNSW benchmark needs to record enough detail that someone else, on different hardware and months later, could look at the results and understand exactly what was measured, under what conditions, and whether their own results are actually comparable — anything less than this turns a benchmark from a reusable, checkable record into a number nobody can verify or build on.

What environment and software details need recording?

The specific CPU or GPU used, the amount of available memory, whether NUMA effects were a factor, and which instruction sets were available all belong in a trustworthy record, since as covered in this site’s page on common benchmarking mistakes, running two configurations on different hardware and presenting the result as a fair comparison is one of the most common ways a benchmark misleads. The software version or commit being tested, and the thread count used during both construction and querying, matter just as much — a result reported without a thread count attached can’t be meaningfully compared against a similar-looking result run with a different one.

What dataset and metric details need recording?

The dataset itself needs identifying precisely enough that someone else could reproduce it — its size, its dimensionality, and, ideally, exactly which version or snapshot was used if the underlying data can change over time. The distance metric and any normalization applied before indexing need recording explicitly too, since, as covered in this site’s page on measuring distance and similarity between vectors, comparing results computed under different metrics produces numbers that were never actually measuring the same thing in the first place. Whether queries were drawn from the same distribution as the indexed data or deliberately out-of-distribution is worth stating plainly as well, since this single detail changes what a given recall number actually demonstrates.

What build and query parameters need recording, and why does insertion order belong on this list?

M, efConstruction, and efSearch obviously belong in any complete record, but two details are easy to forget and just as important: the random seed used for level assignment, and the order in which vectors were inserted during construction. As covered in this site’s page on common benchmarking mistakes, building an index with an artificially sorted insertion order rather than a realistic one can produce a graph with meaningfully different properties, and reporting a result without recording insertion order makes that variable invisible to anyone trying to reproduce or trust the result later. Recording multiple seeds, along with the resulting spread rather than a single number, is what actually lets a reader distinguish a genuine effect from ordinary build-to-build variance, as this site’s page on tuning methodology covers in more depth.

What does recording all of this actually buy you?

A benchmark manifest this thorough turns a single, static result into something that can actually be trusted, reproduced, and extended by someone other than whoever originally ran it — including, often, the very same person returning to their own results months later without any memory of the exact conditions under which they were produced. Every category listed on this page corresponds directly to a specific way benchmarks go wrong, covered throughout this site’s evaluation-and-benchmarking section; treating this list as a checklist to fill out before publishing or acting on any benchmark result is a small amount of discipline that prevents a large amount of wasted effort chasing conclusions that never actually held up.

This appendix closes out the structured curriculum this site has worked through from foundational concepts to production concerns, and from here the glossary sections cover every individual term encountered along the way — vector and distance math, HNSW-specific parameters, systems and hardware concepts, and everything else — as standalone, direct-answer reference pages worth consulting whenever a specific term needs a quick, focused explanation on its own.