What common mistakes make HNSW benchmarks misleading?

Common mistakes make HNSW benchmarks misleading not by being technically wrong in isolation, but by quietly comparing things that were never actually comparable in the first place — an inconsistent ground truth, a different hardware environment, an unrealistically easy query set, or a single cherry-picked number can each make a benchmark look rigorous while still leading to a conclusion that doesn’t hold up in production.

What ground-truth and data mistakes silently invalidate a benchmark before it even starts?

An incorrect or inconsistently computed ground truth undermines every recall number built on top of it, and this happens more easily than it sounds — using a slightly different distance metric to compute ground truth than the one the index actually searches with, or computing ground truth against a stale copy of the dataset that no longer matches what was actually indexed, both quietly poison every later measurement without necessarily producing an obviously wrong-looking result. Comparing recall@1 for one method against recall@10 for another is a related but more obvious version of the same category of mistake: the two numbers measure genuinely different things, and placing them side by side in a results table implies a comparison that was never actually made.

What environment inconsistencies make two “comparable” results actually incomparable?

Running two configurations on different CPU instruction sets, with different thread counts, or on hardware with different memory bandwidth characteristics, and then presenting the results as a fair head-to-head comparison, conflates the effect of the environment with the effect of whatever was actually meant to be tested. Memory reporting is a particularly easy place for this to go wrong: including raw vector storage in one method’s reported memory footprint while excluding it from another’s, or reporting one method’s peak build-time memory against another’s steady-state memory after construction finishes, produces numbers that look directly comparable in a table while actually describing different things entirely.

Why does testing with queries that are already in the dataset flatter the results?

A query vector that’s also a member of the dataset being searched will always find itself as a distance-zero exact match, which makes recall look artificially good in a way that has nothing to do with how the system would perform on genuinely new queries it has never seen before. A related issue arises from insertion order during construction: building an index by inserting vectors in a convenient, artificially sorted order rather than a realistic, effectively random one can produce a graph with different — sometimes better, sometimes worse — properties than the same data would produce under a realistic insertion pattern, which is exactly why insertion order deserves to be recorded and reported as part of a benchmark’s manifest rather than left as an undocumented implementation detail. A cold-cache benchmark run immediately after a warm-cache one, without deliberately flushing or accounting for what’s still sitting in cache from the previous run, can likewise flatter whichever configuration happens to run second purely because of leftover cache state rather than anything about the configuration itself.

Why is reporting a single operating point often the most misleading choice of all?

Because recall, latency, memory, and build time all trade off against each other through adjustable parameters, reporting one single number for each — the equivalent of a single point plucked off what should be a whole trade-off curve — invites cherry-picking, whether or not that’s the intent. A configuration that looks dramatically better at one specific recall target might look considerably worse at a different, equally reasonable target, and a reader shown only the favorable point has no way to know this. Reporting a full recall-versus-latency curve across a meaningful range of parameter settings, rather than one hand-picked configuration, is what actually lets a reader judge whether a claimed improvement holds up generally or only at the one specific point that happened to get reported.

With evaluation and benchmarking now covered end to end, the next part of this site turns from measuring HNSW to actually using it — starting with how it functions inside recommendation systems, one of its most common real-world applications. Anyone setting up their own benchmark should treat the methodology and pitfalls covered across this part of the site as a joint checklist, since a technically sound method applied to a poorly chosen dataset, or a well-chosen dataset measured with a flawed method, both still produce results not worth trusting.