Which benchmark suites and datasets should you use, and for what purpose?

Which benchmark suite and dataset make sense depends entirely on the question being asked: a quick, classical million-vector dataset is enough to sanity-check an implementation or compare two parameter settings, while understanding how HNSW behaves on realistic modern embeddings, at billion-vector scale, or under filtering and streaming updates each require datasets specifically built to expose those particular conditions.

Why do classical datasets like SIFT and GIST still get used for quick, comparable testing?

Datasets built from classical, hand-engineered image descriptors — a million-vector collection commonly called SIFT1M, and a larger-dimensional counterpart called GIST1M — became an early standard specifically because they’re small enough to experiment with quickly, well understood, and widely used enough that a result reported against them is instantly comparable to a large body of prior published work. A GloVe word-embedding dataset serves a related purpose while testing a different geometry: word embeddings tend to behave differently under cosine or angular similarity than the classical image descriptors above do, making GloVe a useful, still-small-enough-to-iterate-on stand-in for the angular, semantic-similarity workloads much closer to what modern applications actually search over. None of these datasets should be mistaken for a proxy for every kind of data an index might eventually serve — they’re best understood as fast, cheap, historically comparable baselines rather than a final word on real-world performance.

Why do modern embedding workloads need different benchmark data than classical descriptors?

Vectors produced by modern learned embedding models frequently have different statistical properties than hand-engineered descriptors from image processing — different intrinsic dimensionality, different clustering behavior, and often deliberately normalized magnitudes that change which distance function makes sense. A billion-scale dataset built from deep, learned descriptors rather than classical hand-engineered ones, generally known as DEEP1B, was introduced specifically because conclusions drawn from classical descriptor datasets didn’t reliably transfer to this newer, more representative kind of data — a direct demonstration that a technique performing well on one data distribution doesn’t automatically perform equally well on another, even at comparable scale. This is the core motivation behind newer benchmark efforts built specifically around contemporary embedding models, in-distribution and out-of-distribution query sets, and both CPU and GPU execution, rather than continuing to lean entirely on classical descriptor datasets that predate the embedding models most current applications actually use.

What does testing at billion-scale actually add beyond a million-vector test?

Conclusions that hold at a million vectors don’t automatically hold at a billion — memory pressure, cache behavior, and the sheer number of distance computations involved all change qualitatively once a dataset grows by three orders of magnitude, in ways a smaller-scale test simply can’t reveal. Billion-scale benchmark collections, generally referred to collectively as big-scale ANN benchmarks, bundle several such large datasets together with specialized tracks covering filtered search, out-of-distribution queries, and sparse or streaming workloads, specifically to expose systems-level limits — memory ceilings, build-time practicality, and update throughput — that remain comfortably invisible at smaller scale. Testing exclusively at a convenient, fast, small scale risks drawing conclusions that quietly stop applying the moment a real deployment grows past that scale.

Why do filtered, streaming, and out-of-distribution benchmarks exist as their own categories?

Each of these targets a specific real-world condition that a plain, static, unfiltered nearest-neighbor benchmark doesn’t exercise at all. A filtered benchmark measures how badly, or how gracefully, a system’s recall degrades as a metadata filter grows more selective, directly testing the concerns covered elsewhere on this site about filtering and hybrid search. A streaming benchmark measures behavior under continuous inserts and deletes rather than a single static build, testing exactly the update-and-delete difficulties covered in this site’s page on graph repair. An out-of-distribution benchmark deliberately draws queries from a different distribution than the indexed data, testing whether a system’s tuning — and its underlying graph structure — remains reasonable when real usage doesn’t perfectly resemble whatever data the system happened to be built or tuned on. Each of these represents a genuine production concern that a classical, static, in-distribution benchmark simply cannot surface.

Choosing the right dataset only solves half the reproducibility problem — the next page covers the other half: the specific, easy-to-make mistakes that turn a technically correct benchmark into a misleading one. For synthetic datasets that let a single geometric property, like intrinsic dimensionality, be varied in isolation while everything else stays fixed, this site’s high-dimensional-geometry glossary section explains the underlying concept these datasets are built to test.