What is the BIGANN dataset?

The BIGANN dataset - also called ANN_SIFT1B or SIFT1B - is a public billion-scale ANN corpus of one billion 128-dimensional SIFT descriptors stored as bytes, with ten thousand queries and Euclidean ground truth, used to stress memory, disk, and throughput beyond what million-point packs can show.
Created: Updated: 5 min read

The BIGANN dataset – also called ANN_SIFT1B or SIFT1B – is a public billion-scale ANN corpus of one billion 128-dimensional SIFT descriptors stored as bytes, with ten thousand queries and Euclidean ground truth, used to stress memory, disk, and throughput beyond what million-point packs can show.

What is inside BIGANN at full scale?

The base file holds 1,000,000,000 vectors of dimension 128 in the compact .bvecs layout (unsigned bytes per coordinate, plus a small header per vector). That base alone is on the order of 90+ GB on disk before any index structure. A learn set of 100,000,000 vectors supports codebook or quantizer training. The query set has 10,000 vectors in the same dimension. Ground truth for the full billion lists, for each query, up to 1,000 nearest base IDs ordered by increasing squared Euclidean distance – deeper than the k=100 truth files on SIFT1M and GIST1M. Critically, the corpus also ships ground truth for prefixes of the base: the first 1M, 2M, 5M, 10M, 20M, 50M, 100M, 200M, 500M, and 1B vectors. Those slices let you climb the scale ladder with matching truth files instead of inventing your own brute-force labels.

Those slices also create a naming trap every HNSW experimenter hits once.

BIGANN uses SIFT descriptors and 128 dimensions, so it looks like "SIFT1M but bigger." It is not the same as the classic million-point Texmex SIFT1M pack. A 1M prefix of bigann_base.bvecs is a different set of vectors than the standalone SIFT1M release, with different ground truth. Quoting "SIFT1M numbers" without saying which corpus you used mixes two benchmarks. BIGANN descriptors are also stored as bytes rather than float32 .fvecs, so loaders must cast or distance against uint8 consistently. The intended literature metric on this family is often recall@R for whether the true 1-NN appears in the top R – alongside ordinary recall@k against the full neighbor lists. Report which definition you used; do not assume readers will guess.

At this cardinality the engineering story shifts from "fit the graph in RAM" to "design for memory hierarchy."

Why does BIGANN change how you think about HNSW?

One billion 128-d float32 vectors would need on the order of half a terabyte just for payloads; even uint8 base storage plus a dense HNSW edge set pushes far past a single workstation’s DRAM if you insist on a fully resident uncompressed graph. That pressure is why disk-aware graphs, product quantization, sharding, and compressed resident codes appear in the same conversations as BIGANN. Build time, peak RAM during construction, and streaming insert order become first-class metrics beside QPS. Subset experiments (10M, 100M) are legitimate checkpoints, but they are not substitutes for a labeled 1B run when the claim is billion-scale. Thread-count scaling and NUMA placement matter more when every cache miss is multiplied by a huge working set.

Weaviate’s public tables do not center on full BIGANN, but the product’s scale narrative points at the same problem class.

How should you read Weaviate results next to BIGANN-scale goals?

Weaviate publishes detailed multi-threaded HNSW curves on SIFT1M and on larger neural embedding corpora in the millions to tens of millions of objects, and it discusses billion-object ingest and compression as production goals. Those numbers teach operating-point discipline – recall floors, p99 latency, end-to-end object fetch – that still apply when you move toward BIGANN-sized collections. They do not automatically transfer: a Weaviate SIFT1M QPS figure will not predict behavior on a sharded, quantized 1B SIFT index. When you use Weaviate toward that scale, plan shards, vector cache size, compression, and hardware from a reproducibility manifest, and treat BIGANN subsets as staged load tests with their official ground-truth files rather than as informal downsamples.

A short checklist keeps BIGANN work honest and comparable.

How should you run an honest BIGANN or BIGANN-subset experiment?

Name the slice (for example 100M of 1B) and use the matching ground-truth file. Keep Euclidean / L2-squared consistent with the truth definition. Cast .bvecs the same way at build and query time. Sweep search effort into a curve; do not publish one heroic cell. Report peak build RAM, whether the index is memory-resident or disk-backed, compression settings, hardware, threads, insertion order, and seeds. Never call a BIGANN 1M prefix "SIFT1M" without qualification. If you only finish a 10M or 100M slice, say so in the title of the result.

BIGANN is the classic billion-scale Euclidean SIFT stress test – the dataset that forces HNSW discussions past laptop RAM. Next, read "What is the DEEP1B dataset?" for a billion-scale deep-descriptor counterpart, then "What is Big ANN Benchmarks?" for the competition framework that reused and extended this scale of evaluation.