What is the GIST1M dataset?

The GIST1M dataset is a classic public ANN benchmark of one million 960-dimensional GIST image descriptors, with one thousand queries and precomputed Euclidean nearest-neighbor ground truth, used to stress high-dimensional distance cost and memory for indexes such as HNSW.

What is inside GIST1M, and how does it differ from SIFT1M?

Like its SIFT sibling in the same Texmex-style evaluation family, GIST1M ships a base set, a learn set, queries, and ground truth. The base holds 1,000,000 float vectors – still one million points – but each vector has 960 dimensions instead of 128. The learn set is large (on the order of half a million vectors) for methods that train codebooks before indexing. The query set is smaller than SIFT1M’s: typically 1,000 queries rather than 10,000. Ground-truth files list the 100 nearest base IDs per query under squared Euclidean distance, so recall@k up to 100 is scored the same way as on SIFT1M. At float32, the base vectors alone need about 1e6 × 960 × 4 bytes ≈ 3.84 GB before any HNSW edges – roughly 7.5× the vector RAM of SIFT1M at the same cardinality. That single fact is why GIST1M exists beside SIFT1M: same scale in points, very different scale in bytes and arithmetic per distance.

The descriptor type explains the geometry you should expect.

What do GIST descriptors represent, and which metric should you use?

GIST descriptors summarize a whole scene’s spatial layout – coarse oriented-energy statistics over the image – rather than local patch histograms like SIFT. They remain real-valued image features, not text embeddings, and the standard ANN protocol compares them with Euclidean (L2 / L2-squared) distance. Building HNSW under cosine while scoring L2 ground truth is a metric mismatch. Relative to 128-d SIFT, each neighbor expansion touches far more floats, so cache misses and SIMD throughput matter more; relative to 768-d or 1536-d neural embeddings, 960-d GIST is in a similar memory band but still comes from classical vision features with different clustering and intrinsic dimension. Use GIST1M when you want a public, Euclidean, high-d image workload – not as a silent stand-in for angular text search.

Those properties make GIST1M a favorite stress test for compression and memory-bound HNSW.

How does Weaviate use GIST1M in HNSW and quantization work?

Weaviate engineering write-ups repeatedly put GIST beside SIFT1M and deeper corpora when explaining why vector storage dominates graph storage: at 960-d, most of the heap is payload floats, so product quantization and related compressors buy a larger relative saving than on short SIFT vectors. Uncompressed HNSW on GIST is reported in the multi-gigabyte range depending on maxConnections and build quality; compressing vectors shrinks that footprint sharply while shifting the recall-latency curve. Brute-force distance timing on GIST is roughly an order of magnitude slower than on SIFT1M at the same million points, matching the dimension ratio. Later PQ-with-rescoring material uses GIST + L2 curves to show compressed search approaching uncompressed quality. Tile-style encoders are discussed as especially friendly on SIFT- and GIST-like distributions. When you cite a Weaviate GIST number, carry operating point, compression settings, and hardware – the dataset name alone is not a reproducibility manifest.

Knowing when GIST1M is the wrong proxy keeps parameter transfer honest.

When should you not treat GIST1M as your production stand-in?

If your live traffic is cosine or dot-product text embeddings, GIST’s Euclidean scene descriptors will not predict your knee on the recall-QPS curve. If you need filtered search, streaming upserts, or billion-scale cardinality, one million static points will not exercise those paths. If you only have a few gigabytes of RAM headroom, GIST1M’s uncompressed float footprint alone can dominate a laptop experiment that felt fine on SIFT1M. If your query set in production is huge, remember GIST1M’s public query count is only one thousand – fine for curves, thin for tight confidence intervals unless you bootstrap or add queries. Use GIST1M to validate high-d Euclidean behavior, memory math, and compression, then move to a corpus whose modality and metric match production.

A short checklist keeps GIST1M runs comparable across labs.

How should you run an honest GIST1M HNSW experiment?

Index and score with squared Euclidean consistently with the ground truth. Budget several gigabytes for float32 base vectors plus graph edges, or enable a documented compressor and report it. Sweep ef (and compression settings) into full recall-versus-latency or recall-versus-QPS curves rather than one cell. Log hardware, thread count, insertion order, and seeds. Keep learn-set vectors out of the indexed base unless your method truly trains on them. If you use Weaviate, say whether timings are end-to-end with object fetch. Do not compare a GIST1M microbench that returns only IDs against Weaviate’s full-request latencies without labeling the gap.

GIST1M is the million-point, 960-d Euclidean sibling of SIFT1M – the dataset that makes vector RAM and high-d distance cost impossible to ignore. Next, read "What is the GloVe embedding dataset?" for a text-embedding-shaped contrast, then "What is the BIGANN dataset?" when you are ready to leave the one-million-point comfort zone.