What is vector quantization?

Vector quantization is lossy compression of embedding vectors: each full-precision coordinate (or group of coordinates) is replaced by a coarser code so the index stores and compares far fewer bits per vector while still approximating distances well enough for approximate nearest-neighbor search.

Why does HNSW care so much about how vectors are stored?

An HNSW index keeps a graph of neighbor links and, for search, repeatedly loads candidate vectors to compute distances. Graph edges are relatively compact; the vectors themselves often dominate RAM. A 768-dimensional float32 embedding is about three kilobytes before any graph overhead – multiply by tens of millions of objects and memory, not CPU cycles alone, becomes the scaling wall. Every distance in the walk also moves those bytes through caches and DRAM. Shrinking each vector means more of the working set fits in memory, more candidates fit in a cache line budget, and more comparisons complete per second for the same bandwidth. Quantization attacks that cost directly without throwing away the graph topology that makes HNSW fast.

The compression is approximate by design, so search quality and speed shift together rather than one alone.

What does quantization change inside an HNSW search?

During traversal, the index compares the query to neighbor vectors using the compressed codes – or a fast distance estimate derived from them – instead of full float32 (or float64) arithmetic on every hop. That estimate ranks candidates for the beam (efSearch). Because codes are imperfect, the ranking can mis-order some neighbors relative to true distance, which is quantization error showing up as recall loss unless you compensate. Common compensations include raising efSearch so the walk examines more candidates, and two-stage retrieval: over-fetch with cheap quantized scores, then rerank a shortlist with full-precision vectors kept on the side. Construction can use the same codes (compression-aware build) or build on full precision and quantize afterward; each choice trades build cost, memory peak, and how well edges match the distances search will use later.

“Vector quantization” is the family name; concrete methods differ in how they carve up the space.

How do the main quantization families differ at a high level?

Scalar approaches map each dimension independently onto a small integer alphabet – int8 buckets, sometimes after a transform that spreads energy across coordinates. Product quantization splits the vector into subspaces and replaces each subspace with a codebook index, achieving aggressive size cuts when codebooks fit the data. Binary schemes collapse each dimension (or a rotated view of it) toward a bit, maximizing compression and making distance a fast bitwise operation at the cost of coarser geometry. Lower-bit floating formats such as FP16 or BF16 sit between full precision and integer codes: less dramatic savings, milder error. Systems choose among these based on memory budget, whether a training pass over data is acceptable, and how much recall drop the application can absorb. Vector databases such as Weaviate expose several of these options on HNSW indexes specifically so operators can cut the in-memory vector footprint while keeping the same graph-based search path.

Picking a scheme is only half the decision; you still have to validate it on your vectors and queries.

How should you evaluate quantization before relying on it in production?

Measure recall at fixed efSearch before and after compression, then retune efSearch (and any rescore limit) until you recover an acceptable operating point on the recall-versus-latency curve. Watch memory: quantized vectors shrink, but codebooks, graph edges, and optional full-precision sidecar storage still count. Check that distance semantics stay consistent – cosine versus dot product versus Euclidean – because some code paths quantize in a space that assumes normalized inputs. For trained methods, use a representative sample; a codebook fit on the wrong slice of the collection systematically biases distances. Quantization is not a substitute for good embeddings or sane HNSW parameters; it is a storage and bandwidth lever that reshapes the same search trade-offs you already tune.

Vector quantization compresses embeddings into compact codes so HNSW can hold more data in memory and score candidates faster, at the cost of approximate distances you must validate and often compensate for. From here, scalar quantization and product quantization unpack the two most common trained schemes, int8 and FP16/BF16 cover precision ladders, full-precision reranking and two-stage retrieval show how to claw recall back, and the rest of the compression chapter works through error, adaptive policies, and compression-aware graph construction.