What is quantization error?

Quantization error is the geometric distortion introduced when a full-precision embedding is replaced by a coarser code – the gap between the true vector (or true distance) and what the quantized representation can express – and it is the reason compressed HNSW search can mis-rank candidates even when the graph walk itself is sound.

What exactly is being “wrong” when we talk about quantization error?

In reconstruction terms, error is how far the decoded approximate vector sits from the original float coordinates, often summarized with mean squared error or a similar norm over dimensions or subspaces. In search terms, what matters more is distance error: the difference between the exact metric (L2, cosine, inner product) and the approximate score computed from codes or reduced-precision values. A tiny reconstruction error that still preserves neighbor order may be harmless; a modest reconstruction error that swaps two nearly tied candidates can drop recall at top-k. Product quantization‘s subspace centroids, scalar quantization‘s bucket centers, int8 scales, and FP16/BF16 rounding each create a different error shape – but all of them inject noise into the comparisons HNSW uses to grow its beam.

That noise is not uniform across the collection or across queries.

When does quantization error get worse in practice?

Error grows when the codebook or scale was trained on an unrepresentative sample, when new inserts drift from the training distribution, or when aggressiveness rises – fewer PQ segments, binary codes, or very coarse buckets. High intrinsic dimension and many near-duplicate neighbors leave less margin: small score perturbations reorder the heap more easily. Metric mismatch compounds the problem – for example normalizing for cosine in float32 then comparing int8 codes that assume a different scaling. Build-versus-query mismatch is another amplifier: edges chosen with exact distances, then searched with noisy estimates, mean the graph’s idea of “near” disagrees with the scorer’s. Under-trained PQ centroids and clipped scalar ranges are classic operational causes of sudden error spikes after a compression rollout.

Inside HNSW, those spikes show up as search behavior changes, not only as a number on a lab plot.

How does quantization error affect HNSW recall and latency tuning?

Noisy scores cause the walk to expand the wrong neighbors first or to evict true near neighbors from a tight efSearch candidate list. Operators often “pay” for error by raising efSearch or over-fetching for full-precision reranking – buying back recall with more distance computations and, sometimes, more memory traffic. Mild error can be invisible at high efSearch and glaring at low efSearch, which is why evaluating compression only at one operating point misleads. End-to-end recall mixes graph approximation with distance approximation; measuring ranking agreement between exact and approximate scores on fixed candidate sets isolates quantization error from coverage failures. Vector databases such as Weaviate pair quantized HNSW with rescoring limits because they treat residual error as expected, not as a rare bug.

You cannot eliminate the error if you compress, but you can bound and budget for it.

How should you measure and manage quantization error before production?

Track reconstruction or distance-error histograms on a held-out set, plus recall@k against an exact baseline at several efSearch values. Retune over-fetch depth whenever you change quantizer aggressiveness. Prefer compression-aware graph construction when build/search distance mismatch dominates. Revisit training limits after major corpus shifts. Choose a milder scheme (FP16, careful SQ) when error budgets are tight; choose PQ or binary when memory dominates and you can afford deeper rescoring. Quantization error is the price tag on vector compression – read it in recall curves, not only in bytes saved.

Quantization error is the distortion between true vectors or distances and their compressed stand-ins, and it is what pushes HNSW to need wider beams and full-precision reranking after compression. From here, approximate distance computation covers how scores are formed, full-precision reranking and two-stage retrieval show how to correct ranking, compression-aware graph construction reduces build/search mismatch, and scalar versus product quantization explain which knobs most strongly move the error.