What is int8 quantization?

Int8 quantization is representing each vector dimension (or each product-quantization code component) as an 8-bit integer instead of a 32-bit float, cutting vector storage roughly fourfold and letting HNSW distance loops run on compact integer data that modern CPUs process efficiently with SIMD.

What does “int8” actually mean for an embedding coordinate?

A float32 value uses four bytes and a wide dynamic range. An int8 value uses one signed byte spanning -128 to 127, or a uint8 spanning 0 to 255 – 256 distinct levels in either case. Quantizing to int8 means choosing a scale (and often a zero-point or per-dimension bucket map) that sends real-valued coordinates into that discrete grid, then storing only the integers. At query time, distance kernels either work directly in integer arithmetic or dequantize on the fly to an approximate float. The same 8-bit width appears inside scalar quantization‘s per-dimension buckets and as the usual width of each product-quantization centroid ID when codebooks have 256 entries. “Int8 quantization” names the precision and storage format; SQ and related schemes name how the mapping into those 256 levels is chosen.

That format choice is popular because hardware and memory systems reward it unusually well.

Why is int8 a sweet spot for HNSW serving hardware?

Four times fewer bytes per vector means four times more embeddings fit in the same RAM and cache footprint, which matters when HNSW keeps vectors hot for repeated neighbor comparisons. Memory bandwidth – often the limiter once many parallel searches run – delivers more candidates per second when each candidate is an int8 row instead of float32. SIMD units on contemporary CPUs pack more int8 lanes into a single instruction than float32 lanes, so a carefully written distance kernel can finish more dimensions per cycle even before counting the bandwidth win. The quantization error from 256 levels is usually far milder than one-bit binary codes, so recall often stays recoverable with modest efSearch increases or a short full-precision rescore. Vector databases such as Weaviate lean on 8-bit schemes (scalar, rotational, and related compressors) for HNSW for exactly this balance of size, speed, and accuracy.

Int8 is still lossy, and the mapping into those 256 bins determines how much geometry you keep.

How do scale choices and asymmetric comparison affect search quality?

A single global scale for all dimensions is simple but wasteful when coordinates have different spreads; per-tensor or per-dimension scales (as in scalar quantization) allocate levels where the data actually lives. Signed versus unsigned domains, clamping of outliers, and whether the query stays in float32 while database vectors are int8 (asymmetric distance) all change both speed and error. Asymmetric setups are common in ANN: keep the query rich, compare against compact codes, optionally rescore top candidates with stored full-precision vectors. If build-time edges were chosen with float distances and query-time ranking uses int8 estimates without compensation, the graph and the scorer disagree – a frequent source of quiet recall loss. Measuring quantization error on your metric (cosine, L2, inner product) beats assuming a generic “4× compression” claim.

Int8 is not the only reduced-precision option, and it is not always enough alone.

How should you place int8 among FP16, binary, and product codes?

Prefer int8 when you want a clear memory cut, strong SIMD support, and simpler mental models than multi-subspace PQ – often as the payload format behind SQ or an 8-bit rotational scheme. Prefer FP16 or BF16 when you need still-floating values with less aggressive discretization and smaller savings. Prefer PQ when even int8 rows are too large and you accept codebook complexity for higher compression. Prefer binary when extreme compression dominates and coarser ranking is acceptable. In all cases, retune HNSW search width after switching precision, and validate recall on held-out queries before calling the migration done. Int8 quantization is the workhorse precision layer under much of modern compressed HNSW.

Int8 quantization stores embedding values as 8-bit integers so HNSW indexes shrink and distance kernels run leaner, at the cost of discrete approximation error you must measure and often mitigate with rescoring. From here, scalar quantization explains one common way to fill those 256 levels, FP16 and BF16 cover milder floating alternatives, product quantization shows when 8-bit codes index subspaces instead of raw dimensions, and full-precision reranking covers the usual fix when int8 ranking is almost – but not quite – good enough.