What is SIMD (single instruction, multiple data)?

SIMD (single instruction, multiple data) is a CPU feature that applies one arithmetic operation to several values at once in wide registers, which is how HNSW implementations accelerate high-dimensional distance calculations on a single core.
Created: Updated: 5 min read

SIMD, short for single instruction, multiple data, is a CPU design in which one instruction performs the same operation on several values at once — for example multiplying eight floats in a single step — instead of processing those values one at a time in a scalar loop.

How does a SIMD instruction actually differ from ordinary scalar code?

A scalar loop that computes a dot product loads one dimension from each vector, multiplies, adds into an accumulator, then moves to the next dimension. A SIMD implementation packs several dimensions into wide registers, issues one multiply and one add that touch every lane in those registers together, then advances by that many dimensions. The program still expresses the same mathematics; the hardware simply applies each arithmetic step across a small batch of elements in parallel inside one core. How many lanes fit in a register depends on the instruction set and the element size: narrower numbers (16-bit floats, 8-bit integers) pack more lanes into the same register width than 32-bit floats, which is one reason reduced-precision distances often go hand in hand with SIMD. The compiler may emit these instructions automatically for simple loops, or implementations may use explicit intrinsics when the gain matters enough to hand-tune.

That pattern maps almost perfectly onto the distance calculations HNSW performs on every candidate it scores.

Why is SIMD such a natural fit for HNSW distance computations?

Every hop that considers a neighbor needs a full distance or similarity between the query and that neighbor’s vector — Euclidean distance, inner product, or cosine via normalized inner product — and each of those reduces to walking corresponding dimensions with multiplies and adds. The work is uniform, branch-light, and long enough in high dimensions that amortizing loop overhead across SIMD lanes pays off. Speeding up that inner kernel does not reduce how many candidates the graph search visits, but it lowers the cost of each visit, which is often a large fraction of total CPU time when the working set is already warm in cache. Production HNSW stacks, including distance paths used in systems like Weaviate, routinely ship architecture-specific SIMD kernels for exactly this reason: the graph algorithm is the same, while the bytes-per-cycle burned inside each comparison change with the instruction set.

SIMD only helps once the data is available to the core; it does not erase the memory hierarchy costs that dominate cold graph hops.

If each next vector address depends on a cache-missing load from a random neighbor, the core spends its time waiting on main-memory latency rather than on arithmetic throughput. In that regime a faster distance kernel sits idle behind stalled loads, and profiles show memory waits rather than busy SIMD units. SIMD shines when vectors are already in L1/L2, when layout is contiguous enough for wide loads to hit, and when the search is comparing enough candidates that arithmetic volume is real — typical of warm queries, higher efSearch with a hot working set, or construction searches that revisit regions already pulled in. Alignment and contiguous storage matter: SIMD loads want consecutive dimensions packed without awkward splits across poorly aligned boundaries. Lower-precision formats help twice when they fit the pipeline — less bandwidth per vector and more lanes per instruction — but they trade accuracy unless a later full-precision rerank step corrects the ranking.

SIMD is also not the same thing as multithreading: one is parallelism inside a core’s registers, the other is parallelism across cores.

How should you think about SIMD together with threads and recall tuning?

Using SIMD speeds a single distance call on one thread; adding threads runs many searches or many construction workers at once. Both can be necessary for throughput, and both can contend for memory bandwidth when overdone. From a tuning perspective, SIMD is usually a one-time implementation win you enable and keep, while efSearch and M remain the knobs that trade recall for how much work — and therefore how many SIMD-accelerated distances — each query performs. If latency is high and counters show the core compute-bound in the distance loop, SIMD and precision choices are the right layer. If the core is bound on cache misses and pointer chasing, memory layout and hop count matter more than another instruction-set variant. Knowing which side you are on prevents polishing the arithmetic while the graph walk waits on DRAM.

SIMD lets one instruction crush several vector dimensions at once, which is why it underpins fast HNSW distance kernels once data is on-chip. From here, the page on vectorized distance computation applies this idea directly to metric loops, memory bandwidth and the roofline model explain when arithmetic can actually feed, the CPU-optimization chapter walks through distance and layout choices in depth, and FP16 / int8 glossary entries cover the precision formats that widen SIMD lanes.