What is the trade-off between network cost and computation cost?
The trade-off between network cost and computation cost is the design tension in distributed vector search between spending time and bandwidth on moving queries, vectors, and candidate lists across machines versus spending CPU (or GPU) cycles on local distance work, graph walks, and merges — end-to-end latency is the sum of both, so optimizing only one often makes the other worse.
What counts as network cost versus computation cost in ANN?
Network cost covers every byte and round-trip the query path needs: shipping the query vector to shards, returning neighbor IDs and distances (or full vectors), coordinating a coordinator’s fan-out and top-k merge, and — in RDMA or disaggregated-memory setups — remote reads of adjacency lists and embeddings. Computation cost covers everything done once data is local: distance evaluations, HNSW beam expansion, quantization decode or asymmetric scoring, filter evaluation, and the final heap merge. A configuration can look “cheap” on CPU while drowning in serial round-trips, or look “light” on the wire while burning cores on redundant probes. Good distributed ANN design measures both in the same budget: wall-clock time under a target recall.
Sharding makes that budget explicit on every query.
How does sharding and fan-out force the trade-off?
When an index is split across nodes — Weaviate’s usual horizontal path — a query is typically fanned out to one or more shards, each runs local HNSW (or another index), and a coordinator merges partial top-k lists. Probing more shards raises recall when neighbors are scattered, but multiplies network messages and parallel compute; probing fewer shards saves both, at the risk of missing the true neighborhood (recall decomposition). Random sharding tends to force wider fan-out for the same recall; cluster-based sharding plus a routing or meta-index can shrink the probe set so you pay less network and less aggregate compute — until routing mistakes send you back to more probes. Replication flips the ledger: you pay network and storage to keep copies, then often cut query network by serving reads from a local replica. The “right” shard count is rarely the one that minimizes either dollars of RAM or dollars of bandwidth alone; it is the Pareto point for QPS, latency, and recall under your skew.
Research systems push the same trade-off into finer-grained graph walks.
When should you ship work instead of shipping vectors?
Naïve remote graph search pulls full neighbor payloads over the network for every hop — high bandwidth, serial latency, and wasted transfer of neighbors the heap will discard. Near-data scoring flips the choice: keep vectors next to CPUs on the storage hosts, send only IDs (or short keys), compute distances where the data lives, and return compact scores. That raises remote compute slightly and cuts bytes dramatically — often several times less traffic than shipping embeddings. Batching multiple candidate fetches into one request amortizes RTT; pruning hopeless neighbors before the reply shrinks payloads further. Conversely, if interconnect is extremely fast and local CPUs are saturated, shipping compressed vectors (or PQ codes) to spare compute can win. Quantization itself is a micro version of the same trade: fewer bits on the wire or in cache, more arithmetic to compare in the compressed space. Always ask whether the bottleneck is RTT count, bandwidth, or FLOPs before choosing “move data” or “move work.”
Production systems like Weaviate usually stay on the “compute next to the shard” side of that line.
How does Weaviate typically land on this spectrum?
Weaviate keeps each shard self-contained — object store, inverted index, and vector index together — so distance work and graph traversal happen on the node that owns the data. Query-time network is mostly the query itself plus ranked candidates for a merge, not a remote pointer-chase of the HNSW graph. That keeps network cost predictable and computation local, which matches datacenter NICs and mature shared-nothing ops better than chatty disaggregated walks. You still feel the trade-off when you add shards (more fan-out merge work and messages), raise ef (more local compute per shard), enable compression (less memory, different compute per compare), or place clients far from the cluster (application RTT dominating database compute). Vertical scale (more CPU/RAM per node) buys compute headroom without extra fan-out; horizontal shard scale buys capacity and import parallelism, not free query latency. Treat cross-zone or multi-region links as expensive network: prefer fewer probes, local replicas for reads, and avoid architectures that need many serial remote hops per query.
Choosing a side of the trade-off is easier once you know your bottleneck.
How do you decide which cost to pay?
If traces show waiting on sockets while CPUs are idle, cut round-trips: better routing, fewer shards probed, larger batched fetches, scores instead of vectors, or RDMA with bulk reads rather than chatty PEER hops. If CPUs are pegged and the NIC is quiet, reduce work: lower ef carefully, tighten routing, use quantization, or add replicas/shards so each node sees less of the corpus. If both are hot, you are undersized — scale out with a plan for merge cost, or compress and keep the graph local. Prefer paying modest extra local compute to avoid cross-AZ chatter; prefer paying a little network to avoid rebuilding a huge in-process graph when disaggregation or disk-backed layouts already own the vectors. Re-benchmark after every change to shard count, compression, and replica factor; the trade-off surface moves with hardware and query mix.
Network-versus-computation cost is the ledger behind every distributed ANN choice — fan-out width, near-data scoring, quantization, and Weaviate’s shard-local HNSW are all placements on that ledger. Next, read distributed graph traversal for how walks cross machine boundaries, query-aware data loading for fetching only what a query needs, and the earlier pages on fan-out, top-k merge, and RDMA to see each cost item in isolation.