What is random sharding?

Random sharding assigns each object to a shard with a content-blind rule – usually a hash of its ID – so placement is statistically even and independent of vector similarity, at the cost of sending most global ANN queries to every shard and merging their local top-k results.

How does random (hash) sharding place objects?

On insert, the system takes a stable key for the object, runs a fast hash, and maps the digest onto one of S physical shards (often through a larger set of virtual shards that bucket into physical ones). In Weaviate, that key is the object’s UUID, the function is 64-bit Murmur-3, and the strategy is literally named hash in the collection’s sharding config – a textbook random partition with respect to embedding geometry. Nearby vectors in cosine or L2 space are no more likely to share a shard than distant ones; UUIDs that happen to collide in the hash ring do. Virtual shards sit between the hash and the physical machines so that later rebalancing can move smaller units with less data churn, but the placement philosophy stays the same: ignore the vector, trust the hash for balance. Multi-tenant layouts that give each tenant its own shard are a different axis of isolation; within a single-tenant collection, random hashing is the default way Weaviate spreads objects across desiredCount shards.

That blindness to similarity is exactly why the query path looks the way it does.

Why do ANN queries fan out under random sharding?

True nearest neighbors of a query can land on any shard with equal chance. Therefore a correct global search must probe all shards (or accept systematic recall loss). Each shard runs its local HNSW (or other ANN) search for some k’ ≥ k candidates; a coordinator merges distances and returns the global top-k. Adding shards shrinks each local graph and can speed per-shard work and imports, but it does not remove the fan-out: total coordination and merge cost grow with S, and end-to-end latency is often dominated by the slowest shard. Weaviate’s horizontal-scaling guidance matches this: more shards help dataset size and ingest parallelism; they do not by themselves raise query QPS the way replicas do. Random sharding preserves high recall when fan-out is complete and per-shard k’ is generous enough for the merge – the failure mode is under-fetching locally, not “missing the right shard,” because every shard is always in scope.

Operators choose random sharding for operational simplicity as much as for balance.

What are the strengths and limits of random placement?

Strengths: even object counts and memory footprints across shards when IDs are well distributed; no training of centroids; no hot shards caused by a popular region of embedding space; deterministic routing for get-by-ID; easy mental model for capacity planning. Limits: unavoidable query broadcast for unfiltered vector search; network and merge overhead that scales with shard count; no pruning of irrelevant partitions; resharding an HNSW-backed collection remains expensive even with virtual shards, so shard count should be planned up front. Random sharding also does not fix load skew from multi-tenancy or from a few huge filtered subsets – those need tenant placement or filter-aware design. Compared with cluster-based sharding, you trade selective routing for predictability: you never wonder whether the router probed enough centroids, because there is no router beyond “all shards.”

Knowing when random is enough clarifies when to look at locality-aware alternatives.

When should you keep random sharding versus consider cluster-based schemes?

Keep hash/random sharding when cluster ops simplicity matters, when shard counts are modest, when replication already handles QPS and HA, and when you are fine paying full fan-out for recall – the default Weaviate single-tenant path. Consider cluster-based or semantic sharding research designs when shard counts are large, cross-node latency dominates, and embeddings form tight clusters so a routing index can send each query to a small subset of shards without tanking recall. Even then, many systems still hash partition IDs onto nodes so popular geometric regions do not pin all traffic on one machine – random placement of partitions, clustered content inside partitions. Always validate recall after changing shard count or local k’, watch p99 waiting on straggler shards, and remember that filtered search can change the picture if filters correlate with shard keys (they usually do not under UUID hashing).

Random sharding is content-blind hash placement that balances load and forces all-shard ANN fan-out. Next, read cluster-based sharding for locality-aware placement, then routing indexes and fan-out / top-k merge for how selective and broadcast queries are executed, and revisit the general sharding page whenever you set desiredCount on a Weaviate collection.