What is sharding in a search index?

Sharding in a search index partitions one logical collection into multiple self-contained pieces (shards), each with its own data and local indexes, so storage, memory, and ingest work can be spread across CPUs or cluster nodes instead of living in a single monolithic index.

What is a shard, and what does it contain?

A shard is the unit of horizontal data placement. In a vector search system such as Weaviate, each shard typically owns a subset of the collection’s objects plus the indexes needed to serve that subset on its own: an object store, inverted indexes for filters and keyword fields, and a vector index (commonly an HNSW graph for that shard’s vectors). Because the vector graph sits beside the objects for the same IDs, a node holding a shard can answer local ANN and filtered queries without borrowing another shard’s adjacency lists. Multiple shards make up one collection index; they may all sit on one machine (to parallelize imports across cores) or be spread across a cluster (to grow beyond one node’s RAM). In multi-tenant collections, Weaviate often maps each tenant to its own shard so isolation is physical as well as logical – deletes and index pressure stay inside that tenant’s boundary.

That packaging exists because a single graph does not stretch forever.

Why do search indexes shard instead of growing one giant graph?

In-memory HNSW cost scales with vectors and edges. Past tens or hundreds of millions of high-dimensional embeddings, one node’s RAM, rebuild time, and insert latency become the bottleneck. Sharding cuts the working set: each shard’s graph is smaller, imports can proceed in parallel into different shards, and adding nodes can host additional shards when the dataset outgrows vertical scaling. Disk-oriented indexes reduce the need to shard purely for memory, but sharding still helps ingest throughput and operational isolation. The flip side is query path complexity: a similarity search over the whole collection must reach every relevant shard (a fan-out), run local top-k, then merge candidates – more shards means more coordination, not automatically more QPS. Weaviate’s own guidance is explicit: more shards help dataset size and import speed; replication, not sharding, is what raises query throughput and availability by copying the same data.

How objects land on shards determines whether those benefits stay balanced.

How are objects assigned to shards?

Assignment uses a sharding key and a deterministic function. In Weaviate’s single-tenant collections, the key is the object’s UUID and the function is a 64-bit Murmur-3 hash into a virtual-shard space that maps onto a fixed number of physical shards. That hash routing is essentially random with respect to vector similarity – nearby embeddings are not deliberately co-located – which keeps load statistically even but means every global ANN query must consult all shards (or a routing layer that still covers the space). Cluster-based or learned partitioning schemes try to put similar vectors together so some queries hit fewer shards; those appear later in this glossary as their own topics. Shard count is usually chosen at collection creation and treated as immutable for the index lifetime, because splitting an HNSW graph afterward is expensive. Virtual shards exist so future rebalancing can move finer units with less data churn, but operators still plan physical shard count for expected growth rather than assuming cheap resharding.

Day-to-day operations follow from those assignment and query rules.

How should you size and operate sharded indexes?

Estimate peak corpus size and memory per HNSW (or choose a more disk-friendly index), then set shard count so you can add nodes later without re-creating the collection – for example more shards than initial nodes. Even on one node, several shards can speed bulk import by using multiple CPUs. Watch for load skew: hash sharding is usually even, but multi-tenant setups can skew if a few tenants dominate; tenant-aware placement and offloading idle tenants become part of capacity planning. Do not confuse sharding with replication: replicas are extra copies of the same shard for HA and read scale; shards are disjoint pieces of the keyspace. Measure end-to-end recall after fan-out merges, because per-shard top-k that is too small can drop true neighbors that only rank highly after a global merge. When a node fails, sharded-only layouts lose the shards they hosted unless those shards also have replicas elsewhere – fault tolerance is a replication concern layered on top of the partition plan.

Sharding splits one search index into many self-contained pieces so size and ingest can scale out; it does not by itself multiply query capacity or survive node loss. Next, read replication for how copies add HA and QPS, then random versus cluster-based sharding for how placement policies change fan-out, and the fan-out / top-k merge pages for what a distributed ANN query does after the shards are chosen.