What is load skew?

Load skew in distributed search is uneven pressure across shards or nodes – some hold far more vectors, absorb far more queries, or burn far more CPU and RAM than others – so cluster capacity is gated by hot partitions while cold ones sit idle.

What kinds of imbalance count as load skew?

Skew is not one metric. Size skew means one shard’s HNSW graph and object store dwarf the rest (max vectors ÷ median vectors climbing above healthy ratios). Query skew means traffic concentrates on a few partitions even when sizes look even – a viral product category, a dominant language cluster, or a handful of active SaaS tenants. Resource skew is what operators feel: CPU, memory bandwidth, disk I/O, and GC pauses pile onto the hot node while siblings idle. In fan-out ANN, size and query skew both become latency skew: every global search waits on the slowest arm, so one overloaded shard raises p99 for the whole collection. Replication can mask some read skew by spreading copies, but a single oversized primary shard still hurts ingest, memory, and any consistency path that must touch that replica set.

Different partitioning strategies create different skew failure modes.

How do hash, tenant, and cluster-based layouts produce skew?

Random hash sharding on object IDs – Weaviate’s Murmur-3 UUID placement – usually keeps object counts statistically even, so pure size skew is rare unless IDs are pathological. Multi-tenant collections invert that story: each tenant is one shard, and real tenant sizes are Zipfian – one customer may hold hundreds of thousands of vectors while thousands of others are tiny. That elephant tenant pins an entire HNSW graph on whichever node hosts it; Weaviate places new tenants on the least-loaded node but does not automatically reshuffle existing ones when growth diverges. Cluster-based (semantic) sharding co-locates similar vectors, which invites geometric hotspots: dense regions of embedding space become fat partitions and popular query neighborhoods hammer the same few shards. A common mitigation in research systems is to keep semantic partitions for routing but hash those partition IDs onto nodes so geometric neighbors are not forced onto one host – balance placement without giving up locality-aware probe sets.

Skew shows up in metrics before it shows up as an outage.

How should you detect and quantify load skew?

Track per-shard and per-node vector counts, RSS/HNSW memory, ingest QPS, query QPS, and p95/p99 latency side by side. Alert when max/median size or QPS exceeds a chosen factor (many teams start watching around 2×). Weaviate query profiles that tag each shard with its executing node make stragglers visible on multi-node fan-out. For multi-tenancy, rank tenants by object count and by request rate separately – a medium-sized tenant with huge QPS is as dangerous as a huge idle tenant filling RAM. Watch import queues and async indexing backlogs on hot shards; they often lead latency symptoms. Correlate with product events (launches, crawls, model refreshes) that suddenly thicken one region of the keyspace or one tenant.

Mitigation is operational as much as algorithmic.

How do you reduce skew without breaking the index?

For hash-sharded single-tenant collections, ensure enough shards relative to nodes so placement can spread, and use replica movement (Weaviate supports moving or copying shard replicas between nodes when enabled) to rebalance after scale-out or to drain a hot machine. For multi-tenant skew, replicate hot tenants for read scale, activate/inactive/offload cold tenants so they stop holding hot memory, and plan node capacity around the largest expected tenant because one shard cannot split across nodes. For semantic partitions, combine locality-aware routing with load-aware placement, selectively replicate hot partitions, and recluster when distribution drift recreates elephants. Vertical scale buys time; it does not fix a Zipfian tenant map. Always remeasure recall and p99 after moves – rebalancing changes which fan-out arms are slow, not the merge math itself.

Load skew is uneven data or traffic that turns a few shards into bottlenecks for the whole cluster. Next, read fault tolerance for when hot-node failure coincides with skew, fan-out queries for why one slow shard dominates latency, and random versus cluster-based sharding for which placement policy you are actually fighting.