What is a meta-index over shards?

A meta-index over shards is a higher-level index – or coordinated index hierarchy – that sits above per-shard ANN structures so a query can discover which partitions to touch, how to traverse levels of centroids, and how to assemble a global answer without treating every shard as an opaque full broadcast target.

How is a meta-index different from a plain shard list or a routing index?

Every distributed search system has some catalog of shards: where they live, which replicas exist, how to contact them. That membership map is necessary but not a meta-index. A routing index is narrower still: a compact ANN over partition signatures that emits a probe set. A meta-index is the broader layer that may include routing but also encodes hierarchical structure – centroids of centroids, multi-level graphs, ID translation, and merge policy – so search descends through levels instead of only broadcasting sideways. In research systems that recursively cluster until the root fits in one machine’s RAM, each upper level is literally an index over the level below: vectors at level L are centroids of partitions at level L−1. The leaf shards hold base vectors and local HNSW (or disk graphs); the meta-index is everything above that makes selective descent possible. Call it a “index over indexes”: its keys are partition representatives, its values are pointers to child indexes or shard endpoints.

That hierarchy changes both build and query shape.

How do hierarchical meta-indexes get built and searched?

Build usually proceeds bottom-up. Partition the base set at a chosen granularity, store each partition’s data on its shard (often SSD-backed for scale), take partition centroids (or richer representatives), and if that centroid set is still too large, cluster again until a root index fits in memory. Query proceeds top-down: search the root meta-index for the nearest upper-level centroids, open only the child partitions those centroids own, repeat until leaf shards run local ANN, then merge candidates upward. Probe width at each level (how many children to open) is the recall dial – the same nprobe idea, stacked. Accuracy-preserving designs tune each level so end-to-end recall targets are met, not merely local recall at one layer. Cross-node work concentrates on level transitions rather than on a full mesh of leaf-to-leaf graph edges, which is why meta-indexes appear when naive sharding of one giant HNSW would create too many cut edges or too much fan-out.

Production vector databases sit on a spectrum of how much meta-structure they expose.

What does Weaviate do instead of a semantic meta-index across shards?

In Weaviate, a collection “index” is already a wrapper over one or more shards, each with its own object store, inverted indexes, and vector index. For default hash sharding, there is no geometric meta-ANN that picks a subset of shards: the coordinator fans the vector query out to the shards that own the keyspace and merges results – orchestration without semantic routing. That wrapper still matters operationally (schema, shard placement, replication), but it is not a centroid hierarchy over embedding space. Closer to a true meta-index pattern inside one shard is HFresh: an in-memory HNSW over posting centroids routes into on-disk posting lists, which is a two-level index (centroid meta-layer + leaf postings) even when the collection has only one shard. Multi-tenant layouts give each tenant a shard; the “meta” concern there is tenant routing and isolation, not nearest-centroid search across tenants. So when this glossary says meta-index over shards, think distributed ANN architecture; when you operate Weaviate day to day, expect hash fan-out plus optional in-shard cluster indexes, not a user-facing multi-level shard router.

The trade-offs explain when the extra layer is worth building.

When is a meta-index worth the complexity?

Build one when leaf count is large, full fan-out dominates latency, and you can keep the upper levels small enough to search quickly while preserving recall with level-wise probes. Skip it when shard counts are modest, hash broadcast is cheap, and operational simplicity wins – Weaviate’s default path. Costs include maintaining centroid freshness under updates, tuning probe budgets per level, avoiding hotspots when popular branches of the hierarchy land on few nodes, and debugging recall that fails at the meta layer versus inside a leaf HNSW. Measure separately: router/meta hit rate, leaf ANN recall given a hit, and merge correctness. A meta-index that is stale or under-probed silently drops neighbors no leaf can recover.

A meta-index over shards is the hierarchical coordination and search layer above leaf ANN partitions – broader than a membership list, often richer than a single routing table. Next, read fan-out queries and top-k merge for the broadcast path used when no selective meta-index applies, recall decomposition for how meta misses combine with leaf misses, and the routing-index page for the compact ANN that often forms the top of that hierarchy.