What is database-aware index selection?

Database-aware index selection is the practice – by a query planner, database defaults, or deliberate API choice – of picking which physical access path to use for a request: inverted/BM25, HNSW (or another ANN graph), flat brute-force over vectors, hybrid fusion of lexical and dense arms, or a filtered variant with allow-lists and cutoffs – based on query shape, selectivity, and collection size rather than always running the same index.

Why does one forever index choice fail in a real vector database?

HNSW is excellent for semantic nearest neighbors at scale, but it is the wrong first tool for "find SKU X exactly," for a ten-object tenant that fits in a flat scan, or for a filter that leaves only dozens of eligible ids where constrained graph search would thrash. BM25 is precise on literals and weak on paraphrase. Hybrid costs both arms. Flat search is exact among a small set and linear in that set’s size. A database that stores inverted indexes beside vector graphs can – and should – route each query to the path whose asymptotic and practical costs match the request. Selection is what turns a pile of indexes into a query engine instead of a single ANN demo.

The decision inputs are observable properties of the query and the data.

What signals should drive which index path runs?

Query intent: natural-language or cross-modal similarity favors vector or hybrid; controlled identifiers and citations favor keyword; mixed user search often starts at hybrid. Filter selectivity: tiny allow-lists favor flat distance over matching ids; large soft filters favor allow-list HNSW with an appropriate filter strategy; unfiltered semantic search stays on the graph. Correlation between predicate and embedding neighborhood changes whether sweeping or ACORN-style traversal is cheaper for the same match rate. Collection or tenant size: small corpora or small tenants can stay on flat indexes; past a threshold, HNSW (or a dynamic switch from flat to HNSW) wins on latency. Modality: image- or audio-only queries need vector paths. Second-stage tools (boost, rerank) only reorder candidates the first path retrieved – they do not replace index selection when the first shortlist is wrong.

Some choices are automatic inside the engine; others remain API-level product decisions.

Where does selection happen automatically versus in application code?

Automatic examples include flat-search cutoffs on filtered vector queries when the allow-list is small enough that HNSW would become a poor deal, dynamic vector indexes that start flat and convert to HNSW after an object-count threshold, and filter strategies that adapt exploration under sparse matches. Application- or agent-level selection chooses among bm25, nearText / nearVector, hybrid, filters, alpha, and fusion type from product rules or a query agent. Both layers are "database-aware" when they use statistics the database can see – cardinality, index presence, shard layout – not when they hard-code HNSW for every string. Skipping inverted indexes on properties you never filter or search is the build-time half of the same idea: do not pay for paths you will never select.

Weaviate co-locates the indexes that make these plans possible.

How does Weaviate support database-aware selection in practice?

Each shard can hold inverted indexes for BM25 and filters next to an HNSW, flat, dynamic, or other vector index. Filtered vector search builds an allow-list then runs HNSW or falls back via flatSearchCutoff; ACORN versus sweeping picks how the graph explores under predicates. Hybrid runs BM25 and vector arms then fuses with ranked or relative-score methods and an alpha weight. Dynamic indexes let small tenants stay flat until growth justifies HNSW. Operators still choose the search API per request class – keyword for legal citations, hybrid for storefront typeahead, nearObject for recommendations – while letting cutoffs and strategies handle mechanical branches. Configure only the indexes your selected paths need so memory tracks the plans you actually run.

Wrong selection shows up as latency cliffs or missing exact hits, not as abstract planner theory.

How should you validate that the right indexes are being chosen?

Label production query classes and expected paths; sample whether traffic hits BM25, hybrid, or pure vector as intended. Bucket filtered queries by allow-list size and confirm flat cutoff engages on the tiny end. After catalog growth, revisit dynamic thresholds and HNSW parameters. Use query profiling where available to see per-shard time in inverted versus vector work. A/B alpha and fusion only after the correct arms are selected – tuning fusion cannot fix a missing keyword path. Treat planner and API choices as part of capacity planning when predicate mixes shift overnight.

Database-aware index selection is routing each query to BM25, HNSW, flat, or hybrid for the right reasons – selectivity, intent, and scale. This closes the filtered and hybrid glossary; next, start Failure modes and debugging with "What causes a recall regression?" when the chosen path still returns the wrong neighbors, or revisit "What is filter selectivity?" when cutoff and strategy decisions keep misfiring.