What is aggressive pre-filtering?
Aggressive pre-filtering is the strategy of building the full set of predicate-matching objects first, then running similarity search only inside that subset – often by brute-forcing the allow-list or by refusing to use non-matching HNSW nodes even as bridges – so every ranked neighbor is guaranteed legal, at the cost of connectivity and latency when the subset is large or scattered.
How does aggressive pre-filtering differ from naive post-filtering?
Naive post-filtering runs unfiltered ANN, then drops illegal hits. Aggressive pre-filtering inverts the order: structured indexes (or a scan of properties) produce the eligible id set before any distance ranking that matters for the final top-k. In its simplest "aggressive" form, the system then compares the query vector to every allow-listed embedding and sorts – true flat search over the filtered subset. A related hard variant keeps the HNSW graph but treats non-matching nodes as removed for that query: they cannot enter the result set and cannot be stepped through. Soft allow-list traversal is different: edges may still cross non-matching nodes so the walk stays connected, while only matching ids are admitted as results. "Aggressive" names the extreme that prioritizes predicate purity and subset-only scoring over preserving the full graph’s highways.
That extreme solves empty shortlists – and creates new failure modes when selectivity rises.
When does searching only the allow-list help, and when does it hurt?
When the filter is highly selective – tens or hundreds of ids out of millions – flat search over the allow-list is often fastest and exact among legal objects: HNSW would otherwise spend most of its budget evaluating neighbors it cannot return. As the allow-list grows into tens or hundreds of thousands, linear scan cost climbs with subset size; a graph walk constrained by an allow-list (but still using bridges) usually wins. If you aggressively delete non-matching nodes from the traversal itself, sparse matching sets under low vector/predicate correlation can fragment the usable graph: the entry point may sit in a region where cheap bridges were the only path to distant matching clusters, and recall collapses even though the allow-list is non-empty. Positive correlation – matching objects clustered near the query – makes subset-only search kinder; negative correlation ("diamond rings" with a harsh low-price cap) is where aggressiveness without a flat cutoff or smarter filter strategy bleeds latency or quality.
Production systems therefore treat aggressiveness as a plan choice keyed on selectivity, not a single forever policy.
What does a selectivity-aware pre-filter plan look like?
Estimate or measure how many objects pass the predicate. Below a cutoff, skip the graph and brute-force the allow-list. Above it, run HNSW with the allow-list as a result constraint while still following edges through non-matching nodes when needed for connectivity. Filter-aware strategies (for example multi-hop exploration and extra matching entry points) further reduce wasted distance work under low correlation without requiring you to physically excise non-matching nodes from the walk. Oversampling belongs mainly to post-filter pipelines; pre-filter plans instead tune ef, cutoff thresholds, and filter strategy. The goal is predictable filled-k among legal objects without paying full-corpus HNSW cost when the subset is tiny, and without paying full-subset scan cost when the subset is huge.
Weaviate’s filtered path is pre-filter oriented with that kind of cutoff, not application-side discard.
How does Weaviate implement pre-filtering without the brittle extreme?
Inverted indexes build an allow-list of eligible document ids before vector search. HNSW receives that list and only adds allow-listed ids to the result set, while still moving along edges – including through nodes that fail the filter – so graph integrity for traversal stays intact. When the allow-list is small enough that constrained HNSW would become near-exhaustive, flatSearchCutoff switches to flat search over the matching subset – the beneficial half of aggressive pre-filtering. Filter strategies such as sweeping versus ACORN change how the walk handles non-matching neighbors and low-correlation queries; ACORN can multi-hop and seed matching entry points so the search reaches the filtered zone faster without rebuilding the index for every predicate. Operators tune cutoff and strategy per collection against real selectivity mixes. Property filters attach to nearText, nearVector, nearObject, and related queries so business rules are allow-lists, not post-hoc hope.
Mis-tuned aggressiveness shows up as either slow scans or mysterious recall gaps.
How should you diagnose aggressive pre-filtering gone wrong?
If restrictive filters are slow, check whether the plan stayed on HNSW when a flat scan of a tiny allow-list would win – raise confidence in cutoff settings. If moderate filters are slow, the opposite may be true: flat search over a huge allow-list, or a walk that effectively behaves as if non-matching bridges were cut. Compare filtered recall to exact neighbors restricted to the allow-list; sudden drops under sparse, anti-correlated predicates suggest connectivity loss from over-aggressive exclusion during traversal. Track allow-list size alongside p95 latency and filled-k. After schema or filter-strategy changes, re-sweep a selectivity ladder before declaring victory. Prefer measuring on production-like predicates – region, stock, price bands – not only random bit masks.
Aggressive pre-filtering is filter-first, subset-only ranking taken to the extreme – exact among legal ids when the subset is small, risky when it severs HNSW bridges or scans a huge allow-list. Next, read "What is filter selectivity?" and "What is vector/predicate correlation?" for the two knobs that choose between flat, constrained HNSW, and softer filter-aware walks, or "What is filter-aware graph traversal?" for the connectivity-preserving middle path.