What causes a latency spike in search?

A latency spike in search is a sudden jump in p95/p99 (or average) query time for ANN or hybrid retrieval – often HNSW – caused by deeper graph exploration, hostile filters, heavier query shapes, resource contention, or a plan that stopped using the fast path, without necessarily changing the public API the client calls.

Why do HNSW queries sometimes jump from tens of milliseconds to hundreds?

Unfiltered HNSW cost tracks how many distance evaluations the walk performs. That count grows with ef (or dynamic ef derived from limit), with a poorly connected or tombstone-heavy graph that forces longer paths, and with cold caches that miss vector pages. A deploy that raises exploration for recall, a client that starts requesting huge limits, or a reload that restores a high fixed ef can all look like "search got slow overnight." Hybrid queries pay both BM25 and vector arms plus fusion. Generative or rerank stages after retrieval add model time that shows up in end-to-end latency even when the HNSW walk itself is fine. Spikes are therefore a mixture of algorithmic depth and everything wrapped around the index.

Filtered search is the other classic spike factory.

How do selective or low-correlation filters inflate latency?

Pre-filtered HNSW builds an allow-list then walks the graph while only admitting matching ids to the result set. When few objects match – or matching objects sit far from the query in embedding space – the walk rejects most neighbors and may explore a huge fraction of the graph before filling k. That pathological case approaches exhaustive cost on a billion-scale corpus even though the allow-list is tiny; flat search over the matching ids would have been cheaper. Loose filters behave almost like unfiltered search with a membership check. Mid-size allow-lists without a sensible flat-search cutoff, or sweeping-style exploration under negative correlation, are where p99 blows up. Traffic mix shifts – more bargain-price or rare-tenant queries – produce fleet-wide spikes without any code change.

Infrastructure and concurrency sit beside algorithm choice.

What non-algorithm issues create search latency spikes?

CPU saturation from concurrent heavy imports, cleanup, or compaction steals cycles from query threads. Memory pressure and vector-cache thrashing turn distance kernels into memory stalls. Synchronous HNSW updates on the write path can contend with reads; large async indexing queues mainly hurt freshness, but cleanup and segment merges still compete for IO. Deep reference resolution and wide nested fetches after the ANN shortlist add graph-database-style cost on top of vector search. Distance kernels that fall back from vectorized paths to scalar loops (build or deployment mishap) raise every query uniformly. Multi-shard scatter-gather waits on the slowest shard – one hot or overloaded shard defines the spike.

Weaviate exposes knobs that target each of these classes.

How do you diagnose and mitigate spikes on a Weaviate deployment?

Separate vector-walk time from filter build, hybrid fusion, rerank, and generative stages – query profiling helps. Plot latency by filter match-rate band and by correlation; if restrictive anti-correlated queries dominate the tail, enable or confirm ACORN, tune flatSearchCutoff, and avoid application-side oversampling that forces huge ef. Cap dynamic ef with dynamicEfMax when limit inflation is the culprit; set an explicit ef for stable SLOs. Check CPU, memory, cache hit rate, and indexing/cleanup load during the spike window. Prefer filters inside Weaviate so the engine can choose flat versus graph plans. After embedding or catalog growth, re-benchmark – the same ef visits more work as the graph densifies.

Prevention means budgeting depth and watching the query mix.

How should you keep latency spikes from becoming the default?

Define p95/p99 SLOs per query class (unfiltered, filtered, hybrid). Alert when the share of low-match-rate filters rises or when median ef/nodes-visited climbs. Load-test mixed read/write traffic, not only static snapshot QPS. Document operating points so on-call does not "fix" latency by cutting ef so hard that a recall regression appears next. Capacity-plan shards so one tenant’s graph cannot pin the flotilla.

A latency spike in search is extra work per query – deeper HNSW, hostile filters, or starved hardware – showing up in the tail. Next, read "What causes an out-of-memory error during index build?" when memory, not milliseconds, is the failure mode, or "What is a filtered-query collapse?" when restrictive predicates empty results instead of only slowing them down.