What is the difference between a singleton query and a batch query?
A singleton query is one nearest-neighbor search issued and timed alone — one query vector, one result list — while a batch query submits many query vectors together (or schedules them as one coordinated group) so setup, scheduling, and memory traffic are shared across the set.
What does each mode actually measure?
Singleton latency answers “how long does one user wait on an otherwise quiet (or lightly loaded) path?” You send query 1, wait for completion, then query 2. That isolates per-search work: HNSW walk, distance kernels, object fetch, and network for a single payload. Batch mode answers “how efficiently can the system process a list of queries?” The client (or server) packs Q1…Qn into one request or one internal work item; throughput is often counted as queries completed per second inside that batch, and latency may be reported as time-to-last or mean time-to-each. Mixing the two without labeling them produces fake comparisons: a batched library microbench can look vastly faster than an interactive API that only ever sees singletons.
They are also not the same as “many concurrent clients.”
How do singleton and batch differ from multi-threaded QPS tests?
Concurrent QPS drives many independent in-flight requests — each typically a singleton from the server’s point of view — to saturate cores and expose lock contention. Batching deliberately groups queries so the runtime can amortize: one contiguous buffer of query vectors, one allocation of scratch heaps for the group, SIMD loops that stay on related distance work longer, fewer RPC framing costs per vector. You can run concurrent batches too, which stacks both effects. ANN methodology that only quotes “QPS” must say whether each counted query was a lone RPC, one slot inside a batch of size B, or one of C concurrent singletons. Extrapolating singleton latency × core count to production QPS fails for the same reason Weaviate warns against it: real concurrency does not scale linearly, and batching changes the cost model further.
Hardware and product shape push you toward one or the other.
When is singleton the right benchmark, and when is batch?
Interactive search, autocomplete, and per-request RAG retrieval are singleton-shaped: the user issues one embedding and needs that answer’s latency percentiles. Offline evaluation, embedding-model sweeps, and bulk “search these 10,000 held-out queries” jobs are batch-shaped: wall-clock for the whole set and queries-per-second inside the batch matter more than any one tail. GPUs and wide SIMD units often favor batches because they fill parallel lanes; classic CPU HNSW still benefits from batching through better cache reuse and fewer syscalls, but a single walk remains one thread of graph work. If your SLA is “p99 under 20 ms for one user,” optimize and measure singletons under realistic concurrent load. If your SLA is “score this nightly query file before morning,” measure large batches on warm caches.
Weaviate’s own numbers and APIs reflect both worlds.
How does Weaviate relate singleton search to batched work?
Each vector search is fundamentally single-threaded graph traversal; multiple searches at once use multiple threads — the basis of multi-threaded QPS in public ANN tables, which are concurrent request measurements, not silent batching of query vectors into one call. Client and server batching appear most prominently on the write path (batch import objects/vectors), where amortizing overhead raises update throughput; read paths are usually one query per request unless you build an application-level batcher. Monitoring exposes both query latency histograms and batch operation durations, underscoring that “batch” in ops often means writes. When comparing a Weaviate deployment to an in-process ANN library, check whether the library’s headline QPS used batched queries in one process with no network — a singleton HTTP/gRPC search will not match that number even at equal recall.
State the mode explicitly in every result table.
How should you report singleton versus batch results?
Label batch size B (B=1 means singleton), concurrency C, and whether latency is per-query or end-of-batch. Keep recall, ef, dataset, and cache warmth fixed when swapping modes. For product capacity, run concurrent singletons that mirror real traffic; use batches as an additional column if your pipeline can group work. Do not advertise batch QPS as interactive latency. If batch helps throughput but hurts time-to-first-result, say so — those are different user pains.
Singleton versus batch is a load-shape choice: one-at-a-time user waits versus grouped processing efficiency — and conflating them is a classic HNSW benchmarking pitfall. Next, read “What is a confidence interval in benchmarking?” so you can tell whether a singleton/batch gap is statistically real, then “What is a reproducibility manifest?” to lock down batch size and concurrency for anyone repeating your run.