What is a cherry-picked benchmark result?

A cherry-picked benchmark result is a published number — QPS, latency, or recall — taken from the single most flattering operating point, seed, or setup, while weaker points on the same trade-off curve or less friendly conditions are left unreported.

Why is cherry-picking especially tempting with HNSW?

Approximate search has many dials: ef, build quality, compression, concurrency, cache warmth, and random seeds. Somewhere in that grid there is almost always a heroic cell — sky-high QPS at mediocre recall, or near-perfect recall at a latency nobody would accept in production. Quoting only that cell lets a slide claim “100k QPS” or “99.9% recall” without saying at what cost. Readers who never see the Pareto frontier cannot tell whether the headline is the feasible region or a corner case. Cherry-picking is not always fraud; sometimes it is incomplete reporting. The effect on decisions is the same: capacity plans and algorithm bake-offs get anchored on a point that does not survive contact with the SLA.

Several recurring patterns show up again and again.

What are common forms of cherry-picked ANN results?

Single operating point without a curve: one (ef, recall, QPS) triple with no neighbors. Best-of-N seeds or insertion orders: rebuild until recall looks good, publish the winner. Warm-cache-only latency after discarding cold starts. Comparing your method’s best point to a rival’s default or untuned point. Mismatched k, metrics, or ground truth between systems. Counting ID-only in-process search against end-to-end database latency. Reporting mean latency while p99 is terrible. Extrapolating singleton latency to multi-core QPS without measuring concurrency. Each pattern produces a number that is “true” in a narrow sense and misleading as a general claim.

Honest suites make cherry-picking harder by design.

How do full curves and manifests block cherry-picking?

Requiring a recall-versus-latency or recall-versus-QPS sweep shows whether the advertised point sits on the frontier or is an interior oddity. Declaring a named operating point for regressions stops teams from quietly sliding ef until the chart looks green. Reproducibility manifests expose seed, insertion order, hardware, and cache policy so “lucky” runs can be challenged. Confidence intervals reveal when a 0.5% recall edge is noise. Pre-registering the comparison rule — e.g. QPS at ≥95% recall@10, p99 ≤ X — removes post-hoc point selection.

Weaviate’s public ANN material leans toward that fuller picture.

How does Weaviate present results in a way that resists cherry-picking?

Official tables and plots show families of HNSW parameter combinations with recall, QPS, and latency together, not one orphaned headline. “Recommended configuration” rows are labeled as balanced overviews, while the surrounding charts still expose the trade-off. Documentation stresses that higher recall generally costs throughput, that multi-threaded QPS must be measured (not extrapolated from single-thread latency), and that end-to-end timing includes object fetch — closing common apples-to-oranges gaps versus library microbenches. When you cite Weaviate numbers, carry the operating point and dataset with them; stripping those details recreates cherry-picking even if the source did not.

Readers and reviewers need a short sniff test.

How can you spot or avoid cherry-picked claims?

Ask: Where is the rest of the curve? What recall floor was fixed? Was the cache warm? How many seeds? Is k and the metric identical across methods? Does the reproducibility manifest match production? If the answer is missing, treat the number as marketing until a sweep appears. For your own reports, publish the frontier, mark the operating point, and include at least one unfavorable condition (cold start, filtered query, or higher k) so stakeholders see the sensitivity.

A cherry-picked benchmark result is a true measurement wrapped in selective silence — useful for slides, dangerous for systems. Next, read “What is a recall-versus-latency trade-off curve?” for the plot that makes cherry-picking obvious, then “What is thread-count scaling?” so concurrency claims are measured across cores instead of inferred from one flattering run.