What is duplicate detection?
Duplicate detection finds exact or near-identical items in a corpus – copies, paraphrases, resized images, or republished passages – often by embedding each object and flagging neighbors in an HNSW index whose distance falls inside a calibrated similarity threshold.
Why do teams use vectors for duplicates instead of only hashes?
Cryptographic hashes catch bit-identical files and nothing else. Perceptual or locality-sensitive hashes catch some visual or token-level near-copies, but struggle with paraphrase, translation, reordering, or mild edits that still mean the same thing to a user. Embedding-based detection maps each document or image into a space where semantic or visual near-duplicates land close together even when bytes differ. You then ask: for each new or existing item, are there neighbors closer than threshold t? That question is exact pairwise comparison for tiny sets and approximate nearest-neighbor search for large ones. HNSW answers it in milliseconds so ingest-time gates, catalog cleanup jobs, and RAG chunk hygiene can run continuously without a quadratic scan.
The workflow is simple in outline and sharp in the details that decide false positives.
How does HNSW-based duplicate detection typically run?
Embed every object with a model suited to the modality – text encoders for articles and tickets, vision encoders for photos, or multimodal models when formats mix. Normalize if your metric expects unit vectors. Index embeddings in HNSW under the same distance used for scoring (cosine or squared L2 are common). At insert or batch time, query nearObject / nearVector for the candidate, exclude self, and inspect the top neighbors’ distances. If the best neighbor is inside the duplicate threshold, merge, skip, or queue for human review; otherwise accept the insert. Online systems probe before write so the graph never accumulates twins; offline jobs walk the collection in batches and build connected components of near-duplicate clusters. Exact byte duplicates can still be short-circuited with hashes before the vector path to save cost.
Weaviate is a natural place to host that similarity index.
How would you implement duplicate detection with Weaviate?
Store each item with its embedding (vectorizer on ingest or bring-your-own vectors). Before creating a new object, query with nearVector or nearText/nearImage as appropriate, request distance or certainty metadata, and apply your threshold in application logic – Weaviate returns neighbors; your policy decides duplicate versus related-but-distinct. Use nearObject when re-checking an existing UUID after an update that re-embeds content. Scope searches with filters (tenant, collection, language) so cross-tenant nearness never merges the wrong worlds. Tune query ef high enough that true near-duplicates are not missed by ANN approximation – missed twins are silent data-quality bugs. After embedding-model changes, re-embed and re-scan; old and new spaces are not comparable. Optional second-stage checks (string diff, image pixel compare, or a small adjudicator model) resolve the ambiguous band just below a hard auto-merge cutoff.
Threshold choice and failure modes decide whether dedup helps or hurts.
What goes wrong when duplicate detection is mis-tuned?
Too-loose thresholds collapse distinct but thematically similar items – product variants, related support tickets, or articles that share a template. Too-tight thresholds leave paraphrases and lightly cropped images in place. ANN recall that is fine for search UI can be too low for dedup if a true twin never enters the shortlist; raise ef or verify with exact search on a sample. Updates that rewrite text and re-embed can drift an object onto a different twin – re-probe after updates, not only on create. Mixing metrics or unnormalized vectors with a cosine-trained model invents false geometry. Measure precision and recall of duplicate pairs on labeled sets, track merge-error complaints, and keep an audit log of automated merges so bad clusters can be undone.
Duplicate detection is ANN used as a similarity gate – HNSW finds the suspicious neighbors; thresholds and policy decide what counts as the same thing. Next, read "What is cross-modal retrieval?" for search across text and images, or revisit image-similarity and face-instance pages when your duplicates are visual rather than textual.