How is HNSW used for image and multimodal search?

HNSW is used for image and multimodal search the same way it’s used for text: a model converts an image, a face, or even a piece of text into a vector positioned so that visually or semantically similar inputs land near each other, and finding similar images then becomes an ordinary nearest-neighbor search over those vectors rather than anything specific to pixels or image formats.

How does an image become searchable the same way a vector of numbers is?

An image embedding model is trained to map an image to a fixed-length vector such that images a human would judge as visually or conceptually similar end up positioned close together in that vector space, while dissimilar images end up far apart. Once this mapping exists, “find images similar to this one” becomes exactly the nearest-neighbor problem covered throughout this site — embed the query image, search an index of already-embedded catalog images for the closest vectors, and return whichever images those vectors correspond to. Nothing about the graph traversal, the layered structure, or the parameter tuning changes because the underlying data happens to be images rather than text; the only difference is which model produced the vectors in the first place.

What extra care does face and instance-level retrieval require?

Face retrieval and instance-level retrieval — finding not just similar-looking images but images of the same specific person or the same specific object — generally demand a stricter notion of similarity than looser, more general visual search, since two different people’s faces can otherwise sit uncomfortably close together in an embedding space trained on more general visual similarity. This tends to call for embedding models trained specifically to separate distinct identities or instances more aggressively than a general-purpose visual similarity model would, coupled with a higher recall target and often a lower tolerance for false positives, since a face-matching system’s practical usefulness depends heavily on how rarely it confuses two different people rather than merely how well it clusters visually similar images in general.

How does HNSW get used for duplicate and near-duplicate detection?

Detecting duplicate or near-duplicate images — the same photograph re-encoded, cropped, or lightly edited — is a direct application of the same nearest-neighbor search, since near-duplicates produce embedding vectors that land extremely close together, typically much closer than genuinely distinct images do. Running every new image against an existing index and checking whether anything comes back within an unusually tight distance threshold is a practical, efficient way to catch duplicates at a scale where comparing every new image against every existing one directly would be far too slow. This same pattern extends naturally to cleaning large datasets used for training other models, where removing near-duplicate examples before training can meaningfully improve the quality of what gets learned, and to spotting mislabeled examples by finding cases where an item’s nearest neighbors in embedding space belong to a different label than the item itself does.

What does it mean for text and images to share one joint embedding space?

Some models are trained specifically to place both images and their text descriptions into the same shared vector space, so that an image and a caption describing it well end up positioned close together even though one input was a picture and the other was a sentence. This joint space enables searches that cross between modalities entirely — finding images using a text description as the query, or finding a relevant caption using an image as the query — without needing separate, incompatible indexes for text and images. From HNSW’s perspective, none of this cross-modal capability changes anything about how the graph itself works; it only changes what the embedding model upstream is doing, since the graph simply searches whatever vectors it’s given regardless of which modality, or combination of modalities, originally produced them. This does introduce one genuine wrinkle worth being aware of: a query and the dataset it’s searching don’t always come from perfectly matched distributions in a joint space — a text query and an image being searched were produced by fundamentally different kinds of input even if they share a vector space, which is exactly the kind of out-of-distribution query behavior covered elsewhere on this site.

Image and multimodal search show that HNSW’s usefulness has nothing to do with the specific content being searched, only with whether that content can be embedded as a vector — the next page extends this same idea to text specifically, covering semantic search and retrieval-augmented generation in depth. For the distribution-mismatch concern raised above, this site’s page on in-distribution and out-of-distribution queries covers that concept in its own right.