What is intrinsic dimensionality?

Intrinsic dimensionality is a measure of how many genuine degrees of freedom a dataset actually varies along, as opposed to how many coordinates its vectors happen to be stored with. A collection of points can have an ambient dimension of a thousand — a thousand numbers per vector — while still having an intrinsic dimensionality of only a dozen or so, if those thousand-number vectors all happen to trace out variation along just a dozen meaningful directions, with the rest of the apparent freedom being redundant, correlated, or simply unused by the actual data. It’s the difference between how large a space a dataset is stored in and how large a space it actually occupies in any meaningful sense.

How can data stored with many coordinates end up using so few of them meaningfully?

Picture a dataset of points that all happen to lie exactly on a two-dimensional plane, but that plane is tilted and embedded inside a three-dimensional space, and every point is stored using all three of those coordinates. The ambient dimension of that data is three, since that’s how many numbers each point uses, but the intrinsic dimensionality is really two, because knowing a point’s position within that plane — two numbers’ worth of information — is enough to pin it down exactly, with the third coordinate always following predictably from a fixed relationship to the other two rather than adding any genuinely new information. Real high-dimensional datasets exhibit a much more elaborate version of this same idea: rather than lying on a flat plane, points often cluster near a curved, lower-dimensional surface winding through the larger space, but the same basic principle holds — far fewer independent quantities are needed to describe where a point sits than the raw coordinate count would suggest.

This happens in real embedding data for an intuitive reason: the underlying things being represented — words, images, product descriptions — don’t actually vary along every conceivable direction independently. A model trained to represent images, for instance, produces vectors that reflect real visual structure like color, shape, and texture, and those underlying visual properties are correlated with each other in the way real images actually look, rather than varying completely independently across every one of the model’s output dimensions. The model’s output ends up occupying only a fraction of the full geometric freedom that its ambient dimension would technically allow, because the real-world data it was trained on never used that much freedom in the first place.

Why does intrinsic dimensionality matter so much for how well HNSW actually performs?

Recall the curse of dimensionality‘s central practical concern: distance concentration and the general breakdown of spatial structure that makes nearest-neighbor search harder as dimensionality grows. Crucially, the severity of those effects tracks a dataset’s intrinsic dimensionality far more closely than its ambient dimension. A dataset with an ambient dimension of a thousand but an intrinsic dimensionality of twenty behaves, for search purposes, much more like a genuinely twenty-dimensional problem than a thousand-dimensional one, because the meaningful variation driving distance calculations is really happening along only those twenty effective directions. This is a large part of why HNSW, and graph-based approximate nearest-neighbor methods generally, perform far better in practice on real embedding datasets than a worst-case theoretical analysis based purely on ambient dimension would predict — real embedding data is rarely as adversarially high-dimensional in effective terms as its raw vector length suggests.

This also explains a pattern that shows up repeatedly in practical benchmarking: two datasets with identical ambient dimension can require noticeably different HNSW parameter settings to hit the same recall target, purely because they differ in intrinsic dimensionality. A dataset with lower intrinsic dimensionality tends to be more forgiving — its proximity graph can be built with fewer connections per node and searched with a narrower beam while still achieving strong recall, since the effective search problem is genuinely simpler than the raw coordinate count implies. A dataset with higher intrinsic dimensionality, closer to genuinely using all the freedom its ambient dimension provides, tends to need a more generously connected graph and a wider search to reach the same recall, since there’s less redundant structure for the search to exploit.

Can intrinsic dimensionality actually be measured, or is it purely a conceptual idea?

It can be estimated, though not computed exactly the way ambient dimension can simply be counted. A range of statistical techniques exist for estimating a dataset’s intrinsic dimensionality from the data itself, generally by examining how the number of nearby points grows as the radius around a given point increases — a pattern that reveals how much genuine geometric freedom the local neighborhood actually has, independent of how many raw coordinates the vectors are stored with. These global estimates are useful for characterizing a dataset as a whole, but the more locally-focused version of this idea, local intrinsic dimensionality, turns out to matter even more directly for understanding why some individual queries against an HNSW index are harder than others, since a dataset’s effective dimensionality can vary meaningfully from one region of the space to another rather than being a single fixed number everywhere.

Having established the distinction between ambient and intrinsic dimensionality and why the gap between them matters so much for real-world search performance, the local intrinsic dimensionality glossary page is the natural next stop, since it refines this same idea down to the level of individual neighborhoods rather than a dataset-wide average. From there, the manifold hypothesis page explains the broader theoretical framework — the idea that real high-dimensional data tends to lie near lower-dimensional structures in the first place — that intrinsic dimensionality is really trying to measure a concrete instance of.