How should an HNSW binary format be specified for stable persistence?

An HNSW binary format should be specified precisely enough, byte for byte, that a loader written independently from the original writer could still parse it correctly — a looser, implicit format that merely “works” for whichever code originally wrote and read it tends to break silently the moment anything about that code changes, which is exactly the failure mode a careful specification exists to prevent.

What should the file header contain, byte for byte?

As covered in this site’s page on persisting and recovering an index safely, a well-designed format opens with a short, fixed magic value whose only job is letting a loader immediately confirm it’s actually looking at the expected file type, followed by an explicit format version number recorded as its own fixed-width field. After that, the header should record every global property needed to interpret the rest of the file correctly: the number of vectors, the vector dimensionality, the distance metric used, the construction parameters M and mL, the current entry point’s identifier, and the topmost occupied layer. Recording these as fixed, predictably positioned fields — rather than inferring them from the rest of the file’s contents — is what lets a loader validate a file’s basic shape before attempting to parse the potentially much larger body that follows.

How should the graph’s variable-length neighbor lists actually be laid out on disk?

Because different nodes have different numbers of connections at different layers, a format needs an explicit way to know where one node’s data ends and the next begins, rather than relying on implicit assumptions about fixed sizes. A common approach records, immediately before each node’s neighbor list, an explicit count of how many neighbor identifiers follow, so a reader always knows exactly how many bytes to consume next without needing to guess or scan ahead. As discussed in this site’s page on memory layout, storing each node’s own vector data, layer memberships, and neighbor lists contiguously — matching the same locality principle that improves in-memory cache behavior — also tends to produce a format that’s naturally efficient to read sequentially from disk, since related data ends up physically close together in the file the same way it benefits from being close together in memory.

How should endianness and numeric precision be handled explicitly?

Endianness — the byte order used to store multi-byte numbers — needs to be fixed explicitly by the format rather than left to whatever the writing machine’s native byte order happens to be, since a file written on one architecture and read on another with different native endianness will otherwise be silently misinterpreted rather than cleanly rejected. The numeric precision used for stored vectors, whether standard floating-point, a compact half-precision format, or a quantized integer representation, needs recording explicitly in the header too, since a loader has no way to correctly interpret the raw bytes of a vector without first knowing how many bytes each individual number occupies and how those bytes should be decoded.

What makes a format change backward-compatible versus a hard break?

Adding new optional fields at the end of an existing structure, in a way older loaders can safely ignore, generally preserves backward compatibility — an older loader that doesn’t understand a new trailing field can simply stop reading where it expects the structure to end, without needing to know the new field exists at all. Changing the meaning, size, or position of an existing field, on the other hand, is a hard break that demands a new version number, since any loader written against the old version would otherwise misinterpret the new file’s contents without any obvious warning that something has gone wrong. A version number recorded prominently in the header, checked before anything else in the file gets parsed, is precisely what allows a loader to detect this kind of hard break immediately and refuse to proceed, rather than silently producing a corrupted, seemingly functional graph from a file it was never actually designed to read.

This is the final appendix in this site’s structured curriculum, and from here the glossary sections take over as the place to look up any individual term encountered along the way — starting with vector and distance math, then moving through HNSW‘s own parameters, systems and hardware concepts, and every other topic this site has touched on, each as its own focused, direct-answer page.