What is on-disk record layout?

On-disk record layout is the precise arrangement of fields, lengths, offsets, and padding in a persisted HNSW file — which bytes mean the header, where vector blocks begin, how neighbor lists are encoded — so a reader can reload or mmap the index without guessing.

Why does a durable HNSW file need an explicit record layout at all?

In RAM, pointers and language structs define where data lives. On disk those pointers are meaningless; only offsets and sizes survive. A record layout fixes the grammar of the file: magic bytes and version, endianness, section table, then the payload sections themselves. Without that contract, two builds cannot share snapshots, mmap cannot treat file offsets as array indices, and a checksum cannot cover well-defined regions. Layout is therefore part of the index’s public persistence API, not an internal accident of whatever fwrite order the writer used on Tuesday. The appendix-style binary format work on this site exists because HNSW is useless as a long-lived index if the bytes on disk are ambiguous.

Most practical layouts separate a small self-describing header from large contiguous bulk sections that mirror how search wants to read memory.

What sections does a typical HNSW on-disk layout contain?

A header records format version, distance metric, dimensionality, N, M, entry point, maximum layer, and file offsets (or lengths) of each section. A vector section stores coordinates in a fixed stride — often the same contiguous arena shape used at query time — with element type and alignment documented. An edges section stores neighbor IDs densely, either as per-node fixed-capacity slots or as offset/length pairs into a shared ID blob. A levels or metadata section holds each node’s maximum layer and flags. Optional sections cover tombstones, deleted-slot freelists, checksums, and external-ID maps. Keeping bulk vectors and edges contiguous makes serialized size predictable and lets mmap expose search-ready arrays when alignment matches. Production systems that snapshot HNSW state, including paths used in databases like Weaviate, depend on this kind of sectional discipline so reopen is a map-and-validate step rather than a full graph rebuild.

Field-level detail — widths, padding, and order — decides whether the file is both compact and safely readable across machines.

Which field-level choices make or break an on-disk HNSW layout?

Neighbor-ID width (32- vs 64-bit), vector element type, and endianness must be stated in the header and held constant for that version. Padding should be explicit so alignment for SIMD loads is preserved under mmap, not left as “whatever the compiler did on the writer host.” Length prefixes need fixed-width integers so a corrupt length cannot send the reader off the end without checks. Avoid embedding absolute virtual addresses. Prefer relative offsets from section bases. Document whether edges are bidirectional duplicates or stored once. Reserve bytes for future fields so minor versions can extend without breaking old readers that skip unknown trailing header space. These choices couple directly to serialized index size and to whether a cold open can search in place.

Layout also has to explain evolution: how a newer writer remains readable, and how a failed write is detected.

How should versioning and integrity interact with the record layout?

A monotonic format version in the header lets readers reject unsupported files cleanly. Breaking changes (ID width, metric encoding, section order) require a new version and a migration tool, not silent reinterpretation. Checksums over each section catch truncation and bitrot before search returns nonsense neighbors. Atomic snapshot practice — write to a temp file, fsync, then rename — keeps readers from mapping a half-written layout. Under concurrent upserts, the on-disk layout usually represents a point-in-time freeze, while the live graph continues in RAM until the next checkpoint. Designing the record layout with those operational rules in mind is what makes persistence trustworthy rather than a debug dump.

On-disk record layout is the byte-level blueprint of a persisted HNSW index — header, vectors, edges, metadata — that makes reload, mmap, and format evolution possible. From here, serialized index size quantifies the footprint, mmap explains serving from the file, format versioning and checksum entries deepen integrity, and the appendix on specifying an HNSW binary format walks through a complete persistence contract.