What is an atomic snapshot?
An atomic snapshot is a point-in-time on-disk image of an index that becomes visible to readers only as a whole — typically by writing a complete temporary file, syncing it, and renaming it into place — so a crash can never leave a half-written snapshot that the next startup mistakes for a valid graph.
Why must snapshots be published atomically?
A snapshot of an HNSW index is a large binary artifact: metadata, checksummed blocks, packed neighbor lists. Writing it in place under the final filename means that at every moment until the last byte is flushed, the path that recovery will open contains incomplete data. A power loss mid-write would leave a truncated file that might fail checksums — or, worse in weaker layouts, pass superficial checks while missing nodes. Atomic publish separates “under construction” from “current truth”: readers and recovery only ever see the previous good snapshot or the new complete one, never a torn intermediate. That property is what lets compactors replace commit logs with confidence after a snapshot lands.
The usual implementation is older than vector search, but it remains the right tool.
How does write-temp-then-rename achieve atomicity?
The writer creates a file under a temporary name in the same directory (so the rename stays on one filesystem), streams the full snapshot — headers, blocks, checksums — then fsyncs the file (and often the directory) so bytes and metadata hit stable storage. Only then does it rename the temp path to the canonical snapshot name. On POSIX filesystems a rename that replaces an existing file is atomic with respect to readers: they see either the old inode or the new one. Orphaned temp files left by a crash are ignored or deleted on the next startup because they never became the published name. Compacted commit logs follow the same pattern when they are sealed as standalone artifacts. Without the sync-before-rename step, atomic rename alone can still publish a file whose data blocks were still in the page cache — durability requires both.
Atomicity also governs when older files may disappear.
When is it safe to delete the previous snapshot and covered logs?
Only after the new snapshot is durably published. Until that moment the previous snapshot plus any commit-log delta remain the recovery set. Deleting early creates a window where a crash leaves neither a complete new image nor the logs needed to rebuild it. Once the new file is renamed into place and synced, the engine may remove the superseded snapshot and the commit logs it fully covers, keeping on-disk size proportional to the live index rather than to every historical edge edit. During assembly, disk usage spikes: old snapshot, logs being merged, and the growing temp file coexist — operators should size capacity for that peak, not only the steady state after cleanup.
Weaviate’s HNSW compactor follows exactly this publish-then-retire discipline.
How does Weaviate use atomic snapshots for HNSW?
Weaviate writes HNSW snapshots and compacted commit logs to a temporary path and atomically renames them into place so an interrupted write cannot be mistaken for a complete file. Orphaned temporaries are cleaned up on the next startup. A new snapshot replaces the commit logs it covers only after that durable publish; the leftover logs are solely the delta since the snapshot. Snapshots themselves are checksummed block images — if validation fails after publish, Weaviate will not load a partial graph or “repair” the file by truncation; restore from backup instead. That pairing matters: atomic rename prevents half-written finals; checksums detect corruption after a good rename; together they make automatic snapshot-centric storage safe enough that covered logs can be deleted. LSM segment flushes use related ideas (immutable segments appear as whole units), but HNSW’s snapshot is the clearest “one file, all or nothing” case in the vector path.
Atomic publish is necessary but not sufficient for a trustworthy recovery story.
What else must sit beside an atomic snapshot?
Format versioning so readers know the layout; endianness so multi-byte fields decode correctly; checksum validation so bit rot surfaces; and a write-ahead or commit log so mutations since the last snapshot are not lost. Atomicity without checksums can still publish a complete-looking file that was corrupted in transit to disk. Checksums without atomicity can leave a final path that never finished writing. Backups should capture published snapshots, not in-progress temp names. When testing disaster recovery, crash the process during snapshot creation and confirm the node restarts from the previous snapshot plus logs — proof that atomicity held.
An atomic snapshot is how a multi-gigabyte HNSW image becomes the new source of truth in one indivisible step. Next, read “What is a write-ahead log (WAL)?” for the append-only stream snapshots eventually fold in, then “What is an incremental checkpoint?” for related ideas about advancing durable state without rewriting everything every time.