What is index persistence?

Index persistence is the set of mechanisms that write a vector index‘s durable state to disk — and reload it after a restart — so approximate structures like HNSW survive crashes and shutdowns instead of existing only as an in-memory build that vanishes when the process dies.

Why can’t a vector index live only in RAM?

Building an HNSW graph is expensive: each insert decides layer membership, searches for neighbors, and rewires edges. An in-memory-only library can dump a snapshot file and call that “saved,” but production systems need every acknowledged write to outlive the process that computed it. Without persistence, a crash after hours of import loses the graph; without incremental durability, you must rebuild from raw vectors on every boot. Index persistence separates the living structure (fast RAM for queries) from the recovery story (append-only logs, compacted snapshots, checksummed blocks) so the next startup reconstitutes the same neighborhood links the last process held. Object and inverted data usually persist with their own LSM segments; the vector index often uses a parallel but distinct path because graph edges are not row-shaped segments.

The workhorse of that path is almost always a write-ahead or commit log.

How do commit logs make incremental graph changes durable?

A write-ahead log (WAL) records the result of expensive index mutations — new node placement, neighbor lists, deletes marked with tombstones — in an append-only file that is cheap to sync. The costly geometry work happens in memory; the log stores outcomes so a replay can rebuild the same graph without re-running neighbor search from scratch. Logs rotate when they reach a size threshold; completed rotations become candidates for compaction. If a crash tears the final record, recovery truncates back to the last valid entry rather than loading a corrupt mid-write. Until the log is safely on disk, a well-behaved database should not claim the write succeeded. That contract is what turns ANN from a bench tool into something you can restart overnight without re-embedding the world.

Full replay still gets slow as logs accumulate — which is why snapshots exist.

How do snapshots shorten recovery without losing durability?

A snapshot is a point-in-time encoding of the index state: the compacted neighbor graph (and related metadata) as it stood when the snapshot was sealed. On startup the engine loads the newest good snapshot, then replays only commit-log entries written afterward. Startup cost tracks the delta since the snapshot, not the full lifetime of the collection — often an order-of-magnitude win on multi-million-vector graphs. Compactors merge rotated logs, fold them into a new snapshot, and remove superseded logs once the new file is durably renamed into place. Atomic rename from a temporary path prevents a half-written snapshot from being treated as complete; checksummed blocks detect silent corruption on read. Disk briefly spikes while old snapshot, merging logs, and new snapshot coexist — operators should size headroom for that peak, not only the steady-state footprint.

Weaviate’s HNSW store is a concrete example of this log-plus-snapshot design.

How does Weaviate persist its vector indexes?

Each Weaviate shard keeps object and inverted data in LSM stores with their own WALs and segments, while the HNSW vector index keeps a separate commit log of graph mutations. Successful ingests durable enough for LSM WAL acknowledgment still depend on the HNSW commit path for the index side; both participate in crash safety. HNSW snapshots (automatic in recent versions) load first at startup, then only post-snapshot commit logs replay. A background commit-log compactor owns compaction and snapshot creation; after a successful snapshot, covered logs are deleted so disk stays proportional to index size rather than to every historical edge edit. Incomplete commit-log tails self-heal by truncation; a bad snapshot is not silently patched — restore from backup for that shard rather than serving a partial graph. Flat and HFresh indexes persist through their own segment or posting layouts, but the same principle holds: durable units on disk, reconstructible state after restart, clear failure modes when a unit is corrupt.

Knowing what persistence guarantees — and what it does not — keeps operations realistic.

What should operators expect from durable indexes?

Expect acknowledged writes to survive process death, startups that load snapshot plus delta rather than rebuilding from embeddings, and brief disk/CPU cost during compaction. Do not expect persistence alone to replace backups: a checksum failure or lost snapshot directory still needs an external restore. Plan lazy versus eager shard loading around restart latency; large HNSW shards without recent snapshots (or with huge uncompacted deltas) dominate boot time. Watch commit-log rotation size: too small means chatty files and weaker compaction batches; too large means longer replay windows and more memory during merge. Treat tombstone cleanup as part of freshness — deletes hide immediately in memory but disk state catches up as cleanup and compaction run. Persistence is infrastructure for the vector index, not a substitute for choosing the right index type or compression for your RAM budget.

Index persistence turns an in-RAM ANN graph into a restartable database component through commit logs, snapshots, and careful atomic writes. Next, read “What is a binary index format?” and “What is format versioning?” for how those on-disk bytes are laid out and evolved, then “What is a write-ahead log (WAL)?” and “What is an atomic snapshot?” for deeper dives into the two pillars this page introduced.