What is format versioning?
Format versioning is the practice of embedding an explicit version identifier in an on-disk binary layout — and defining how readers and writers behave when that identifier is older, current, or newer — so index files can evolve (new fields, block layouts, compression metadata) without silently misinterpreting bytes from another release.
Why do binary indexes need versions at all?
A binary index format is a contract: byte 0 might mean “format version,” the next words might be checksums and sizes, then node records follow a fixed packing. The first time you ship that file, one layout is enough. The second time you need packed connections, fixed-size checksummed blocks, or sparse node encoding, the old byte offsets stop meaning what new code assumes. Without a version field, a reader either crashes or — worse — decodes neighbor IDs as floats and serves nonsense recall. Versioning turns “this blob of bytes” into “schema N of this index type,” which is the minimum requirement for upgrades, rolling clusters, and restoring backups taken months earlier.
Good versioning is a policy, not just an integer in a header.
What rules make format versions safe?
Writers always stamp the version they emit. Readers check the version before trusting the body: accept known versions, migrate when a supported upgrade path exists, and hard-fail on unknown or corrupt versions instead of guessing. Backward compatibility means a new binary can still open files written by older writers (possibly after an automatic rewrite). Forward compatibility is rarer: an old binary usually must refuse files from a newer major format rather than skip unfamiliar fields and hope. Breaking changes bump the version and ship a one-way migration; additive changes may keep the same major version only if old readers ignore trailing data by design — which most tight ANN formats do not. Document which versions coexist during a rolling upgrade so two nodes briefly speaking different formats do not exchange incompatible shard files.
Vector indexes feel these rules acutely because persistence has several cooperating formats.
How does format versioning show up in HNSW persistence?
Commit logs and snapshots often version independently. A snapshot might move from a legacy metadata header to a layout with version, checksum, metadata size, then fixed-size body blocks — each step a new snapshot version that old readers must either understand or reject. Commit-log record types may gain new opcodes for compression settings or multi-vector nodes while keeping older opcodes readable. Compaction is where versions often meet: a compactor loads an older snapshot, replays logs, and writes the newest snapshot version, retiring obsolete files only after the new artifact is durable. That rewrite is the migration. Skipping it — copying a data directory from a much older engine into a much newer one without the supported upgrade path — is how clusters fail to start or silently drop graph connectivity.
Weaviate’s release history is a practical tour of those migrations.
How does Weaviate handle evolving on-disk formats?
Weaviate’s custom HNSW store has grown commit logs, then optional snapshots, then automatic snapshot-centric storage with checksummed block formats — each stage changing what lives on disk and how startup loads it. Software version upgrades (for example introducing multi-shard layouts or Raft metadata) sometimes require automatic on-disk migrations at first boot; others recommend reimport when older HNSW files would be weaker than a fresh build. Snapshot behavior itself shifted across minors: configurable periodic snapshots gave way to always-on compaction-owned snapshots, with old environment variables ignored but still recognized so configs do not hard-crash. Operators are steered to upgrade one minor at a time with backups first — not only for API compatibility, but because format readers and migrators are tested along that path. Inactive tenants may keep older on-disk shapes until activation triggers compaction into the current format.
Version discipline only helps if you treat files as versioned artifacts in ops too.
What should operators do when formats change?
Backup before every Weaviate minor upgrade. Prefer the documented stepwise upgrade over jumping many versions so intermediate migrators run. Do not hand-edit binary index files or mix shard directories from mismatched majors. After upgrade, allow compactors to rewrite snapshots before judging disk savings or startup time. If a snapshot fails checksum or version checks, restore from backup rather than forcing a partial load. When building custom tooling that reads commit logs or snapshots, gate every parser on the format version byte and fail closed on unknowns — the same rule the database itself should follow.
Format versioning is how binary index layouts survive product evolution without corrupting graphs. Next, read “What is endianness?” and “What is checksum validation?” for the other header contracts that travel with version bytes, then “What is an atomic snapshot?” for how new versioned files are published safely during compaction.