Durability and crash recovery

mushroomdb persists with a write-ahead log (WAL) plus periodic V9 snapshots. Every committed write is appended to the WAL and fsynced per the configured policy; a snapshot is a self-contained, memory-mappable image of the whole store with per-section CRCs.

How recovery works

On open, the store is reconstructed as snapshot (if present) + WAL tail:

• With a snapshot — the V9 image is memory-mapped (open-to-first-query in milliseconds even at 100k nodes; derived edges, HNSW vector indexes, provenance and the full-text index are all restored from the image), then any WAL frames written after the snapshot are replayed. This is the normal path and it is fast.
• WAL-only, from genesis — if the store has never been snapshotted, the entire history replays from the beginning. Node/edge/property writes replay cheaply, but rules re-derive their edges, and for VectorSimilar rules that means rebuilding the ANN (HNSW/IVF) index from scratch. On a large, vector-rule store this can take minutes. This is the worst case, and it only happens when no snapshot exists.

Keeping recovery fast

The tooling already bounds WAL-only-from-genesis exposure:

• ingest-git snapshots itself. A first ingest always writes one before it returns; a later ingest-git writes one when the store has no snapshot at all, or when the WAL has grown past 4 MiB since the last one. Measured on a 435-file repository, that is the difference between a 288 ms open and a 173 ms one — paid on every hook, every MCP start and every CLI call until something snapshots.
• mushroomdb mcp snapshots a store whose WAL has grown. When it opens a store whose WAL is past 4 MiB, and again when its session ends, it takes the same snapshot: the WAL is archived, not dropped, so history reads keep their reach. A store used only as an assistant’s memory is snapshotted by nothing else, and without this every open replayed everything it had ever written. The one at open runs before the server’s first answer. Skipped when another process holds the write lock, and by the server mushroomdb doctor starts.
• The server snapshots on graceful shutdown (SIGINT/SIGTERM), so a clean restart always recovers from an image.
• mushroomdb serve … --snapshot-every <secs> snapshots periodically, so even after a hard crash the replay is only the WAL tail since the last snapshot.
• mushroomdb snapshot <db> takes one on demand (e.g. before an upgrade).

Guidance: for a long-running vector-rule deployment, set --snapshot-every (or snapshot on a schedule). The cost of a periodic snapshot is far smaller than a from-genesis rebuild, and it caps how much WAL a crash can leave to replay.

For the operational side of all this — which directory to mount, how to back it up, and how serve --restore-from seeds a fresh volume on boot — see Running it as a service.

A snapshot does not cost you the past

Folding the WAL into a snapshot is what makes the next open fast, but the WAL is also what node_history, edge_history, was_linked and asof read. Dropping it would leave a store that cannot say how any of its nodes came to be.

So every snapshot mushroomdb takes on its own — the ingest’s, a --snapshot-every tick, the shutdown one — and mushroomdb snapshot <db> with no flags archive the WAL rather than dropping it. The frames are renamed to wal.<N>.archive, which the history reads still scan and the open path does not. You end up with a store that opens fast and remembers. An automatic snapshot keeps every archive: history is the thing the archives exist for, and nothing deletes it unless you ask; see Disk below for what that costs and how to bound it.

Three flags choose otherwise:

--truncate is the destructive one and nothing chooses it for you. It discards the tail, deletes any archives an earlier snapshot left, and moves the store’s floor to where that history ended, so the commits it discarded keep their numbers and are refused by name from then on.

Disk. One snapshot moves bytes rather than adding them: the archive holds what wal.bin was holding anyway. A series of them accumulates, because each one leaves another wal.<N>.archive behind and nothing deletes it. On a repository that syncs on every commit that is one new archive per 4 MiB of WAL churn, indefinitely.

So retention is opt-in, not automatic — nothing prunes unless a caller asks:

Left alone, the directory grows with churn: one new archive per 4 MiB of WAL on a repository that syncs on every commit, indefinitely. mushroomdb stats and mushroomdb doctor both print where history currently starts, so a growing directory is never a surprise. mushroomdb snapshot <db> --retention 8 is how to bound it once that trade is worth making. Pruning is not free: it advances the history horizon, so node_history, edge_history and was_linked stop reaching commits below it, and because the first prune breaks the wal.genesis chain, asof from then on answers for commits after the last snapshot rather than replaying into the archives.

The other added cost is one snapshot.bin, which is rewritten in place rather than accumulated. A snapshot is an expanded image rather than a log, so it is sized by the data it holds and not by the WAL it replaces: a store of 3.7 MB of string properties spread over eight columns snapshots to 5.2 MB, 1.39× its property payload. A snapshot carries one string table, not one per column — before 0.6.5 the same store wrote eight copies of the table and came to 36.8 MB, 9.82×. That change moves the format to V9; see Recovery vs. refresh below.

Recovery vs. refresh

Two different things read the WAL tail, and they are easy to confuse.

Recovery happens at open. The snapshot is loaded and the WAL is replayed on top of it — all of it, except the part a snapshot that kept its WAL already holds. A torn trailing frame — the signature of a crash mid- append — is dropped, and with repair_wal on (the default) the valid prefix is written back over it. That truncation is correct crash recovery: the frame was never fsynced, so no caller was ever told it committed.

Recovery is also where the snapshot format is upgraded, and that upgrade is one-way: a store snapshotted by 0.6.5 is format V9, and an earlier binary cannot open it — it refuses with snapshot: unsupported version 9 rather than reading it wrongly. The first read-write open of a V5–V8 store rewrites snapshot.bin at V9 and keeps the original beside it as snapshot.bin.bak until the next clean open, so the way back is to restore that file with the older binary.

Refresh happens while the store is open, and it is not recovery. A handle tracks how much of the WAL it has applied and, on refresh(), decodes only what another process has appended since. An incomplete trailing frame here means a peer is still writing, not that anything crashed, so it is left alone and picked up next time. Nothing is written to disk.

Noticing that there is a tail to read is a stat, so refresh inherits whatever the filesystem says: on a local disk a peer’s fsynced write is visible to the next check, while a mount that caches attributes (NFS, SMB, some container layers) can hide it for as long as its attribute timeout. See Concurrency.

The distinction matters for unattended readers: a reader that treated a live writer’s half-written frame as a torn tail, and repaired it, would destroy a frame that writer is about to make durable. That is why repair_wal: false and read_only: true exist, and why the hooks use them. See Concurrency.

The LOCK file

A store directory holds one extra file for cross-process coordination: LOCK, always empty. It exists only to carry an advisory OS lock that writers hold while they commit. It is created on the first write, never removed, and carries no data — deleting it while no process has the store open is harmless, and deleting it while one does defeats the coordination for handles opened afterwards.

It is not part of the on-disk format: snapshots and WAL frames are unchanged by its presence, and a store copied without it works normally.

Integrity

mushroomdb verify <db> validates a snapshot end to end: per-section CRC32 and a structural (rkyv bytecheck) pass over the sections the hot read path otherwise reads unchecked. Run it before restoring a snapshot from an untrusted source — CRC32 alone cannot detect a maliciously crafted image (an attacker can recompute the checksum), but the structural pass rejects out-of-bounds pointers.

Format compatibility matrix

Every historical snapshot format is permanently readable. A format version is stamped in the 6-byte header (magic GDB1 + 2-byte little-endian version).

Patch-stability promise: a snapshot written by any release in the same minor series opens correctly on any other binary in that series without migration. Format version is only bumped on minor or major releases.

What the golden pins prove: crates/core-api/tests/fixtures/ contains a committed binary fixture for each version (golden_v5.bin … golden_v9.bin). The golden_v{5..9}_pin tests in the snapshot integration-test suite load each fixture into a real GraphDb::open() call and assert node count, edge count, and specific property values. A change to any decoder that silently corrupts data will fail its pin, not just the new-data tests.

V5 limitation: the V5 encoder was removed when V6 shipped. The V5 golden pin contains 2 nodes and 1 edge (the minimal shape that exercises the decoder); richer content coverage is not possible without a V5 encoder.

format-compat CI job: the format-compat job runs cargo test -p mushroomdb --test snapshot --test migrate, which exercises every version pin and every migration path on every CI run. A failure in that job names the format-compat category explicitly rather than surfacing as a generic test failure.

Backup and export from Rust and the CLI

Backup (Rust API)

#![allow(unused)]
fn main() {
let report = db.backup_to(Path::new("/backup/dir"))?;
// report.files: Vec<String>  — files copied, sorted
// report.bytes: u64          — total bytes copied
// report.verified: bool      — true when post-copy CRC check passed
}

backup_to copies the store files and verifies the copy by opening it read-only. The &self borrow excludes concurrent writers within the same process, but it provides no cross-process guarantee. Do not call backup_to via the Rust API against a directory that a separate mushroomdb serve process is writing to. Use POST /backup instead.

Backup (CLI)

mushroomdb backup <db-dir> <backup-dir>

WARNING: unsafe against a concurrently running mushroomdb serve process. For live-served stores use POST /backup.

Export (Rust API)

#![allow(unused)]
fn main() {
let nodes: Vec<NodeInfo>     = db.all_nodes_for_export();
let edges: Vec<ExportEdge>   = db.all_edges_for_export();
// ExportEdge { edge_type, src, dst, derived: bool, rule: Option<String>, weight: Option<f64> }
}

Nodes are sorted by key. Edges are sorted by (edge_type, src, dst). Derived edges include the rule name in rule; manually-written edges have rule: None.

Export (CLI)

mushroomdb export <db-dir> <dest> [--format jsonl|parquet|graphml]

Default format: jsonl.

JSONL format (default): writes nodes.jsonl, edges.jsonl, and rules.jsonl to <dest> (a directory). Each line is a JSON object. JSONL output is stable, deterministic, and byte-identical between two runs on the same store.

Parquet format (--format parquet): writes nodes.parquet, edges.parquet, and rules.parquet to <dest> (a directory) with Snappy compression (parquet-rs default). Parquet output is not byte-identical between parquet-rs library versions — use JSONL if you need a stable byte checksum.

GraphML format (--format graphml): writes a single GraphML file — nodes and edges only, since rules have no GraphML analogue — for import into generic graph viewers and analysis tools. If <dest> is an existing directory, the file is <dest>/graph.graphml; otherwise <dest> is the file path itself. Node elements carry a label attribute plus every property (lists and maps are JSON-encoded as string-typed attributes); edge elements carry type and derived, plus rule and weight when present. weight is the creating rule’s declared weight_prop, read off the edge. GraphML output is byte-identical between two runs on the same store.

GraphML attribute types: int properties (mushroomdb’s Value::Int is a 64-bit i64) declare attr.type="long", not "int" — GraphML’s informal convention treats "int" as 32-bit, so "long" is the interoperable choice for the full range. If nodes disagree on a property’s type (one has an int under score, another a string), the key declares attr.type="string" for every node rather than risk a value that doesn’t fit a narrower type.

Float handling: Value::Float fields that are NaN or ±Inf are exported as JSON null (JSONL/Parquet) or an empty <data> element (GraphML). Normal finite floats round-trip correctly.

Derived edges: the derived field is true for rule-derived edges; rule is the rule name (present for derived edges, null/omitted for manual edges).

Roadmap

A future optimization can make rule-derived edges first-class replayable WAL records (and persist incremental ANN state) so that even WAL-only-from-genesis recovery skips re-derivation. It is deferred because the snapshot path already delivers fast recovery for the common case, and the change touches the crash-recovery correctness core.