brainy/docs/ADR-001-generational-mvcc.md

# ADR-001: Generational MVCC storage and the immutable Db API

**Status:** Accepted (ships in 8.0)
**Date:** 2026-06-10

## Context

Before 8.0, Brainy carried two overlapping version-control subsystems: a
copy-on-write branching layer (`fork`/`checkout`/`commit`/`branches`) and a
separate versioning subsystem (`versions.save/list/compare/restore/prune`),
plus a read-only historical adapter for commit-based time travel. Together
they were ~5,100 LOC of mechanism for one product need: *read a consistent
past state while the store keeps moving, and snapshot/restore cheaply.*

Neither subsystem gave a precise isolation guarantee. Reads raced in-place
JSON overwrites, so a "snapshot" was only as immutable as the bytes it
happened to share with the live store.

8.0 replaces both with **one mechanism**: generational MVCC over immutable,
generation-stamped records, exposed through a Datomic-style immutable
database value (`Db`). The same model is implemented natively by versioned
index providers (LSM snapshots), so semantics are identical on the pure-JS
path and the native path.

## Decision

### The model

- A **monotonic u64 generation counter** is the store's logical clock. It
  advances once per committed `transact()` batch and once per
  single-operation write (`add`/`update`/`remove`/`relate`/…), so
  `brain.generation()` is always a meaningful watermark. It is persisted in
  `_system/generation.json` and never reissued for anything durable.
- `brain.now()` **pins** the current generation in O(1) and returns a `Db` —
  an immutable view. Pins are refcounted; `db.release()` (with a
  `FinalizationRegistry` backstop for leaked values) ends the pin.
- `brain.transact(ops, { meta, ifAtGeneration })` commits a declarative
  batch atomically as **exactly one generation**, with whole-store
  compare-and-swap (`ifAtGeneration` → `GenerationConflictError`) and
  reified transaction metadata appended to `_system/tx-log.jsonl`.
- `brain.asOf(generation | Date | snapshotPath)` opens past state;
  `db.with(ops)` layers a speculative in-memory overlay (never touching
  disk, the counter, or index providers); `db.persist(path)` cuts an
  instant snapshot; `brain.restore(path, { confirm: true })` replaces state
  from one; `Brainy.load(path)` opens a snapshot read-only with the full
  query surface.

### Persisted layout

All paths are storage-root-relative:

```
_system/generation.json          { generation, updatedAt }       atomic tmp+rename
_system/manifest.json            { version, generation,          atomic tmp+rename
                                   committedAt, horizon }        (the commit point)
_system/tx-log.jsonl             one line per committed          append-only
                                 transact: { generation,
                                 timestamp, meta? }
_generations/<N>/tx.json         the generation-N delta:         immutable
                                 touched noun/verb ids + meta
_generations/<N>/prev/<id>.json  before-image of <id> as of      immutable
                                 commit N (raw stored bytes;
                                 null parts = file was absent)
```

**Why per-generation deltas instead of a global `id → latest generation`
map in the manifest:** a global map makes every commit O(all ids) — the
whole map must be rewritten to swap it atomically. The delta layout makes a
commit O(ids touched) and keeps the manifest a fixed-size watermark, while
point-in-time resolution stays correct (see "Read resolution" below). The
trade is that resolution at a pinned generation scans the deltas of later
commits — bounded by the number of commits since the pin, which is exactly
the window compaction keeps short.

Before-images are deliberately the *only* per-id records. They serve both
roles the layer needs — the crash-recovery undo log and the point-in-time
read source. After-images would duplicate state that is already readable
(the canonical entity files hold the latest bytes; earlier states resolve
from later before-images) and would double record I/O per commit.

### Commit protocol (durability)

`transact()` commits under a store-wide mutex:

1. **CAS check.** A stale `ifAtGeneration` throws `GenerationConflictError`
   before anything is staged.
2. **Reserve** generation `N` (counter increment).
3. **Stage the undo log:** write the before-image of every touched id plus
   `tx.json` under `_generations/N/`, then **fsync** the files and their
   directories. From this point, any crash is recoverable to the exact
   pre-transaction bytes.
4. **Execute** the planned batch through the TransactionManager (which has
   its own operation-level rollback for in-flight failures).
5. **Commit point:** persist the counter, then write `_system/manifest.json`
   via atomic tmp+rename and fsync it. The rename *is* the commit: a
   generation directory is committed if and only if `N ≤
   manifest.generation`.
6. Append the tx-log line (advisory metadata — a crash between 5 and 6
   keeps the transaction).

**Crash recovery (on open):** any `_generations/<N>` directory with
`N > manifest.generation` is an uncommitted transaction. Its before-images
are restored to the canonical paths (idempotently — recovery itself can
crash and rerun) and the directory is removed. Because recovery runs before
any index is built, and a recovery that rolled something back forces a full
index rebuild, derived indexes never observe rolled-back state. Reader-mode
instances skip recovery (readers never write; the next writer repairs).

A failed (non-crash) transaction takes the same staging directory down the
abort path: the TransactionManager rolls back applied operations, the
staging directory is removed, and the generation reservation is returned —
a failed batch leaves the generation counter unchanged.

### Read resolution at a pinned generation

The state of id X at pinned generation G is:

- the before-image stored by the **first committed generation after G that
  touched X**, or
- the live canonical bytes, when nothing after G touched X.

While nothing has committed past G, *every* read on the `Db` delegates to
the live fast paths untouched — `now()` adds no read overhead until history
actually moves.

**Two read paths, one result set.** `get()`, metadata-level `find()`, and
filter-based `related()` resolve directly through the record layer at any
reachable pinned generation — no extra cost beyond scanning the deltas of
later commits. Index-accelerated dimensions (semantic/vector search, graph
traversal, cursors, aggregation) are served by **at-generation index
materialization**: the first such query on a historical `Db` copies the
exact at-G record set (live bytes for ids untouched since the pin,
before-images for the rest; a final reconciliation pass runs under the
commit mutex so transactions racing the copy cannot skew it) into an
ephemeral in-memory store and opens a read-only engine over it — the same
vector/metadata/graph index classes the live brain uses, sharing the host's
embedder and aggregate definitions. The handle is cached on the `Db` and
freed by `release()`.

**Cost, stated plainly:** materialization is O(n at G) time and memory,
once per `Db`. That is the open-core price of historical index queries. A
native `VersionedIndexProvider` (`isGenerationVisible()` + pins over
retained LSM segments) serves the same reads with no rebuild at all — the
materializer is the correctness baseline, the provider is the accelerator.

**The one remaining boundary.** Speculative `with()` overlays throw
`SpeculativeOverlayError` for index-accelerated queries and `persist()`:
overlay entities carry no embeddings (`with()` never invokes the embedder),
so a "full" index query over an overlay would silently exclude the
overlay's own entities. Commit with `transact()` to get the full surface.

**History granularity (Model-B).** EVERY write is its own immutable generation
— `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
Single-ops stage a before-image and are reported by `db.since()`/`asOf()`/
`diff()`/`history()` exactly like transacts; a pin always freezes against later
writes. `transact()` groups several operations into ONE atomic generation.

Single-op history durability is **async group-commit**: the live write hits
canonical storage immediately (acknowledged), while its before-image is buffered
and persisted to disk in one batched fsync on a size/timer trigger (or forced by
`flush()`/`close()`/`transact()`/`compactHistory()`). The buffer participates in
point-in-time resolution exactly like on-disk generations, so the synchronous
`now()` freezes with no forced flush. A hard crash before the flush loses only
the buffered *history* of the last window — never live data — and a crash
*mid-flush* is recovered by **drop-without-restore** (the partial generation's
before-images are discarded, never replayed, because the live write was already
acknowledged; restoring them would silently revert it).

### Pinning, retention, compaction

- Each live `Db` holds one refcounted pin on its generation (plus a
  `pin(generation)` on every registered `VersionedIndexProvider`, whose
  explicit pin lifetime overrides any time-based snapshot retention the
  provider has).
- The constructor **`retention`** knob governs auto-compaction (on `flush()`/
  `close()`): unset → ADAPTIVE (disk/RAM-pressure byte budget, zero-config;
  driven by a coordinator's `budgetBytes` or a local `os.freemem` probe) ·
  `'all'` → unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit
  CAPS. `compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
  manually on the same caps — the oldest unpinned record-sets are reclaimed
  while ANY supplied cap is exceeded.
- A record-set `N` is reclaimed only when `N` is at or below **every** live
  pin — deleting `N` can only break readers pinned *below* `N`, because
  resolution reads before-images from generations strictly greater than the
  pin. Live pins are ALWAYS exempt, in every retention mode.
- The manifest records the **horizon** (highest reclaimed generation).
  Generations below the horizon are unreachable; `asOf()` on them throws
  `GenerationCompactedError`. The horizon itself stays reachable, resolved
  from the record-sets above it. To keep a generation readable forever,
  `persist()` it first — snapshots are self-contained.

### Snapshots and restore

`db.persist(path)` flushes indexes, then cuts the snapshot under the
store's commit mutex (no commit, compaction, or counter write can
interleave). On filesystem storage it is a **hard-link farm**: every data
file is immutable-by-rename, so linking is safe — later rewrites swap
inodes and the snapshot keeps the old bytes. The two exceptions are handled
explicitly: the append-in-place tx-log is byte-copied, and process-local
lock state is excluded. Cross-device targets (and filesystems that refuse
links) fall back to per-file byte copies. In-memory stores serialize to the
same directory layout, so persisting a memory brain produces a real,
durable, loadable store.

`persist()` requires the view to still be the store's latest generation
(a snapshot captures current bytes); a view that history has moved past
throws rather than persisting the wrong state.

`restore(path, { confirm: true })` replaces the store's contents from a
snapshot via byte copy (never links — the snapshot stays independent),
reloads all adapter-internal derived state, rebuilds all indexes, and
floors the generation counter at its pre-restore value so observed
generation numbers are never reissued. Live pins do not survive a restore;
a warning is logged when any exist.

### Versioned index providers

Native index providers may implement the optional 4-method
`VersionedIndexProvider` capability (`generation()`,
`isGenerationVisible()`, `pin()`, `release()` — BigInt generations at the
boundary). The locked consistency model: providers are **post-commit
appliers**. The storage-record commit is the source of truth; provider
index state is derived. On open, a provider behind the committed watermark
replays the gap from storage (or requests a rebuild) — there are no
provider rollback hooks, because uncommitted transactions are repaired at
the storage layer before any index opens. Speculative `with()` overlays
never reach providers.

## Guarantees (and their proofs)

Each stated guarantee has a test that proves it, not merely exercises it
(`tests/integration/db-mvcc.test.ts`, plus
`tests/unit/db/generationStore.test.ts` for the record layer in isolation):

| Guarantee | Proof |
|---|---|
| Snapshot isolation: a pinned `Db` reads exactly its pinned state, forever | proof 1 (200 mutations, including deletes, against a pinned view) |
| Atomicity: a failing batch applies nothing; generation unchanged | proofs 2a/2b/2c (plan-time failure, injected execution-phase storage failure, `ifRev` conflict) |
| Whole-store CAS | proof 3 (`ifAtGeneration` success + conflict with exact expected/actual) |
| Snapshot integrity under source mutation (hard-link safety) | proofs 4a/4b/4c |
| Compaction never breaks a pinned read; release enables reclaim | proof 5 |
| `with()` overlays touch nothing durable | proof 6 |
| Generation monotonicity across close/reopen | proof 7 |
| Crash before the manifest rename recovers to exact pre-transaction state through the real recovery path | proof 8 (fault injection that skips abort cleanup, exactly as a dead process would) |
| Balanced provider pin/release lockstep | proof 9 |

One deliberate softness: single-operation generation bumps persist the
counter coalesced (per write burst), not per write. Durable artifacts —
records, manifests, snapshots — always persist the counter synchronously at
their own commit points, so a crash inside the coalescing window can lose
only counter values that nothing durable ever referenced.

## Failure modes

| Failure | Outcome |
|---|---|
| Crash before staging completes | Partial staging directory > manifest watermark → removed on next open; canonical state untouched. |
| Crash after staging, before/during batch execution | Before-images restored on next open; indexes rebuilt; byte-identical pre-transaction state. |
| Crash after execution, before manifest rename | Same as above — the rename is the only commit point. |
| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. |
| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. |
| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. |
| Index-accelerated query on a `with()` overlay | `SpeculativeOverlayError` — explicit, never silently-incomplete results (overlay entities carry no embeddings). |
| `persist()` of a view history has moved past | `GenerationConflictError` — a snapshot captures current bytes; persist before further writes. |
| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. |

## Lineage

The design is an assembly of well-understood prior art, chosen for being
boring where it counts:

- **Datomic** — the database-as-a-value: an immutable `Db` you query, with
  `with()` for speculation and reified transaction metadata instead of
  commit messages.
- **LMDB** — reader pins: readers never block writers; a reader's view
  stays valid because nothing overwrites the pages (here: records) it
  references; reclamation waits for the last reader.
- **LSM trees / Cassandra** — immutable segments make snapshots hard links
  and make compaction a retention policy instead of a locking problem.

## Consequences

- One mechanism replaces the COW and versioning subsystems (their removal
  is the companion change to this ADR).
- In-place branch switching (`checkout`) is gone by design; the replacement
  is opening a persisted snapshot as a separate instance — a name→path
  mapping where a product needs named branches.
- Every commit pays O(ids touched) extra writes (before-images + delta +
  manifest). Single-operation writes pay only an in-memory counter bump
  with coalesced persistence.
- The full query surface works at every reachable pinned generation.
  Record-path reads (`get`, metadata `find`, filter `related`) are
  effectively free; index-accelerated historical queries pay a one-time
  O(n at G) materialization per `Db` on the open-core path (freed on
  `release()`), and run rebuild-free on a native `VersionedIndexProvider`.