diff --git a/docs/concepts/generation-fact-log.md b/docs/concepts/generation-fact-log.md new file mode 100644 index 00000000..f9b3e974 --- /dev/null +++ b/docs/concepts/generation-fact-log.md @@ -0,0 +1,117 @@ +--- +title: The Generation Fact Log +slug: concepts/generation-fact-log +public: true +category: concepts +template: concept +order: 6 +description: Every committed write also appends a self-verifying "fact" — an after-image commit record — to an append-only log. What facts are, the crash-safety model, the scanFacts() streaming surface, family stamps, and how index providers consume the log for sequential heals. +next: + - concepts/consistency-model + - guides/snapshots-and-time-travel +--- + +# The Generation Fact Log + +Since 8.4.0, every committed generation also appends a **fact** — a compact record of what each +touched entity or relationship *became* — to an append-only, checksummed log under +`_generations/facts/`. Where the generational history answers *"what did things look like +before?"* (before-images, powering `asOf()` and rollback), the fact log answers *"what happened, +in order?"* — one sequential, self-verifying stream of the store's present being written. + +Nothing about querying changes. The fact log exists for three consumers: + +1. **Index heals and rebuilds** — one sequential read in commit order replaces a per-entity + directory walk over millions of files. +2. **Incremental catch-up** — a derived index that knows which generation it reflects reads *just + the gap*, instead of rebuilding from scratch. +3. **Replay and audit tooling** — anything that wants the store's committed timeline as a stream. + +## What a fact is + +One fact per committed generation: + +- **`generation`** and **`timestamp`** — which commit, when. +- **`ops`** — every write in that commit: `{ kind: 'noun' | 'verb', id, record }` where `record` + holds the entity's full after-image (both stored legs), or **`null` for a tombstone** — a + removal carries no body, by design. +- **`meta`** — the transaction metadata `transact()` was submitted with, when present. +- **`blobHashes`** — content-blob references, for exact reclamation accounting. + +Facts accumulate **from the first write after upgrading** — pre-existing history is not +retroactively converted, and consumers fall back to the enumeration walk when no log exists. + +## Crash safety, in one paragraph + +Facts are appended and fsynced **inside the same durability window as the commit itself**, before +the commit point — so after a crash, the log can only ever be *ahead* of committed truth, never +behind it with a hole. On open, the store reconciles the log back to the committed watermark: +torn tails are detected by per-record checksums and cut; whole records beyond the watermark are +truncated. The invariant every reader can rely on: **an absent generation was never committed; a +present fact was.** `transact()` facts are durable the moment `transact()` returns; single-op +facts share the same group-commit flush as the rest of their generation, so a hard kill loses the +fact and the generation *together* — never a torn state. + +## Reading the log + +```typescript +const scan = brain.scanFacts({ fromGeneration: 1 }) +if (scan) { + // Telemetry up front — progress bars get a denominator from second zero. + console.log(scan.headGeneration, scan.segmentCount, scan.approxFactCount) + + for await (const batch of scan.batches()) { + // Each batch: { facts, firstGeneration, lastGeneration, factCount, byteSize, segmentId } + for (const fact of batch.facts) { + for (const op of fact.ops) { + if (op.record === null) { + // a tombstone: op.id was removed in this generation + } + } + } + } + + console.log(scan.summary()) // { factsYielded, segmentsRead } — the cross-check +} +``` + +- `scanFacts()` returns `null` when the store hosts no fact log (older store, or a storage adapter + without binary append support) — fall back to enumerating entities. +- Scans run against a **snapshot**: facts appended after the scan opens never bleed in, each fact + is yielded exactly once, and a detected gap aborts loudly — never a silent skip. +- `brain.factSegmentPaths()` returns the immutable, *sealed* segment files for zero-copy consumers + (the append-mutable tail is excluded — read it through `scanFacts()`). + +## Family stamps: how a projection proves it's current + +Anything derived from the store — an index, the entity file tree itself — carries a **family +stamp**: a small JSON record of *which committed generation the projection reflects* +(`sourceGeneration`) plus the invariants that verify it whole (exact per-file byte sizes for +bounded families; rollup invariants like entity counts for unbounded ones). At open, coherence is +a **comparison**, not a walk: + +- stamp equals the committed watermark and invariants hold → serve; +- stamp is behind → the projection reads just the gap from the fact log; +- invariants fail → loud, named divergence — `brain.repairIndex()` rebuilds from canonical and + re-stamps. + +The verifier is exported (`verifyFamilyStamp`) so every projection — TypeScript or native — runs +literally the same check. + +## For plugin authors: the storage capability + +Index providers receive the storage adapter, not the brain — so the host wires the log onto it. +Feature-detect and prefer the stream; fall back to enumeration: + +```typescript +const scan = storage.scanFacts?.({ fromGeneration: stamp.sourceGeneration + 1 }) +if (scan) { + // sequential catch-up from the log +} else { + // enumeration walk (older store or adapter) +} +const committed = storage.committedGeneration?.() // the watermark stamps compare against +``` + +Providers must never construct their own reader over the log's files — the open path belongs to +the single writer (it reconciles the log at open); the capability is the sanctioned seam. diff --git a/docs/guides/external-backups-and-sparse-storage.md b/docs/guides/external-backups-and-sparse-storage.md new file mode 100644 index 00000000..f28dcfd7 --- /dev/null +++ b/docs/guides/external-backups-and-sparse-storage.md @@ -0,0 +1,99 @@ +--- +title: External Backups & Sparse Storage +slug: guides/external-backups +public: true +category: guides +template: guide +order: 10 +description: How to back up a brain directory with external tools (tar, rsync, cp) without exploding sparse files — why a store can show 100+ GB "apparent" size on a small disk, which files are sparse, and how persist()/restore() handle it for you. +next: + - guides/snapshots-and-time-travel + - concepts/storage-adapters +--- + +# External Backups & Sparse Storage + +The built-in snapshot path — [`db.persist()` and `brain.restore()`](/docs/guides/snapshots-and-time-travel) — +already handles everything on this page for you. Read this when you back up a brain directory with +**external tools**: `tar`, `rsync`, `cp`, `scp`, or a filesystem-level backup agent. + +## The one-sentence rule + +> **Always use the sparse-aware flag**: `tar czSf` (capital `S`), `rsync --sparse`, +> `cp --sparse=always`. A naive copy can turn a 2 GB store into a 100+ GB one — or fail +> the disk entirely. + +## Why: some files are sparse + +When a native accelerator plugin is active, parts of the index live in **memory-mapped files** +created at a large fixed virtual size — the file's *apparent* size — while the filesystem only +allocates blocks that were actually written. A brand-new id-mapper file can report tens of +gigabytes in `ls -l` while occupying a few megabytes on disk. + +Check the difference yourself: + +```bash +ls -lh brain-data/_id_mapper/ # APPARENT size (can be huge) +du -sh brain-data/ # ALLOCATED size (the real footprint) +``` + +The sparse candidates in a brain directory: + +| Path | What it is | +|---|---| +| `_id_mapper/` | The native id-mapper's mmap files (large fixed virtual size) | +| `_blobs/` | Native index files (vector base, segments) — may be mmap-backed | + +Everything else (entities, `_system`, `_generations`, `_cas` content blobs) is ordinary dense data. + +## Doing it right + +**tar** — the `S` flag detects holes and stores only real data: + +```bash +tar czSf brain-backup.tgz /data/brain +# restore preserves the holes: +tar xzSf brain-backup.tgz -C /data/ +``` + +**rsync**: + +```bash +rsync -a --sparse /data/brain/ backup-host:/backups/brain/ +``` + +**cp**: + +```bash +cp -a --sparse=always /data/brain /backups/brain +``` + +**What goes wrong without the flag:** the copy *materializes* every hole as real zero bytes. +A store whose apparent size exceeds the target disk fails with `ENOSPC` partway through — and a +copy that *does* fit silently costs the full apparent size in storage and transfer time. + +## What the built-in paths do (so you don't have to) + +- **`db.persist(path)`** snapshots via **hard links** — instant and space-shared, since every data + file is immutable-by-rename. The handful of append-in-place files (the transaction log, the + commit fact log's tail segment) and mmap-mutated directories (`_id_mapper/`) are **byte-copied** + instead, so a post-snapshot write can never reach through a shared inode into your backup. +- **`brain.restore(path, { confirm: true })`** is **non-destructive and sparse-aware**: the snapshot + is copied into a staging area *before* any live data is touched (all-zero blocks stay holes), and + only after the copy fully succeeds does an atomic swap move it into place. A failed copy — + including `ENOSPC` — leaves the live store exactly as it was. A crash mid-swap completes forward + on the next open. + +## Live-store caveats for external tools + +1. **Prefer snapshotting a `persist()` output, not the live directory.** `persist()` produces a + crash-consistent, immutable snapshot; running `tar` against a live, actively-written directory + can capture a torn mid-write state. If you must archive live, stop writes first (or accept that + the archive is only as consistent as the moment's flush state). +2. **Never prune or "clean up" files inside a brain directory.** Index files that look stale or + redundant are load-bearing; the store protects its declared index families from in-process + deletion, but an external `rm` bypasses that fence. If space is the concern, `du -sh` first — + the allocated size is usually far smaller than it looks. +3. **Verify restores by opening them.** `Brainy.load(path)` opens any snapshot or restored + directory read-only — the store verifies its own coherence at open and reports loudly if + anything is missing or torn. diff --git a/docs/guides/snapshots-and-time-travel.md b/docs/guides/snapshots-and-time-travel.md index 6f7b150a..56c49044 100644 --- a/docs/guides/snapshots-and-time-travel.md +++ b/docs/guides/snapshots-and-time-travel.md @@ -9,6 +9,7 @@ description: Recipes for the Db API — instant backups with persist(), restore, next: - concepts/consistency-model - guides/optimistic-concurrency + - guides/external-backups --- # Snapshots & Time Travel @@ -41,6 +42,10 @@ bytes. Cross-device targets fall back to per-file byte copies, and persisting an in-memory brain serializes it to the same directory layout — a real, durable store. +> Archiving a brain directory with **external tools** (`tar`, `rsync`, `cp`)? +> Some index files are sparse and can explode to their apparent size under a +> naive copy — see [External Backups & Sparse Storage](/docs/guides/external-backups). + Two things to know: - `persist()` requires the view to still be the store's **latest** diff --git a/src/brainy.ts b/src/brainy.ts index d5d4bc33..383b1195 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -8280,6 +8280,14 @@ export class Brainy implements BrainyInterface { * replacement. Release them first; a warning is logged when live pins * exist. * + * NON-DESTRUCTIVE STAGING + SPARSE-AWARE: the snapshot is copied into a + * staging area BEFORE any live data is touched (all-zero blocks stay + * holes, so a sparse mmap store restores at its true allocated size — not + * its apparent size). Any copy failure, ENOSPC included, removes only the + * staging and throws; the live store is untouched. Only after the copy + * succeeds does an atomic per-entry swap move it into place; a crash + * mid-swap completes FORWARD on the next open. + * * @param path - Snapshot directory to restore from. * @param options - Must be `{ confirm: true }` — an explicit acknowledgment * that current state is destroyed. diff --git a/src/db/db.ts b/src/db/db.ts index 62c3de0e..ac927fc5 100644 --- a/src/db/db.ts +++ b/src/db/db.ts @@ -945,6 +945,13 @@ export class Db { * {@link SpeculativeOverlayError} (commit them with `brain.transact()` * first). * + * SPARSE FILES: a native accelerator's mmap index files can be sparse — + * huge apparent size, small allocated size. `persist()` handles them + * correctly (hard links share the allocation). But if you then archive the + * snapshot with EXTERNAL tools, use the sparse-aware flags (`tar czSf`, + * `rsync --sparse`, `cp --sparse=always`) or the copy materializes every + * hole — see docs/guides/external-backups-and-sparse-storage.md. + * * @param path - Absolute directory for the snapshot (created; must be * empty or absent). * @throws GenerationConflictError when this view is no longer the latest