feat: two-tier history reads + the repacker + generationDigest — D1+D3 wired end-to-end

The packed tier goes live inside GenerationStore:

- Two-tier reads: getDelta / readBeforeImage / readGenerationRecords
  fall through live-tier → sealed segments (live-tier-wins: a crash
  mid-fold leaves a duplicate representation, never a gap). Cold-open
  seeds committedRanges from the segment manifest via interval merge —
  packed generations resolve without their directories existing.
- repackHistory({timeBudgetMs, batchGenerations}): folds cold
  generations (older than the newest 1024) oldest-first into sealed
  segments, deleting per-generation directories only after segment +
  manifest are durable. Public API + automatic time-bounded pass at
  close() (before compaction, so reclaim can drop whole segments);
  re-representation only — the sole history transform under the
  archival profile. Early stop = consistent prefix, next pass resumes.
- compact(): packed generations reclaim logically in the loop and
  physically at whole-segment boundaries via dropSegmentsBelow (the
  frozen partial-segments-wait rule).
- generationDigest(g) (D8): deterministic content digest through g —
  sealed-segment checksum chain + live-tier delta hashes; O(segments +
  live window); RangeError out of range, GenerationCompactedError
  below the horizon (a gate can never silently pin reclaimed history).

Four end-to-end pins: asOf answers byte-identical across fold + cold
reopen with folded dirs physically gone; repack+reclaim composition;
digest reopen-stability/divergence/loud-horizon; budget no-op+resume.

.github/workflows/ci.yml

@@ -0,0 +1,40 @@
+name: CI
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  node:
+    name: Node ${{ matrix.node-version }}
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        node-version: ['22', '24']
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: ${{ matrix.node-version }}
+          cache: npm
+      - run: npm ci
+      - run: npm run test:unit
+
+  bun:
+    name: Bun (latest)
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+          cache: npm
+      - uses: oven-sh/setup-bun@v2
+        with:
+          bun-version: latest
+      - run: npm ci
+      # test:bun imports the built dist/, so build first.
+      - run: npm run build
+      # Bun as a runtime is the supported Bun story (`bun add` / `bun run`).
+      - run: npm run test:bun

.gitignore

@@ -93,9 +93,14 @@ docs/internal/
 # Rust/Cargo build artifacts
 src/embeddings/candle-wasm/target/
 src/embeddings/candle-wasm/Cargo.lock
-src/embeddings/wasm/pkg/
 
-# But keep the pre-built WASM (committed for users without Rust)
+# Ignore the wasm-pack output dir's CONTENTS (note the `/*`, not `/`, so the
+# re-includes below can take effect — git cannot re-include a file whose parent
+# DIRECTORY is excluded). Keep the pre-built WASM committed: it ships in the npm
+# package anyway, it lets consumers + CI build without a Rust/wasm-pack toolchain,
+# and versioning it makes the shipped artifact reproducible (not "whatever the
+# maintainer last built").
+src/embeddings/wasm/pkg/*
 !src/embeddings/wasm/pkg/*.wasm
 !src/embeddings/wasm/pkg/*.js
 !src/embeddings/wasm/pkg/*.d.ts

.versionrc.json

@@ -1,24 +0,0 @@
-{
-  "types": [
-    {"type": "feat", "section": "✨ Features"},
-    {"type": "fix", "section": "🐛 Bug Fixes"},
-    {"type": "docs", "section": "📚 Documentation"},
-    {"type": "refactor", "section": "♻️ Code Refactoring"},
-    {"type": "perf", "section": "⚡ Performance Improvements"},
-    {"type": "test", "section": "✅ Tests"},
-    {"type": "build", "section": "🔧 Build System"},
-    {"type": "ci", "section": "🔄 CI/CD"},
-    {"type": "style", "hidden": true},
-    {"type": "chore", "hidden": true}
-  ],
-  "compareUrlFormat": "https://github.com/soulcraftlabs/brainy/compare/{{previousTag}}...{{currentTag}}",
-  "commitUrlFormat": "https://github.com/soulcraftlabs/brainy/commit/{{hash}}",
-  "issueUrlFormat": "https://github.com/soulcraftlabs/brainy/issues/{{id}}",
-  "userUrlFormat": "https://github.com/{{user}}",
-  "releaseCommitMessageFormat": "chore(release): {{currentTag}}",
-  "issuePrefixes": ["#"],
-  "header": "# Changelog\n\nAll notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.\n",
-  "scripts": {
-    "postbump": "echo '✅ Version bumped to' $(node -p \"require('./package.json').version\")"
-  }
-}

CHANGELOG.md

@@ -2,6 +2,347 @@
 
 All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
 
+### [8.9.0](https://github.com/soulcraftlabs/brainy/compare/v8.8.2...v8.9.0) (2026-07-19)
+
+- docs: measured performance envelopes v1 (per-op p50/p95 at 1k and 10k, pure-JS floor) (5cabd78)
+- fix: release drains in-flight writer-lock heartbeat — no phantom lock after unlink (70e4bc8)
+- feat: flush() never compacts — history maintenance moves to close() with bounded passes (300d9f2)
+
+
+### [8.8.2](https://github.com/soulcraftlabs/brainy/compare/v8.8.1...v8.8.2) (2026-07-19)
+
+- fix: one field-resolution law across aggregation hooks, source.where, removeMany, and find() spellings (945d92d)
+- chore: push public docs to the soulcraft.com ingest door on release (42037d0)
+
+
+### [8.8.1](https://github.com/soulcraftlabs/brainy/compare/v8.8.0...v8.8.1) (2026-07-18)
+
+- fix: O(1) adaptive retention accounting + historyStats fleet audit (6207e48)
+- fix: import dedup off-switch honesty + brain-owned lifecycle for the background pass (4fcef7b)
+
+
+### [8.8.0](https://github.com/soulcraftlabs/brainy/compare/v8.7.1...v8.8.0) (2026-07-17)
+
+- feat: OS-limit detection for pool-scale deployments (16a73b8)
+
+
+### [8.7.1](https://github.com/soulcraftlabs/brainy/compare/v8.7.0...v8.7.1) (2026-07-17)
+
+- fix: race-proof writer-lock acquisition + machine-readable conflict through init (01a3b46)
+
+
+### [8.7.0](https://github.com/soulcraftlabs/brainy/compare/v8.6.0...v8.7.0) (2026-07-17)
+
+- feat: scaled transact budgets + labeled timeout diagnostics + envelope docs (6ef9fcb)
+
+
+### [8.6.0](https://github.com/soulcraftlabs/brainy/compare/v8.5.2...v8.6.0) (2026-07-17)
+
+- feat: brain.auditGraph() — read-only graph-truth audit (2a03fae)
+
+
+### [8.5.2](https://github.com/soulcraftlabs/brainy/compare/v8.5.1...v8.5.2) (2026-07-17)
+
+- fix: exception-safe aggregation backfill + generation-verified adoption + loud open-path guards (a77b064)
+
+
+### [8.5.1](https://github.com/soulcraftlabs/brainy/compare/v8.5.0...v8.5.1) (2026-07-17)
+
+- fix: aggregation state adoption on reopen + single-flight backfill + query-cap ratchet removal (da55be7)
+- docs: external-backups/sparse-storage guide + generation fact log concept (593bb8b)
+
+
+### [8.5.0](https://github.com/soulcraftlabs/brainy/compare/v8.4.0...v8.5.0) (2026-07-15)
+
+- test: tolerant timing assertion in the execution-time measure test (4dc0a92)
+- feat: committedGeneration capability + pinned durability/stability contracts (d1ecee1)
+- docs: RELEASES.md entry for 8.5.0 (provider fact-log access + shared verifier) (e4f37cd)
+- feat: provider access to the fact log + shared stamp verifier via internals (352e356)
+
+
+### [8.4.0](https://github.com/soulcraftlabs/brainy/compare/v8.3.3...v8.4.0) (2026-07-15)
+
+- docs: RELEASES.md entry for 8.4.0 (generation fact log + family stamp) (4a60b43)
+- feat: entity-tree family stamp — sourceGeneration + rollup coherence at open (2888ae6)
+- feat: generation fact log — after-image commit records, dual-written at every commit point (38b0041)
+
+
+### [8.3.3](https://github.com/soulcraftlabs/brainy/compare/v8.3.2...v8.3.3) (2026-07-15)
+
+- docs: RELEASES.md entry for 8.3.3 (rename containment fix + repair) (c3feafd)
+- test: lens-consistency regression — combined vs subtype-only vs canonical ground truth (4fb41f9)
+- fix: VFS rename moves the containment edge — no ghost in the old directory (af8c179)
+
+
+### [8.3.2](https://github.com/soulcraftlabs/brainy/compare/v8.3.1...v8.3.2) (2026-07-14)
+
+- docs: RELEASES.md entry for 8.3.2 (honest counters) (0932ecd)
+- fix: honest counters — removal never re-reads the removed record + repairIndex recounts and persists all rollups (2e2ba9c)
+
+
+### [8.3.1](https://github.com/soulcraftlabs/brainy/compare/v8.3.0...v8.3.1) (2026-07-14)
+
+- docs: RELEASES.md entry for 8.3.1 (full-removal deletes + family-scoped gate) (c0c68ac)
+- fix: full-removal canonical deletes + family-scoped migration gate (366f9a9)
+- docs: cite the cross-layer integrity contract generically in comments and notes (1d26988)
+
+
+### [8.3.0](https://github.com/soulcraftlabs/brainy/compare/v8.2.8...v8.3.0) (2026-07-13)
+
+- docs: RELEASES.md entry for 8.3.0 (heal-cost + cross-layer integrity contract) (7692c6f)
+- perf: parallel + id-only canonical enumeration (heal-cost dominant term) (ec5b933)
+- feat: registered-blob family contract — declared index blobs are undeletable (bfa1762)
+- feat: validateIndexConsistency delegates to provider invariants (6bcb54f)
+
+
+### [8.2.8](https://github.com/soulcraftlabs/brainy/compare/v8.2.7...v8.2.8) (2026-07-13)
+
+- fix: honest index readiness — no silently-empty queries on a cold index (d0f69c7)
+
+
+### [8.2.7](https://github.com/soulcraftlabs/brainy/compare/v8.2.6...v8.2.7) (2026-07-13)
+
+- fix: restore loadBinaryBlob fault-propagation (native column-store lockstep) (b6c7039)
+
+
+### [8.2.6](https://github.com/soulcraftlabs/brainy/compare/v8.2.5...v8.2.6) (2026-07-13)
+
+- docs: RELEASES.md entry for 8.2.6 (write/index-spine hardening) (a873852)
+- chore: hold loadBinaryBlob fault-propagation for the cortex column-store lockstep (36c10c1)
+- fix: aggregation surfaces materialize/state-load failures loudly (02eff64)
+- fix: surface a degraded derived index on reads instead of serving it silently (ba958d9)
+- fix: saveBinaryBlob never acks a durable write that stored nothing (7feba49)
+- fix: refuse writes when single-op history cannot be made durable (54c1836)
+- fix: clear() wipes the full native/derived footprint, not a subset (d8301f8)
+- fix: surface segment/entity read faults loudly instead of masking as absent (af5d2f3)
+- fix: spine hardening pass 1 (part) — count symmetry, honest partial-load, flush durability, read-fault propagation (119087a)
+- test: pin the read-your-writes contract under the single writer (eb9c4eb)
+
+
+### [8.2.5](https://github.com/soulcraftlabs/brainy/compare/v8.2.4...v8.2.5) (2026-07-12)
+
+- docs: RELEASES.md entry for 8.2.5 (honest rollback-failure response) (a7c7aa5)
+- fix: honest response when a transaction rollback cannot complete (711d2f0)
+
+
+### [8.2.4](https://github.com/soulcraftlabs/brainy/compare/v8.2.3...v8.2.4) (2026-07-12)
+
+- docs: RELEASES.md entry for 8.2.4 (non-destructive restore) (4574695)
+- fix: non-destructive, crash-resumable restore (a2f4f6a)
+
+
+### [8.2.3](https://github.com/soulcraftlabs/brainy/compare/v8.2.2...v8.2.3) (2026-07-12)
+
+- docs: RELEASES.md entry for 8.2.3 (transact durability barrier) (be5ce0b)
+- fix: transact durability barrier — committed transactions are durable on return (3b8fa51)
+
+
+### [8.2.2](https://github.com/soulcraftlabs/brainy/compare/v8.2.1...v8.2.2) (2026-07-11)
+
+- docs: RELEASES.md entry for 8.2.2 (transaction timeout rollback) (ed97006)
+- fix: transaction timeout rolls back applied operations (no torn state) (508a8e3)
+
+
+### [8.2.1](https://github.com/soulcraftlabs/brainy/compare/v8.2.0...v8.2.1) (2026-07-10)
+
+- test: update graph-index operation constructors to the VerbEndpointInts signature (62a449d)
+- docs: RELEASES.md entry for 8.2.1 (transact forward-ref parity fix) (7089782)
+- fix: transact forward references resolve graph endpoint ints at execute time (a175406)
+
+
+### [8.2.0](https://github.com/soulcraftlabs/brainy/compare/v8.1.0...v8.2.0) (2026-07-10)
+
+- docs: RELEASES.md entry for 8.2.0 (temporal VFS) (98ceadc)
+- feat: temporal VFS — file content joins the Model-B immutability model (a3467e1)
+- docs: pin the write-path invariant in the plugin contract (the onChange change-feed guarantee) (4af8fb3)
+
+
+### [8.1.0](https://github.com/soulcraftlabs/brainy/compare/v8.0.17...v8.1.0) (2026-07-10)
+
+- docs: RELEASES.md entry for 8.1.0 (brain.onChange change feed) (4e9be08)
+- feat: brain.onChange — the in-process change feed for every committed mutation (fd5edb5)
+
+
+### [8.0.17](https://github.com/soulcraftlabs/brainy/compare/v8.0.16...v8.0.17) (2026-07-08)
+
+- docs: RELEASES.md entry for 8.0.17 (canonical count recovery + dead-machinery sweep) (6b8b9cb)
+- fix: count recovery scans the canonical layout; remove the dead 7.x hnsw sharding machinery (352e2da)
+
+
+### [8.0.16](https://github.com/soulcraftlabs/brainy/compare/v8.0.15...v8.0.16) (2026-07-08)
+
+- docs: RELEASES.md entry for 8.0.16 (atomic ifAbsent/upsert + exact blob refCounts) (54e7c0e)
+- fix: atomic ifAbsent/upsert inserts + exact blob reference counts under concurrency (867939e)
+
+
+### [8.0.15](https://github.com/soulcraftlabs/brainy/compare/v8.0.14...v8.0.15) (2026-07-08)
+
+- docs: RELEASES.md entry for 8.0.15 (atomic ifRev CAS) (b1fe25a)
+- fix: ifRev CAS is atomic — the revision check now runs under the commit mutex (9a3d1bd)
+
+
+### [8.0.14](https://github.com/soulcraftlabs/brainy/compare/v8.0.13...v8.0.14) (2026-07-07)
+
+- docs: RELEASES.md entry for 8.0.14 (migration preserves branch-scoped non-entity state) (64188a3)
+- fix: 7→8 migration preserves branch-scoped non-entity state instead of deleting it (a93bb4e)
+
+
+### [8.0.13](https://github.com/soulcraftlabs/brainy/compare/v8.0.12...v8.0.13) (2026-07-07)
+
+- docs: RELEASES.md entry for 8.0.13 (accurate boot log for established stores) (38e8de5)
+- fix: an established store no longer boot-logs "New installation" (3086916)
+
+
+### [8.0.12](https://github.com/soulcraftlabs/brainy/compare/v8.0.11...v8.0.12) (2026-07-07)
+
+- docs: RELEASES.md entry for 8.0.12 (7→8 VFS recovery, zero-rebuild cold open, strict query operators) (d9017e7)
+- fix: recover VFS content blobs stranded by a 7→8 upgrade, in place on open (c0f6ccd)
+- fix: validate where-operators and align the in-memory matcher to the documented set (6821e19)
+- docs: correct rc-era time-travel staleness + record the embedding-model ordering constraint (68da660)
+- fix: cold-open no longer re-derives durable indexes — complete the readiness contract for all three providers (61c247c)
+- docs: RELEASES.md entry for 8.0.11 (exit-hang class closed for every op shape) (4fde94b)
+
+
+### [8.0.11](https://github.com/soulcraftlabs/brainy/compare/v8.0.10...v8.0.11) (2026-07-02)
+
+- fix: no script shape can hang on brainy's internals — unref every maintenance timer + one-shot beforeExit (30eacbd)
+- docs: RELEASES.md entry for 8.0.10 (clean process exit after close) (2da2736)
+
+
+### [8.0.10](https://github.com/soulcraftlabs/brainy/compare/v8.0.9...v8.0.10) (2026-07-02)
+
+- fix: a bare script now exits cleanly after close() — release every process keep-alive (c540d63)
+
+
+### [8.0.9](https://github.com/soulcraftlabs/brainy/compare/v8.0.8...v8.0.9) (2026-07-02)
+
+- feat: guarded plugin auto-detection — installing @soulcraft/cor is the opt-in (588267b)
+
+
+### [8.0.8](https://github.com/soulcraftlabs/brainy/compare/v8.0.7...v8.0.8) (2026-07-02)
+
+- docs: plugins are explicit opt-in — correct the README scale section and plugins config comment (e420369)
+
+
+### [8.0.7](https://github.com/soulcraftlabs/brainy/compare/v8.0.1...v8.0.7) (2026-07-02)
+
+- docs: GA version is 8.0.7 — npm retired 8.0.0-8.0.6 (January dev-cycle unpublishes) (5db2c41)
+- chore(release): 8.0.1 (48bea9e)
+- docs: flagship README for the 8.0 GA; GA version is 8.0.1 (e44620e)
+- docs: rename the native provider to @soulcraft/cor across public docs and JSDoc (bf4a333)
+- chore(release): 8.0.0 (a3c2717)
+- docs: RELEASES.md 8.0.0 GA entry (RC notes become history) (4584d0b)
+- feat: promote the 8.0 u64-id line to main for the 8.0.0 GA (55d57f8)
+- fix(8.0): byte-copy _id_mapper/* in the pre-upgrade backup (cor's write-new nuance) (a30ed72)
+- chore(release): 7.33.5 (29b9d5f)
+- fix: metadata cold-read guard — no more silent [] on cold find({where}) (7.33.5) (9dc4c5e)
+- fix(8.0): metadata cold-read guard — no more silent [] on cold find({where}) (79e8709)
+- docs(8.0): add module JSDoc to typeValidation.ts (the one file missing a module block) (ab53fa0)
+- feat(8.0): auto pre-upgrade backup — hard-link snapshot before the 7.x→8.0 migration (1aad1f6)
+- fix(ci): commit the prebuilt wasm pkg + build before test:bun (green CI on fresh clone) (ed178e2)
+- chore(release): 7.33.4 (2be3d0f)
+- fix: never serve a silent [] from find({connected}) on a cold-loaded graph (fd699d0)
+- chore(release): 7.33.3 (d1665bb)
+- fix: re-validate find() results against the predicate (index-integrity guard) (7b5db0d)
+- chore(release): 7.33.2 (9593a27)
+- fix: graph adjacency cold-load consistency guard — no more silent [] on connected (1694f68)
+- chore(release): 7.33.1 (811c7da)
+- fix: getNouns cursor pagination re-scanned the first page forever (permanent CPU loop) (6721c52)
+- chore(release): 7.33.0 (526aaad)
+- feat: visibility tier (public/internal/system) on nouns + verbs (3a62445)
+- chore(release): 7.32.2 (c53dd61)
+- refactor: rename BackupData → PortableGraph (the type is interchange, not a backup) (89036de)
+- chore(release): 7.32.1 (5e7379d)
+- fix: getNouns().totalCount reports true total, not page size; quiet benign mmap-vector log (edff637)
+- chore(release): 7.32.0 (adec0ba)
+- feat: portable graph export()/import() (BackupData v1) on brain.data() (a408d37)
+- chore(release): 7.31.8 (89c6d04)
+- fix: query-cap memory misread (MemAvailable + floor) + rootDirectory getter for native mmap fast-path (3f8e097)
+- chore(release): 7.31.7 (4f8159c)
+- fix: vfs.rename() issues a metadata-only update + rollback of fresh adds removes them (ac29b0e)
+- chore(release): 7.31.6 (9b52629)
+- fix: remap reserved fields from update() metadata patches to their canonical location (67e5fc8)
+- chore(release): 7.31.5 (e5ec658)
+- fix: feature-detect setVectorBackend before wiring the mmap-vector backend (a537b36)
+- chore(release): 7.31.4 (a8cbab6)
+- fix: feature-detect setConnectionsCodec before wiring the connections codec (747ab97)
+- chore(release): 7.31.3 (cfb051c)
+- fix: mmap-vector backend capacity NaN at the provider FFI boundary (eade6ff)
+
+
+### [8.0.0-rc.9](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.8...v8.0.0-rc.9) (2026-07-01)
+
+- docs(8.0): RELEASES.md — rc.9 (migration LOCK + 6x cosine + ES2023/Node22 floor) (3a33987)
+- chore(8.0): ES2023 target + drop DOM lib + downlevelIteration (config truth-up) (cf74c25)
+- perf(8.0): allocation-free distance loops (6x cosine) — evidence-revised Fork X (b5bc73f)
+- feat(8.0): #18 coordinated migration LOCK — block-and-queue the 7.x→8.0 auto-upgrade (67bbf69)
+- chore(8.0): modernize toolchain + position Bun as a runtime (ca9129a)
+
+
+### [8.0.0-rc.8](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.7...v8.0.0-rc.8) (2026-06-30)
+
+- docs(8.0): RELEASES.md — rc.8 (no-freeze online whole-brain auto-upgrade) (5af48a9)
+- feat(8.0): no-freeze auto-upgrade hooks — isMigrating() deference + stampBrainFormat() + brain-format export (b6b9198)
+
+
+### [8.0.0-rc.7](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.6...v8.0.0-rc.7) (2026-06-30)
+
+- docs(8.0): RELEASES.md — rc.7 (cold-graph self-heal + billion-scale RAM + version handshake) (1ddc786)
+- perf(8.0): bound per-id generation history chains (O(W+L) resident, was O(N)) (a859d6e)
+- feat(8.0): eager graphIndex.init() before the isReady() rebuild gate (8f4787b)
+- feat(8.0): version-handshake marker (formatInfo + indexEpoch) for whole-brain auto-upgrade (fc7f110)
+- fix(8.0): never serve a silent [] from find({connected}) on a cold-loaded graph (229b067)
+- perf(8.0): represent the committed-generation ledger as an interval set (93f61db)
+- perf(8.0): drop O(N)-resident id-keyed storage caches; source counts from the record (b6beb7f)
+
+
+### [8.0.0-rc.6](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.5...v8.0.0-rc.6) (2026-06-29)
+
+- docs(8.0): RELEASES.md — rc.6 (perf + native-provider contract + test hygiene) (6daa70e)
+- feat(8.0): wire the two cor-confirmed metadata-provider contract additions (8b19122)
+- test(8.0): re-home orphaned test files into the gate + guard against recurrence (3f9f140)
+- perf(8.0): negation/absence where-operators via roaring-bitmap difference (5f974ab)
+- perf(8.0): HNSW removeItem is O(in-degree) via a reverse-adjacency index (72df557)
+
+
+### [8.0.0-rc.5](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.4...v8.0.0-rc.5) (2026-06-29)
+
+- docs(8.0): RELEASES.md — rc.5 hardening + the breaking operator removal (6c9a438)
+- refactor(8.0): remove the 4 deprecated query-operator aliases (clean break) (ddcc0c7)
+- refactor(8.0): remove dead/deprecated code (legacy sweep) (b9369f2)
+- refactor(8.0): API-surface + quality polish from the readiness audit (a52dba2)
+- fix(8.0): close GA-blocking correctness gaps from the readiness audit (47e8031)
+- docs(8.0): correct public docs to the real 8.0 API + honest perf claims (40d2cd5)
+- fix(8.0): re-validate find() results against the predicate (index-integrity guard) (3d11619)
+
+
+### [8.0.0-rc.4](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.3...v8.0.0-rc.4) (2026-06-24)
+
+- docs(8.0): drop the DeletedItemsIndex section + pseudo-code from index-architecture (e7b50cf)
+- fix(8.0): gate native graph analytics on the provider readiness flag (d321cf5)
+- refactor(8.0): remove dead, unreachable, and unwired modules (bf0afe8)
+- build(8.0): clean dist before every build so stale artifacts never ship (03d6540)
+- feat(8.0): #35 part-3 — supply at-gen candidate vectors for the native exact-rerank (c9e2169)
+
+
+### [8.0.0-rc.3](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.2...v8.0.0-rc.3) (2026-06-23)
+
+- test(8.0): de-flake the VFS path-cache timing assertion (0e8972c)
+- feat(8.0): asOf at-gen vector defer — provider-served historical semantic search (#35) (1c363e8)
+- perf(8.0): bound find({ where, orderBy }) sort to the page (CTX-BR-FIND-ORDERBY) (450084b)
+- feat(8.0): brain.graph.subgraph(query) query→expand fusion (#61) (82dde92)
+- feat(8.0): vector allowedIds predicate-pushdown into find() (#46) (dd325f2)
+- feat(8.0): graph analytics — brain.graph.rank / communities / path (632d90a)
+- fix(8.0): restore() reloads a native entity-id mapper before graphIndex.rebuild() (4d0b64f)
+- test(8.0): cover pending-tier range queries + setRetentionBudget adaptive reclaim (3783e61)
+- feat(8.0): Model-B per-write generation-stamping + adaptive retention knob (5c3bb2c)
+- test(8.0): Model-B write-perf + scalability spike harnesses (afac7f9)
+- perf(8.0): per-id history chains for O(log) historical reads + bounded delta cache (ceed70d)
+- refactor(8.0): graph analytics contract — intent names, not algorithm names (f3e6911)
+- docs(8.0): RELEASES — native provider is @soulcraft/cor 3.0 (fix cortex 3.0 self-contradiction) (96d9c0b)
+- test(8.0): cover the native graph seam + make provider resolution factory-tolerant (29410bc)
+
+
 ### [8.0.0-rc.2](https://github.com/soulcraftlabs/brainy/compare/v8.0.0-rc.1...v8.0.0-rc.2) (2026-06-21)
 
 - docs(8.0): RELEASES rc.2 additions — graph engine + additive wins + correctness fixes (18f27cb)

README.md

@@ -1,439 +1,217 @@
-# Brainy
-
 <p align="center">
-  <img src="https://raw.githubusercontent.com/soulcraftlabs/brainy/main/brainy.png" alt="Brainy Logo" width="200">
+  <img src="https://raw.githubusercontent.com/soulcraftlabs/brainy/main/brainy.png" alt="Brainy" width="180">
 </p>
 
-[![npm version](https://badge.fury.io/js/%40soulcraft%2Fbrainy.svg)](https://www.npmjs.com/package/@soulcraft/brainy)
-[![npm downloads](https://img.shields.io/npm/dm/@soulcraft/brainy.svg)](https://www.npmjs.com/package/@soulcraft/brainy)
-[![Documentation](https://img.shields.io/badge/docs-soulcraft.com-blue.svg)](https://soulcraft.com/docs)
-[![MIT License](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
-[![TypeScript](https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg)](https://www.typescriptlang.org/)
+<h1 align="center">Brainy</h1>
 
-**Three database paradigms. One API. Zero configuration.**
+<p align="center">
+  <b>Three database paradigms. One API. Zero configuration.</b><br>
+  The in-process knowledge database for TypeScript — vector search, graph traversal,<br>
+  and metadata filtering unified in a single query.
+</p>
 
-Built because we were tired of stitching together Pinecone + Neo4j + MongoDB and spending weeks on configuration before writing a single line of business logic. Brainy unifies vector search, graph traversal, and metadata filtering so you don't have to choose.
+<p align="center">
+  <a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/v/@soulcraft/brainy.svg" alt="npm version"></a>
+  <a href="https://www.npmjs.com/package/@soulcraft/brainy"><img src="https://img.shields.io/npm/dm/@soulcraft/brainy.svg" alt="npm downloads"></a>
+  <a href="https://github.com/soulcraftlabs/brainy/actions/workflows/ci.yml"><img src="https://github.com/soulcraftlabs/brainy/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
+  <a href="https://soulcraft.com/docs"><img src="https://img.shields.io/badge/docs-soulcraft.com-blue.svg" alt="Documentation"></a>
+  <a href="LICENSE"><img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="MIT License"></a>
+  <a href="https://www.typescriptlang.org/"><img src="https://img.shields.io/badge/%3C%2F%3E-TypeScript-%230074c1.svg" alt="TypeScript"></a>
+</p>
 
-**New here?** → **[What is Brainy? — plain-language overview, no jargon](docs/eli5.md)**
+<p align="center">
+  <a href="#quick-start">Quick start</a> ·
+  <a href="#one-query-three-engines">One query</a> ·
+  <a href="#feature-tour">Features</a> ·
+  <a href="#from-laptop-to-hundreds-of-millions">Scale with Cor</a> ·
+  <a href="#documentation">Docs</a>
+</p>
 
 ---
 
-## Install
+Built because we were tired of stitching a vector store to a graph database to a document store — and spending weeks on plumbing before writing a line of business logic. Brainy indexes every fact **three ways at once** and lets one call query them together:
+
+| You write | Brainy indexes it as | You query it with |
+|---|---|---|
+| `data: 'Ada wrote the first program'` | a **384-dim vector** (local embedding — no API key) | `find({ query: 'computing pioneers' })` |
+| `metadata: { field: 'CS', year: 1843 }` | **structured fields** (O(1) exact, O(log n) range) | `find({ where: { year: { lessThan: 1900 } } })` |
+| `relate({ from: ada, to: babbage })` | a **typed, directed graph edge** | `find({ connected: { to: babbage, depth: 2 } })` |
+
+It runs **inside your process** — no server, no Docker, nothing to operate — and persists to plain files you can snapshot with a hard link.
+
+**New here?** → **[What is Brainy? — plain-language overview, no jargon](docs/eli5.md)**
+
+## Quick start
 
 ```bash
-npm install @soulcraft/brainy
+bun add @soulcraft/brainy        # Bun ≥ 1.1 — recommended
+npm install @soulcraft/brainy    # Node.js ≥ 22
 ```
 
-## Quick Start
-
 ```javascript
 import { Brainy, NounType, VerbType } from '@soulcraft/brainy'
 
-const brain = new Brainy()
+const brain = new Brainy()                       // in-memory; one line swaps to disk
 await brain.init()
 
-// Add knowledge — text auto-embeds, metadata auto-indexes
-const reactId = await brain.add({
+// Text auto-embeds locally; metadata auto-indexes
+const react = await brain.add({
   data: 'React is a JavaScript library for building user interfaces',
   type: NounType.Concept,
+  subtype: 'library',
   metadata: { category: 'frontend', year: 2013 }
 })
 
-const nextId = await brain.add({
-  data: 'Next.js framework for React with server-side rendering',
+const next = await brain.add({
+  data: 'Next.js is a React framework with server-side rendering',
   type: NounType.Concept,
-  metadata: { category: 'framework', year: 2016 }
+  subtype: 'framework',
+  metadata: { category: 'frontend', year: 2016 }
 })
 
-// Create a relationship
-await brain.relate({ from: nextId, to: reactId, type: VerbType.BuiltOn })
-
-// Query all three paradigms at once
-const results = await brain.find({
-  query: 'modern frontend frameworks',            // Vector similarity
-  where: { year: { greaterThan: 2015 } },         // Metadata filtering
-  connected: { to: reactId, depth: 2 }            // Graph traversal
-})
+await brain.relate({ from: next, to: react, type: VerbType.DependsOn, subtype: 'runtime' })
 ```
 
-**[Full API Reference](docs/api/README.md)** | **[soulcraft.com/docs](https://soulcraft.com/docs)**
-
----
-
-## Three Indexes, One Query
-
-Every piece of knowledge lives in three indexes simultaneously:
-
-- **`data`** → **Vector index** — Content for semantic search. Strings auto-embed into 384-dim vectors. Queried with `find({ query: '...' })`.
-- **`metadata`** → **Metadata index** — Structured fields for filtering. O(1) lookups. Queried with `find({ where: { ... } })`.
-- **`relate()`** → **Graph index** — Typed, directed relationships between entities. Traversed with `find({ connected: { ... } })`.
-
-```javascript
-// Data → vector index (semantic search)
-const articleId = await brain.add({
-  data: 'A deep dive into transformer architectures',
-  type: NounType.Document,
-  metadata: { author: 'Dr. Chen', year: 2024, tags: ['AI'] }  // → metadata index
-})
-
-// Relationships → graph index
-await brain.relate({ from: authorId, to: articleId, type: VerbType.Authored })
-
-// Query all three at once
-brain.find({
-  query: 'attention mechanisms',                  // Vector similarity
-  where: { year: { greaterThan: 2023 } },         // Metadata filter
-  connected: { from: authorId, depth: 1 }         // Graph traversal
-})
-```
-
-**[Data Model Reference](docs/DATA_MODEL.md)** | **[Query Operators](docs/QUERY_OPERATORS.md)**
-
----
-
-## Features
-
-### Triple Intelligence
-
-Vector search + graph traversal + metadata filtering in every query. No stitching services together — one `find()` call combines all three.
+## One query, three engines
 
 ```javascript
 const results = await brain.find({
-  query: 'machine learning',
-  where: { department: 'engineering', level: 'senior' },
-  connected: { from: teamLeadId, via: VerbType.WorksWith, depth: 2 }
+  query: 'modern frontend frameworks',           // vector — what it means
+  where: { year: { greaterThan: 2015 } },        // metadata — what it is
+  connected: { to: react, depth: 2 }             // graph — what it touches
 })
 ```
 
-### Hybrid Search
+Every clause is optional; any combination composes. Under the hood Brainy plans the query across an HNSW vector index, a roaring-bitmap field index, and an adjacency graph index — and re-validates every result against your predicate before returning it, so a corrupt index can never hand you a wrong answer.
 
-Automatically combines keyword (text) and semantic (vector) search. No configuration needed.
+## Feature tour
+
+### The database is a value
+
+Pin it, rewind it, fork it. Snapshot isolation without a server.
 
 ```javascript
-await brain.find({ query: 'David Smith' })             // Auto: text + semantic
-await brain.find({ query: 'AI concepts', searchMode: 'semantic' })  // Semantic only
-await brain.find({ query: 'exact id', searchMode: 'text' })         // Text only
+const db = brain.now()                           // pin current state — O(1)
+
+await brain.transact([                           // atomic all-or-nothing, CAS-guarded
+  { op: 'update', id: order, metadata: { status: 'paid' } },
+  { op: 'relate', from: invoice, to: order, type: VerbType.References, subtype: 'billing' }
+], { ifAtGeneration: db.generation })
+
+await db.get(order)                              // still 'pending' — pinned forever
+await brain.get(order)                           // 'paid' — live
+
+const lastWeek = await brain.asOf(Date.now() - 7 * 86_400_000)   // full query surface, past state
+const whatIf = await db.with([{ op: 'remove', id: order }])      // speculative — never touches disk
+await brain.now().persist('/backups/today')                      // instant hard-link snapshot
 ```
 
-### Query Operators
+**[Consistency model](docs/concepts/consistency-model.md)** · **[Snapshots & time travel](docs/guides/snapshots-and-time-travel.md)**
 
-Filter metadata with equality, comparison, array, existence, pattern, and logical operators:
+### Local embeddings — no API keys
+
+Strings embed on-device with a bundled MiniLM model (WASM). Semantic search works offline, in CI, and on air-gapped machines, at zero cost per call. Hybrid keyword + semantic ranking is the default:
 
 ```javascript
-await brain.find({
-  where: {
-    status: 'active',                          // Exact match
-    score: { greaterThan: 90 },                // Comparison
-    tags: { contains: 'ai' },                  // Array
-    anyOf: [{ role: 'admin' }, { role: 'owner' }]  // Logical OR
-  }
-})
+await brain.find({ query: 'David Smith' })                            // auto: text + semantic
+await brain.find({ query: 'AI concepts', searchMode: 'semantic' })    // semantic only
 ```
 
-**[Query Operators Reference](docs/QUERY_OPERATORS.md)** — all operators with indexed/in-memory matrix
+### A typed graph, not a bag of edges
 
-### Graph Relationships
-
-Typed, directed edges between entities. Traverse connections at any depth.
+42 entity types × 127 relationship types form a shared vocabulary for any domain — healthcare (`Patient → diagnoses → Condition`), finance (`Account → transfers → Transaction`), yours. Your own taxonomy layers on with `subtype`, enforced at write time:
 
 ```javascript
-await brain.relate({ from: personId, to: projectId, type: VerbType.WorksOn })
+await brain.add({ data: 'Avery Brooks', type: NounType.Person, subtype: 'employee' })
 
-const results = await brain.find({
-  connected: { from: personId, via: VerbType.WorksOn, depth: 3 }
-})
+brain.counts.bySubtype(NounType.Person)          // O(1) — { employee: 12, customer: 847 }
+brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true })
 ```
 
-### Database as a Value
+**[Type system](docs/architecture/noun-verb-taxonomy.md)** · **[Subtypes & facets](docs/guides/subtypes-and-facets.md)**
 
-The whole database, pinned as an immutable value. Snapshot isolation, time travel, atomic transactions, instant hard-link snapshots.
+### Graph analytics built in
 
 ```javascript
-const db = brain.now()                         // Pin current state — O(1)
-
-// Atomic multi-write transaction (all-or-nothing, with CAS)
-await brain.transact([
-  { op: 'update', id: orderId, metadata: { status: 'paid' } },
-  { op: 'relate', from: invoiceId, to: orderId, type: VerbType.References, subtype: 'billing' }
-], { meta: { author: 'billing-service' }, ifAtGeneration: db.generation })
-
-await db.get(orderId)                          // Still 'pending' — pinned, forever
-await brain.get(orderId)                       // 'paid' — live
-
-// Time travel: full query surface at any past state
-const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000))
-const past = await yesterday.find({ query: 'unpaid orders' })
-
-// What-if: speculative writes, nothing touches disk
-const whatIf = await db.with([{ op: 'remove', id: orderId }])
-
-// Instant backup: hard-link snapshot, opens read-only with Brainy.load()
-await brain.now().persist('/backups/today')
+await brain.graph.rank()                          // which entities matter most (centrality)
+await brain.graph.communities()                   // natural clusters
+await brain.graph.path(a, b)                      // how two things connect
+await brain.graph.subgraph([seed], { depth: 2 })  // bounded neighborhood → { nodes, edges }
+await brain.graph.export()                        // whole graph, one O(N+E) streaming pass
 ```
 
-**[Consistency Model](docs/concepts/consistency-model.md)** | **[Snapshots & Time Travel](docs/guides/snapshots-and-time-travel.md)**
+### Write-time aggregations
 
-### Virtual Filesystem
+`SUM` / `COUNT` / `AVG` / `MIN` / `MAX` with `GROUP BY` and time windows, maintained incrementally on every write — reads are O(1) lookups, not scans. **[Aggregation guide](docs/guides/aggregation.md)**
 
-File operations with semantic search built in.
-
-```javascript
-const vfs = brain.vfs
-
-await vfs.writeFile('/docs/readme.md', 'Project documentation')
-const content = await vfs.readFile('/docs/readme.md')
-const tree = await vfs.getTreeStructure('/docs', { maxDepth: 3 })
-
-// Semantic file search
-const matches = await vfs.search('React components with hooks')
-```
-
-**[VFS Quick Start](docs/vfs/QUICK_START.md)** | **[Common Patterns](docs/vfs/COMMON_PATTERNS.md)**
-
-### Import Anything
-
-CSV, Excel, PDF, URLs — auto-detected format, auto-classified entities.
+### Import anything
 
 ```javascript
 await brain.import('customers.csv')
-await brain.import('sales-data.xlsx')                              // all sheets processed
-await brain.import('research-paper.pdf')                           // tables extracted automatically
+await brain.import('sales.xlsx')                              // every sheet
+await brain.import('research-paper.pdf')                      // tables extracted
 await brain.import('https://api.example.com/data.json')
-await brain.import('./handbook.md', { vfsPath: '/imports/handbook' })  // preserve in VFS
 ```
 
-**[Import Guide](docs/guides/import-anything.md)**
+Entities auto-classify on the way in; `brain.extractEntities(text)` exposes the same NER ensemble directly. **[Import guide](docs/guides/import-anything.md)**
 
-### Entity Extraction
-
-AI-powered named entity recognition with 4-signal ensemble scoring.
+### A filesystem that understands content
 
 ```javascript
-const entities = await brain.extractEntities('John Smith founded Acme Corp in New York')
-// [
-//   { text: 'John Smith', type: NounType.Person, confidence: 0.95 },
-//   { text: 'Acme Corp', type: NounType.Organization, confidence: 0.92 },
-//   { text: 'New York', type: NounType.Location, confidence: 0.88 }
-// ]
+await brain.vfs.writeFile('/docs/readme.md', 'Project documentation')
+await brain.vfs.search('React components with hooks')          // semantic file search
 ```
 
-**[Neural Extraction Guide](docs/neural-extraction.md)**
+**[VFS quick start](docs/vfs/QUICK_START.md)**
 
-### Plugin System
+### Operations-grade by default
 
-Optional native acceleration via `@soulcraft/cortex` — SIMD distance calculations, CRoaring bitmaps, Candle ML embeddings.
+- **Single-writer, many-reader** — an exclusive lock protects the data directory; `Brainy.openReadOnly()` and the `brainy inspect` CLI examine a live brain from another process, safely.
+- **Self-upgrading data files** — a 7.x brain opens under 8.x and migrates itself behind an observable lock (`getIndexStatus().migration`), with an automatic pre-upgrade backup. No migration scripts.
+- **No silent wrong answers** — cold-open guards self-heal or throw typed errors (`MetadataIndexNotReadyError`, `GraphIndexNotReadyError`); they never return `[]` for data that exists.
+
+**[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)**
+
+## From laptop to hundreds of millions
+
+Brainy's TypeScript engines take you a long way. When you outgrow them, add the native engine — **the API doesn't change**:
+
+```bash
+npm install @soulcraft/cor
+```
 
 ```javascript
-const brain = new Brainy({ plugins: ['@soulcraft/cortex'] })
-await brain.init()
+const brain = new Brainy({ storage: { type: 'filesystem', path: './data' } })
+await brain.init()   // @soulcraft/cor detected — same code, native engines underneath
 ```
 
-Plugins are opt-in. Brainy never auto-imports packages unless listed in `plugins`.
+Installing the package is the opt-in: if `@soulcraft/cor` is present, it loads and announces itself in the init log; if it's present but broken, `init()` **throws** — an installed accelerator never silently vanishes behind the JS engines. Opt out with `plugins: []`, or pin exactly what loads with `plugins: ['@soulcraft/cor']`. [`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate.
 
-**[Plugin Documentation](docs/PLUGINS.md)**
+Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is licensed and funds both.
 
----
+## Performance
 
-## Type System
+- JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41).
+- Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan.
+- Full numbers and capacity planning: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)**
 
-42 noun types and 127 verb types form a universal knowledge protocol:
+## Use cases
 
-```
-42 Nouns × 127 Verbs = 5,334 base relationship combinations
-```
-
-Model any domain — healthcare (`Patient → diagnoses → Condition`), finance (`Account → transfers → Transaction`), education (`Student → completes → Course`), or your own.
-
-### Subtypes — sub-classification within a NounType *or* VerbType
-
-Both noun types and verb types are intentionally coarse. Use the top-level `subtype` field to sub-classify entities AND relationships within a type — flat string, no hierarchy, your choice of vocabulary:
-
-```javascript
-// Nouns: sub-classify entities
-await brain.add({
-  data: 'Avery Brooks — runs the AI lab',
-  type: NounType.Person,
-  subtype: 'employee'                 // 'customer', 'vendor', 'contractor', …
-})
-
-// Verbs: sub-classify relationships
-await brain.relate({
-  from: ceoId,
-  to: vpId,
-  type: VerbType.ReportsTo,
-  subtype: 'direct'                   // 'dotted-line', 'matrix', …
-})
-
-// Filter on the fast path — column-store hit, not metadata fallback:
-const employees = await brain.find({ type: NounType.Person, subtype: 'employee' })
-const directReports = await brain.related({ from: ceoId, subtype: 'direct' })
-
-// O(1) counts via the persisted rollups:
-brain.counts.bySubtype(NounType.Person)
-// → { employee: 12, customer: 847, vendor: 34 }
-
-brain.counts.byRelationshipSubtype(VerbType.ReportsTo)
-// → { direct: 12, 'dotted-line': 3 }
-```
-
-**Enforce the pairing.** Register a vocabulary per type or turn on brain-wide strict mode to ensure every entity AND relationship has both `type` AND `subtype`:
-
-```javascript
-// Per-type rule with a closed vocabulary
-brain.requireSubtype(NounType.Person, { values: ['employee', 'customer'], required: true })
-
-// 8.0 default: every write requires a subtype. Exempt genuine catch-all types…
-const brain = new Brainy({ requireSubtype: { except: [NounType.Thing] } })
-
-// …or opt out while migrating pre-8.0 data, then audit and back-fill:
-const legacy = new Brainy({ requireSubtype: false })
-await legacy.audit()                                      // gaps, grouped by type
-await legacy.fillSubtypes({ [NounType.Person]: 'unspecified' })  // close them
-```
-
-For other facets you want counted (`status`, `source`, `role`), register them with `brain.trackField(name)`. Renaming an existing convention to `subtype`? Use `brain.migrateField({from, to, entityKind: 'both'})` to walk nouns AND verbs in one pass. Full guide: **[Subtypes & Facets](docs/guides/subtypes-and-facets.md)**.
-
-**[Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)** | **[Stage 3 Canonical Reference](docs/STAGE3-CANONICAL-TAXONOMY.md)**
-
----
-
-## Storage: Memory and Filesystem
-
-The same API at every scale. Change one config line to go from prototype to production.
-
-### Development — Zero Config
-
-```javascript
-const brain = new Brainy()
-```
-
-### Production — Filesystem (gzip compression on by default)
-
-```javascript
-const brain = new Brainy({
-  storage: { type: 'filesystem', path: './data' }
-})
-```
-
-### Backups and Portability — Snapshots
-
-```javascript
-const db = brain.now()
-await db.persist('/backups/2026-06-11')   // instant hard-link snapshot
-await db.release()
-
-const snapshot = await Brainy.load('/backups/2026-06-11')  // read-only Db
-const hits = await snapshot.find({ query: 'quarterly invoices' })
-await snapshot.release()
-```
-
-A snapshot directory is self-contained — copy it to another machine, open it with `Brainy.load()`, or restore it wholesale with `brain.restore(path, { confirm: true })`.
-
-Performance benchmarks and capacity planning in **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)**.
-
-**[Capacity Planning](docs/operations/capacity-planning.md)**
-
----
-
-## Use Cases
-
-- **AI agents** — Persistent memory with semantic recall and relationship tracking
-- **Knowledge bases** — Auto-linking, semantic search, relationship-aware navigation
-- **Semantic search** — Find by meaning across codebases, documents, or media
-- **Enterprise knowledge** — CRM, product catalogs, institutional memory
-- **Interactive experiences** — Game worlds, NPCs, and characters that remember
-- **Content platforms** — Similarity-based discovery, intelligent tagging
-
----
+**AI agent memory** — persistent semantic recall with relationship tracking · **Knowledge bases** — auto-linking and meaning-aware navigation · **Semantic search** over codebases, documents, media · **Enterprise data** — CRM, catalogs, institutional memory · **Games & simulations** — worlds and characters that remember.
 
 ## Documentation
 
-### Start Here
-
-- **[Brainy explained simply](docs/eli5.md)** — Plain-language overview, no jargon, no code
-
-### Core
-
-- **[API Reference](docs/api/README.md)** — Every method with parameters, returns, and examples
-- **[Data Model](docs/DATA_MODEL.md)** — Entity structure, data vs metadata
-- **[Query Operators](docs/QUERY_OPERATORS.md)** — All BFO operators with examples
-- **[Find System](docs/FIND_SYSTEM.md)** — Natural language find() and hybrid search
-
-### Architecture
-
-- **[Architecture Overview](docs/architecture/overview.md)** — System design and components
-- **[Triple Intelligence](docs/architecture/triple-intelligence.md)** — Vector + graph + metadata unified query
-- **[Noun-Verb Taxonomy](docs/architecture/noun-verb-taxonomy.md)** — Universal type system
-- **[Data Storage Architecture](docs/architecture/data-storage-architecture.md)** — Type-aware indexing and HNSW
-
-### Virtual Filesystem
-
-- **[VFS Quick Start](docs/vfs/QUICK_START.md)** — Build file explorers that never crash
-- **[VFS Core](docs/vfs/VFS_CORE.md)** — Full VFS API reference
-- **[Semantic VFS](docs/vfs/SEMANTIC_VFS.md)** — AI-powered file navigation
-
-### Guides
-
-- **[Import Anything](docs/guides/import-anything.md)** — CSV, Excel, PDF, URLs
-- **[Framework Integration](docs/guides/framework-integration.md)** — React, Vue, Angular, Svelte
-- **[Natural Language Queries](docs/guides/natural-language.md)** — Master the find() method
-
-### Operations
-
-- **[Capacity Planning](docs/operations/capacity-planning.md)** — Memory, storage, and scaling
-- **[Performance](docs/PERFORMANCE.md)** — Benchmarks and architecture details
-
----
+| Start | Core | Going deeper |
+|---|---|---|
+| [Brainy explained simply](docs/eli5.md) | [API reference](docs/api/README.md) | [Architecture overview](docs/architecture/overview.md) |
+| [Installation](docs/guides/installation.md) | [Data model](docs/DATA_MODEL.md) | [Consistency model](docs/concepts/consistency-model.md) |
+| [Natural-language queries](docs/guides/natural-language.md) | [Query operators](docs/QUERY_OPERATORS.md) | [Multi-process model](docs/concepts/multi-process.md) |
+| | [Find system](docs/FIND_SYSTEM.md) | [Scaling](docs/SCALING.md) |
 
 ## Requirements
 
-**Bun 1.0+** (recommended) or **Node.js 22 LTS**
+**Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use.
 
-```bash
-bun install @soulcraft/brainy    # Bun — best performance
-npm install @soulcraft/brainy    # Node.js — fully supported
-```
+## Contributing & license
 
-> Brainy 8.0 is server-only. Browser support (OPFS storage, Web Workers, in-browser WASM embeddings) was removed in 8.0 — the 7.x line remains available on npm if you need it.
-
-## Single-Writer Model
-
-Brainy is **single-writer, many-reader** on filesystem storage. One writer
-holds an exclusive lock on the data directory; any number of readers can
-inspect it concurrently. Opening a second writer throws with the PID of the
-existing one.
-
-```typescript
-// Live application — writer mode is the default
-const brain = new Brainy({ storage: { type: 'filesystem', path: '/data/brain' } })
-await brain.init()
-
-// Out-of-band diagnostics from a separate process — safe to run while the
-// writer is live
-const reader = await Brainy.openReadOnly({
-  storage: { type: 'filesystem', path: '/data/brain' }
-})
-await reader.requestFlush({ timeoutMs: 5000 })
-const stats = await reader.stats()
-```
-
-For incident debugging, use the `brainy inspect` CLI:
-
-```bash
-brainy inspect stats   /data/brain
-brainy inspect find    /data/brain --where '{"entityType":"booking"}'
-brainy inspect explain /data/brain --where '{"entityType":"booking"}'
-brainy inspect health  /data/brain
-```
-
-See [the multi-process model](docs/concepts/multi-process.md) and the
-[inspection guide](docs/guides/inspection.md) for the full story, including
-stale-lock detection and the cross-process flush RPC.
-
-## Contributing
-
-We welcome contributions! See **[CONTRIBUTING.md](CONTRIBUTING.md)** for guidelines.
-
-## License
-
-MIT © Brainy Contributors
+Contributions welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)**. MIT © Brainy Contributors.

docs/ADR-001-generational-mvcc.md

@@ -149,12 +149,22 @@ overlay entities carry no embeddings (`with()` never invokes the embedder),
 so a "full" index query over an overlay would silently exclude the
 overlay's own entities. Commit with `transact()` to get the full surface.
 
-**History granularity.** Generation *records* are written per `transact()`
-batch only. Single-operation writes advance the counter (so watermarks and
-CAS stay sound) but do not stage before-images: they remain visible through
-earlier pins and are not reported by `db.since()`. Code that needs pinned
-isolation across its writes uses `transact()` — that is the documented 8.0
-contract, stated rather than papered over.
+**History granularity (Model-B).** EVERY write is its own immutable generation
+— `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
+Single-ops stage a before-image and are reported by `db.since()`/`asOf()`/
+`diff()`/`history()` exactly like transacts; a pin always freezes against later
+writes. `transact()` groups several operations into ONE atomic generation.
+
+Single-op history durability is **async group-commit**: the live write hits
+canonical storage immediately (acknowledged), while its before-image is buffered
+and persisted to disk in one batched fsync on a size/timer trigger (or forced by
+`flush()`/`close()`/`transact()`/`compactHistory()`). The buffer participates in
+point-in-time resolution exactly like on-disk generations, so the synchronous
+`now()` freezes with no forced flush. A hard crash before the flush loses only
+the buffered *history* of the last window — never live data — and a crash
+*mid-flush* is recovered by **drop-without-restore** (the partial generation's
+before-images are discarded, never replayed, because the live write was already
+acknowledged; restoring them would silently revert it).
 
 ### Pinning, retention, compaction
 
@@ -162,11 +172,17 @@ contract, stated rather than papered over.
   `pin(generation)` on every registered `VersionedIndexProvider`, whose
   explicit pin lifetime overrides any time-based snapshot retention the
   provider has).
-- `compactHistory({ retainGenerations?, retainMs? })` reclaims record-sets.
-  A record-set `N` is reclaimed only when `N` is at or below **every** live
+- The constructor **`retention`** knob governs auto-compaction (on `flush()`/
+  `close()`): unset → ADAPTIVE (disk/RAM-pressure byte budget, zero-config;
+  driven by a coordinator's `budgetBytes` or a local `os.freemem` probe) ·
+  `'all'` → unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit
+  CAPS. `compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
+  manually on the same caps — the oldest unpinned record-sets are reclaimed
+  while ANY supplied cap is exceeded.
+- A record-set `N` is reclaimed only when `N` is at or below **every** live
   pin — deleting `N` can only break readers pinned *below* `N`, because
   resolution reads before-images from generations strictly greater than the
-  pin. With both retention options supplied, both must allow the reclaim.
+  pin. Live pins are ALWAYS exempt, in every retention mode.
 - The manifest records the **horizon** (highest reclaimed generation).
   Generations below the horizon are unreachable; `asOf()` on them throws
   `GenerationCompactedError`. The horizon itself stays reachable, resolved

docs/BATCHING.md

@@ -78,10 +78,10 @@ for (const [id, metadata] of metadataMap) {
 - ✅ Direct O(1) path construction from ID (no type lookup needed!)
 - ✅ Sharding preservation (all paths include `{shard}/{id}`)
 - ✅ Write-cache coherent (read-after-write consistency)
-- ✅ 40x faster than v5.x type-first architecture
+- ✅ O(1) path construction — eliminates the per-entity type search of the old type-first layout
 
 **Performance:**
-- ~1ms per 100 entities (consistent, no cache misses!)
+- Constant-time path construction per ID — no type-cache misses
 - Filesystem: parallel reads bounded by IOPS
 - No type search delays — every ID maps directly to storage path
 
@@ -121,7 +121,7 @@ for (const [sourceId, verbs] of results) {
 - Bulk export of relationship data
 
 **Performance:**
-- Memory storage: <10ms for 1000 relationships
+- Memory storage: single in-memory pass over the metadata index
 - Filesystem storage: parallel reads through the metadata index
 
 ---
@@ -171,7 +171,7 @@ entities/verbs/{SHARD}/{ID}/metadata.json
 
 **Direct O(1) Path Construction:**
 ```typescript
-// Every ID maps directly to exactly ONE path - 40x faster!
+// Every ID maps directly to exactly ONE path - O(1), no type search
 const id = 'abc-123'
 const shard = getShardIdFromUuid(id) // → 'ab' (first 2 hex chars)
 const path = `entities/nouns/${shard}/${id}/metadata.json`
@@ -183,9 +183,9 @@ const path = `entities/nouns/${shard}/${id}/metadata.json`
 ```
 
 **Benefits:**
-- **40x faster** path lookups (eliminates 42-type sequential search)
+- **O(1)** path lookups (eliminates the 42-type sequential search the old type-first layout required)
 - **Simpler code** - removed 500+ lines of type cache complexity
-- **Scalable** - works at billion-scale without type tracking overhead
+- **Scalable** - works at large scale without type tracking overhead
 
 ---
 
@@ -220,28 +220,22 @@ await db.release()
 
 ---
 
-## Performance Benchmarks
+## Why Batching Is Faster
 
-### VFS Operations (12 Files)
+Batching's advantage is structural, not a fixed multiplier (the actual speedup
+depends on storage backend, IOPS, and batch size):
 
-| Storage | Before optimization | After optimization | Improvement |
-|---------|---------------------|--------------------|-------------|
-| **Memory** | 150ms | 50ms | **67% faster** |
-| **Filesystem** | ~500ms | ~80ms | **84% faster** |
+- **N+1 elimination** — N sequential reads collapse into a single parallel pass
+  (`Promise.all` over the batch).
+- **O(1) path construction** — every ID maps directly to one storage path, with
+  no per-type cache lookup.
+- **One metadata round-trip** — relationship batches fetch all sources' metadata
+  in a single pass instead of one query per source.
 
-### Entity Batch Retrieval (100 Entities)
-
-| Storage | Individual Gets | Batch Get | Improvement |
-|---------|-----------------|-----------|-------------|
-| **Memory** | 180ms | 15ms | **92% faster** |
-| **Filesystem** | ~1.2s | ~120ms | **90% faster** |
-
-### Throughput (Entities/Second)
-
-| Storage | Individual | Batch | Improvement |
-|---------|------------|-------|-------------|
-| **Memory** | 556 ent/s | 6667 ent/s | **12x** |
-| **Filesystem** | ~80 ent/s | ~800 ent/s | **10x** |
+The integration test `tests/integration/storage-batch-operations.test.ts`
+exercises batch vs. individual reads and asserts that batch retrieval is not
+slower than the per-entity loop for large batches; it does not pin a specific
+multiplier, since that is hardware- and IOPS-dependent.
 
 ---
 
@@ -306,7 +300,7 @@ const results = await brain.batchGet(ids)
 const entities = Array.from(results.values())
 ```
 
-**Performance Gain:** 10-20x faster on filesystem storage.
+**Performance Gain:** Replaces N sequential reads with a single batched pass — no fixed multiplier, it scales with storage IOPS.
 
 ---
 
@@ -332,7 +326,7 @@ for (const verbs of results.values()) {
 }
 ```
 
-**Performance Gain:** 5-10x faster due to batched metadata fetches.
+**Performance Gain:** One batched metadata fetch instead of one query per source entity.
 
 ---
 
@@ -450,8 +444,8 @@ return await Promise.all(resolvedPaths.map(path => this.read(path)))
 - `storage.getVerbsBySourceBatch(sourceIds, verbType?)` - Batch relationship queries
 
 **Performance Improvements:**
-- VFS operations: 90%+ faster than the naive per-entity loop
-- Entity retrieval: 10-20x throughput improvement
+- VFS operations: single batched pass instead of N sequential reads
+- Entity retrieval: N+1 reads collapsed into one batched pass
 - Zero N+1 query patterns
 
 **Compatibility:**

docs/FIND_SYSTEM.md

@@ -22,7 +22,7 @@ Brainy's `find()` method is the most advanced query system in any vector databas
 ### 1. Vector Intelligence (HNSW Index)
 - **Purpose**: Semantic similarity search using embeddings
 - **Algorithm**: Hierarchical Navigable Small World (HNSW)
-- **Performance**: O(log n) search, ~1.8ms typical
+- **Performance**: O(log n) search
 - **Data Structure**: Multi-layer graph with 16 connections per node
 - **Use Cases**: "Find similar documents", "Content like this"
 
@@ -36,14 +36,14 @@ Brainy's `find()` method is the most advanced query system in any vector databas
 ### 3. Metadata Intelligence (Incremental Indices)
 - **Purpose**: Fast filtering on structured data
 - **Algorithm**: HashMap for exact matches, Sorted arrays for ranges
-- **Performance**: O(1) exact, O(log n) ranges, <1ms typical
+- **Performance**: O(1) exact, O(log n) ranges
 - **Data Structure**: `Map<field:value, Set<id>>` + sorted value arrays
 - **Use Cases**: "Documents from 2023", "Status equals active"
 
 ### 4. Graph Intelligence (Adjacency Maps)
 - **Purpose**: Relationship traversal and connection analysis
 - **Algorithm**: Pure O(1) neighbor lookups via Map operations
-- **Performance**: O(1) per hop, ~0.1ms typical
+- **Performance**: O(1) per hop (measured <1 ms per neighbor lookup, validated up to 1M relationships — `tests/performance/graph-scale-performance.test.ts:238`)
 - **Data Structure**: `Map<sourceId, Set<targetId>>`
 - **Use Cases**: "Papers connected to MIT", "Authors who collaborated"
 
@@ -373,36 +373,40 @@ return results.slice(offset, offset + limit)
 
 ### Query Performance by Type
 
-| Query Type | Index Used | Performance | Example |
-|------------|------------|-------------|---------|
-| **Semantic Search** | HNSW Vector | O(log n), ~1.8ms | `"AI research papers"` |
-| **Exact Metadata** | HashMap | O(1), ~0.8ms | `{status: "published"}` |
-| **Range Metadata** | Sorted Array | O(log n), ~0.6ms | `{year: {greaterThan: 2020}}` |
-| **Graph Traversal** | Adjacency Map | O(1), ~0.1ms | `{connected: {to: "mit"}}` |
-| **Type Detection** | Pre-embedded Types | O(t), ~0.3ms | `"documents"` → `NounType.Document` |
-| **Field Matching** | Field Embeddings | O(f), ~0.1ms | `"by"` → `"author"` |
-| **Combined Query** | All Indices | O(log n), ~1.8ms | NLP + filters + graph |
+| Query Type | Index Used | Complexity | Example |
+|------------|------------|------------|---------|
+| **Semantic Search** | HNSW Vector | O(log n) | `"AI research papers"` |
+| **Exact Metadata** | HashMap | O(1) | `{status: "published"}` |
+| **Range Metadata** | Sorted Array | O(log n) | `{year: {greaterThan: 2020}}` |
+| **Graph Traversal** | Adjacency Map | O(1) | `{connected: {to: "mit"}}` |
+| **Type Detection** | Pre-embedded Types | O(t) | `"documents"` → `NounType.Document` |
+| **Field Matching** | Field Embeddings | O(f) | `"by"` → `"author"` |
+| **Combined Query** | All Indices | O(log n) | NLP + filters + graph |
 
 Where:
 - n = number of entities in database
 - t = number of types (169 total: 42 noun + 127 verb)
 - f = number of fields for detected entity type (typically 5-15)
 
+Absolute latencies depend on hardware, embedding model, and storage backend; the columns above describe how each stage scales. The graph stage is the one path with a committed scale benchmark (see [Scalability](#scalability)).
+
 ### Scalability
 
-| Database Size | Vector Search | Metadata Filter | Graph Query | Combined |
-|---------------|---------------|-----------------|-------------|----------|
-| **1K entities** | 0.8ms | 0.3ms | 0.05ms | 1.1ms |
-| **10K entities** | 1.2ms | 0.5ms | 0.08ms | 1.5ms |
-| **100K entities** | 1.8ms | 0.8ms | 0.1ms | 2.1ms |
-| **1M entities** | 2.5ms | 1.2ms | 0.1ms | 2.8ms |
-| **10M entities** | 3.8ms | 1.8ms | 0.1ms | 4.2ms |
+The cost of each query stage is governed by its algorithmic complexity, not a fixed millisecond figure — absolute latency depends on hardware, embedding model, and storage backend. Only the graph adjacency index carries a committed scale assertion:
+
+| Query stage | Complexity | Scaling behavior |
+|-------------|------------|------------------|
+| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
+| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
+| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
+| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
+| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
 
 **Key Performance Notes:**
 - Graph queries stay O(1) regardless of scale
 - Metadata ranges scale as O(log n), not O(n)
 - Vector search degrades gracefully due to HNSW
-- Type-aware NLP adds minimal overhead (~0.4ms)
+- Type-aware NLP adds minimal overhead (single embedding pass, no full scan)
 
 ## Example Query Flows
 
@@ -455,7 +459,7 @@ await brain.find({
  },
  limit: 10
 })
-// Total performance: ~1.2ms for 100K entities
+// Executes as O(1) type filter + O(1)/O(log n) metadata + O(1) graph lookup — no full scan
 ```
 
 ## Filter Syntax Reference
@@ -1001,7 +1005,7 @@ const results = await brain.find({
 })
 ```
 
-**Performance**: O(log n) vector search + O(log n) metadata filters + O(1) graph traversal = ~2-3ms total.
+**Performance**: O(log n) vector search + O(log n) metadata filters + O(1) graph traversal — bounded by the vector stage.
 
 ### Excluding Soft-Deleted Entities
 

docs/PERFORMANCE.md

@@ -2,11 +2,11 @@
 
 ## Performance Characteristics
 
-Brainy achieves industry-leading performance through carefully optimized data structures and algorithms. All performance claims are verified through actual benchmarks on production code.
+Brainy achieves high performance through carefully optimized data structures and algorithms. The tables below describe each component by its **algorithmic complexity** — the durable, defensible guarantee. The example latencies are figures from a single 100-item run on one machine (see [Benchmarks](#benchmarks)); they are illustrative, not a committed benchmark, and vary with hardware, embedding model, and storage backend. The one component with a committed scale assertion is the graph adjacency index (`tests/performance/graph-scale-performance.test.ts:238`).
 
 ### Core Performance Summary
 
-| Component | Operation | Time Complexity | Measured Performance | Data Structure |
+| Component | Operation | Time Complexity | Example latency (100-item run)\* | Data Structure |
 |-----------|-----------|-----------------|---------------------|----------------|
 | **Metadata Index** | Exact match | **O(1)** | 0.8ms | `Map<string, Set<string>>` |
 | **Metadata Index** | Range query | **O(log n) + O(k)** | 0.6ms | Sorted array + binary search |
@@ -17,6 +17,8 @@ Brainy achieves industry-leading performance through carefully optimized data st
 | **Type Detection** | Noun/Verb matching | **O(t)** | 0.3ms | Pre-embedded type vectors |
 | **Triple Intelligence** | Combined query | **O(1) to O(log n)** | 1.8ms | Parallel execution |
 
+\* Illustrative single-run figures at 100 items on one machine — not a committed benchmark. Only the graph index carries an asserted scale bound (measured <1 ms per neighbor lookup up to 1M relationships, `tests/performance/graph-scale-performance.test.ts:238`).
+
 Where:
 - `n` = number of items in index
 - `k` = number of results returned
@@ -26,25 +28,32 @@ Where:
 
 ### brain.get() Metadata-Only Optimization
 
-✨ **Massive Performance Improvement**: `brain.get()` is now **76-81% faster** by default!
+`brain.get()` returns **metadata only by default**, skipping the 384-dimensional
+embedding — the bulk of an entity's payload. Callers that need the vector opt in
+with `{ includeVectors: true }`.
 
-| Operation | Before | After | Speedup | Use Case |
-|-----------|------------------|-----------------|---------|----------|
-| **brain.get() (metadata-only)** | 43ms, 6KB | **10ms, 300 bytes** | **76-81%** | VFS, existence checks, metadata |
-| **brain.get({ includeVectors: true })** | 43ms, 6KB | 43ms, 6KB | 0% | Similarity calculations |
-| **VFS readFile()** | 53ms | **~13ms** | **75%** | File operations |
-| **VFS readdir(100 files)** | 5.3s | **~1.3s** | **75%** | Directory listings |
+| Operation | Default (metadata-only) | With `includeVectors: true` | Use Case |
+|-----------|-------------------------|-----------------------------|----------|
+| **brain.get()** | Skips vector load | Loads full vector | VFS, existence checks, metadata |
+| **VFS readFile() / readdir()** | Inherits metadata-only path | n/a | File operations, directory listings |
 
-**Key Innovation**: Lazy vector loading - only load 384-dimensional embeddings when explicitly needed.
+**Key Innovation**: Lazy vector loading — only load the 384-dimensional embedding when explicitly needed.
+
+The integration test `tests/integration/metadata-only-comprehensive.test.ts:306`
+asserts metadata-only `get()` is faster than the full-entity `get()`
+(`metadataTime < fullTime`). The *magnitude* of the speedup is
+environment-dependent (the percentage assertion in
+`tests/integration/vfs-performance-v5.11.1.test.ts` is intentionally skipped on
+CI for that reason), so no fixed percentage is quoted here.
 
 **Why this matters**:
-- **94% of brain.get() calls** don't need vectors (VFS, admin tools, import utilities, data APIs)
-- **Vector data is 95% of entity size** (6KB vectors vs 300 bytes metadata)
-- **Zero code changes** for most applications - automatic speedup!
+- Most `brain.get()` calls don't need vectors (VFS, admin tools, import utilities, data APIs)
+- The embedding dominates an entity's serialized size, so skipping it is the largest win
+- **Zero code changes** for most applications — automatic by default
 
 **When to use what**:
 ```typescript
-// DEFAULT: Metadata-only (76-81% faster) - use for:
+// DEFAULT: Metadata-only (skips the vector load) - use for:
 const entity = await brain.get(id)
 // - VFS operations (readFile, stat, readdir)
 // - Existence checks: if (await brain.get(id)) ...
@@ -252,28 +261,34 @@ const results = await Promise.all(searchPromises)
 
 ## Benchmarks
 
-### Real-world Performance Test (100 items)
+### Illustrative Single Run (100 items, one machine)
+
+Example output from a single 100-item run — illustrative only, not a committed
+benchmark; absolute numbers vary by hardware. The values feed the
+[Core Performance Summary](#core-performance-summary) example-latency column.
 
 ```
-📊 Metadata exact match: 0.818ms (50 items matched)
-📊 Metadata range query: 0.631ms (40 items in range)
-🔗 Graph neighbor lookup: 0.092ms (2 connections)
-🎯 Vector k-NN search: 1.773ms (10 nearest neighbors)
-🧠 NLP query parsing: 8.906ms (full natural language)
-⚡ Triple Intelligence: 1.830ms (combined query)
+Metadata exact match: 0.818ms (50 items matched)
+Metadata range query: 0.631ms (40 items in range)
+Graph neighbor lookup: 0.092ms (2 connections)
+Vector k-NN search:    1.773ms (10 nearest neighbors)
+NLP query parsing:     8.906ms (full natural language)
+Triple Intelligence:   1.830ms (combined query)
 ```
 
 ### Scaling Characteristics
 
-| Items | Metadata O(1) | Range O(log n) | Graph O(1) | Vector O(log n) |
-|-------|---------------|----------------|------------|-----------------|
-| 100 | 0.8ms | 0.6ms | 0.09ms | 1.8ms |
-| 1,000 | 0.8ms | 0.9ms | 0.09ms | 2.5ms |
-| 10,000 | 0.8ms | 1.2ms | 0.09ms | 3.2ms |
-| 100,000 | 0.8ms | 1.5ms | 0.09ms | 4.1ms |
-| 1,000,000 | 0.8ms | 1.8ms | 0.09ms | 5.0ms |
+Each stage scales by its algorithmic complexity, not a fixed millisecond figure
+— absolute latency depends on hardware, embedding model, and storage backend.
+Only the graph adjacency index carries a committed scale assertion:
 
-*Note: O(1) operations maintain constant time regardless of scale*
+| Query stage | Complexity | Scaling behavior |
+|-------------|------------|------------------|
+| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
+| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
+| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
+| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
+| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
 
 ## Comparison with Other Systems
 
@@ -402,7 +417,7 @@ const brain = new Brainy({
 })
 ```
 
-The default JS index is `JsHnswVectorIndex`. An optional native acceleration package (`@soulcraft/cortex`) can replace it with a higher-performing implementation; the public knobs stay the same.
+The default JS index is `JsHnswVectorIndex`. An optional native acceleration package (`@soulcraft/cor`) can replace it with a higher-performing implementation; the public knobs stay the same.
 
 ### Scale Scenarios
 
@@ -470,8 +485,8 @@ For off-site replication, snapshot `path` from your scheduler (`gsutil rsync`, `
 - **Auto-configuration system** with environment detection
 - **Zero-config operation** with intelligent defaults
 - **Auto-flush and auto-optimize** in indices
-- **Sub-2ms response times** for complex queries
+- **Low-latency Triple Intelligence queries** (O(log n) vector + O(1) metadata/graph)
 
 ## Conclusion
 
-Brainy delivers on its promise of **production-ready Triple Intelligence** with measured, verified performance characteristics. All listed features are fully implemented, tested, and benchmarked. No stubs, no mocks, no theoretical claims - just real, working code with measured performance.
+Brainy delivers on its promise of **production-ready Triple Intelligence** with documented algorithmic-complexity guarantees and a committed graph-scale benchmark (`tests/performance/graph-scale-performance.test.ts`). All listed features are fully implemented and tested. No stubs, no mocks — just real, working code with characterized performance.

docs/PLUGINS.md

@@ -12,7 +12,7 @@ next:
 
 # Plugin Development Guide
 
-Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cortex` provides optional native acceleration, and it's the same system available to any developer.
+Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer.
 
 ## Architecture Overview
 
@@ -23,20 +23,19 @@ Brainy's plugin system uses **named providers** — string keys mapped to implem
 3. The plugin calls `context.registerProvider(key, implementation)` for each subsystem it provides
 4. Brainy checks each provider key and wires the implementation into its internal pipeline
 
-Plugins are **opt-in** — brainy never auto-imports packages. You must explicitly list plugins in the config:
+Installing the first-party accelerator is the opt-in: with the default config, brainy probes for `@soulcraft/cor` and loads it when present. Everything except "not installed" fails **loud** — a present-but-broken accelerator makes `init()` throw rather than silently degrading to the JS engines.
 
 ```typescript
-const brain = new Brainy({
-  plugins: ['@soulcraft/cortex']  // explicitly load the native acceleration package
-})
+const brain = new Brainy()                                  // @soulcraft/cor auto-detected when installed
+const pinned = new Brainy({ plugins: ['@soulcraft/cor'] })  // or pin exactly what loads
+const plain = new Brainy({ plugins: [] })                   // or opt out of detection entirely
 ```
 
 | `plugins` value | Behavior |
 |---|---|
-| `undefined` (default) | No plugins loaded |
-| `false` | No plugins loaded |
-| `[]` | No plugins loaded |
-| `['@soulcraft/cortex']` | Load only the listed packages |
+| `undefined` (default) | Guarded auto-detection of `@soulcraft/cor`: not installed → no plugins, silently; installed → loads + announces; installed-but-broken → `init()` throws |
+| `false` / `[]` | No plugins, no detection (explicit opt-out) |
+| `['@soulcraft/cor']` | Load only the listed packages; a listed plugin that fails to load throws |
 
 Plugins registered programmatically via `brain.use(plugin)` are always activated regardless of the `plugins` config.
 
@@ -70,7 +69,7 @@ export default myPlugin
 
 ### 2. Package exports
 
-Your package must export the plugin as the default export so brainy's auto-detection works:
+Your package must export the plugin as the default export so brainy's plugin loader can resolve it:
 
 ```typescript
 // index.ts
@@ -150,6 +149,17 @@ context.registerProvider('embedBatch', async (texts: string[]) => {
 
 ### Index Providers
 
+> **Write-path invariant (the change-feed contract).** Every canonical
+> mutation flows through Brainy's generation-store commit points — index
+> providers are invoked *inside* that commit and never originate canonical
+> writes of their own. The `brain.onChange` change feed is emitted from those
+> commit points and relies on this: **a plugin must never introduce a write
+> path that bypasses the generation-store commit.** If a future provider ever
+> needs a direct native ingest path, it must either route through the commit
+> or emit equivalent change events — otherwise every `onChange` consumer
+> (live UIs, cache invalidation, realtime sync) silently develops a blind
+> spot.
+
 #### `vector`
 **Type:** `(config: object, distanceFunction: Function, options: object) => VectorIndexProvider-compatible`
 
@@ -182,6 +192,27 @@ context.registerProvider('vector', (config, distanceFn, options) => {
 })
 ```
 
+#### The readiness contract (all three index providers)
+
+A provider that **persists its derived index** should implement the optional readiness
+members so a warm reopen never pays a redundant rebuild-from-canonical:
+
+- **`init?(): Promise<void>`** — eager cold-load. Brainy awaits it once during
+  `brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first)
+  and **before the rebuild gate**.
+- **`isReady?(): boolean`** — honest durability signal. `true` ⇔ the persisted index is
+  loaded (or cheaply demand-loadable) and consistent with what was last persisted. When
+  exposed, the rebuild gate defers to this signal **instead of** the `size() === 0` /
+  `totalEntries === 0` heuristics — a disk-native index may report 0 resident entries
+  while fully durable. Never return `true` if the durable state failed to load: the
+  signal is honest in both directions, and a not-ready provider gets its rebuild even
+  when `size() > 0`.
+- **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background
+  migration); brainy skips its rebuild entirely.
+
+Providers that implement none of these keep the size/count heuristics — correct for
+engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index).
+
 #### `metadataIndex`
 **Type:** `(storage: StorageAdapter) => MetadataIndexManager-compatible`
 
@@ -215,7 +246,7 @@ context.registerProvider('aggregation', (storage) => {
 })
 ```
 
-When provided by an optional native acceleration plugin (such as `@soulcraft/cortex`), this enables:
+When provided by an optional native acceleration plugin (such as `@soulcraft/cor`), this enables:
 - Compiled source filters (vs per-entity JS object traversal)
 - Precise MIN/MAX via sorted data structures (vs lazy recompute)
 - Parallel aggregate rebuild across CPU cores
@@ -251,7 +282,7 @@ Native msgpack encode/decode for SSTable serialization.
 
 ### Analytics Providers (Native-Only)
 
-These provider keys have **no JavaScript fallback** — they represent capabilities that require native code (SIMD, mmap, sub-microsecond latency). They are available when an optional native acceleration plugin (such as `@soulcraft/cortex`) is installed.
+These provider keys have **no JavaScript fallback** — they represent capabilities that require native code (SIMD, mmap, sub-microsecond latency). They are available when an optional native acceleration plugin (such as `@soulcraft/cor`) is installed.
 
 Use `brain.getProvider('analytics:hyperloglog')` to check availability. Returns `undefined` if no plugin provides it.
 
@@ -359,8 +390,8 @@ brainy diagnostics
 When a plugin is active, brainy automatically logs a provider summary after `init()`:
 
 ```
-[brainy] Plugin activated: @soulcraft/cortex
-[brainy] Providers: 8/10 native (@soulcraft/cortex) | default: vector, cache
+[brainy] Plugin activated: @soulcraft/cor
+[brainy] Providers: 8/10 native (@soulcraft/cor) | default: vector, cache
 ```
 
 This tells you at a glance how many subsystems are accelerated and which ones are falling back to JavaScript. The log respects `config.silent`.
@@ -381,7 +412,7 @@ If a required provider is missing, the error message tells you exactly what's wr
 
 ```
 [brainy] Required providers using JS fallback: graphIndex.
-Active plugins: @soulcraft/cortex.
+Active plugins: @soulcraft/cor.
 These providers must be supplied by a plugin for this deployment.
 Check plugin installation, license, and native module availability.
 ```

docs/PRODUCTION_SERVICE_ARCHITECTURE.md

@@ -215,9 +215,9 @@ console.log('Server running on http://localhost:3000')
 ```
 
 **Benefits:**
-- ✅ Native Bun performance (~2x faster than Node.js)
+- ✅ Native Bun runtime performance
 - ✅ No framework dependencies
-- ✅ Works with `bun --compile` for single-binary deployment
+- ✅ Pure WASM — no native binaries, bundler-friendly
 - ✅ Built-in TypeScript support
 
 ### Pattern 4: Express/Node.js Middleware (Legacy)

docs/QUERY_OPERATORS.md

@@ -10,8 +10,8 @@ All operators work with `find({ where: { ... } })` and filter on **metadata fiel
 
 | Operator | Alias | Description | Example |
 |----------|-------|-------------|---------|
-| `equals` | `eq`, `is` | Exact match | `{ status: { equals: 'active' } }` |
-| `notEquals` | `ne`, `isNot` | Not equal | `{ status: { notEquals: 'deleted' } }` |
+| `eq` | `equals` | Exact match | `{ status: { eq: 'active' } }` |
+| `ne` | `notEquals` | Not equal | `{ status: { ne: 'deleted' } }` |
 
 **Shorthand:** A bare value is treated as `equals`:
 
@@ -27,10 +27,10 @@ brain.find({ where: { status: { equals: 'active' } } })
 
 | Operator | Alias | Description | Example |
 |----------|-------|-------------|---------|
-| `greaterThan` | `gt` | Greater than | `{ age: { greaterThan: 18 } }` |
-| `greaterEqual` | `gte` | Greater or equal | `{ score: { greaterEqual: 90 } }` |
-| `lessThan` | `lt` | Less than | `{ price: { lessThan: 100 } }` |
-| `lessEqual` | `lte` | Less or equal | `{ rating: { lessEqual: 3 } }` |
+| `gt` | `greaterThan` | Greater than | `{ age: { gt: 18 } }` |
+| `gte` | `greaterThanOrEqual` | Greater or equal | `{ score: { gte: 90 } }` |
+| `lt` | `lessThan` | Less than | `{ price: { lt: 100 } }` |
+| `lte` | `lessThanOrEqual` | Less or equal | `{ rating: { lte: 3 } }` |
 | `between` | — | Inclusive range `[min, max]` | `{ year: { between: [2020, 2025] } }` |
 
 ```typescript
@@ -160,9 +160,9 @@ Brainy's MetadataIndex supports a subset of operators natively for O(1) field lo
 | `equals` / `eq` | Yes | Yes |
 | `notEquals` / `ne` | — | Yes |
 | `greaterThan` / `gt` | Yes | Yes |
-| `greaterEqual` / `gte` | Yes | Yes |
+| `greaterThanOrEqual` / `gte` | Yes | Yes |
 | `lessThan` / `lt` | Yes | Yes |
-| `lessEqual` / `lte` | Yes | Yes |
+| `lessThanOrEqual` / `lte` | Yes | Yes |
 | `between` | Yes | Yes |
 | `oneOf` / `in` | Yes | Yes |
 | `noneOf` | — | Yes |
@@ -234,7 +234,7 @@ const all = await brain.related({
 })
 
 // Graph traversal — subtype filters traversal edges (depth-1 in 7.30 JS path;
-// multi-hop subtype filtering lands on Cortex native)
+// multi-hop subtype filtering lands on Cor native)
 const reports = await brain.find({
   connected: {
     from: ceoId,

docs/RELEASE-GUIDE.md

@@ -119,4 +119,13 @@ gh release create vY.Y.Y --generate-notes
 - **Most releases should be MINOR or PATCH**
 - **Major versions should be RARE**
 - **When in doubt, it's probably MINOR**
-- **NEVER use "BREAKING CHANGE" for internal changes**
+- **NEVER use "BREAKING CHANGE" for internal changes**
+## Hard Ordering Constraints (check before EVERY release)
+
+- **Embedding model changes are SEQUENCED, not free.** No release may change the
+  embedding model (or its quantization/dimensions) before **vector model-version
+  stamping + hard-error-on-mismatch** ships. Stored vectors carry no model version
+  today; mixing vectors from two models silently corrupts every similarity
+  comparison. If a model bump is ever proposed, the stamping work moves ahead of
+  it in the schedule — coordinate with the native provider so both engines stamp
+  and enforce identically. (Registered with the native-provider team 2026-07-07.)

docs/SCALING.md

@@ -37,7 +37,7 @@ The three knobs that matter most:
 1. **`config.vector.recall`** — `'fast'`, `'balanced'`, or `'accurate'` (default `'balanced'`)
 2. **`config.vector.persistMode`** — `'immediate'` for durability, `'deferred'` for throughput
 
-The native vector provider (via the optional `@soulcraft/cortex` package) extends this with a higher-performing index — and its own at-scale acceleration such as on-disk compressed indexing — when installed.
+The native vector provider (via the optional `@soulcraft/cor` package) extends this with a higher-performing index — and its own at-scale acceleration such as on-disk compressed indexing — when installed.
 
 ## Measured Performance
 
@@ -77,7 +77,7 @@ graph, metadata index, and 100k edges).
 **Scale ceiling (open-core).** The pure-JS HNSW *build* cost (~100 inserts/s at 384-dim on
 one core) makes the in-process open-core path most appropriate up to ~10⁵–10⁶ entities.
 *Query* latency stays low well beyond that, but for the 10⁸–10¹⁰ regime install the native
-provider (`@soulcraft/cortex`, on-disk DiskANN) — same API, no code change. _Projected from
+provider (`@soulcraft/cor`, on-disk DiskANN) — same API, no code change. _Projected from
 the two measured points, vector p50 at 1M is ~2 ms; metadata-heavy composition grows with
 match-set size and is the path to move onto the native provider first._
 
@@ -213,12 +213,12 @@ const stats = await brain.stats()
 ### Issue: Slow queries
 1. Switch to `vector.recall: 'fast'`
 2. Increase the read cache (`storage.cache.maxSize`)
-3. Consider the optional native vector provider via `@soulcraft/cortex`
+3. Consider the optional native vector provider via `@soulcraft/cor`
 
 ### Issue: Memory pressure
 1. Reduce `storage.cache.maxSize`
 2. Move to `vector.persistMode: 'deferred'` to batch writes
-3. Consider the optional native vector provider via `@soulcraft/cortex` for at-scale index acceleration
+3. Consider the optional native vector provider via `@soulcraft/cor` for at-scale index acceleration
 
 ### Issue: Slow startup after a crash
 1. Use `vector.persistMode: 'immediate'` so the index file stays in sync with storage

docs/api/README.md

@@ -413,9 +413,9 @@ Brainy uses clean, readable operators (BFO — Brainy Field Operators):
 | `equals` / `eq` | Exact match | `{age: {equals: 25}}` |
 | `notEquals` / `ne` | Not equal | `{status: {notEquals: 'deleted'}}` |
 | `greaterThan` / `gt` | Greater than | `{age: {greaterThan: 18}}` |
-| `greaterEqual` / `gte` | Greater or equal | `{score: {greaterEqual: 90}}` |
+| `gte` / `greaterThanOrEqual` | Greater or equal | `{score: {gte: 90}}` |
 | `lessThan` / `lt` | Less than | `{price: {lessThan: 100}}` |
-| `lessEqual` / `lte` | Less or equal | `{rating: {lessEqual: 3}}` |
+| `lte` / `lessThanOrEqual` | Less or equal | `{rating: {lte: 3}}` |
 | `between` | Inclusive range | `{year: {between: [2020, 2025]}}` |
 | `oneOf` / `in` | In array | `{color: {oneOf: ['red', 'blue']}}` |
 | `noneOf` | Not in array | `{status: {noneOf: ['deleted']}}` |
@@ -901,16 +901,18 @@ that generation), once per `Db`, freed on `release()`.
 
 **Throws:** `GenerationCompactedError` when the generation's records were reclaimed by `compactHistory()`.
 
-**History granularity:** only `transact()` batches produce historical
-records; single-operation writes advance the clock but stay visible through
-earlier pins. See the [consistency model](../concepts/consistency-model.md).
+**History granularity:** every write is its own immutable generation —
+`transact()` batches AND single-operation writes — so a pin always freezes and
+every write is addressable via `asOf()`. See the
+[consistency model](../concepts/consistency-model.md).
 
 ---
 
 ### `transactionLog(options?)` → `Promise<TxLogEntry[]>`
 
-Read the reified transaction log — one entry per committed `transact()`
-batch, newest first: `{ generation, timestamp, meta? }`.
+Read the reified transaction log — one entry per committed generation (every
+`transact()` AND single-op write), newest first: `{ generation, timestamp,
+meta? }`. Single-op generations carry no `meta` (it is a `transact()`-only field).
 
 ```typescript
 const [latest] = await brain.transactionLog({ limit: 1 })
@@ -921,13 +923,15 @@ latest.meta // { author: 'order-service', requestId: 'req-9f2' }
 
 ### `compactHistory(options?)` → `Promise<CompactHistoryResult>`
 
-Reclaim historical record-sets that no retention rule and no live `Db` pin
-protects. Pinned reads stay correct across compaction, always.
+Reclaim historical record-sets that no retention cap and no live `Db` pin
+protects. Pinned reads stay correct across compaction, always. (Auto-compaction
+on `flush()`/`close()` is governed by the constructor `retention` knob — unset →
+adaptive, `'all'` → unbounded, `{ … }` → explicit caps.)
 
 ```typescript
 await brain.compactHistory({
-  retainGenerations: 100,            // keep the 100 most recent commits
-  retainMs: 7 * 24 * 60 * 60 * 1000  // and everything from the last 7 days
+  maxGenerations: 100,               // keep at most the 100 most recent generations
+  maxAge: 7 * 24 * 60 * 60 * 1000    // and only those from the last 7 days
 })
 ```
 

docs/architecture/aggregation.md

@@ -33,7 +33,7 @@
 │             ▼                                            │
 │  ┌──────────────────────┐                                │
 │  │  AggregationProvider │  (optional, registered by      │
-│  │  ├─ incrementalUpdate│   plugin like @soulcraft/cortex)│
+│  │  ├─ incrementalUpdate│   plugin like @soulcraft/cor)   │
 │  │  ├─ rebuildAggregate │                                │
 │  │  ├─ queryAggregate   │                                │
 │  │  └─ serialize/restore│                                │
@@ -135,7 +135,7 @@ M2 is clamped to zero on remove to prevent floating-point drift from producing n
 
 The TypeScript engine uses simple comparison for add operations and marks MIN/MAX as potentially stale on delete (since removing the current min/max value requires a rescan). Stale values are lazily recomputed on the next query.
 
-The Cortex native engine uses a `BTreeMap<OrderedFloat<f64>, u64>` that tracks the exact frequency of every value, providing precise MIN/MAX after any sequence of operations without rescanning.
+The Cor native engine uses a `BTreeMap<OrderedFloat<f64>, u64>` that tracks the exact frequency of every value, providing precise MIN/MAX after any sequence of operations without rescanning.
 
 ### Time Window Bucketing
 

docs/architecture/data-storage-architecture.md

@@ -369,7 +369,10 @@ await brain.restore('/backups/today', { confirm: true })   // replace store stat
   through the metadata index, not the filesystem.
 - **`_system/`** holds singletons plus 256 hash buckets of index state.
 - **`_generations/` + `manifest.json` + `tx-log.jsonl`** implement
-  generational MVCC; `transact()` is the unit of history.
+  generational MVCC. History is per-write: every `add()`/`update()`/`remove()`/
+  `relate()` gets its own generation, and `transact()` groups several ops into
+  one atomic generation. (Single-op retention has been the model since 8.0;
+  you never need to route a write through `transact()` just to keep its history.)
 - **`_column_index/` + `_blobs/`** hold the columnar metadata runs and binary
   blobs (VFS content included).
 - **`locks/`** coordinates the single writer and reader flush requests, and

docs/architecture/index-architecture.md

@@ -183,7 +183,7 @@ async getIdsForMultipleFields(pairs: [{field, value}, ...]): Promise<string[]> {
 - Memory usage: **90% reduction** (17.17 MB → 2.01 MB for 100K entities)
 - Hardware acceleration: SIMD instructions make bitmap operations nearly free
 
-**Benchmark Results** (1,000 queries on various dataset sizes):
+**Benchmark Results** — example output from a single run of `tests/performance/roaring-bitmap-benchmark.ts` (1,000 queries per size, one machine; absolute times vary by hardware, the relative speedup and memory savings are the durable signal):
 | Dataset Size | Operation | Set Time | Roaring Time | Speedup | Memory Savings |
 |--------------|-----------|----------|--------------|---------|----------------|
 | 10,000 entities | 3-field intersection | 3.74ms | 1.14ms | **3.3x faster** | 90% |
@@ -402,7 +402,7 @@ const DEFAULT_EXCLUDE_FIELDS = [
 
 **Purpose**: O(log n) semantic similarity search using vector embeddings.
 
-The default JS implementation is `JsHnswVectorIndex`; an optional native acceleration package (`@soulcraft/cortex`) can register a higher-performing `VectorIndexProvider` through the plugin system. The public API stays the same either way.
+The default JS implementation is `JsHnswVectorIndex`; an optional native acceleration package (`@soulcraft/cor`) can register a higher-performing `VectorIndexProvider` through the plugin system. The public API stays the same either way.
 
 ### Internal Architecture
 
@@ -578,16 +578,6 @@ const reachable = await this.graphIndex.traverse({
 // Complexity: O(V + E) breadth-first search, but each neighbor lookup is O(1)
 ```
 
-## Notes on Other Indexes
-
-### DeletedItemsIndex (Not Currently Used)
-
-**Status**: Utility class exists in `src/utils/deletedItemsIndex.ts` but is **not instantiated** in the Brainy class.
-
-**Purpose** (if used): O(1) tracking of soft-deleted items without removing data.
-
-**Current Behavior**: Brainy uses hard deletes via `storage.deleteNoun()` and `storage.deleteVerb()`. Soft-delete functionality is not currently integrated.
-
 ## Shared Memory Management: UnifiedCache
 
 All three main indexes share a single **UnifiedCache** instance for coordinated memory management.
@@ -684,9 +674,6 @@ async find(query: FindQuery): Promise<Result[]> {
     results = results.filter(r => connectedIds.includes(r.id))
   }
 
-  // Step 4: Filter deleted items
-  results = results.filter(r => !this.deletedItemsIndex.isDeleted(r.id))
-
   return results
 }
 ```
@@ -728,9 +715,6 @@ async stats(): Promise<Statistics> {
     relationships: this.graphIndex.getTotalRelationshipCount(),
     relationshipTypes: this.graphIndex.getRelationshipCountsByType(),
 
-    // From deleted items index
-    deletedItems: this.deletedItemsIndex.getDeletedCount(),
-
     // From vector index
     vectorIndexSize: this.index.getSize()
   }
@@ -858,14 +842,15 @@ Where:
 
 ### Scalability
 
-All indexes scale gracefully:
+All indexes scale gracefully. The cost of each stage is governed by its algorithmic complexity, not a fixed millisecond figure — absolute latency depends on hardware, embedding model, and storage backend. Only the graph adjacency index carries a committed scale assertion:
 
-| Database Size | Metadata Filter | Vector Search | Graph Hop | Combined Query |
-|---------------|----------------|---------------|-----------|----------------|
-| **1K entities** | 0.3ms | 0.8ms | 0.05ms | 1.1ms |
-| **10K entities** | 0.5ms | 1.2ms | 0.08ms | 1.5ms |
-| **100K entities** | 0.8ms | 1.8ms | 0.1ms | 2.1ms |
-| **1M entities** | 1.2ms | 2.5ms | 0.1ms | 2.8ms |
+| Query stage | Complexity | Scaling behavior |
+|-------------|------------|------------------|
+| Metadata filter (exact) | O(1) | Constant — independent of dataset size |
+| Metadata filter (range) | O(log n) + O(k) | Sub-linear; k = matching results |
+| Vector search (HNSW) | O(log n) | Degrades gracefully via hierarchical layers |
+| Graph hop | O(1) | Measured <1 ms per neighbor lookup, validated up to 1M relationships (`tests/performance/graph-scale-performance.test.ts:238`) |
+| Combined query | O(log n) | Bounded by the vector stage; metadata and graph stages stay O(1)/O(log n) |
 
 **Key observations**:
 - Graph queries stay O(1) regardless of scale

docs/architecture/multiprocess-storage-mixin.md

@@ -22,7 +22,7 @@ requestFlushOverFilesystem(timeoutMs)
 
 They live on `BaseStorage` as no-op defaults and are overridden on
 `FileSystemStorage` with real implementations. Any adapter extending
-`FileSystemStorage` (e.g. Cortex's `MmapFileSystemStorage`) inherits the
+`FileSystemStorage` (e.g. Cor's `MmapFileSystemStorage`) inherits the
 real ones for free.
 
 This works correctly today. The question is whether the methods *belong*
@@ -83,7 +83,7 @@ Benefits:
 ## The case against doing it now
 
 - Breaking change for any adapter that already overrides these methods.
-  `FileSystemStorage` is the only one in-tree, but Cortex's
+  `FileSystemStorage` is the only one in-tree, but Cor's
   `MmapFileSystemStorage` inherits from it — interface relocation would
   ripple through the plugin ecosystem.
 - The current state works. The real failure modes seen in the field

docs/architecture/overview.md

@@ -48,7 +48,7 @@ Pluggable vector index (`VectorIndexProvider`) for efficient nearest-neighbor se
 - **Configurable recall**: `fast` / `balanced` / `accurate` presets trade recall for latency
 - **Scalable**: Handles millions of vectors per process
 - **Persistent**: Serializable to storage
-- **Swappable**: Replace with a native implementation (such as `@soulcraft/cortex`) via the plugin system without changing application code
+- **Swappable**: Replace with a native implementation (such as `@soulcraft/cor`) via the plugin system without changing application code
 
 ### Metadata Index Manager
 High-performance field indexing system:

docs/architecture/triple-intelligence.md

@@ -23,29 +23,37 @@ Traditional databases force you to choose between vector search, graph traversal
 
 ### Unified Query Structure
 
+`find()` accepts a single `FindParams` object (or a natural-language string). One
+object combines all three intelligences:
+
 ```typescript
-interface TripleQuery {
-  // Vector/Semantic search
-  like?: string | Vector | any
-  similar?: string | Vector | any
-  
-  // Graph/Relationship search  
+interface FindParams {
+  // Vector intelligence — semantic similarity
+  query?: string                 // Natural-language / semantic query (embedded, matched via HNSW + text index)
+  vector?: number[]              // Pre-computed embedding for direct vector search
+
+  // Metadata intelligence — structured field filters
+  type?: NounType | NounType[]   // Filter by entity type
+  subtype?: string | string[]    // Filter by per-product subtype
+  where?: Record<string, any>    // Field predicates with bare operators (gte, lt, in, contains, exists…)
+
+  // Graph intelligence — relationship traversal
   connected?: {
-    to?: string | string[]
-    from?: string | string[]
-    type?: string | string[]
-    depth?: number
+    to?: string                  // Reachable to this entity
+    from?: string                // Reachable from this entity
+    via?: VerbType | VerbType[]  // Relationship type(s) to traverse (alias: type)
+    depth?: number               // Max traversal depth (default: 1)
     direction?: 'in' | 'out' | 'both'
   }
-  
-  // Field/Attribute search
-  where?: Record<string, any>
-  
-  // Advanced options
-  limit?: number
-  boost?: 'recent' | 'popular' | 'verified' | string
-  explain?: boolean
-  threshold?: number
+
+  // Proximity — nearest neighbours of a known entity
+  near?: { id: string; threshold?: number }
+
+  // Control
+  limit?: number                 // Max results (default: 10)
+  offset?: number                // Skip N results
+  orderBy?: string               // Field to sort by (e.g. 'createdAt')
+  order?: 'asc' | 'desc'         // Sort direction
 }
 ```
 
@@ -74,10 +82,10 @@ const results = await brain.find("machine learning concepts")
 #### Combined Intelligence Query
 ```typescript
 const results = await brain.find({
-  like: "neural networks",
+  query: "neural networks",
   where: {
     category: "research",
-    year: { $gte: 2023 }
+    year: { gte: 2023 }
   },
   connected: {
     to: "deep-learning-team",
@@ -109,8 +117,8 @@ All three search types execute simultaneously:
 ```typescript
 // Parallel execution for balanced query
 const results = await brain.find({
-  like: "AI research",           // ~1000 potential matches
-  where: { type: "paper" },       // ~500 potential matches
+  query: "AI research",          // ~1000 potential matches
+  where: { kind: "paper" },       // ~500 potential matches
   connected: { to: "stanford" }   // ~200 potential matches
 })
 // All three execute in parallel, results fused
@@ -126,7 +134,7 @@ Operations chain for maximum efficiency:
 // Progressive execution for selective query
 const results = await brain.find({
   where: { userId: "user123" },   // Very selective (1-10 matches)
-  like: "recent posts",            // Applied to filtered set
+  query: "recent posts",           // Applied to filtered set
   limit: 5
 })
 // Metadata filter first, then vector search on results
@@ -166,10 +174,10 @@ const results = await brain.find(
 )
 // Automatically converts to:
 // {
-//   like: "AI papers",
-//   where: { 
+//   query: "AI papers",
+//   where: {
 //     institution: "Stanford",
-//     published: { $gte: "2024-01-01" }
+//     published: { gte: "2024-01-01" }
 //   }
 // }
 ```
@@ -188,10 +196,10 @@ The NLP processor identifies query intent:
 
 Successful execution plans are cached:
 ```typescript
-// First query: 50ms (plan generation + execution)
+// First call parses the natural-language query and builds an execution plan
 await brain.find("machine learning papers")
 
-// Subsequent similar queries: 10ms (cached plan)
+// A structurally similar query reuses that plan, skipping plan generation
 await brain.find("deep learning papers")
 ```
 
@@ -213,50 +221,45 @@ Triple Intelligence leverages all available indexes:
 
 ### Explain Mode
 
-Understand how your query was executed:
+Diagnose how a query's `where` fields map to the index. Run `brain.explain()`
+first whenever `find()` returns surprising or empty results:
 
 ```typescript
-const results = await brain.find({
-  like: "quantum computing",
-  where: { category: "research" },
-  explain: true
+const plan = await brain.explain({
+  query: "quantum computing",
+  where: { category: "research" }
 })
 
-console.log(results[0].explanation)
-// {
-//   plan: "field-first-progressive",
-//   timing: {
-//     fieldFilter: 2,
-//     vectorSearch: 8,
-//     fusion: 1
-//   },
-//   selectivity: {
-//     field: 0.1,
-//     vector: 0.3
-//   }
-// }
+console.log(plan.fieldPlan)
+// [
+//   { field: 'category', path: 'column-store', notes: '...' }
+// ]
+
+console.log(plan.warnings)
+// e.g. ['Field "category" has no index entries. find() will return [] silently...']
 ```
 
-### Boosting
+### Result Ordering
 
-Apply custom ranking boosts:
+Sort results by any stored field with `orderBy` / `order`:
 
 ```typescript
 const results = await brain.find({
-  like: "news articles",
-  boost: 'recent',  // Boost recent items
-  where: { verified: true }
+  query: "news articles",
+  where: { verified: true },
+  orderBy: 'createdAt',  // Newest first
+  order: 'desc'
 })
 ```
 
-### Threshold Control
+### Similarity Threshold
 
-Set minimum similarity thresholds:
+Find the nearest neighbours of a known entity and keep only close matches with
+`near`:
 
 ```typescript
 const results = await brain.find({
-  like: "exact match needed",
-  threshold: 0.9,  // Only very similar results
+  near: { id: anchorId, threshold: 0.9 },  // Only results >= 0.9 similarity
   limit: 10
 })
 ```
@@ -283,8 +286,8 @@ const results = await brain.find({
 ```typescript
 // Find similar content with constraints
 const results = await brain.find({
-  like: query,
-  where: { 
+  query: searchText,
+  where: {
     status: 'published',
     language: 'en'
   }
@@ -295,10 +298,10 @@ const results = await brain.find({
 ```typescript
 // Find items related to a specific item
 const results = await brain.find({
-  connected: { 
+  connected: {
     to: itemId,
     depth: 2,
-    type: 'similar'
+    via: VerbType.RelatedTo
   },
   limit: 20
 })
@@ -309,10 +312,11 @@ const results = await brain.find({
 // Recent items matching criteria
 const results = await brain.find({
   where: {
-    timestamp: { $gte: Date.now() - 86400000 }
+    timestamp: { gte: Date.now() - 86400000 }
   },
-  like: "trending topics",
-  boost: 'recent'
+  query: "trending topics",
+  orderBy: 'timestamp',
+  order: 'desc'
 })
 ```
 

docs/architecture/zero-config.md

@@ -84,7 +84,7 @@ const brain = new Brainy({
 ```
 
 The default JS index is `JsHnswVectorIndex`. An optional native acceleration
-provider (the `@soulcraft/cortex` package) can replace it with a
+provider (the `@soulcraft/cor` package) can replace it with a
 higher-performing implementation; the public knobs stay the same. Quantization
 and other index-internal acceleration are the native provider's concern, not a
 Brainy configuration option.

docs/concepts/consistency-model.md

@@ -212,10 +212,12 @@ Historical records cost disk space, so retention is explicit:
 - Every live `Db` holds a refcounted **pin**; a record-set is never
   reclaimed while any pin could need it — pinned reads stay correct across
   compaction, always.
-- `brain.compactHistory({ retainGenerations?, retainMs? })` reclaims
-  everything no retention rule and no pin protects, and records the
-  **horizon** — `asOf()` below it throws `GenerationCompactedError`,
-  explicitly, never partial data.
+- The **`retention`** knob governs auto-compaction (on every `flush()`/
+  `close()`): unset → ADAPTIVE (disk/RAM-pressure, zero-config) · `'all'` →
+  unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit caps.
+  `brain.compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
+  manually on the same caps, and records the **horizon** — `asOf()` below it
+  throws `GenerationCompactedError`, explicitly, never partial data.
 - To keep a state readable forever, `persist()` it first: snapshots are
   self-contained and unaffected by compaction of the source store.
 
@@ -357,9 +359,13 @@ Stated plainly, so nothing surprises you in production:
   ([multi-process model](./multi-process.md)). Transactions are atomic
   within one writer process — there is no distributed or cross-process
   transaction coordination.
-- **History granularity.** Only `transact()` batches produce historical
-  records; single-operation writes between commits stay visible through
-  earlier pins (see above).
+- **History granularity.** Every write is its own immutable generation —
+  `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
+  A pin always freezes against later writes, and every write is addressable
+  via `asOf()`. (`transact()` groups several operations into ONE atomic
+  generation; durability of single-op history is batched via async
+  group-commit — a hard crash can lose only the last un-flushed window's
+  *history*, never live data.)
 - **Compacted history is gone.** `asOf()` below the compaction horizon
   fails explicitly; persist what you must keep.
 - **Counter persistence is coalesced for single-operation writes.** Durable

docs/concepts/generation-fact-log.md

@@ -0,0 +1,117 @@
+---
+title: The Generation Fact Log
+slug: concepts/generation-fact-log
+public: true
+category: concepts
+template: concept
+order: 6
+description: Every committed write also appends a self-verifying "fact" — an after-image commit record — to an append-only log. What facts are, the crash-safety model, the scanFacts() streaming surface, family stamps, and how index providers consume the log for sequential heals.
+next:
+  - concepts/consistency-model
+  - guides/snapshots-and-time-travel
+---
+
+# The Generation Fact Log
+
+Since 8.4.0, every committed generation also appends a **fact** — a compact record of what each
+touched entity or relationship *became* — to an append-only, checksummed log under
+`_generations/facts/`. Where the generational history answers *"what did things look like
+before?"* (before-images, powering `asOf()` and rollback), the fact log answers *"what happened,
+in order?"* — one sequential, self-verifying stream of the store's present being written.
+
+Nothing about querying changes. The fact log exists for three consumers:
+
+1. **Index heals and rebuilds** — one sequential read in commit order replaces a per-entity
+   directory walk over millions of files.
+2. **Incremental catch-up** — a derived index that knows which generation it reflects reads *just
+   the gap*, instead of rebuilding from scratch.
+3. **Replay and audit tooling** — anything that wants the store's committed timeline as a stream.
+
+## What a fact is
+
+One fact per committed generation:
+
+- **`generation`** and **`timestamp`** — which commit, when.
+- **`ops`** — every write in that commit: `{ kind: 'noun' | 'verb', id, record }` where `record`
+  holds the entity's full after-image (both stored legs), or **`null` for a tombstone** — a
+  removal carries no body, by design.
+- **`meta`** — the transaction metadata `transact()` was submitted with, when present.
+- **`blobHashes`** — content-blob references, for exact reclamation accounting.
+
+Facts accumulate **from the first write after upgrading** — pre-existing history is not
+retroactively converted, and consumers fall back to the enumeration walk when no log exists.
+
+## Crash safety, in one paragraph
+
+Facts are appended and fsynced **inside the same durability window as the commit itself**, before
+the commit point — so after a crash, the log can only ever be *ahead* of committed truth, never
+behind it with a hole. On open, the store reconciles the log back to the committed watermark:
+torn tails are detected by per-record checksums and cut; whole records beyond the watermark are
+truncated. The invariant every reader can rely on: **an absent generation was never committed; a
+present fact was.** `transact()` facts are durable the moment `transact()` returns; single-op
+facts share the same group-commit flush as the rest of their generation, so a hard kill loses the
+fact and the generation *together* — never a torn state.
+
+## Reading the log
+
+```typescript
+const scan = brain.scanFacts({ fromGeneration: 1 })
+if (scan) {
+  // Telemetry up front — progress bars get a denominator from second zero.
+  console.log(scan.headGeneration, scan.segmentCount, scan.approxFactCount)
+
+  for await (const batch of scan.batches()) {
+    // Each batch: { facts, firstGeneration, lastGeneration, factCount, byteSize, segmentId }
+    for (const fact of batch.facts) {
+      for (const op of fact.ops) {
+        if (op.record === null) {
+          // a tombstone: op.id was removed in this generation
+        }
+      }
+    }
+  }
+
+  console.log(scan.summary()) // { factsYielded, segmentsRead } — the cross-check
+}
+```
+
+- `scanFacts()` returns `null` when the store hosts no fact log (older store, or a storage adapter
+  without binary append support) — fall back to enumerating entities.
+- Scans run against a **snapshot**: facts appended after the scan opens never bleed in, each fact
+  is yielded exactly once, and a detected gap aborts loudly — never a silent skip.
+- `brain.factSegmentPaths()` returns the immutable, *sealed* segment files for zero-copy consumers
+  (the append-mutable tail is excluded — read it through `scanFacts()`).
+
+## Family stamps: how a projection proves it's current
+
+Anything derived from the store — an index, the entity file tree itself — carries a **family
+stamp**: a small JSON record of *which committed generation the projection reflects*
+(`sourceGeneration`) plus the invariants that verify it whole (exact per-file byte sizes for
+bounded families; rollup invariants like entity counts for unbounded ones). At open, coherence is
+a **comparison**, not a walk:
+
+- stamp equals the committed watermark and invariants hold → serve;
+- stamp is behind → the projection reads just the gap from the fact log;
+- invariants fail → loud, named divergence — `brain.repairIndex()` rebuilds from canonical and
+  re-stamps.
+
+The verifier is exported (`verifyFamilyStamp`) so every projection — TypeScript or native — runs
+literally the same check.
+
+## For plugin authors: the storage capability
+
+Index providers receive the storage adapter, not the brain — so the host wires the log onto it.
+Feature-detect and prefer the stream; fall back to enumeration:
+
+```typescript
+const scan = storage.scanFacts?.({ fromGeneration: stamp.sourceGeneration + 1 })
+if (scan) {
+  // sequential catch-up from the log
+} else {
+  // enumeration walk (older store or adapter)
+}
+const committed = storage.committedGeneration?.() // the watermark stamps compare against
+```
+
+Providers must never construct their own reader over the log's files — the open path belongs to
+the single writer (it reconciles the log at open); the capability is the sanctioned seam.

docs/concepts/multi-process.md

@@ -55,20 +55,20 @@ process opening the same directory:
 The fix is the lock: refuse to open a second writer. SQLite has done the same
 since the late 1990s (`SQLITE_BUSY`).
 
-## What about Cortex?
+## What about Cor?
 
-Brainy + Cortex compose cleanly under this model:
+Brainy + Cor compose cleanly under this model:
 
-- Cortex stores its column-index segments inside the same `rootDir` (under
+- Cor stores its column-index segments inside the same `rootDir` (under
   `indexes/_column_index/{field}/`).
-- Segments (`*.cidx` files) are **immutable** once written. Cortex mmaps them
+- Segments (`*.cidx` files) are **immutable** once written. Cor mmaps them
   read-only.
 - The `MANIFEST.json` per field is updated via atomic rename — readers see
   either the old or new manifest, never a torn file.
 
-A reader process can safely mmap Cortex segments alongside a live writer
+A reader process can safely mmap Cor segments alongside a live writer
 without coordination. The single Brainy writer lock at
-`<rootDir>/locks/_writer.lock` covers Cortex too, because Cortex segment
+`<rootDir>/locks/_writer.lock` covers Cor too, because Cor segment
 writes happen on the writer's side.
 
 ## Stale-lock detection
@@ -141,7 +141,7 @@ The CLI `brainy inspect` subcommands all do this for you by default
   processes can both succeed at `init()` in writer mode and clobber each
   other's writes. A best-effort warning is logged in writer mode against a
   non-filesystem backend.
-- **Long-running readers** do not automatically pick up new Cortex segments
+- **Long-running readers** do not automatically pick up new Cor segments
   the writer publishes. One-shot inspector calls re-open the store and see
   fresh segments; a reader that stays open for hours sees its column store
   as-of the time it opened.
@@ -150,4 +150,4 @@ The CLI `brainy inspect` subcommands all do this for you by default
 
 - `Brainy.openReadOnly()` — [API reference](../api/brainy.md)
 - `brainy inspect` — [inspection guide](../guides/inspection.md)
-- Cortex columnar storage — see `node_modules/@soulcraft/cortex/README.md`
+- Cor columnar storage — see `node_modules/@soulcraft/cor/README.md`

docs/concepts/storage-adapters.md

@@ -33,7 +33,7 @@ BaseStorage                (type-statistics, lifecycle helpers, generation
 FileSystemStorage          (real filesystem I/O, writer-lock implementation,
                             flush-request watcher, atomic writes)
         ↑
-<your plugin's storage>    (Cortex's MmapFileSystemStorage, etc.)
+<your plugin's storage>    (Cor's MmapFileSystemStorage, etc.)
 ```
 
 When you `extend FileSystemStorage`, your adapter inherits every method on

docs/eli5.md

@@ -11,7 +11,7 @@ next:
   - getting-started/quick-start
 ---
 
-# Brainy and Cortex — Explained Simply
+# Brainy and Cor — Explained Simply
 
 *A plain-language guide for anyone who wants to understand what this thing actually does.*
 
@@ -65,13 +65,13 @@ Brainy can narrow any result set down by exact labels or ranges in the same brea
 
 ---
 
-## What is Cortex?
+## What is Cor?
 
-Cortex is a turbocharger for Brainy.
+Cor is a turbocharger for Brainy.
 
 Same car. Same controls. Same fuel. You just swap in a faster engine under the hood, and everything that used to take a moment now happens instantly.
 
-Technically, Cortex is an optional plugin written in Rust — a lower-level language that runs much closer to the raw metal of your processor. It plugs into Brainy and takes over the most compute-intensive work: the distance calculations that power meaning search, the number-crunching behind live aggregates, and the set operations that drive label filtering.
+Technically, Cor is an optional plugin written in Rust — a lower-level language that runs much closer to the raw metal of your processor. It plugs into Brainy and takes over the most compute-intensive work: the distance calculations that power meaning search, the number-crunching behind live aggregates, and the set operations that drive label filtering.
 
 You install it with one line, register it with one call, and Brainy automatically uses it everywhere it can help.
 
@@ -83,9 +83,9 @@ Plain language:
 
 - **Searches** go from "the blink of an eye" to "faster than a blink." The overall speedup is **5.2× on average** across all operations.
 - **Live aggregates** are rebuilt using all CPU cores in parallel, so re-indexing large datasets takes a fraction of the time.
-- **Analytics** that aren't even possible in pure JavaScript — real-time anomaly detection, streaming percentile estimates, approximate unique counts — become available because Cortex brings the native capabilities required to run them efficiently.
+- **Analytics** that aren't even possible in pure JavaScript — real-time anomaly detection, streaming percentile estimates, approximate unique counts — become available because Cor brings the native capabilities required to run them efficiently.
 
-If Brainy is what makes knowledge fast, Cortex is what makes Brainy feel instant.
+If Brainy is what makes knowledge fast, Cor is what makes Brainy feel instant.
 
 ---
 
@@ -135,7 +135,7 @@ Brainy is the only row with every box checked. And it runs all of them in a sing
 
 Brainy scales from a quick experiment to serious production datasets without changing a line of code. Small datasets live entirely in memory. Larger ones spill to disk, where Brainy shards and compresses everything automatically. Need a backup or a copy? Snapshots are instant — the same API the whole way.
 
-Add Cortex and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows.
+Add Cor and you also unlock memory-mapped storage — aggregate state lives directly in the operating system's memory with zero serialization overhead, as fast as the hardware allows.
 
 ---
 

docs/guides/aggregation.md

@@ -11,7 +11,13 @@ No batch jobs. No scheduled recalculations. Aggregates stay current with every w
 **Defining over existing data:** if you define an aggregate on a store that already holds
 matching entities, Brainy backfills it from those entities on the first query (a one-time scan,
 then purely incremental). So `defineAggregate()` behaves the same whether you define it before
-or after the data exists — including when a persisted brain reopens already populated.
+or after the data exists.
+
+**Reopening a persisted brain:** aggregate state persists across restarts. Re-defining the
+same aggregate at boot (the normal declarative pattern) adopts the persisted state directly —
+no rescan. A backfill scan runs only when the definition actually changed, when no persisted
+state exists, or when the state failed to load; and however many aggregates need backfilling,
+they share a single scan.
 
 ## Quick Start
 
@@ -487,7 +493,7 @@ Aggregate definitions and running state are automatically persisted:
 
 ## Native Acceleration
 
-When [Cortex](https://github.com/soulcraftlabs/cortex) is installed as a plugin, the aggregation engine automatically uses Rust-accelerated computation:
+When [Cor](https://www.npmjs.com/package/@soulcraft/cor) is installed as a plugin, the aggregation engine automatically uses Rust-accelerated computation:
 
 - Incremental updates run in Rust with BTreeMap-backed precise MIN/MAX
 - Welford's online stddev/variance computed natively
@@ -496,7 +502,7 @@ When [Cortex](https://github.com/soulcraftlabs/cortex) is installed as a plugin,
 
 ```typescript
 const brain = new Brainy({
-  plugins: ['@soulcraft/cortex']
+  plugins: ['@soulcraft/cor']
 })
 await brain.init()
 
@@ -577,7 +583,7 @@ brain.defineAggregate({
 
 Aggregation complexity per write is O(A x G x M) where A = matching aggregates, G = groupBy dimensions, M = metrics. For typical configurations (2-5 aggregates, 1-3 dimensions, 3-5 metrics), this is effectively O(1).
 
-With Cortex native acceleration:
+With Cor native acceleration:
 
 | Operation | Throughput | Latency |
 |-----------|-----------|---------|

docs/guides/enterprise-for-everyone.md

@@ -174,13 +174,13 @@ await brain.syncWith({
 const brain = new Brainy()
 
 // 1 → ~1M vectors: pure-JS HNSW, zero extra setup
-// 1M → 10B+ vectors: install @soulcraft/cortex for the native DiskANN provider
+// 1M → 10B+ vectors: install @soulcraft/cor for the native DiskANN provider
 ```
 
 **Scaling model:**
 - **Single process, no cluster**: Brainy runs in one process — no coordinator,
   no peer discovery, no consensus to operate
-- **Optional native provider**: install `@soulcraft/cortex` to back the index with
+- **Optional native provider**: install `@soulcraft/cor` to back the index with
   on-disk DiskANN that scales to 10B+ vectors on one machine
 - **Per-tenant pools**: isolate tenants by giving each its own Brainy instance and
   storage directory

docs/guides/external-backups-and-sparse-storage.md

@@ -0,0 +1,99 @@
+---
+title: External Backups & Sparse Storage
+slug: guides/external-backups
+public: true
+category: guides
+template: guide
+order: 10
+description: How to back up a brain directory with external tools (tar, rsync, cp) without exploding sparse files — why a store can show 100+ GB "apparent" size on a small disk, which files are sparse, and how persist()/restore() handle it for you.
+next:
+  - guides/snapshots-and-time-travel
+  - concepts/storage-adapters
+---
+
+# External Backups & Sparse Storage
+
+The built-in snapshot path — [`db.persist()` and `brain.restore()`](/docs/guides/snapshots-and-time-travel) —
+already handles everything on this page for you. Read this when you back up a brain directory with
+**external tools**: `tar`, `rsync`, `cp`, `scp`, or a filesystem-level backup agent.
+
+## The one-sentence rule
+
+> **Always use the sparse-aware flag**: `tar czSf` (capital `S`), `rsync --sparse`,
+> `cp --sparse=always`. A naive copy can turn a 2 GB store into a 100+ GB one — or fail
+> the disk entirely.
+
+## Why: some files are sparse
+
+When a native accelerator plugin is active, parts of the index live in **memory-mapped files**
+created at a large fixed virtual size — the file's *apparent* size — while the filesystem only
+allocates blocks that were actually written. A brand-new id-mapper file can report tens of
+gigabytes in `ls -l` while occupying a few megabytes on disk.
+
+Check the difference yourself:
+
+```bash
+ls -lh  brain-data/_id_mapper/    # APPARENT size (can be huge)
+du -sh  brain-data/               # ALLOCATED size (the real footprint)
+```
+
+The sparse candidates in a brain directory:
+
+| Path | What it is |
+|---|---|
+| `_id_mapper/` | The native id-mapper's mmap files (large fixed virtual size) |
+| `_blobs/` | Native index files (vector base, segments) — may be mmap-backed |
+
+Everything else (entities, `_system`, `_generations`, `_cas` content blobs) is ordinary dense data.
+
+## Doing it right
+
+**tar** — the `S` flag detects holes and stores only real data:
+
+```bash
+tar czSf brain-backup.tgz /data/brain
+# restore preserves the holes:
+tar xzSf brain-backup.tgz -C /data/
+```
+
+**rsync**:
+
+```bash
+rsync -a --sparse /data/brain/ backup-host:/backups/brain/
+```
+
+**cp**:
+
+```bash
+cp -a --sparse=always /data/brain /backups/brain
+```
+
+**What goes wrong without the flag:** the copy *materializes* every hole as real zero bytes.
+A store whose apparent size exceeds the target disk fails with `ENOSPC` partway through — and a
+copy that *does* fit silently costs the full apparent size in storage and transfer time.
+
+## What the built-in paths do (so you don't have to)
+
+- **`db.persist(path)`** snapshots via **hard links** — instant and space-shared, since every data
+  file is immutable-by-rename. The handful of append-in-place files (the transaction log, the
+  commit fact log's tail segment) and mmap-mutated directories (`_id_mapper/`) are **byte-copied**
+  instead, so a post-snapshot write can never reach through a shared inode into your backup.
+- **`brain.restore(path, { confirm: true })`** is **non-destructive and sparse-aware**: the snapshot
+  is copied into a staging area *before* any live data is touched (all-zero blocks stay holes), and
+  only after the copy fully succeeds does an atomic swap move it into place. A failed copy —
+  including `ENOSPC` — leaves the live store exactly as it was. A crash mid-swap completes forward
+  on the next open.
+
+## Live-store caveats for external tools
+
+1. **Prefer snapshotting a `persist()` output, not the live directory.** `persist()` produces a
+   crash-consistent, immutable snapshot; running `tar` against a live, actively-written directory
+   can capture a torn mid-write state. If you must archive live, stop writes first (or accept that
+   the archive is only as consistent as the moment's flush state).
+2. **Never prune or "clean up" files inside a brain directory.** Index files that look stale or
+   redundant are load-bearing; the store protects its declared index families from in-process
+   deletion, but an external `rm` bypasses that fence. If space is the concern, `du -sh` first —
+   the allocated size is usually far smaller than it looks.
+3. **Verify restores by opening them.** `Brainy.load(path)` opens any snapshot or restored
+   directory read-only — the store verifies its own coherence at open and reports loudly if
+   anything is missing or torn.

docs/guides/find-limits.md

@@ -40,6 +40,10 @@ Brainy picks `maxLimit` from the first of these that's available:
 
 Worked example: a 4 GB Cloud Run container picks priority 3 → `floor(4 GB × 0.25 / 25 KB) = floor(40 960) = 40 000` results. A 900 MB free-memory box on priority 4 gets `floor(900 MB / 25 KB) = ~36 000`.
 
+The cap is fixed at construction and never changes at runtime. Query timing is recorded
+for diagnostics only — a burst of slow queries cannot silently shrink the cap, and the
+auto-detected tiers (3 and 4) never go below a floor of 10 000.
+
 > **Calibration note.** Pre-7.30.2 used 100 KB per result instead of 25 KB, which produced caps that were 4× too tight for typical workloads (an 8 KB / result reality). 7.30.2 recalibrated to match observed entity sizes; existing `limit: 10_000` safety patterns now pass silently on any reasonably-sized box.
 
 ## What happens when you exceed the cap

docs/guides/import-anything.md

@@ -300,7 +300,10 @@ await brain.import(data, {
  // Deduplication
  enableDeduplication: true, // Check for duplicate entities (default: true)
  deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85)
- // Note: Auto-disabled for imports >100 entities
+ // Notes: false disables BOTH the inline merge and the background pass that
+ // runs ~5 min after the last import (merged duplicates are deleted).
+ // The inline pass auto-disables for imports >100 entities (O(n²) cost);
+ // the background pass still covers those unless the flag is false.
 
  // Performance
  chunkSize: 100, // Batch size for processing (default: varies by operation)

docs/guides/import-quick-reference.md

@@ -86,11 +86,22 @@ await brain.import(file, {
 
 ```typescript
 await brain.import(file, {
-  enableDeduplication: true,         // Check for duplicates (default: false)
+  enableDeduplication: true,         // Check for duplicates (default: true)
   deduplicationThreshold: 0.85       // Similarity threshold (default: 0.85)
 })
 ```
 
+Deduplication merges entities judged duplicates — the non-primary records are
+**deleted**. Set `enableDeduplication: false` to disable it entirely: the flag
+gates both the inline merge during import and the background pass that runs
+about 5 minutes after the last import.
+
+```typescript
+await brain.import(file, {
+  enableDeduplication: false         // No merging, inline or background
+})
+```
+
 ### Import Tracking
 
 Track and organize imports by project:

docs/guides/inspection.md

@@ -166,6 +166,34 @@ brainy inspect diff /data/brain-prod /data/brain-staging
 Sample-based — for a full diff, dump both with `inspect dump` and compare
 the JSONL.
 
+## Auditing graph-read truth
+
+`brain.auditGraph()` (8.6.0+) proves — or disproves — that relationship reads
+return canonical truth on a given brain, without mutating anything. It walks
+every stored relationship record, asks the same read path your application
+uses (`related()`, VFS `readdir`) with every visibility tier included, and
+classifies every discrepancy:
+
+```typescript
+const report = await brain.auditGraph()
+
+report.coherent              // true = related()/readdir can be trusted on this brain
+report.missingFromReadsCount // records the read path omits — stale index
+report.danglingEndpointsCount // relationships whose endpoint entity is gone
+report.readOnlyCount         // read-path edges with NO stored record — ghosts
+report.visibilityHiddenCount // internal/system edges hidden by design (not a fault)
+```
+
+Counts are always exact; the example lists (`missingFromReads`,
+`danglingEndpoints`, `readOnlyVerbIds`) are capped at `maxExamples`
+(default 100) and `truncatedExamples` says so when they are.
+
+Run it after any engine upgrade, restore, or migration. If it reports
+discrepancies, run `brain.repairIndex()` and audit again — a `coherent`
+report after the repair is the verified statement that the heal worked.
+Cost: one relationship-record walk plus one indexed read per distinct
+source entity — safe on a live brain.
+
 ## Repairing a corrupted store
 
 If invariants fail and you suspect index corruption, `inspect repair`

docs/guides/installation.md

@@ -45,20 +45,20 @@ console.log('Brainy ready.')
 
 ## Native Acceleration (Optional)
 
-For production workloads, add Cortex for Rust-accelerated SIMD distance calculations and native embeddings:
+For production workloads, add Cor for Rust-accelerated SIMD distance calculations and native embeddings:
 
 ```bash
-npm install @soulcraft/cortex
+npm install @soulcraft/cor
 ```
 
 ```typescript
 import { Brainy } from '@soulcraft/brainy'
 
-const brain = new Brainy({ plugins: ['@soulcraft/cortex'] })
+const brain = new Brainy({ plugins: ['@soulcraft/cor'] })
 await brain.init()  // native providers registered during init
 ```
 
-Cortex registers native (Rust/SIMD) vector, metadata, and graph engines behind the same Brainy `find()` API — no code changes, an optional dependency for production-scale workloads.
+Cor registers native (Rust/SIMD) vector, metadata, and graph engines behind the same Brainy `find()` API — no code changes, an optional dependency for production-scale workloads.
 
 ## Server-only since 8.0
 

docs/guides/model-loading.md

@@ -35,15 +35,18 @@ This single WASM file contains everything needed for sentence embeddings.
 
 ### Bun (Recommended)
 
-```typescript
-// Works with Bun runtime
+```bash
+# Bun as a runtime — supported and recommended
+bun add @soulcraft/brainy
 bun run server.ts
-
-// Works with bun --compile (single binary deployment!)
-bun build --compile --target=bun server.ts
-./server  // Self-contained binary with embedded model
 ```
 
+Brainy is pure WebAssembly with no native binaries, so the module graph stays
+bundler-friendly. Single-binary `bun build --compile` is **not a supported
+target** at present: Bun 1.3.10 has a `--compile` codegen regression
+(`__promiseAll is not defined`) triggered by top-level `await` in the bundled
+graph. Run Brainy under the Bun runtime (above) instead.
+
 ### Node.js
 
 ```typescript
@@ -164,7 +167,7 @@ await brain.init()
 
 **What's new:**
 - Faster initialization
-- Works with `bun --compile`
+- Bundler-friendly (pure WASM, no native binaries)
 - No network requirements
 
 ### From Custom Embedding Functions
@@ -216,9 +219,8 @@ export { brain }
 
 ### Deployment
 ```bash
-# Option 1: Bun compile (single binary)
-bun build --compile server.ts
-./server  # Contains everything
+# Option 1: Bun runtime
+bun run server.ts
 
 # Option 2: Docker
 docker build -t my-app .

docs/guides/optimistic-concurrency.md

@@ -180,3 +180,35 @@ Brainy 8.0 has exactly two write-coordination counters, at two granularities:
 They compose: a `transact()` batch can carry per-entity `ifRev` checks *and* a whole-store `ifAtGeneration`; any failed check rejects the entire batch before anything is staged. Generations also power snapshots and time travel (`brain.now()`, `brain.asOf()`, `db.persist()`) — see the [consistency model](../concepts/consistency-model.md) and [Snapshots & Time Travel](./snapshots-and-time-travel.md).
 
 A snapshot or historical view captures each entity *including* its `_rev` at that moment, so reading the past and writing back with `ifRev` against the live state works exactly as you'd hope: the write fails if the entity moved since the state you copied from.
+
+## The transact envelope: batch size, budget, and bulk imports
+
+`transact()` applies its batch atomically under one commit — which means the whole batch
+shares one **apply budget**. Since 8.7.0 the budget scales with the batch:
+`max(30 s, opCount × 2 s)`, or exactly what you pass as `timeoutMs`. A tripped budget rolls
+the entire batch back (nothing partial survives) and throws a retryable
+`TransactionTimeoutError` that names the operation it stopped at, the batch size, and the
+elapsed vs budgeted time — a diagnosis, not just a failure:
+
+```
+Transaction timed out at operation 41/120 ('add') — 246012ms elapsed, budget 240000ms.
+The batch rolled back atomically; retry with a higher timeoutMs or a smaller batch.
+```
+
+Practical envelope guidance for bulk work:
+
+1. **Precompute embeddings outside the commit path.** Embedding inside `transact()` spends
+   the budget on model inference. Use `brain.embedBatch(texts)` and pass each vector via
+   the op's `vector` field — the commit then pays only storage costs, and a retried batch
+   never re-pays inference. (The win is *where* the inference happens, not raw embedding
+   throughput: on the default WASM engine, batch and sequential embedding measure
+   comparably, ~160 ms/text; native embedding providers may batch faster.)
+2. **Chunk very large imports** into batches of a few hundred ops with one `transact()`
+   each. You lose whole-import atomicity but keep per-chunk atomicity, bounded memory, and
+   resumability — pair with `ifAbsent` upserts so a retried chunk is idempotent.
+3. **Slow disks change the math, not the contract.** On network-attached storage a single
+   op can cost ~2 s (canonical write + fsync + index maintenance). The scaled default
+   absorbs that; pass an explicit `timeoutMs` only when you know better than the scale.
+4. **`addMany`/`relateMany` are the convenience tier** — they chunk and batch-embed for
+   you, with per-item error reporting instead of batch atomicity. Choose by what you need:
+   atomic-all-or-nothing → `transact()`; resilient bulk load → `addMany`.

docs/guides/quick-start.md

@@ -60,17 +60,17 @@ const nextId: string = await brain.add({
 await brain.relate({
   from: nextId,
   to: reactId,
-  type: VerbType.BuiltOn
+  type: VerbType.DependsOn
 })
 ```
 
 ## 5. Query with Triple Intelligence
 
 ```typescript
-import type { FindResult } from '@soulcraft/brainy'
+import type { Result } from '@soulcraft/brainy'
 
 // All three search paradigms in one call
-const results: FindResult[] = await brain.find({
+const results: Result[] = await brain.find({
   query: 'modern frontend frameworks',    // Vector similarity search
   where: { year: { greaterThan: 2015 } }, // Metadata filtering
   connected: { to: reactId, depth: 2 }   // Graph traversal

docs/guides/reacting-to-changes.md

@@ -0,0 +1,117 @@
+---
+title: Reacting to Changes
+slug: guides/reacting-to-changes
+public: true
+category: guides
+template: guide
+order: 11
+description: Subscribe to every committed mutation with `brain.onChange` — the in-process change feed behind live UIs, cache invalidation, and realtime sync. Covers the event shape, delivery guarantees, and catch-up patterns.
+next:
+  - guides/optimistic-concurrency
+  - guides/snapshots-and-time-travel
+---
+
+# Reacting to Changes
+
+`brain.onChange(cb)` is Brainy's in-process change feed: subscribe once and
+receive one event per committed mutation — **every** mutation, regardless of
+how it happened. Direct calls, batch methods, `transact()`, imports, and
+Virtual Filesystem writes all funnel through the same commit point the feed is
+emitted from, so nothing slips past it.
+
+```ts
+const off = brain.onChange((e) => {
+  if (e.kind === 'entity') {
+    console.log(`${e.op} ${e.entity?.type} ${e.id} @ generation ${e.generation}`)
+  }
+})
+
+await brain.add({ data: 'Ada Lovelace', type: 'person' })
+// → "add person 0198... @ generation 42"
+
+off() // unsubscribe when done
+```
+
+## The event
+
+```ts
+interface BrainyChangeEvent {
+  kind: 'entity' | 'relation' | 'store'
+  op: 'add' | 'update' | 'remove' | 'relate' | 'unrelate' | 'updateRelation'
+    | 'clear' | 'restore'
+  id?: string
+  entity?:   { id: string; type: string; subtype?: string;
+               metadata: Record<string, unknown>; service?: string }
+  relation?: { id: string; from: string; to: string; type: string;
+               metadata?: Record<string, unknown> }
+  generation?: number
+  timestamp: number
+}
+```
+
+- **Entity events** (`add` / `update` / `remove`) carry the post-commit indexed
+  view — `type`, `subtype`, and the full custom `metadata`, so you can match
+  your own `where`-style filters against events without a read.
+- **Deletes are fully described.** A `remove` or `unrelate` event carries the
+  record's *last committed state* (sourced from the commit's own history
+  record), not just an id.
+- **Batches emit per item.** `addMany` / `updateMany` / `relateMany` /
+  `removeMany` emit one event per affected record; a `transact()` batch emits
+  one event per item, all sharing the batch's single `generation`.
+- **Cascades are visible.** Removing an entity also emits `unrelate` for each
+  relationship the delete cascaded to.
+- **Store-level events** (`kind: 'store'`) fire for the two wholesale
+  operations — `clear()` and `restore()` — and mean *"everything may have
+  changed; refetch what you care about."*
+
+## Delivery guarantees
+
+- **Post-commit only.** An aborted write — a losing
+  [`ifRev` compare-and-swap](optimistic-concurrency.md), a rejected
+  transaction — never emits. If you received the event, the write is durable.
+- **Commit-ordered.** Events arrive in the order writes committed;
+  `generation` is monotonic.
+- **Asynchronous, never blocking.** Delivery happens in a microtask after the
+  write completes. A slow listener cannot delay a write; a throwing listener
+  is logged and isolated from other listeners.
+- **Zero overhead when unused.** With no subscribers, the write path does no
+  event work at all.
+- **Fire-and-forget.** There is no replay or backpressure. For catch-up after
+  a disconnect, use the `generation` on each event together with
+  [`asOf()` / the transaction log](snapshots-and-time-travel.md): record the
+  last generation you processed, and on reconnect diff from there. For file
+  content specifically, `vfs.readFile(path, { asOf })` and
+  `vfs.history(path)` are the temporal read — see
+  [Snapshots & Time Travel](snapshots-and-time-travel.md).
+
+## Patterns
+
+**Cache invalidation** — drop cached reads for whatever changed:
+
+```ts
+brain.onChange((e) => {
+  if (e.kind === 'store') return cache.clear()
+  if (e.id) cache.delete(e.id)
+})
+```
+
+**Live queries (notify-and-refetch)** — re-run a query when a relevant change
+lands, rather than diffing incrementally:
+
+```ts
+brain.onChange((e) => {
+  if (e.kind === 'entity' && e.entity?.type === 'order') {
+    refreshOpenOrdersView() // debounce as needed
+  }
+})
+```
+
+**Forwarding to other processes** — the feed is in-process by design. To push
+changes to browsers or other services, forward events through your own
+transport (WebSocket, SSE) from the process that owns the brain.
+
+## Lifecycle
+
+`onChange` returns an unsubscribe function — call it when tearing down a
+subscriber (for example, when evicting a pooled instance). `brain.close()`
+drops all listeners; no events are delivered for or after `close()`.

docs/guides/snapshots-and-time-travel.md

@@ -9,6 +9,7 @@ description: Recipes for the Db API — instant backups with persist(), restore,
 next:
   - concepts/consistency-model
   - guides/optimistic-concurrency
+  - guides/external-backups
 ---
 
 # Snapshots & Time Travel
@@ -41,6 +42,10 @@ bytes. Cross-device targets fall back to per-file byte copies, and
 persisting an in-memory brain serializes it to the same directory layout —
 a real, durable store.
 
+> Archiving a brain directory with **external tools** (`tar`, `rsync`, `cp`)?
+> Some index files are sparse and can explode to their apparent size under a
+> naive copy — see [External Backups & Sparse Storage](/docs/guides/external-backups).
+
 Two things to know:
 
 - `persist()` requires the view to still be the store's **latest**
@@ -135,10 +140,12 @@ await after.release()
 
 Three things to remember:
 
-- History granularity is `transact()` commits — single-operation writes
-  advance the clock but do not produce historical records (see the
-  [consistency model](../concepts/consistency-model.md)). Use `transact()`
-  for writes you want to travel back through.
+- History granularity is per-write: EVERY write — `transact()` AND a
+  single-operation `add`/`update`/`remove`/`relate` — is its own immutable
+  generation, so a pin always freezes against later writes and every write is
+  individually addressable via `asOf()` (see the
+  [consistency model](../concepts/consistency-model.md)). Use `transact()` when
+  you want several operations to share ONE atomic generation.
 - The first index-accelerated query (semantic search, traversal, cursors,
   aggregation) at a historical generation builds an in-memory index
   materialization — O(n at that generation), once per `Db`, freed on
@@ -336,20 +343,95 @@ For per-entity write coordination (rather than whole-store history), the
 
 ## Keeping history bounded
 
-Historical records cost disk space. Reclaim what no live pin protects:
+Under Model-B every write is a generation, so history can grow quickly —
+Brainy auto-compacts at `close()` (time-bounded per pass) under the
+**`retention`** knob (configured on the constructor). Since 8.9.0, `flush()`
+never compacts: flushing is durability work and costs only what the current
+window's writes cost, regardless of history backlog. A long-lived writer that
+never closes keeps its history until its next explicit `compactHistory()` —
+schedule one in your maintenance window if you run bounded retention:
 
 ```typescript
-await brain.compactHistory({
-  retainGenerations: 100,            // keep the 100 most recent commits
-  retainMs: 7 * 24 * 60 * 60 * 1000  // and everything from the last 7 days
-})
+// Zero-config: ADAPTIVE — keep as much history as free disk/RAM allows,
+// reclaiming oldest-first under pressure. (This is the default.)
+new Brainy({ /* retention unset */ })
+
+// Unbounded — never reclaim history (opt in explicitly):
+new Brainy({ retention: 'all' })
+
+// Explicit CAPS — reclaim oldest-unpinned generations while ANY cap is exceeded:
+new Brainy({ retention: { maxGenerations: 1000, maxAge: 7 * 86_400_000, maxBytes: 512 * 1024 ** 2 } })
+```
+
+Reclaim manually at any time (the same caps, plus an optional per-pass time
+budget for maintenance windows — an early stop is a consistent prefix and the
+next pass resumes):
+
+```typescript
+await brain.compactHistory({ maxGenerations: 100, maxAge: 7 * 24 * 60 * 60 * 1000 })
+await brain.compactHistory({ maxBytes: 512 * 1024 ** 2, timeBudgetMs: 10_000 })
 ```
 
 Compaction never breaks a pinned read — record-sets are reclaimed only when
-no live `Db` could need them. Release views you are done with (including the
-ones `transact()` returns), and `persist()` any generation you want to keep
-beyond the retention window: snapshots are self-contained and unaffected by
-compaction.
+no live `Db` could need them (live pins are ALWAYS exempt). Release views you
+are done with (including the ones `transact()` returns), and `persist()` any
+generation you want to keep beyond the retention window: snapshots are
+self-contained and unaffected by compaction.
+
+## Time travel for files (the VFS)
+
+Since 8.2.0, time travel covers Virtual Filesystem **content**, not just
+entity records. File bytes are retention-protected: a content blob referenced
+by any generation inside the retention window is never reclaimed, so reading
+the past always returns the exact bytes — never a stale field or a
+dangling hash.
+
+**`vfs.readFile(path, { asOf })`** takes a generation number or a `Date` and
+returns the file's exact bytes as they stood then. It resolves the path's
+current entity, then materializes its state at the target generation — so it
+answers *"what did the file at this path hold at that point?"* It bypasses
+the content cache; the `encoding` option still applies. Asking about a
+generation before the file existed throws the usual not-found error, and
+asking past the retention window's compaction horizon throws a
+compacted-generation error.
+
+**`vfs.history(path)`** returns the file's versions inside the retention
+window, oldest first — one `FileVersion` per generation that wrote the file,
+the newest entry being the current state:
+
+```typescript
+// A CMS page evolves…
+await brain.vfs.writeFile('/pages/home.json', '{"title":"Launch"}')
+await brain.vfs.writeFile('/pages/home.json', '{"title":"Launch v2"}')
+await brain.vfs.writeFile('/pages/home.json', '{"title":""}') // bad deploy!
+
+// Every version is listed and readable:
+const versions = await brain.vfs.history('/pages/home.json')
+// → [{ generation, timestamp, hash, size, mimeType? }, …] ascending
+
+const good = versions[versions.length - 2]
+const bytes = await brain.vfs.readFile('/pages/home.json', {
+  asOf: good.generation
+})
+
+// Restore = write the old bytes back. This is a NEW write (a new
+// generation) — history is never rewritten, so the bad version stays
+// visible in the audit trail.
+await brain.vfs.writeFile('/pages/home.json', bytes)
+```
+
+Two lifecycle consequences worth stating plainly:
+
+- **Deleting or overwriting a file no longer frees its bytes immediately.**
+  Old content lives until history compaction reclaims the generations that
+  reference it — the same `retention` budget that bounds all Model-B history
+  (and pinned views are exempt, exactly as above). Size your `retention` for
+  the file-version depth you want; `retention: 'all'` keeps every version of
+  every file forever.
+- **After `compactHistory()` reclaims a generation, its file versions are
+  gone** and their bytes are physically reclaimed. (This also fixed a
+  pre-8.2.0 defect where overwritten content was never reclaimed at all — an
+  unbounded silent leak.)
 
 ## From branches to values
 

docs/guides/subtypes-and-facets.md

@@ -332,7 +332,7 @@ await brain.updateRelation({ id: relationId, weight: 0.5, confidence: 0.9 })
 `find({ connected, subtype })` filters traversal edges by their subtype. Composes with `via` (verb-type filter):
 
 ```typescript
-// All direct reports two hops deep (will be supported in Cortex; for now depth-1)
+// All direct reports two hops deep (will be supported in Cor; for now depth-1)
 const directChain = await brain.find({
   connected: {
     from: ceoId,
@@ -343,7 +343,7 @@ const directChain = await brain.find({
 })
 ```
 
-Multi-hop subtype filtering (`depth > 1`) lights up on the Cortex native path; the JS path throws today rather than return incorrect partial results.
+Multi-hop subtype filtering (`depth > 1`) lights up on the Cor native path; the JS path throws today rather than return incorrect partial results.
 
 ### Counts
 

docs/guides/upgrading-7-to-8.md

@@ -0,0 +1,186 @@
+---
+title: Upgrading from 7.x to 8.0
+slug: guides/upgrading-7-to-8
+public: true
+category: guides
+template: guide
+order: 10
+description: What the one-time 7→8 on-disk migration does, how 8.0 automatically recovers Virtual Filesystem content that older layouts stored in the removed copy-on-write area, and how to verify (or force) that recovery.
+next:
+  - guides/storage-adapters
+  - guides/inspection
+---
+
+# Upgrading from 7.x to 8.0
+
+Opening a 7.x on-disk store with Brainy 8.0 runs a **one-time, in-place layout
+migration** the first time the store is opened. It is automatic (`autoMigrate`
+defaults to `true`), it runs once, and it stamps a marker so every later open is
+a no-op.
+
+Almost everything about the upgrade is transparent: your entities, relationships,
+metadata, and indexes migrate and rebuild without any action on your part. This
+guide covers the **one case that needs attention** — Virtual Filesystem (VFS)
+content — and how 8.0 recovers it for you.
+
+## TL;DR
+
+- **Just upgrade to `@soulcraft/brainy@8.0.12` (or later) and open the store.**
+  If a previous upgrade left VFS content stranded, 8.0.12 **heals it on open**,
+  with no operator action.
+- Want to force or script it? Call **`await brain.vfs.adoptOrphanedBlobs()`**.
+- The recovery is **non-destructive and idempotent** — it copies, never moves,
+  and running it twice is a no-op.
+
+## What the migration does
+
+The 7.x on-disk layout stored entities under a per-branch path
+(`branches/<head>/entities/...`). 8.0 uses a flat layout (`entities/...`). On
+first open, Brainy:
+
+1. Collapses `branches/<head>/entities/*` into the flat `entities/*` layout.
+2. Rebuilds the derived indexes (vector, graph, metadata) and count rollups from
+   the canonical entities.
+3. Stamps `_system/migration-layout.json` so re-opening is a no-op.
+4. Takes an automatic **pre-upgrade backup** of the directory before it starts,
+   and removes it once the upgrade is verified complete (see
+   [The safety net](#the-safety-net-pre-upgrade-backup) below).
+
+The log line to expect (once per store):
+
+```
+[brainy] Migrating a 7.x branch layout (branches/main) to the 8.0 flat layout
+  in place — 13175 entity files. This runs once; back up the directory first if
+  you need a rollback (8.0 does not keep the old layout).
+```
+
+## The one thing that needs recovery: VFS content
+
+If your application uses the Virtual Filesystem (VFS) to store file content (for
+example, a CMS that keeps page documents at paths like
+`/pages/homepage/page.json`), that content is held as **content blobs**.
+
+7.x kept those blobs in the branch system's **copy-on-write area** (`_cow/`).
+8.0 removed the branch/copy-on-write system, and its content blobs live in the
+content-addressed store (`_cas/`). The layout migration moves entities — it does
+**not** move the VFS content blobs. Left alone, a 7.x store's VFS blobs would
+remain in `_cow/`, and a read of one would throw:
+
+```
+VFS: Cannot read blob for /pages/homepage/page.json:
+  Blob metadata not found: 6b87cfb71ad2f04602a5c157214dc42000...
+```
+
+### 8.0.12 recovers them automatically
+
+Brainy 8.0.12 adds an **on-open recovery pass** that runs right after the layout
+migration. It scans `_cow/`, and for every content blob whose `blob:` +
+`blob-meta:` pair is not already in `_cas/`, it copies **both** across into the
+8.0 store. It:
+
+- **Heals a fresh 7→8 upgrade** (where `_cow/` still holds the blobs), **and**
+- **Heals a store already upgraded by an earlier 8.0.x** that stranded them —
+  the recovery is gated on the presence of `_cow/` and its own marker, not on
+  the layout-migration marker, so an already-migrated store is still healed.
+- Is a **cheap no-op** on a native-8.0 or fresh store (there is no `_cow/`), and
+  on a store already healed (a marker records completion so later opens skip the
+  scan).
+
+So the operator action for a stranded store is simply: **upgrade to 8.0.12 and
+open it.**
+
+```ts
+import { Brainy } from '@soulcraft/brainy'
+
+// Opening the store is all that is required — recovery runs during init().
+const brain = new Brainy({ storage: { type: 'filesystem', path: '/data/my-store' } })
+await brain.init()
+
+// The previously-failing read now succeeds.
+const page = await brain.vfs.readFile('/pages/homepage/page.json')
+```
+
+On a store that needed recovery you will see:
+
+```
+[brainy] Recovered 337 VFS content blob(s) stranded by a 7→8 upgrade
+  (adopted _cow/ → _cas/ in place).
+```
+
+### Forcing recovery explicitly
+
+If you would rather run the recovery deliberately (for example, in an upgrade
+script that asserts a clean result before flipping traffic), call it directly:
+
+```ts
+const result = await brain.vfs.adoptOrphanedBlobs()
+// → { cowBlobs, adopted, alreadyPresent, incomplete }
+console.log(`adopted ${result.adopted}, already present ${result.alreadyPresent}`)
+if (result.incomplete > 0) {
+  // One or more _cow/ blobs are missing their bytes or metadata — investigate
+  // _cow/ before discarding your own backup. See "The safety net" below.
+}
+```
+
+`adoptOrphanedBlobs()` self-initializes the brain, so it is safe to call
+immediately after construction. It returns:
+
+| Field            | Meaning                                                          |
+| ---------------- | ---------------------------------------------------------------- |
+| `cowBlobs`       | Distinct content-blob hashes found in `_cow/`.                   |
+| `adopted`        | Newly copied into `_cas/` on this call.                          |
+| `alreadyPresent` | Already in `_cas/` (a prior open or run adopted them).           |
+| `incomplete`     | A `_cow/` blob missing its bytes *or* its metadata — **skipped** rather than half-adopted. |
+
+## Verifying recovery
+
+After opening under 8.0.12, confirm a previously-failing path reads:
+
+```ts
+const content = await brain.vfs.readFile('/pages/homepage/page.json')
+console.log(content.toString().slice(0, 80))
+```
+
+If you scripted it with `adoptOrphanedBlobs()`, a clean result is
+`incomplete === 0` and `adopted + alreadyPresent === cowBlobs`.
+
+## The safety net: pre-upgrade backup
+
+Brainy takes an automatic pre-upgrade backup and removes it once the upgrade is
+**verified complete**. In 8.0.12 that verification includes VFS content: if the
+blob recovery reports `incomplete > 0`, the completion marker is **not** stamped
+(the next open retries) and the **pre-upgrade backup is retained** so you still
+have a rollback while blobs remain unaccounted for. You will see:
+
+```
+[brainy] VFS blob recovery adopted N blob(s) but M orphaned _cow/ blob(s) are
+  missing their bytes or metadata and were left in place. The pre-upgrade backup
+  is being retained; inspect _cow/ before discarding it.
+```
+
+The recovery never deletes anything from `_cow/`, so the original blobs stay in
+place for inspection or a manual rollback regardless.
+
+## Who is affected
+
+**Any 7.x store that used the VFS to store file content** (so it has a `_cow/`
+area) is a candidate for stranded blobs on a 7→8 upgrade. Stores that never used
+the VFS have no `_cow/` content blobs and are unaffected.
+
+Because the recovery is **gated on the presence of `_cow/`**, it is
+self-selecting and safe to roll out everywhere:
+
+- On a store with stranded blobs → it adopts them.
+- On a native-8.0, fresh, or non-VFS store → it is a no-op on the existence
+  check.
+
+There is no configuration to set and nothing to opt into. Upgrading to 8.0.12
+and opening each store is sufficient.
+
+## Rollback
+
+The recovery is copy-only, so no rollback of the recovery itself is ever needed.
+If you need to roll back the **whole** 7→8 upgrade, restore the directory from
+your pre-upgrade backup (retained automatically while recovery is incomplete, or
+your own snapshot) and pin `@soulcraft/brainy@7.x`. 8.0 does not keep the old
+branch layout in place, so a directory-level restore is the rollback path.

docs/performance-envelopes.md

@@ -0,0 +1,83 @@
+---
+title: Performance Envelopes
+slug: guides/performance-envelopes
+public: true
+category: guides
+template: guide
+order: 40
+description: Measured per-operation latency envelopes at stated scales — what to expect, on what hardware, and exactly how each number was produced.
+next:
+  - guides/find-limits
+---
+
+# Performance Envelopes
+
+Every number on this page is **measured, never projected** — produced by the script
+cited at the bottom, against the built package (the artifact you install), on the stated
+hardware. Each entry says what was measured, at what scale, on which storage backend.
+When a release touches a measured path, that operation is re-measured and this page
+updates in the same release.
+
+Two scopes to keep straight:
+
+- **These envelopes are the pure-JS engine** (no native accelerator registered) on
+  filesystem storage. This is the floor every deployment gets from `npm install` alone.
+- **Accelerated deployments** (the optional native provider) publish their own numbers —
+  this page never claims them.
+
+## Read operations
+
+Reads are where the architecture pays off: after the write path has done its indexing
+work, queries answer from purpose-built indexes without scanning.
+
+| Operation | 1,000 entities | 10,000 entities | Notes |
+|---|---|---|---|
+| `get(id)` (warm) | p50 < 0.1ms | p50 < 0.1ms | served from cache/metadata index |
+| `find` (metadata: indexed equality + range, limit 100) | p50 1.0ms · p95 1.8ms | p50 7.0ms · p95 8.9ms | column-store bitmap paths |
+| `related(id)` (per-node adjacency) | p50 < 0.1ms · p95 0.2ms | p50 < 0.1ms | LSM adjacency index — O(degree), scale-independent |
+| `find` (semantic: embed + HNSW, 1k docs) | p50 178ms · p95 393ms | — | dominated by WASM query embedding (measured on a machine under concurrent load — treat the p95 as an upper bound); the vector search itself is single-digit ms |
+
+## Write operations
+
+Under Model-B **every write is its own durable generation** — a single-op `add` pays
+serialization, before-image staging, and fsync before it acks. That durability is priced
+into the write path visibly, by design:
+
+| Operation | 1,000 entities | 10,000 entities | Notes |
+|---|---|---|---|
+| `add` (single-op) | p50 167ms · p95 171ms | p50 165ms · p95 172ms | full durable generation per write — flat across scale |
+| `addMany` (bulk) | ~163ms/entity | ~187ms/entity | **currently per-item commits** — see the honest note below |
+| `relateMany` | ~0.8ms/edge | ~0.9ms/edge | edges batch efficiently today |
+| `flush` (steady-state, 1 pending write) | p50 8ms · p95 10ms | p50 45ms · p95 52ms | durability-only since 8.9.0 — cost no longer depends on history backlog or retention mode |
+
+**The honest note on bulk writes:** `addMany` today commits each item as its own
+generation (the same durability as single-op `add`, serialized by the single-writer
+lock), so bulk-load cost is N × single-op cost. Batched chunk commits (one generation
+and one fsync window per chunk, as `removeMany` already does) are designed into the
+unified-commit work on the current roadmap. Until that ships, size bulk imports
+accordingly — 10k entities is minutes, not seconds, on filesystem storage.
+
+## Open / close
+
+| Operation | 1,000 entities | 10,000 entities | Notes |
+|---|---|---|---|
+| `open` (empty store) | ~560ms | ~190ms | includes embedder initialization |
+| `open` (warm, populated, clean shutdown) | 763ms | 4.9s | pure-JS vector index load dominates and grows with entity count; the native accelerator exists precisely to remove this |
+| `close` | bounded | bounded | auto-compaction pass is time-bounded (~5s max) since 8.9.0 |
+
+A store that was NOT cleanly closed pays index rebuilds on top of the warm-open
+number (tens of seconds at 10k) — clean shutdown is worth engineering for.
+
+## How these were produced
+
+- **Hardware**: Intel Core i9-14900HX (32 threads), 62GB RAM, NVMe, Linux, Node v22.
+- **Backend**: `storage: { type: 'filesystem' }`, pure JS (no native providers).
+- **Embeddings**: deterministic stub for non-semantic ops (isolates engine cost);
+  the real WASM embedder for the semantic row (that's what you'll run).
+- **Method**: p50/p95 over 50–200 samples per op against the built `dist/`;
+  the measuring script ships in the repo history and re-runs per release.
+
+Numbers on different hardware will differ; the *shape* (sub-2ms indexed reads,
+~160ms embedding-bound semantic queries, durability-priced writes) is the envelope
+you should hold your deployment against. If your measurements diverge from these
+shapes by an order of magnitude, something is wrong — file it.

docs/vfs/PROJECTION_STRATEGY_API.md

@@ -289,7 +289,7 @@ export class SizeProjection extends BaseProjectionStrategy {
       where: {
         vfsType: 'file',
         size: {
-          greaterEqual: min,
+          gte: min,
           lessThan: max
         }
       },
@@ -305,7 +305,7 @@ export class SizeProjection extends BaseProjectionStrategy {
       where: {
         vfsType: 'file',
         size: {
-          greaterEqual: min,
+          gte: min,
           lessThan: max
         }
       },
@@ -359,7 +359,7 @@ export class StatusProjection extends BaseProjectionStrategy {
       const results = await brain.find({
         where: {
           vfsType: 'file',
-          modified: { greaterEqual: oneDayAgo },
+          modified: { gte: oneDayAgo },
           reviewStatus: { missing: true }  // No review status set
         },
         limit: 1000
@@ -387,7 +387,7 @@ export class StatusProjection extends BaseProjectionStrategy {
         vfsType: 'file',
         anyOf: [
           { reviewStatus: { exists: true } },
-          { modified: { greaterEqual: Date.now() - 86400000 } }
+          { modified: { gte: Date.now() - 86400000 } }
         ]
       },
       limit
@@ -415,7 +415,7 @@ Projection strategies use **Brainy Field Operators** (BFO), not MongoDB-style op
 { size: { $gte: 1000, $lte: 5000 } }
 
 // ✅ BFO style (CORRECT)
-{ size: { greaterEqual: 1000, lessEqual: 5000 } }
+{ size: { gte: 1000, lte: 5000 } }
 ```
 
 ### Logical Operators
@@ -451,9 +451,9 @@ Projection strategies use **Brainy Field Operators** (BFO), not MongoDB-style op
 // Comparison
 { field: value }                          // Exact match
 { field: { greaterThan: 10 } }           // >
-{ field: { greaterEqual: 10 } }          // >=
+{ field: { gte: 10 } }          // >=
 { field: { lessThan: 10 } }              // <
-{ field: { lessEqual: 10 } }             // <=
+{ field: { lte: 10 } }             // <=
 { field: { not: value } }                // !=
 
 // Logical
@@ -485,7 +485,7 @@ All metadata fields are automatically indexed. Use direct equality or range quer
 ```typescript
 // ✅ Fast: Direct index lookup (O(log n))
 { priority: 'high' }
-{ size: { greaterEqual: 1000 } }
+{ size: { gte: 1000 } }
 
 // ⚠️ Slower: Must scan results
 { path: { matches: /complex-regex/ } }
@@ -668,7 +668,7 @@ async resolve(brain, vfs, period: string) {
 
   const results = await brain.find({
     where: {
-      modified: { greaterEqual: since }
+      modified: { gte: since }
     }
   })
   return this.extractIds(results)

docs/vfs/SEMANTIC_VFS.md

@@ -127,7 +127,7 @@ await vfs.readFile('/as-of/2024-03-15/auth.ts')
 // the path only resolves if auth.ts was modified that day
 ```
 
-**How it works:** Tracks the `modified` timestamp on every file and runs a range query (`greaterEqual`/`lessEqual`) over one 24-hour window for O(log n) performance. The VFS does not store historical file contents — `/as-of/` filters by *when a file last changed*; reads return the current bytes. For point-in-time state, use the Db API (`brain.asOf(generation)`).
+**How it works:** Tracks the `modified` timestamp on every file and runs a range query (`gte`/`lte`) over one 24-hour window for O(log n) performance. The VFS does not store historical file contents — `/as-of/` filters by *when a file last changed*; reads return the current bytes. For point-in-time state, use the Db API (`brain.asOf(generation)`).
 
 **Status:** ✅ Fully implemented and tested at 10K file scale
 

examples/directory-import-with-caching.ts

@@ -1,237 +0,0 @@
-/**
- * Directory Import with Entity Extraction Caching Example
- *
- * Demonstrates:
- * - Importing directories with progress tracking
- * - Entity extraction caching for performance
- * - Relationship detection with confidence scores
- * - Cache statistics monitoring
- */
-
-import { Brainy, NounType, VerbType } from '../src/brainy.js'
-import { DirectoryImporter } from '../src/vfs/importers/DirectoryImporter.js'
-import { ProgressTracker, formatProgress } from '../src/types/progress.types.js'
-import { detectRelationshipsWithConfidence } from '../src/neural/relationshipConfidence.js'
-import { NeuralEntityExtractor } from '../src/neural/entityExtractor.js'
-
-async function main() {
-  console.log('🧠 Brainy 3.21.0 - Directory Import with Caching Example\n')
-
-  // Initialize Brainy
-  const brain = new Brainy({ verbose: false })
-  await brain.init()
-
-  console.log('✅ Brainy initialized\n')
-
-  // The entity extractor (and its extraction cache) is constructed directly.
-  const extractor = new NeuralEntityExtractor(brain)
-
-  // Example 1: Import directory with entity extraction caching
-  console.log('📁 Example 1: Import Directory with Caching\n')
-
-  const vfs = brain.vfs
-  const importer = new DirectoryImporter(vfs, brain)
-
-  // Progress tracking
-  const tracker = ProgressTracker.create(100)
-  tracker.start()
-
-  try {
-    // Import with progress (using async generator)
-    console.log('Importing directory...')
-
-    let filesProcessed = 0
-    for await (const progress of importer.importStream('./examples', {
-      batchSize: 10,
-      recursive: true,
-      generateEmbeddings: true,
-      extractMetadata: true
-    })) {
-      if (progress.type === 'progress') {
-        filesProcessed = progress.processed
-        const trackedProgress = tracker.update(progress.processed, progress.current)
-        console.log(`  ${formatProgress(trackedProgress)}`)
-      } else if (progress.type === 'complete') {
-        console.log(`\n✅ Import complete! Processed ${progress.processed} files\n`)
-      } else if (progress.type === 'error') {
-        console.error(`❌ Error: ${progress.error?.message}`)
-      }
-    }
-
-    tracker.complete({ filesProcessed })
-
-  } catch (error) {
-    console.error('Import failed:', error)
-  }
-
-  // Example 2: Entity extraction with caching
-  console.log('\n📝 Example 2: Entity Extraction with Caching\n')
-
-  const sampleText = `
-    John Smith created the user authentication system for the application.
-    The authentication system uses JWT tokens and bcrypt for password hashing.
-    Mary Johnson manages the backend team that maintains the system.
-    The system was built using Node.js and PostgreSQL database.
-  `
-
-  console.log('First extraction (cache miss):')
-  const startTime1 = Date.now()
-  const entities1 = await extractor.extract(sampleText, {
-    types: [NounType.Person, NounType.Service, NounType.Technology],
-    confidence: 0.7,
-    cache: {
-      enabled: true,
-      ttl: 7 * 24 * 60 * 60 * 1000,  // 7 days
-      invalidateOn: 'hash'
-    }
-  })
-  const time1 = Date.now() - startTime1
-  console.log(`  Extracted ${entities1.length} entities in ${time1}ms`)
-  console.log(`  Entities: ${entities1.map(e => e.text).join(', ')}\n`)
-
-  console.log('Second extraction (cache hit):')
-  const startTime2 = Date.now()
-  const entities2 = await extractor.extract(sampleText, {
-    types: [NounType.Person, NounType.Service, NounType.Technology],
-    confidence: 0.7,
-    cache: {
-      enabled: true,
-      invalidateOn: 'hash'
-    }
-  })
-  const time2 = Date.now() - startTime2
-  console.log(`  Extracted ${entities2.length} entities in ${time2}ms`)
-  console.log(`  Speedup: ${Math.round(time1 / time2)}x faster!\n`)
-
-  // Show cache statistics
-  const cacheStats = extractor.getCacheStats()
-  console.log('📊 Cache Statistics:')
-  console.log(`  Hits: ${cacheStats.hits}`)
-  console.log(`  Misses: ${cacheStats.misses}`)
-  console.log(`  Hit Rate: ${(cacheStats.hitRate * 100).toFixed(1)}%`)
-  console.log(`  Total Entries: ${cacheStats.totalEntries}`)
-  console.log(`  Avg Entities per Entry: ${cacheStats.averageEntitiesPerEntry}\n`)
-
-  // Example 3: Relationship detection with confidence
-  console.log('🔗 Example 3: Relationship Detection with Confidence\n')
-
-  const relationships = detectRelationshipsWithConfidence(
-    entities1,
-    sampleText,
-    {
-      minConfidence: 0.6,
-      maxDistance: 100,
-      useProximityBoost: true,
-      usePatternMatching: true,
-      useStructuralAnalysis: true
-    }
-  )
-
-  console.log(`Detected ${relationships.length} relationships:\n`)
-  for (const rel of relationships.slice(0, 5)) {  // Show top 5
-    console.log(`  ${rel.sourceEntity.text} --[${rel.verbType}]--> ${rel.targetEntity.text}`)
-    console.log(`    Confidence: ${(rel.confidence * 100).toFixed(1)}%`)
-    console.log(`    Evidence: ${rel.evidence.reasoning}`)
-    console.log(`    Method: ${rel.evidence.method}`)
-    console.log(`    Source: "${rel.evidence.sourceText?.substring(0, 60)}..."\n`)
-  }
-
-  // Example 4: Create relationships in graph with confidence
-  console.log('📊 Example 4: Creating Relationships in Graph\n')
-
-  const createdRelations = []
-  for (const rel of relationships.slice(0, 3)) {  // Create top 3
-    try {
-      // Add entities to brain
-      const sourceId = await brain.add({
-        data: rel.sourceEntity.text,
-        type: rel.sourceEntity.type,
-        metadata: {
-          confidence: rel.sourceEntity.confidence,
-          extractedFrom: 'sample text'
-        }
-      })
-
-      const targetId = await brain.add({
-        data: rel.targetEntity.text,
-        type: rel.targetEntity.type,
-        metadata: {
-          confidence: rel.targetEntity.confidence,
-          extractedFrom: 'sample text'
-        }
-      })
-
-      // Create relationship with confidence
-      const relationId = await brain.relate({
-        from: sourceId,
-        to: targetId,
-        type: rel.verbType,
-        confidence: rel.confidence,
-        evidence: rel.evidence,
-        metadata: {
-          autoDetected: true,
-          detectedAt: new Date().toISOString()
-        }
-      })
-
-      createdRelations.push(relationId)
-      console.log(`  ✅ Created: ${rel.sourceEntity.text} → ${rel.targetEntity.text}`)
-    } catch (error) {
-      console.error(`  ❌ Failed to create relationship:`, error)
-    }
-  }
-
-  console.log(`\n✅ Created ${createdRelations.length} relationships in knowledge graph`)
-
-  // Example 5: Query relationships by confidence
-  console.log('\n🔍 Example 5: Query High-Confidence Relationships\n')
-
-  const allRelations = await brain.getRelations({
-    limit: 100
-  })
-
-  const highConfidence = allRelations.filter(r => (r.confidence || 0) >= 0.7)
-  console.log(`Found ${highConfidence.length} high-confidence relationships (≥70%):\n`)
-
-  for (const rel of highConfidence.slice(0, 5)) {
-    console.log(`  ${rel.from} → ${rel.to} (${rel.type})`)
-    console.log(`    Confidence: ${((rel.confidence || 0) * 100).toFixed(1)}%`)
-    if (rel.evidence) {
-      console.log(`    Method: ${rel.evidence.method}`)
-      console.log(`    Reasoning: ${rel.evidence.reasoning}\n`)
-    }
-  }
-
-  // Example 6: Cache management
-  console.log('🧹 Example 6: Cache Management\n')
-
-  console.log('Cache operations:')
-
-  // Cleanup expired entries
-  const cleaned = extractor.cleanupCache()
-  console.log(`  Cleaned ${cleaned} expired entries`)
-
-  // Invalidate specific cache entry
-  const invalidated = extractor.invalidateCache('hash:abc123')
-  console.log(`  Invalidated entry: ${invalidated}`)
-
-  // Get final stats
-  const finalStats = extractor.getCacheStats()
-  console.log(`  Final cache size: ${finalStats.totalEntries} entries`)
-  console.log(`  Memory used: ~${Math.round(finalStats.cacheSize / 1024)}KB`)
-
-  // Clear all cache (optional)
-  // extractor.clearCache()
-  // console.log('  Cleared entire cache')
-
-  console.log('\n✨ Example complete!')
-  console.log('\n📚 Key Takeaways:')
-  console.log('  • Entity extraction caching provides 10-100x speedup on repeated content')
-  console.log('  • Progress tracking gives real-time feedback for long operations')
-  console.log('  • Relationship confidence helps filter low-quality connections')
-  console.log('  • Evidence tracking makes relationships explainable and debuggable')
-  console.log('  • All features are opt-in and backward compatible')
-}
-
-// Run example
-main().catch(console.error)

package.json

@@ -1,6 +1,6 @@
 {
   "name": "@soulcraft/brainy",
-  "version": "8.0.0-rc.2",
+  "version": "8.9.0",
   "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.",
   "main": "dist/index.js",
   "module": "dist/index.js",
@@ -56,16 +56,22 @@
       "import": "./dist/internals.js",
       "types": "./dist/internals.d.ts"
     },
+    "./brain-format": {
+      "import": "./dist/storage/brainFormat.js",
+      "types": "./dist/storage/brainFormat.d.ts"
+    },
     "./embeddings/wasm": {
       "import": "./dist/embeddings/wasm/index.js",
       "types": "./dist/embeddings/wasm/index.d.ts"
     }
   },
   "engines": {
-    "node": "22.x",
-    "bun": ">=1.0.0"
+    "node": ">=22",
+    "bun": ">=1.1.0"
   },
   "scripts": {
+    "clean": "node -e \"require('fs').rmSync('dist',{recursive:true,force:true})\"",
+    "prebuild": "npm run clean",
     "build": "npm run build:types:if-needed && npm run build:patterns:if-needed && tsc && tsc -p tsconfig.cli.json && npm run build:copy-wasm",
     "build:copy-wasm": "node -e \"const fs=require('fs');const src='src/embeddings/wasm/pkg';const dst='dist/embeddings/wasm/pkg';const skip=new Set(['package.json','.gitignore']);if(fs.existsSync(src)){fs.mkdirSync(dst,{recursive:true});fs.readdirSync(src).filter(f=>!skip.has(f)).forEach(f=>fs.copyFileSync(src+'/'+f,dst+'/'+f));console.log('Copied WASM pkg to dist')}\"",
     "build:types": "tsx scripts/buildTypeEmbeddings.ts",
@@ -88,8 +94,7 @@
     "test:ci-unit": "CI=true vitest run --config tests/configs/vitest.unit.config.ts",
     "test:ci-integration": "NODE_OPTIONS='--max-old-space-size=16384' CI=true vitest run --config tests/configs/vitest.integration.config.ts",
     "test:ci": "npm run test:ci-unit && npm run test:ci-integration",
-    "test:bun": "bun tests/integration/bun-compile-test.ts",
-    "test:bun:compile": "bun build tests/integration/bun-compile-test.ts --compile --outfile /tmp/brainy-bun-test && /tmp/brainy-bun-test",
+    "test:bun": "bun tests/integration/bun-runtime-test.ts",
     "test:wasm": "npx vitest run tests/integration/wasm-embeddings.test.ts",
     "typecheck": "tsc --noEmit",
     "lint": "eslint --ext .ts,.js src/",
@@ -100,11 +105,7 @@
     "release:patch": "./scripts/release.sh patch",
     "release:minor": "./scripts/release.sh minor",
     "release:major": "./scripts/release.sh major",
-    "release:dry": "./scripts/release.sh patch --dry-run",
-    "release:standard-version": "standard-version",
-    "release:standard-version:patch": "standard-version --release-as patch",
-    "release:standard-version:minor": "standard-version --release-as minor",
-    "release:standard-version:major": "standard-version --release-as major"
+    "release:dry": "./scripts/release.sh patch --dry-run"
   },
   "keywords": [
     "ai-database",
@@ -151,14 +152,10 @@
     "boolean": "3.2.0"
   },
   "devDependencies": {
-    "@rollup/plugin-commonjs": "^28.0.6",
-    "@rollup/plugin-node-resolve": "^16.0.1",
-    "@rollup/plugin-replace": "^6.0.2",
-    "@rollup/plugin-terser": "^0.4.4",
     "@testcontainers/redis": "^11.5.1",
     "@types/js-yaml": "^4.0.9",
     "@types/mime": "^3.0.4",
-    "@types/node": "^20.11.30",
+    "@types/node": "^22",
     "@types/uuid": "^10.0.0",
     "@types/ws": "^8.18.1",
     "@typescript-eslint/eslint-plugin": "^8.0.0",
@@ -166,7 +163,7 @@
     "@vitest/coverage-v8": "^3.2.4",
     "jspdf": "^3.0.3",
     "minio": "^8.0.5",
-    "standard-version": "^9.5.0",
+    "prettier": "^3.9.4",
     "testcontainers": "^11.5.1",
     "tsx": "^4.19.2",
     "typescript": "^5.4.5",
@@ -204,33 +201,5 @@
     "tabWidth": 2,
     "trailingComma": "none",
     "useTabs": false
-  },
-  "eslintConfig": {
-    "root": true,
-    "extends": [
-      "eslint:recommended",
-      "plugin:@typescript-eslint/recommended"
-    ],
-    "parser": "@typescript-eslint/parser",
-    "plugins": [
-      "@typescript-eslint"
-    ],
-    "rules": {
-      "@typescript-eslint/no-explicit-any": "off",
-      "no-unused-vars": "off",
-      "@typescript-eslint/no-unused-vars": [
-        "warn",
-        {
-          "args": "after-used",
-          "argsIgnorePattern": "^_"
-        }
-      ],
-      "semi": "off",
-      "@typescript-eslint/semi": [
-        "error",
-        "never"
-      ],
-      "no-extra-semi": "off"
-    }
   }
 }

scripts/push-docs.js

@@ -0,0 +1,116 @@
+#!/usr/bin/env node
+/**
+ * @module scripts/push-docs
+ * @description Push this repo's PUBLIC docs to the soulcraft.com docs ingest
+ * door after an npm publish (VENUE-DOCS-RELEASE-PUSH — retires the old
+ * build-time docs sync).
+ *
+ * Contract (mirrors the reference implementation on the serving side):
+ *   POST {base}/api/docs/ingest
+ *   headers: x-service-secret: $DOCS_INGEST_SECRET, Content-Type: application/json
+ *   body: { docs: [{ slug, title, markdown, nav: { order, section } }] }
+ *   batches of 10, idempotent per slug.
+ *
+ * A doc is public iff its frontmatter has `public: true` AND a `slug`. The
+ * frontmatter is stripped; `category` → nav.section, `order` → nav.order.
+ *
+ * Deliberately NOT pushed: the combined /docs landing index. It spans BOTH
+ * engine corpora (this repo's and the native accelerator's), so a per-repo
+ * push would clobber the union — the index is authored on the serving side.
+ *
+ * Env: DOCS_INGEST_SECRET (required), DOCS_INGEST_BASE (default
+ * https://soulcraft.com). Exits 0 with a LOUD warning when the secret is
+ * absent (the npm publish has already happened; the serving side runs its
+ * interim sync on request) and exits 1 when a push actually fails — the docs
+ * site would silently trail npm otherwise, and that must be visible.
+ */
+import * as fs from 'node:fs'
+import * as path from 'node:path'
+
+const BASE = (process.env.DOCS_INGEST_BASE || 'https://soulcraft.com').replace(/\/+$/, '')
+const SECRET = process.env.DOCS_INGEST_SECRET
+const DOCS_DIR = path.join(path.dirname(new URL(import.meta.url).pathname), '..', 'docs')
+const BATCH = 10
+
+if (!SECRET) {
+  console.warn(
+    '⚠️  DOCS PUSH SKIPPED: DOCS_INGEST_SECRET is not set.\n' +
+      '    soulcraft.com/docs now TRAILS this npm release until docs are pushed.\n' +
+      '    Either export DOCS_INGEST_SECRET and re-run `node scripts/push-docs.js`,\n' +
+      '    or ping venue on VENUE-DOCS-RELEASE-PUSH for the interim sync.'
+  )
+  process.exit(0)
+}
+
+/** Minimal frontmatter split — returns [meta, body] or [null, raw]. */
+function parseFrontmatter(raw) {
+  const m = raw.match(/^---\n([\s\S]*?)\n---\n([\s\S]*)$/)
+  if (!m) return [null, raw]
+  const meta = {}
+  for (const line of m[1].split('\n')) {
+    const kv = line.match(/^(\w[\w-]*):\s*(.*)$/)
+    if (kv) meta[kv[1]] = kv[2].trim().replace(/^["']|["']$/g, '')
+  }
+  return [meta, m[2]]
+}
+
+const docs = []
+;(function walk(dir) {
+  for (const entry of fs.readdirSync(dir, { withFileTypes: true })) {
+    const full = path.join(dir, entry.name)
+    if (entry.isDirectory()) walk(full)
+    else if (entry.name.endsWith('.md')) {
+      const [meta, body] = parseFrontmatter(fs.readFileSync(full, 'utf-8'))
+      if (!meta || meta.public !== 'true' || !meta.slug) continue
+      docs.push({
+        slug: meta.slug,
+        title: meta.title || meta.slug,
+        markdown: body.trim(),
+        nav: {
+          order: Number.parseInt(meta.order || '99', 10) || 99,
+          section: meta.category || 'guides'
+        }
+      })
+    }
+  }
+})(DOCS_DIR)
+
+if (docs.length === 0) {
+  console.error('❌ DOCS PUSH FAILED: zero public docs collected — refusing to push an empty corpus.')
+  process.exit(1)
+}
+docs.sort((a, b) => a.slug.localeCompare(b.slug))
+console.log(`Pushing ${docs.length} public docs to ${BASE}/api/docs/ingest …`)
+
+let failed = false
+for (let i = 0; i < docs.length; i += BATCH) {
+  const batch = docs.slice(i, i + BATCH)
+  try {
+    const res = await fetch(`${BASE}/api/docs/ingest`, {
+      method: 'POST',
+      headers: {
+        'x-service-secret': SECRET,
+        'Content-Type': 'application/json',
+        'User-Agent': 'brainy-docs-push/1.0'
+      },
+      body: JSON.stringify({ docs: batch }),
+      signal: AbortSignal.timeout(120_000)
+    })
+    if (!res.ok) {
+      throw new Error(`HTTP ${res.status}: ${(await res.text()).slice(0, 300)}`)
+    }
+    console.log(`  batch ${i / BATCH + 1}: ${batch.map((d) => d.slug).join(', ')} → ok`)
+  } catch (err) {
+    failed = true
+    console.error(`  batch ${i / BATCH + 1} FAILED: ${err instanceof Error ? err.message : err}`)
+  }
+}
+
+if (failed) {
+  console.error(
+    '❌ DOCS PUSH INCOMPLETE — soulcraft.com/docs may trail npm. ' +
+      'Re-run `node scripts/push-docs.js` or ping venue on VENUE-DOCS-RELEASE-PUSH.'
+  )
+  process.exit(1)
+}
+console.log('✅ Docs pushed.')

scripts/release.sh

@@ -196,6 +196,18 @@ else
 fi
 echo -e "${GREEN}✅ GitHub release created${NC}\n"
 
+# Step 12: Push public docs to the soulcraft.com docs ingest door
+# (VENUE-DOCS-RELEASE-PUSH). Skips with a loud warning when
+# DOCS_INGEST_SECRET is unset; fails loudly (without undoing the publish —
+# that already happened) when a push errors, so the docs site never
+# silently trails npm.
+echo -e "${BLUE}1️⃣2️⃣ Pushing public docs to soulcraft.com/docs...${NC}"
+if node scripts/push-docs.js; then
+  echo -e "${GREEN}✅ Docs push step done${NC}\n"
+else
+  echo -e "${RED}❌ Docs push FAILED — soulcraft.com/docs trails npm until re-run or interim sync${NC}\n"
+fi
+
 echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
 echo -e "${GREEN}🎉 Release ${NEW_VERSION} complete!${NC}"
 echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"

src/aggregation/AggregationIndex.ts

@@ -29,6 +29,7 @@ import { matchesMetadataFilter } from '../utils/metadataFilter.js'
 import { compareCodePoints } from '../utils/collation.js'
 import { bucketTimestamp } from './timeWindows.js'
 import { NounType } from '../types/graphTypes.js'
+import { prodLog } from '../utils/logger.js'
 
 /** Persistence key for aggregate definitions */
 const DEFINITIONS_KEY = '__aggregation_definitions__'
@@ -87,10 +88,18 @@ function matchesSource(entity: Record<string, unknown>, source: AggregateDefinit
     if (entity.service !== source.service) return false
   }
 
-  // Metadata where filter — match against the entity's metadata sub-object
+  // Where filter — resolve each filtered field through resolveEntityField,
+  // the SAME single source of truth groupBy uses (top-level standard fields
+  // + custom metadata). Matching only the metadata sub-object made
+  // where:{subtype}/{visibility}/… a silent no-op: reserved fields never
+  // live in the custom bag, so those filters could never match anything.
   if (source.where && Object.keys(source.where).length > 0) {
-    const metadata = (entity.metadata ?? entity) as Record<string, unknown>
-    if (!matchesMetadataFilter(metadata, source.where)) return false
+    const e = entity as unknown as HNSWNounWithMetadata
+    const resolved: Record<string, unknown> = {}
+    for (const key of Object.keys(source.where)) {
+      resolved[key] = resolveEntityField(e, key)
+    }
+    if (!matchesMetadataFilter(resolved, source.where)) return false
   }
 
   return true
@@ -269,7 +278,7 @@ function recomputeMinMaxFromCounts(state: MetricState): void {
 /**
  * Exact percentile over a value multiset, using linear interpolation between closest ranks
  * (numpy 'linear' / type-7): `rank = p·(n−1)`, interpolating between the floor and ceil
- * positions of the sorted expansion. Mirrors Cortex's `MetricState::percentile` bit-for-bit.
+ * positions of the sorted expansion. Mirrors Cor's `MetricState::percentile` bit-for-bit.
  */
 function computePercentile(valueCounts: Record<string, number>, count: number, p: number): number {
   if (count === 0) return 0
@@ -327,6 +336,30 @@ export class AggregationIndex {
   /** Track aggregates with stale MIN/MAX (need lazy recompute) */
   private staleMinMax = new Map<string, Set<string>>()
 
+  /** Resolves when init() has finished loading persisted definitions/state. */
+  private initPromise: Promise<void> | null = null
+
+  /** True once init() has settled (success or failure). */
+  private initDone = false
+
+  /**
+   * Aggregates registered by the app before init() finished loading persisted
+   * state, awaiting reconciliation: init() adopts the persisted state when the
+   * definition hash matches; anything left unadopted when init settles resolves
+   * to a backfill. Deciding backfill eagerly at define time was the boot-order
+   * bug that wiped valid persisted state on every restart — the synchronous
+   * defineAggregate() always beats the async init().
+   */
+  private pendingAdopt = new Set<string>()
+
+  /**
+   * In-flight rescan targets. While a name has a staging map, ALL
+   * contributions (the walk's and concurrent write hooks') land there instead
+   * of the live map; the live map keeps serving until {@link finishBackfill}
+   * swaps the staging map in atomically.
+   */
+  private backfillStaging = new Map<string, Map<string, AggregateGroupState>>()
+
   constructor(storage: StorageAdapter, nativeProvider?: AggregationProvider) {
     this.storage = storage
     this.nativeProvider = nativeProvider
@@ -336,21 +369,128 @@ export class AggregationIndex {
 
   /**
    * Initialize: load persisted definitions and state, detect changes, rebuild stale.
+   *
+   * Idempotent — repeated calls return the same promise. Definitions registered
+   * *before* this completes (the normal boot order: `defineAggregate()` is
+   * synchronous and always beats this async load) are reconciled rather than
+   * clobbered: the app's definition wins, and its persisted state is adopted
+   * when the definition hash matches — backfill happens only on a real change.
    */
-  async init(): Promise<void> {
+  init(): Promise<void> {
+    if (!this.initPromise) {
+      this.initPromise = this.loadPersisted().finally(() => {
+        this.resolvePendingAdoptToBackfill()
+        this.initDone = true
+      })
+    }
+    return this.initPromise
+  }
+
+  /**
+   * Await the persisted-state load (if one was started) and settle every
+   * pending adoption decision. After this resolves, `getPendingBackfills()`
+   * is authoritative: a name is listed iff it genuinely needs a rescan.
+   * Query paths must await this before consulting backfill state.
+   */
+  async ready(): Promise<void> {
+    if (this.initPromise) {
+      try {
+        await this.initPromise
+      } catch {
+        // The owner already surfaced the load failure loudly; backfill covers.
+      }
+    }
+    this.resolvePendingAdoptToBackfill()
+  }
+
+  /**
+   * Any definition still awaiting state adoption has no persisted state to
+   * adopt (or init never ran / failed) — it must backfill.
+   */
+  private resolvePendingAdoptToBackfill(): void {
+    if (this.pendingAdopt.size > 0) {
+      prodLog.info(
+        `[Aggregation] no adoptable persisted state for: ${Array.from(this.pendingAdopt).join(', ')} — flagged for backfill`
+      )
+    }
+    for (const name of this.pendingAdopt) this.needsBackfill.add(name)
+    this.pendingAdopt.clear()
+  }
+
+  /**
+   * May this persisted state be ADOPTED? When the store exposes its committed
+   * watermark, the state's `sourceGeneration` must EQUAL it: behind means
+   * later writes are missing from the state (unclean shutdown); ahead means
+   * it counts writes that no longer exist (e.g. a fact-log truncation on a
+   * copied store pulled the watermark back). Either way: one exact rescan,
+   * said out loud — never a silent adopt. Stores without the capability (and
+   * pre-stamp state on them) fall back to hash-only adoption.
+   */
+  private stateGenerationAdoptable(name: string, stateData: unknown): boolean {
+    const committed = this.storage.committedGeneration?.() ?? null
+    if (committed === null) return true
+    const raw = (stateData as Record<string, unknown>).sourceGeneration
+    const stamped = typeof raw === 'number' ? raw : null
+    if (stamped === committed) return true
+    prodLog.warn(
+      `[Aggregation] '${name}': persisted state is at generation ${stamped ?? 'unstamped'} ` +
+        `but the store's committed generation is ${committed} — rescanning instead of adopting`
+    )
+    return false
+  }
+
+  private async loadPersisted(): Promise<void> {
     // Load persisted definitions
     const savedDefs = await this.storage.getMetadata(DEFINITIONS_KEY)
     if (savedDefs && typeof savedDefs === 'object' && savedDefs.definitions) {
       const defs = savedDefs.definitions as Array<AggregateDefinition & { _hash?: string }>
 
       for (const def of defs) {
-        this.definitions.set(def.name, def)
-        const currentHash = hashDefinition(def)
         const savedHash = def._hash || ''
 
-        // Load persisted state
+        if (this.definitions.has(def.name)) {
+          // The app re-registered this aggregate before the load finished.
+          // The app's definition wins — never clobber it with the persisted
+          // copy. Adopt the persisted state when the definition is unchanged
+          // AND no write has landed for it yet (a landed write would be lost
+          // by adoption; the hook flips such names to backfill).
+          const appHash = this.definitionHashes.get(def.name) || ''
+          if (appHash === savedHash && this.pendingAdopt.has(def.name)) {
+            const stateData = await this.storage.getMetadata(`${STATE_KEY_PREFIX}${def.name}__`)
+            if (
+              stateData &&
+              stateData.groups &&
+              this.stateGenerationAdoptable(def.name, stateData)
+            ) {
+              const groupMap = new Map<string, AggregateGroupState>()
+              for (const group of stateData.groups as AggregateGroupState[]) {
+                groupMap.set(serializeGroupKey(group.groupKey), group)
+              }
+              this.states.set(def.name, groupMap)
+              this.pendingAdopt.delete(def.name)
+              this.needsBackfill.delete(def.name)
+              prodLog.info(
+                `[Aggregation] '${def.name}': adopted persisted state (${groupMap.size} groups) — no rescan`
+              )
+            }
+            // No/invalid persisted state: stays in pendingAdopt and resolves
+            // to backfill when init settles.
+          }
+          continue
+        }
+
+        // Not registered this session — restore definition + state from
+        // persistence.
+        this.definitions.set(def.name, def)
+        const currentHash = hashDefinition(def)
+
         const stateData = await this.storage.getMetadata(`${STATE_KEY_PREFIX}${def.name}__`)
-        if (stateData && stateData.groups && savedHash === currentHash) {
+        if (
+          stateData &&
+          stateData.groups &&
+          savedHash === currentHash &&
+          this.stateGenerationAdoptable(def.name, stateData)
+        ) {
           // Definition unchanged — load state
           const groupMap = new Map<string, AggregateGroupState>()
           for (const group of stateData.groups as AggregateGroupState[]) {
@@ -358,6 +498,10 @@ export class AggregationIndex {
             groupMap.set(serialized, group)
           }
           this.states.set(def.name, groupMap)
+          this.needsBackfill.delete(def.name)
+          prodLog.info(
+            `[Aggregation] '${def.name}': restored definition + adopted persisted state (${groupMap.size} groups)`
+          )
         } else {
           // Definition changed or no saved state — start fresh and backfill from
           // existing entities (the owner drains needsBackfill on first query).
@@ -398,14 +542,22 @@ export class AggregationIndex {
     }))
     await this.storage.saveMetadata(DEFINITIONS_KEY, { definitions: defsToSave })
 
-    // Persist dirty states
+    // Persist dirty states, stamped with the committed generation they
+    // reflect. The stamp is what makes reopen-adoption verifiable: state at a
+    // different generation than the store's committed watermark is stale (an
+    // unclean shutdown after later writes) or over-counts (a fact-log
+    // truncation on a copied store pulled the watermark BACK below the
+    // stamp) — either way the answer is one exact rescan, never a silent
+    // adopt. Read the generation after collecting groups so any racing
+    // commit resolves toward rescan, not wrong-adopt.
     for (const name of this.dirty) {
       const stateMap = this.states.get(name)
       if (stateMap) {
         const groups = Array.from(stateMap.values())
+        const sourceGeneration = this.storage.committedGeneration?.() ?? null
         await this.storage.saveMetadata(
           `${STATE_KEY_PREFIX}${name}__`,
-          { groups }
+          sourceGeneration === null ? { groups } : { groups, sourceGeneration }
         )
       }
     }
@@ -452,10 +604,19 @@ export class AggregationIndex {
     this.definitions.set(def.name, def)
     this.definitionHashes.set(def.name, newHash)
 
+    // First sight this session, before init() settled: defer the backfill
+    // decision — init() adopts the persisted state on hash match, and anything
+    // left unadopted resolves to backfill. Deciding eagerly here wiped valid
+    // persisted state on every restart.
+    if (!this.states.has(def.name) && !this.initDone) {
+      this.states.set(def.name, new Map())
+      this.pendingAdopt.add(def.name)
+    }
     // Reset state if definition changed or doesn't exist yet, and flag it for
     // backfill so already-stored entities are counted (write-time hooks only see
     // future writes). The owner drains this on the next query via getPendingBackfills().
-    if (!this.states.has(def.name) || (oldHash && oldHash !== newHash)) {
+    else if (!this.states.has(def.name) || (oldHash && oldHash !== newHash)) {
+      this.pendingAdopt.delete(def.name)
       this.states.set(def.name, new Map())
       this.needsBackfill.add(def.name)
     }
@@ -476,6 +637,8 @@ export class AggregationIndex {
     this.definitionHashes.delete(name)
     this.states.delete(name)
     this.staleMinMax.delete(name)
+    this.pendingAdopt.delete(name)
+    this.needsBackfill.delete(name)
 
     // Notify native provider
     if (this.nativeProvider?.removeAggregate) {
@@ -513,9 +676,17 @@ export class AggregationIndex {
     return Array.from(this.needsBackfill)
   }
 
-  /** Clear an aggregate's state so a full rescan cannot double-count. */
+  /**
+   * Begin a rescan into a STAGING map. The live state is not touched — it
+   * keeps serving (possibly stale, but flagged pending) until the rescan
+   * completes and swaps in atomically. A mid-walk failure drops the staging
+   * map via {@link abortBackfill} and loses nothing: wiping live state before
+   * a scan that could throw was the destructive-before-durable defect.
+   * Contributions (walk + concurrent write hooks) land in staging while it
+   * exists, so the swapped-in result reflects writes that raced the walk.
+   */
   beginBackfill(name: string): void {
-    this.states.set(name, new Map())
+    this.backfillStaging.set(name, new Map())
     // Reset native provider state for this aggregate too, if present.
     const def = this.definitions.get(name)
     if (def && this.nativeProvider?.removeAggregate && this.nativeProvider?.defineAggregate) {
@@ -524,6 +695,15 @@ export class AggregationIndex {
     }
   }
 
+  /**
+   * Abandon an in-flight rescan after a failure: drop the staging map, keep
+   * the live state serving, leave the aggregate flagged as pending so a later
+   * attempt rescans. The failure itself must be surfaced loudly by the owner.
+   */
+  abortBackfill(name: string): void {
+    this.backfillStaging.delete(name)
+  }
+
   /** Feed one already-stored entity into a single aggregate during backfill. */
   backfillEntity(name: string, entity: Record<string, unknown>): void {
     if (isAggregateEntity(entity)) return
@@ -537,14 +717,33 @@ export class AggregationIndex {
     }
   }
 
-  /** Mark an aggregate's backfill complete; rebuilt state persists on next flush(). */
+  /** Swap the rebuilt staging state in atomically; persists on next flush(). */
   finishBackfill(name: string): void {
+    const staged = this.backfillStaging.get(name)
+    if (staged) {
+      this.states.set(name, staged)
+      this.backfillStaging.delete(name)
+    }
     this.needsBackfill.delete(name)
     this.dirty.add(name)
   }
 
   // ============= Write-Time Hooks =============
 
+  /**
+   * A write is landing for an aggregate whose persisted-state adoption is still
+   * pending — adopting after this write would lose its contribution. Settle the
+   * decision now: an exact rescan instead of adoption. The window is the few
+   * milliseconds between a boot-time defineAggregate() and init() completing,
+   * so this rarely fires; when it does, correctness wins over the walk.
+   */
+  private resolveAdoptOnWrite(name: string): void {
+    if (this.pendingAdopt.has(name)) {
+      this.pendingAdopt.delete(name)
+      this.needsBackfill.add(name)
+    }
+  }
+
   /**
    * Called when an entity is added. Updates all matching aggregates.
    */
@@ -553,6 +752,7 @@ export class AggregationIndex {
 
     for (const [name, def] of this.definitions) {
       if (!matchesSource(entity, def.source)) continue
+      this.resolveAdoptOnWrite(name)
 
       if (this.nativeProvider) {
         const results = this.nativeProvider.incrementalUpdate(name, def, entity, 'add')
@@ -579,6 +779,10 @@ export class AggregationIndex {
       const oldMatches = matchesSource(oldEntity, def.source)
       const newMatches = matchesSource(newEntity, def.source)
 
+      if (oldMatches || newMatches) {
+        this.resolveAdoptOnWrite(name)
+      }
+
       if (this.nativeProvider && (oldMatches || newMatches)) {
         const results = this.nativeProvider.incrementalUpdate(name, def, newEntity, 'update', oldEntity)
         this.applyNativeResults(name, results)
@@ -605,6 +809,7 @@ export class AggregationIndex {
 
     for (const [name, def] of this.definitions) {
       if (!matchesSource(entity, def.source)) continue
+      this.resolveAdoptOnWrite(name)
 
       if (this.nativeProvider) {
         const results = this.nativeProvider.incrementalUpdate(name, def, entity, 'delete')
@@ -757,7 +962,7 @@ export class AggregationIndex {
     def: AggregateDefinition,
     entity: Record<string, unknown>
   ): void {
-    const stateMap = this.states.get(aggName)!
+    const stateMap = (this.backfillStaging.get(aggName) ?? this.states.get(aggName))!
 
     // Fan out: an unnest dimension makes one entity contribute to several groups.
     for (const groupKey of computeGroupKeys(entity, def.groupBy)) {
@@ -815,7 +1020,7 @@ export class AggregationIndex {
     def: AggregateDefinition,
     entity: Record<string, unknown>
   ): void {
-    const stateMap = this.states.get(aggName)!
+    const stateMap = (this.backfillStaging.get(aggName) ?? this.states.get(aggName))!
 
     // Fan out: reverse the entity's contribution from every group it joined.
     for (const groupKey of computeGroupKeys(entity, def.groupBy)) {
@@ -871,7 +1076,7 @@ export class AggregationIndex {
    * Apply results from native provider back into the state maps.
    */
   private applyNativeResults(aggName: string, results: AggregateGroupState[]): void {
-    const stateMap = this.states.get(aggName)!
+    const stateMap = (this.backfillStaging.get(aggName) ?? this.states.get(aggName))!
     for (const group of results) {
       const serialized = serializeGroupKey(group.groupKey)
       stateMap.set(serialized, group)

src/aggregation/materializer.ts

@@ -14,6 +14,7 @@ import type {
   AggregateMetricDef
 } from '../types/brainy.types.js'
 import { serializeGroupKey } from './AggregationIndex.js'
+import { prodLog } from '../utils/logger.js'
 
 /**
  * Callback interface for the materializer to create/update Brainy entities.
@@ -86,8 +87,15 @@ export class AggregateMaterializer {
     if (existing) clearTimeout(existing)
 
     this.debounceTimers.set(key, setTimeout(() => {
-      this.materializeOne(key).catch(() => {
-        // Non-fatal — materialization is derived data
+      this.materializeOne(key).catch((err) => {
+        // Materialization is derived data (rebuildable via backfill-on-query), so
+        // a failure is non-fatal — but it leaves the materialized Measurement
+        // entity STALE. Surface it loudly rather than swallow it silently.
+        prodLog.warn(
+          `[Aggregation] Failed to materialize aggregate group '${key}': ` +
+          `${(err as Error).message}. The materialized value is stale until the ` +
+          `next successful materialization or a backfill-on-query.`
+        )
       })
     }, debounceMs))
   }

src/cli/catalog.ts

@@ -1,440 +0,0 @@
-/**
- * Augmentation Catalog for CLI
- * 
- * Displays available augmentations catalog
- * Local catalog with caching support
- */
-
-import chalk from 'chalk'
-import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs'
-import { join } from 'node:path'
-import { homedir } from 'node:os'
-
-const CATALOG_API = process.env.BRAINY_CATALOG_URL || null
-const CACHE_PATH = join(homedir(), '.brainy', 'catalog-cache.json')
-const CACHE_TTL = 24 * 60 * 60 * 1000 // 24 hours
-
-interface Augmentation {
-  id: string
-  name: string
-  description: string
-  category: string
-  status: 'available' | 'coming_soon' | 'deprecated'
-  popular?: boolean
-  eta?: string
-}
-
-interface Category {
-  id: string
-  name: string
-  icon: string
-  description: string
-}
-
-interface Catalog {
-  version: string
-  categories: Category[]
-  augmentations: Augmentation[]
-}
-
-/**
- * Fetch catalog from API with caching
- */
-export async function fetchCatalog(): Promise<Catalog | null> {
-  try {
-    // Check cache first
-    const cached = loadCache()
-    if (cached) return cached
-    
-    // If external catalog API is configured, try to fetch
-    if (CATALOG_API) {
-      const response = await fetch(`${CATALOG_API}/api/catalog/cli`)
-      if (!response.ok) throw new Error('API unavailable')
-      
-      const catalog = await response.json()
-      
-      // Save to cache
-      saveCache(catalog)
-      
-      return catalog
-    }
-    
-    // Fall back to local catalog
-    return getDefaultCatalog()
-  } catch (error) {
-    // Try loading from cache even if expired
-    const cached = loadCache(true)
-    if (cached) {
-      console.log(chalk.yellow('📡 Using cached catalog'))
-      return cached
-    }
-    
-    // Fall back to hardcoded catalog
-    return getDefaultCatalog()
-  }
-}
-
-/**
- * Display catalog in CLI
- */
-export async function showCatalog(options: { 
-  category?: string
-  search?: string
-  detailed?: boolean
-}) {
-  const catalog = await fetchCatalog()
-  if (!catalog) {
-    console.log(chalk.red('❌ Could not load augmentation catalog'))
-    return
-  }
-  
-  console.log(chalk.cyan.bold('🧠 Brainy Augmentation Catalog'))
-  console.log(chalk.gray(`Version ${catalog.version}`))
-  console.log('')
-  
-  // Filter augmentations
-  let augmentations = catalog.augmentations
-  
-  if (options.category) {
-    augmentations = augmentations.filter(a => a.category === options.category)
-  }
-  
-  if (options.search) {
-    const query = options.search.toLowerCase()
-    augmentations = augmentations.filter(a => 
-      a.name.toLowerCase().includes(query) ||
-      a.description.toLowerCase().includes(query)
-    )
-  }
-  
-  // Group by category
-  const grouped = groupByCategory(augmentations, catalog.categories)
-  
-  // Display
-  for (const [category, augs] of Object.entries(grouped)) {
-    if (augs.length === 0) continue
-    
-    const cat = catalog.categories.find(c => c.id === category)
-    console.log(chalk.bold(`${cat?.icon || '📦'} ${cat?.name || category}`))
-    
-    for (const aug of augs) {
-      const status = getStatusIcon(aug.status)
-      const popular = aug.popular ? chalk.yellow(' ⭐') : ''
-      const eta = aug.eta ? chalk.gray(` (${aug.eta})`) : ''
-      
-      console.log(`  ${status} ${aug.name}${popular}${eta}`)
-      if (options.detailed) {
-        console.log(chalk.gray(`     ${aug.description}`))
-      }
-    }
-    console.log('')
-  }
-  
-  // Show summary
-  const available = augmentations.filter(a => a.status === 'available').length
-  const coming = augmentations.filter(a => a.status === 'coming_soon').length
-  
-  console.log(chalk.gray('─'.repeat(50)))
-  console.log(chalk.green(`✅ ${available} available`) + chalk.gray(` • `) + 
-              chalk.yellow(`🔜 ${coming} coming soon`))
-  console.log('')
-  console.log(chalk.dim('Configure augmentations with "brainy augment"'))
-  console.log(chalk.dim('Run "brainy augment info <name>" for details'))
-}
-
-/**
- * Show detailed info about an augmentation
- */
-export async function showAugmentationInfo(id: string) {
-  const catalog = await fetchCatalog()
-  if (!catalog) {
-    console.log(chalk.red('❌ Could not load augmentation catalog'))
-    return
-  }
-  
-  const aug = catalog.augmentations.find(a => a.id === id)
-  if (!aug) {
-    console.log(chalk.red(`❌ Augmentation not found: ${id}`))
-    console.log('')
-    console.log('Available augmentations:')
-    catalog.augmentations.forEach(a => {
-      console.log(`  • ${a.id}`)
-    })
-    return
-  }
-  
-  // Fetch full details from API if available
-  try {
-    if (!CATALOG_API) throw new Error('No external catalog configured')
-    
-    const response = await fetch(`${CATALOG_API}/api/catalog/augmentation/${id}`)
-    const details = await response.json()
-    
-    console.log(chalk.cyan.bold(`📦 ${details.name}`))
-    if (details.popular) console.log(chalk.yellow('⭐ Popular'))
-    console.log('')
-    
-    console.log(chalk.bold('Category:'), getCategoryName(details.category, catalog.categories))
-    console.log(chalk.bold('Status:'), getStatusText(details.status))
-    if (details.eta) console.log(chalk.bold('Expected:'), details.eta)
-    console.log('')
-    
-    console.log(chalk.bold('Description:'))
-    console.log(details.longDescription || details.description)
-    console.log('')
-    
-    if (details.features) {
-      console.log(chalk.bold('Features:'))
-      details.features.forEach((f: string) => console.log(`  ✓ ${f}`))
-      console.log('')
-    }
-    
-    if (details.example) {
-      console.log(chalk.bold('Example:'))
-      console.log(chalk.gray('─'.repeat(50)))
-      console.log(details.example.code)
-      console.log(chalk.gray('─'.repeat(50)))
-      console.log('')
-    }
-    
-    if (details.requirements?.config) {
-      console.log(chalk.bold('Required Configuration:'))
-      details.requirements.config.forEach((c: string) => console.log(`  • ${c}`))
-      console.log('')
-    }
-    
-    if (details.pricing) {
-      console.log(chalk.bold('Available in:'))
-      details.pricing.tiers.forEach((t: string) => console.log(`  • ${t}`))
-      console.log('')
-    }
-    
-    console.log(chalk.dim('To activate: brainy augment activate'))
-  } catch (error) {
-    // Show basic info if API fails
-    console.log(chalk.cyan.bold(`📦 ${aug.name}`))
-    console.log(aug.description)
-    console.log('')
-    console.log(chalk.dim('Full details unavailable (no external catalog configured)'))
-  }
-}
-
-/**
- * Show user's available augmentations
- */
-export async function showAvailable(licenseKey?: string) {
-  // Show local catalog as default
-  const catalog = await fetchCatalog()
-  if (!catalog) {
-    console.log(chalk.red('❌ Could not load augmentation catalog'))
-    return
-  }
-  
-  console.log(chalk.cyan.bold('🧠 Available Augmentations'))
-  console.log('')
-  
-  const available = catalog.augmentations.filter(a => a.status === 'available')
-  const grouped = groupByCategory(available, catalog.categories)
-  
-  for (const [category, augs] of Object.entries(grouped)) {
-    if (augs.length === 0) continue
-    
-    const cat = catalog.categories.find(c => c.id === category)
-    console.log(chalk.bold(`${cat?.icon || '📦'} ${cat?.name || category}`))
-    augs.forEach(aug => {
-      console.log(`  ✅ ${aug.name}`)
-      console.log(chalk.gray(`     ${aug.description}`))
-    })
-    console.log('')
-  }
-  
-  console.log(chalk.green(`✅ ${available.length} augmentations available`))
-  
-  // If external API is configured and license key provided, try to fetch personalized data
-  if (CATALOG_API && licenseKey) {
-    try {
-      const response = await fetch(`${CATALOG_API}/api/catalog/available`, {
-        headers: { 'x-license-key': licenseKey }
-      })
-      
-      if (response.ok) {
-        const data = await response.json()
-        console.log(chalk.gray(`Plan: ${data.plan || 'Standard'}`))
-        
-        if (data.operations) {
-          const used = data.operations.used || 0
-          const limit = data.operations.limit
-          const percent = limit === 'unlimited' ? 0 : Math.round((used / limit) * 100)
-          
-          console.log(chalk.bold('Usage:'))
-          if (limit === 'unlimited') {
-            console.log(`  Unlimited operations`)
-          } else {
-            console.log(`  ${used.toLocaleString()} / ${limit.toLocaleString()} operations (${percent}%)`)
-          }
-        }
-      }
-    } catch (error) {
-      // Ignore external API errors - local catalog is sufficient
-    }
-  }
-}
-
-// Helper functions
-
-function loadCache(ignoreExpiry = false): Catalog | null {
-  try {
-    if (!existsSync(CACHE_PATH)) return null
-    
-    const data = JSON.parse(readFileSync(CACHE_PATH, 'utf8'))
-    
-    if (!ignoreExpiry && Date.now() - data.timestamp > CACHE_TTL) {
-      return null
-    }
-    
-    return data.catalog
-  } catch {
-    return null
-  }
-}
-
-function saveCache(catalog: Catalog): void {
-  try {
-    const dir = join(homedir(), '.brainy')
-    if (!existsSync(dir)) {
-      mkdirSync(dir, { recursive: true })
-    }
-    
-    writeFileSync(CACHE_PATH, JSON.stringify({
-      catalog,
-      timestamp: Date.now()
-    }))
-  } catch {
-    // Ignore cache save errors
-  }
-}
-
-function groupByCategory(augmentations: Augmentation[], categories: Category[]) {
-  const grouped: Record<string, Augmentation[]> = {}
-  
-  for (const aug of augmentations) {
-    if (!grouped[aug.category]) {
-      grouped[aug.category] = []
-    }
-    grouped[aug.category].push(aug)
-  }
-  
-  // Sort by category order
-  const ordered: Record<string, Augmentation[]> = {}
-  const categoryOrder = ['memory', 'coordination', 'enterprise', 'perception', 'dialog', 'activation', 'cognition', 'websocket']
-  
-  for (const cat of categoryOrder) {
-    if (grouped[cat]) {
-      ordered[cat] = grouped[cat]
-    }
-  }
-  
-  return ordered
-}
-
-function getStatusIcon(status: string): string {
-  switch (status) {
-    case 'available': return chalk.green('✅')
-    case 'coming_soon': return chalk.yellow('🔜')
-    case 'deprecated': return chalk.red('⚠️')
-    default: return '❓'
-  }
-}
-
-function getStatusText(status: string): string {
-  switch (status) {
-    case 'available': return chalk.green('Available')
-    case 'coming_soon': return chalk.yellow('Coming Soon')
-    case 'deprecated': return chalk.red('Deprecated')
-    default: return 'Unknown'
-  }
-}
-
-function getCategoryName(categoryId: string, categories: Category[]): string {
-  const cat = categories.find(c => c.id === categoryId)
-  return cat ? `${cat.icon} ${cat.name}` : categoryId
-}
-
-function readLicenseFile(): string | null {
-  try {
-    const licensePath = join(homedir(), '.brainy', 'license')
-    if (existsSync(licensePath)) {
-      return readFileSync(licensePath, 'utf8').trim()
-    }
-  } catch (error) {
-    // License file read failed, return null
-    console.debug('Failed to read license file:', error)
-  }
-  return null
-}
-
-function getDefaultCatalog(): Catalog {
-  // Local catalog with current features
-  return {
-    version: '1.5.0',
-    categories: [
-      { id: 'core', name: 'Core Features', icon: '🧠', description: 'Essential brainy functionality' },
-      { id: 'neural', name: 'Neural API', icon: '🔗', description: 'Semantic similarity and clustering' },
-      { id: 'enterprise', name: 'Enterprise', icon: '🏢', description: 'Business integrations' },
-      { id: 'storage', name: 'Storage', icon: '💾', description: 'Data persistence and caching' }
-    ],
-    augmentations: [
-      {
-        id: 'vector-search',
-        name: 'Vector Search',
-        category: 'core',
-        description: 'High-performance semantic search with HNSW indexing',
-        status: 'available',
-        popular: true
-      },
-      {
-        id: 'neural-similarity',
-        name: 'Neural Similarity API',
-        category: 'neural',
-        description: 'Advanced semantic similarity, clustering, and hierarchy detection',
-        status: 'available',
-        popular: true
-      },
-      {
-        id: 'intelligent-verb-scoring',
-        name: 'Intelligent Verb Scoring',
-        category: 'neural',
-        description: 'Smart relationship scoring with taxonomy understanding',
-        status: 'available'
-      },
-      {
-        id: 'connection-pooling',
-        name: 'Connection Pooling',
-        category: 'enterprise',
-        description: 'Efficient database connection management',
-        status: 'available'
-      },
-      {
-        id: 'batch-processing',
-        name: 'Batch Processing',
-        category: 'enterprise',
-        description: 'High-throughput batch operations with deduplication',
-        status: 'available'
-      },
-      {
-        id: 's3-storage',
-        name: 'S3 Compatible Storage',
-        category: 'storage',
-        description: 'Cloud storage with optimized batch operations',
-        status: 'available'
-      },
-      {
-        id: 'opfs-storage',
-        name: 'OPFS Storage',
-        category: 'storage',
-        description: 'Browser-based persistent storage',
-        status: 'available'
-      }
-    ]
-  }
-}

src/coreTypes.ts

@@ -754,6 +754,39 @@ export interface Change {
   data?: HNSWNounWithMetadata | HNSWVerbWithMetadata
 }
 
+/**
+ * @description A declared derived-index blob FAMILY (the
+ * registered-blob contract). A family names the set of on-disk blobs that a
+ * derived index needs AS A SET (e.g. the vector base = `main.dkann` +
+ * `main.slotmap` + `main.slotrev`): losing ANY member corrupts the index. Once a
+ * family is declared:
+ * - its members are UNDELETABLE through the storage layer — `deleteBinaryBlob` /
+ *   `removeRawPrefix` refuse with a `ProtectedArtifactError`, so an in-process
+ *   GC/sweeper cannot remove a load-bearing file (intentional retirement =
+ *   `unregisterDerivedFamily` first);
+ * - a member missing on open is a loud `DerivedArtifactMissingError` → rebuild.
+ * `COLD ≠ DEAD`: a write-once segment or a recovery-critical archive is
+ * load-bearing even when it has not been touched in a long time.
+ */
+export interface DerivedFamilyDeclaration {
+  /** Stable family id, e.g. `'vector-base'` / `'metadata-sstables'`. */
+  name: string
+  /**
+   * The logical blob keys that make up the family. When {@link namespace} is
+   * set, each entry is a PREFIX protecting every key beneath it (for growing
+   * sets like `seg-*`); otherwise each entry is an exact member key.
+   */
+  members: string[]
+  /** When true, {@link members} are prefixes (protect all keys beneath each). */
+  namespace?: boolean
+  /**
+   * Whether a missing family can be rebuilt from the canonical records (default
+   * `true`). `false` marks an irreplaceable family (a missing member is data
+   * loss, not a rebuild).
+   */
+  rebuildable?: boolean
+}
+
 export interface StorageAdapter {
   init(): Promise<void>
 
@@ -815,7 +848,17 @@ export interface StorageAdapter {
    */
   getNounsByNounType(nounType: string): Promise<HNSWNounWithMetadata[]>
 
-  deleteNoun(id: string): Promise<void>
+  /**
+   * Delete a noun — FULL canonical removal (both legs + the entity container).
+   *
+   * @param id The entity id.
+   * @param priorMetadata OPTIONAL already-known metadata of the entity being
+   *   removed (the caller's pre-delete read). The count decrement must never
+   *   REQUIRE re-reading the record being removed: when the internal read
+   *   returns `null` (replace race, or a ghost left by an earlier version) the
+   *   decrement falls back to this record instead of being silently skipped.
+   */
+  deleteNoun(id: string, priorMetadata?: NounMetadata | null): Promise<void>
 
   /**
    * Save verb - Pure HNSW verb with core fields only
@@ -872,7 +915,15 @@ export interface StorageAdapter {
    */
   getVerbsByType(type: string): Promise<HNSWVerbWithMetadata[]>
 
-  deleteVerb(id: string): Promise<void>
+  /**
+   * Delete a verb — FULL canonical removal (both legs + the container).
+   *
+   * @param id The relationship id.
+   * @param priorMetadata OPTIONAL already-known metadata of the edge being
+   *   removed (the caller's pre-delete read); keeps the count decrement honest
+   *   when the internal read returns `null` (see `deleteNoun`).
+   */
+  deleteVerb(id: string, priorMetadata?: VerbMetadata | null): Promise<void>
 
   /**
    * Save metadata
@@ -888,6 +939,16 @@ export interface StorageAdapter {
    */
   getMetadata(id: string): Promise<NounMetadata | null>
 
+  /**
+   * Delete a system/metadata object previously written with {@link saveMetadata}.
+   * Routes by the same key analysis as save/get (system channel), so callers that
+   * persist their own keyed payloads — e.g. the LSM graph store reclaiming its
+   * compacted-away SSTables — can free the storage instead of orphaning it.
+   * Idempotent: a missing key is a no-op, not an error.
+   * @param id The same key passed to {@link saveMetadata}.
+   */
+  deleteMetadata(id: string): Promise<void>
+
   /**
    * Get multiple metadata objects in batches (prevents socket exhaustion)
    * @param ids Array of IDs to get metadata for
@@ -1039,6 +1100,104 @@ export interface StorageAdapter {
    */
   getBinaryBlobPath(key: string): string | null
 
+  /**
+   * @description OPTIONAL (registered-blob contract). Declare a
+   * derived-index blob {@link DerivedFamilyDeclaration | family} whose members
+   * become UNDELETABLE through this adapter — a subsequent `deleteBinaryBlob` /
+   * `removeRawPrefix` that would remove a declared member throws a
+   * `ProtectedArtifactError`. Providers declare their families on create; the
+   * declaration is persisted so protection survives a reopen. Idempotent per
+   * `name` (re-declaring replaces).
+   * @param family - The family to protect.
+   */
+  registerDerivedFamily?(family: DerivedFamilyDeclaration): Promise<void>
+
+  /**
+   * @description OPTIONAL. Remove a family's protection so its members can be
+   * deleted again — the explicit, auditable step for intentional retirement of a
+   * derived index (the ONLY way a declared member becomes deletable).
+   * @param name - The {@link DerivedFamilyDeclaration.name} to unregister.
+   */
+  unregisterDerivedFamily?(name: string): Promise<void>
+
+  /**
+   * @description OPTIONAL. List the currently-declared derived-index families —
+   * the source of truth for what `clear()` must wipe and what a
+   * missing-on-open check verifies.
+   */
+  listDerivedFamilies?(): Promise<DerivedFamilyDeclaration[]>
+
+  /**
+   * @description OPTIONAL binary raw-byte primitives — the substrate for
+   * append-only log-structured files (the generation fact log's CRC-framed
+   * segments). Feature-detected: an adapter that omits them simply hosts no
+   * fact log (readers fall back to canonical enumeration). Paths are
+   * storage-root-relative and used VERBATIM (no `.gz`/`.bin` suffixing —
+   * unlike the JSON object and blob primitives).
+   *
+   * Append to a raw binary file, creating it (and parent directories) when
+   * absent. Append durability is the CALLER's job via `syncRawObjects` —
+   * matching the commit protocol, which batches fsyncs at its barrier.
+   */
+  appendRawBytes?(path: string, bytes: Uint8Array): Promise<void>
+
+  /**
+   * Read a raw binary file whole. Absent → `null`; a real IO fault throws
+   * (never masked as absence).
+   */
+  readRawBytes?(path: string): Promise<Uint8Array | null>
+
+  /**
+   * Replace a raw binary file atomically (write-new → fsync → rename) — the
+   * reconcile primitive (e.g. truncating a fact-log tail back to committed
+   * truth after a crash).
+   */
+  writeRawBytes?(path: string, bytes: Uint8Array): Promise<void>
+
+  /**
+   * Byte size of a raw binary file, or `null` when absent.
+   */
+  rawByteSize?(path: string): Promise<number | null>
+
+  /**
+   * @description OPTIONAL fact-scan capability — how an index provider that
+   * holds only `storage` reaches the generation fact log (the host brain
+   * wires it at init; providers must never construct their own fact-log
+   * reader — the log's open path is writer-side). Present ⟺ this store hosts
+   * a fact log AND the host wired the capability. Returns a scan handle over
+   * committed facts (heal-grade telemetry included), or `null` when no fact
+   * log exists — callers fall back to the canonical enumeration walk.
+   */
+  scanFacts?(options?: {
+    fromGeneration?: number
+    toGeneration?: number
+    kinds?: Array<'noun' | 'verb'>
+    batchSize?: number
+  }): import('./db/factLog.js').FactScanHandle | null
+
+  /**
+   * @description OPTIONAL (rides the fact-scan capability): the fact log's
+   * head generation — the replay target for `stamp.sourceGeneration + 1`
+   * catch-ups. `null` when no fact log exists.
+   */
+  factLogHeadGeneration?(): number | null
+
+  /**
+   * @description OPTIONAL (rides the fact-scan capability): the COMMITTED
+   * generation watermark — the manifest truth a projection's
+   * `sourceGeneration` compares against. Exposed as a capability so a
+   * provider never parses the store's private manifest format. `null` when
+   * the capability is unwired.
+   */
+  committedGeneration?(): number | null
+
+  /**
+   * @description OPTIONAL (rides the fact-scan capability): the immutable,
+   * sealed fact-segment file paths covering `fromGeneration` — the zero-copy
+   * handoff. The mutable tail is excluded (read it via `scanFacts`).
+   */
+  factSegmentPaths?(options?: { fromGeneration?: number }): string[]
+
   /**
    * Save statistics data
    * @param statistics The statistics data to save
@@ -1128,4 +1287,20 @@ export interface StorageAdapter {
    * @returns Promise that resolves to the total number of verbs
    */
   getVerbCount(): Promise<number>
+
+  /**
+   * OPTIONAL — create a pre-upgrade backup of the whole store and return its
+   * location, or `null` when there is nothing to back up (empty store). On the
+   * filesystem adapter this is a zero-copy hard-link snapshot into a sibling
+   * directory; other backends may omit it. Callers feature-detect. Paired with
+   * {@link removeMigrationBackup}. Used by the 7.x → 8.0 migration lifecycle.
+   */
+  createMigrationBackup?(): Promise<string | null>
+
+  /**
+   * OPTIONAL — remove a backup created by {@link createMigrationBackup}.
+   * Best-effort: a missing `location` is a no-op. Removing a hard-link snapshot
+   * never affects the live store (shared inodes; only the extra links are gone).
+   */
+  removeMigrationBackup?(location: string): Promise<void>
 }

src/db/db.ts

@@ -144,6 +144,29 @@ export interface DbHost<T = any> {
    * caller) and freed via the returned handle's `close()`.
    */
   materializeAt(generation: number): Promise<HistoricalQueryHandle<T>>
+  /**
+   * 8.0 #35 at-gen vector defer: true when the vector index is a
+   * {@link ../plugin.js VersionedIndexProvider} that advertised
+   * `isGenerationVisible(generation)` — i.e. it can serve the at-`generation`
+   * vector leg natively from retained segments, so a filtered semantic read needs
+   * NO O(n@G) materialization. False on the JS index (and whenever a native
+   * provider refuses the generation), so the caller falls back to `materializeAt`.
+   */
+  canServeVectorAtGeneration(generation: number): boolean
+  /**
+   * Run the vector kNN for `params` (semantic or explicit-vector) AS OF
+   * `generation`, restricted to `allowedIds` (the at-gen metadata∩graph universe
+   * the caller resolved from the record layer). Returns `[id, distance]` pairs,
+   * descending relevance. Only reached when {@link DbHost.canServeVectorAtGeneration}
+   * is true — the provider serves the at-gen walk; Brainy composes the metadata
+   * half. `k` is the over-fetch (page + headroom).
+   */
+  vectorSearchAtGeneration(
+    params: FindParams<T>,
+    allowedIds: ReadonlySet<string>,
+    k: number,
+    generation: number
+  ): Promise<Array<[string, number]>>
   /** Add one refcounted pin (store + versioned providers) on `generation`. */
   pinGeneration(generation: number): void
   /** Release one refcounted pin (store + versioned providers) on `generation`. */
@@ -339,6 +362,12 @@ export class Db<T = any> {
     if (this.overlay) {
       this.assertOverlayCompatibleFind(params)
     } else if (findRequiresIndexes(params)) {
+      // 8.0 #35: a FILTERED semantic/vector read at a historical generation can be
+      // served by a native at-gen vector provider (no O(n@G) materialization) — the
+      // metadata∩graph universe comes free from the record-overlay path, the vector
+      // leg from the provider. Falls through to materialization when not available.
+      const native = await this.tryAtGenerationVectorFind(params)
+      if (native !== null) return native
       const materialized = await this.materialize()
       return materialized.find(params)
     }
@@ -346,9 +375,12 @@ export class Db<T = any> {
     const limit = params.limit ?? 10
     const offset = params.offset ?? 0
 
-    // Ids whose live-index answer is NOT valid at this generation.
+    // Ids whose live-index answer is NOT valid at this generation. The upper
+    // bound is the full reserved watermark generation() — NOT committedGeneration()
+    // — so un-flushed single-op (Model-B) writes that changed an id after this
+    // pin are overlaid too (they participate in resolution via the pending tier).
     const changedNouns = historical
-      ? (await this.host.store.changedBetween(this.gen, this.host.store.committedGeneration())).nouns
+      ? (await this.host.store.changedBetween(this.gen, this.host.store.generation())).nouns
       : []
     const overlayNouns = this.overlay?.nouns ?? new Map<string, Entity<T> | null>()
     const excluded = new Set<string>([...changedNouns, ...overlayNouns.keys()])
@@ -413,6 +445,70 @@ export class Db<T = any> {
     }
   }
 
+  /**
+   * @description 8.0 #35 — try to serve a FILTERED semantic/vector read at this
+   * historical generation via the native at-gen vector provider, with no O(n@G)
+   * materialization. Eligible only when: the query needs the vector leg
+   * (`query`/`vector`) WITH a metadata filter (the `allowedIds` source) and NO
+   * other index dimension (`near`/`connected`/`cursor`/`aggregate`/
+   * `includeRelations`), AND the host's vector index can serve this generation
+   * ({@link DbHost.canServeVectorAtGeneration}). The at-gen metadata∩graph universe
+   * is resolved through this view's OWN record-overlay path (the metadata-only
+   * `find`, which costs no materialization); the provider then ranks the vector
+   * leg restricted to that universe, and rows are hydrated from the at-gen universe
+   * entities (so metadata reflects `generation`, not now). Returns `null` when
+   * ineligible — the caller falls back to materialization.
+   */
+  private async tryAtGenerationVectorFind(params: FindParams<T>): Promise<Result<T>[] | null> {
+    const wantsVector = params.query !== undefined || params.vector !== undefined
+    const hasOtherIndexDim =
+      params.near !== undefined ||
+      params.connected !== undefined ||
+      params.cursor !== undefined ||
+      params.aggregate !== undefined ||
+      params.includeRelations === true
+    const hasMetadataFilter = Boolean(
+      params.where || params.type || params.subtype || params.service || params.excludeVFS
+    )
+    if (!wantsVector || hasOtherIndexDim || !hasMetadataFilter) return null
+    if (!this.host.canServeVectorAtGeneration(this.gen)) return null
+
+    // At-gen metadata∩graph universe via the record-overlay path (no materialization):
+    // the same query with the vector legs stripped resolves through the metadata
+    // historical branch. Capped so a pathological filter can't materialize an
+    // unbounded universe; the provider walk is restricted to whatever the cap yields.
+    const universeParams: FindParams<T> = { ...params }
+    delete universeParams.query
+    delete universeParams.vector
+    delete universeParams.searchMode
+    delete universeParams.orderBy
+    delete universeParams.order
+    universeParams.limit = AT_GEN_VECTOR_UNIVERSE_CAP
+    universeParams.offset = 0
+    const universe = await this.find(universeParams)
+    if (universe.length === 0) return []
+
+    const limit = params.limit ?? 10
+    const offset = params.offset ?? 0
+    const allowedIds = new Set(universe.map((r) => r.id))
+    const scored = await this.host.vectorSearchAtGeneration(
+      params,
+      allowedIds,
+      (offset + limit) * 2,
+      this.gen
+    )
+
+    // Compose: each at-gen metadata entity (from the universe) ranked by the
+    // provider's at-gen vector distance.
+    const byId = new Map(universe.map((r) => [r.id, r]))
+    const ranked: Result<T>[] = []
+    for (const [id, distance] of scored) {
+      const row = byId.get(id)
+      if (row) ranked.push({ ...row, score: Math.max(0, Math.min(1, 1 / (1 + distance))) })
+    }
+    return ranked.slice(offset, offset + limit)
+  }
+
   /**
    * @description Serialize part or all of this database value into a portable
    * `PortableGraph` document, read **as of this view's generation**. Because `export`
@@ -480,8 +576,11 @@ export class Db<T = any> {
     const limit = params.limit ?? 100
     const offset = params.offset ?? 0
 
+    // Upper bound is the full reserved watermark generation() (see find()): an
+    // un-flushed single-op write that touched a relationship after this pin is
+    // overlaid too, via the pending tier.
     const changedVerbs = historical
-      ? (await this.host.store.changedBetween(this.gen, this.host.store.committedGeneration())).verbs
+      ? (await this.host.store.changedBetween(this.gen, this.host.store.generation())).verbs
       : []
     const overlayVerbs = this.overlay?.verbs ?? new Map<string, Relation<T> | null>()
     const excluded = new Set<string>([...changedVerbs, ...overlayVerbs.keys()])
@@ -846,6 +945,13 @@ export class Db<T = any> {
    * {@link SpeculativeOverlayError} (commit them with `brain.transact()`
    * first).
    *
+   * SPARSE FILES: a native accelerator's mmap index files can be sparse —
+   * huge apparent size, small allocated size. `persist()` handles them
+   * correctly (hard links share the allocation). But if you then archive the
+   * snapshot with EXTERNAL tools, use the sparse-aware flags (`tar czSf`,
+   * `rsync --sparse`, `cp --sparse=always`) or the copy materializes every
+   * hole — see docs/guides/external-backups-and-sparse-storage.md.
+   *
    * @param path - Absolute directory for the snapshot (created; must be
    *   empty or absent).
    * @throws GenerationConflictError when this view is no longer the latest
@@ -963,9 +1069,10 @@ function indexOnlyFindDimensions(params: FindParams): Array<[string, boolean]> {
     ['aggregation', params.aggregate !== undefined],
     ['relation expansion (includeRelations)', params.includeRelations === true],
     [
-      `search mode '${params.mode ?? params.searchMode}'`,
-      (params.mode !== undefined && params.mode !== 'metadata' && params.mode !== 'auto') ||
-        (params.searchMode !== undefined && params.searchMode !== 'auto')
+      `search mode '${params.searchMode}'`,
+      params.searchMode !== undefined &&
+        params.searchMode !== 'auto' &&
+        params.searchMode !== 'metadata'
     ]
   ]
 }
@@ -980,6 +1087,14 @@ function findRequiresIndexes(params: FindParams): boolean {
   return indexOnlyFindDimensions(params).some(([, present]) => present)
 }
 
+/**
+ * Cap on the at-gen metadata universe materialized for a native at-gen vector find
+ * (#35) — bounds a pathological filter; the provider's vector walk is restricted to
+ * whatever the cap yields. Only the metadata (no vectors) is held, so this is far
+ * cheaper than the full-corpus JS-HNSW rebuild it replaces.
+ */
+const AT_GEN_VECTOR_UNIVERSE_CAP = 100_000
+
 /**
  * @description Build a `Result` row from a record/overlay-resolved entity,
  * mirroring the live metadata-only find path (flattened fields, score `1.0`).

src/db/errors.ts

@@ -159,3 +159,133 @@ export class GenerationCompactedError extends Error {
     this.horizon = horizon
   }
 }
+
+/** One entity/relationship left in an unreconciled state by a failed rollback. */
+export interface UnreconciledRecord {
+  /** The entity or relationship id. */
+  id: string
+  /** `'noun'` (entity) or `'verb'` (relationship). */
+  kind: 'noun' | 'verb'
+  /**
+   * `'orphan'` — the record is durably PRESENT but the transaction aborted
+   * (an add whose delete-undo failed); `'loss'` — the record is durably GONE
+   * or wrong when it should have been restored (a remove/update whose
+   * restore-undo failed — actual data loss).
+   */
+  disposition: 'orphan' | 'loss'
+}
+
+/**
+ * @description Thrown when a transaction's rollback could not be fully applied
+ * and the resulting inconsistency cannot be safely adopted forward — i.e. a
+ * multi-operation batch, or ANY case where a record was lost (a remove/update
+ * whose restore-undo failed). The store is left in a known-inconsistent state:
+ * the {@link records} name every entity/relationship whose canonical state no
+ * longer matches what the aborted transaction should have produced.
+ *
+ * On throw, the brain enters **write-quarantine** — reads continue to work, but
+ * further writes are refused until {@link } `repairIndex()` reconciles the
+ * derived indexes against canonical storage and lifts the quarantine. This is
+ * the honest, loud failure: a visible inconsistency the operator must repair,
+ * never a silent partial write. The generation counter is NOT advanced.
+ *
+ * (A SINGLE-op add whose only damage is a durably-present orphan is NOT this
+ * error: it is adopted forward as a committed generation and returned as a
+ * degraded-but-successful write — the record the caller asked for exists.)
+ *
+ * @example
+ * try {
+ *   await brain.transact(ops)
+ * } catch (err) {
+ *   if (err instanceof StoreInconsistentError) {
+ *     console.error('store inconsistent:', err.records) // ids + dispositions
+ *     await brain.repairIndex() // reconcile + lift the write-quarantine
+ *   }
+ * }
+ */
+export class StoreInconsistentError extends Error {
+  /** Every record left in an unreconciled state by the failed rollback. */
+  public readonly records: UnreconciledRecord[]
+  /** The original error that triggered the (then-failed) rollback. */
+  public override readonly cause: Error
+
+  /**
+   * @param records - The unreconciled entities/relationships (ids + disposition).
+   * @param cause - The error that triggered the rollback.
+   */
+  constructor(records: UnreconciledRecord[], cause: Error) {
+    const orphans = records.filter((r) => r.disposition === 'orphan').length
+    const losses = records.filter((r) => r.disposition === 'loss').length
+    super(
+      `Store left inconsistent by a failed transaction rollback: ` +
+        `${records.length} record(s) could not be reconciled ` +
+        `(${orphans} orphaned, ${losses} lost). ` +
+        `The brain is now WRITE-QUARANTINED (reads still work) — run repairIndex() ` +
+        `to reconcile the derived indexes against canonical storage and lift the ` +
+        `quarantine. Ids: ${records.map((r) => `${r.id}:${r.disposition}`).join(', ')}. ` +
+        `Cause: ${cause.message}`
+    )
+    this.name = 'StoreInconsistentError'
+    this.records = records
+    this.cause = cause
+  }
+}
+
+/**
+ * @description Thrown by a write when the store cannot make single-op
+ * generation **history** durable: the asynchronous group-commit flush that
+ * persists buffered before-images to disk has failed repeatedly (a real
+ * storage fault — a full disk, an EIO, a permission loss — not a transient
+ * blip). Rather than keep accepting writes whose history silently piles up in
+ * memory and is never persisted — an invisible durability loss, and an
+ * unbounded leak — the store LATCHES this failure and refuses further writes
+ * until the pending tier drains.
+ *
+ * This is the loud, honest response to a broken history-durability path:
+ * - Live canonical data is unaffected (single-op live bytes are written before
+ *   the history flush; only the immutable before-image history is stuck).
+ * - The background flush keeps retrying with backoff; when the underlying fault
+ *   clears and a flush succeeds, the latch lifts and writes resume
+ *   automatically. An explicit `flush()` also clears it on success.
+ * - {@link cause} is the underlying storage error from the last failed flush.
+ *
+ * A caller seeing this should treat it exactly like a full disk: stop writing,
+ * resolve the storage fault, and retry. It is NOT a data-corruption error — no
+ * committed generation is lost — it is a refusal to *promise* durability the
+ * store currently cannot deliver.
+ *
+ * @example
+ * try {
+ *   await brain.add({ ... })
+ * } catch (err) {
+ *   if (err instanceof PendingFlushDurabilityError) {
+ *     // History can't be persisted right now (disk fault). Resolve storage,
+ *     // then retry — the store self-heals once a flush succeeds.
+ *     console.error('history durability stalled:', err.cause)
+ *   }
+ * }
+ */
+export class PendingFlushDurabilityError extends Error {
+  /** The underlying storage error from the most recent failed flush. */
+  public override readonly cause: Error
+  /** How many consecutive flush attempts had failed when the latch tripped. */
+  public readonly failedAttempts: number
+
+  /**
+   * @param cause - The storage error from the last failed pending-tier flush.
+   * @param failedAttempts - Consecutive failed flush attempts at latch time.
+   */
+  constructor(cause: Error, failedAttempts: number) {
+    super(
+      `Write refused: single-op generation history could not be made durable ` +
+        `after ${failedAttempts} consecutive flush attempts (${cause.message}). ` +
+        `Live data is intact, but buffered history is not yet persisted — the ` +
+        `store refuses further writes rather than silently accumulate ` +
+        `un-durable history. Resolve the underlying storage fault; the store ` +
+        `self-heals and resumes writes once a flush succeeds. Cause: ${cause.message}`
+    )
+    this.name = 'PendingFlushDurabilityError'
+    this.cause = cause
+    this.failedAttempts = failedAttempts
+  }
+}

src/db/factLog.ts

@@ -0,0 +1,721 @@
+/**
+ * @module db/factLog
+ * @description The generation FACT LOG — an append-only, CRC-framed record of
+ * every committed generation as an AFTER-IMAGE "fact": what each touched
+ * entity/relationship BECAME (or a body-less tombstone when it was removed).
+ * This is the dual-write half of the log-canonical transition: today the
+ * before-image history + canonical tree remain authoritative; the fact log is
+ * appended at the same commit points and reconciled to committed truth at
+ * open, so consumers (index heals, replays, scans) can read one sequential,
+ * self-verifying stream instead of walking the entity tree.
+ *
+ * ## Wire format (frozen; additive-only within a major)
+ *
+ * Fact (msgpack, POSITIONAL array — the segment header's formatVersion
+ * governs the schema):
+ *
+ *   fact := [ generation:u64, timestamp:u64, ops, meta|nil, blobHashes|nil ]
+ *   op   := [ kind:u8 (0=noun, 1=verb), id:bin16 (raw uuid bytes),
+ *             record:[metaLeg, vecLeg] | nil ]        // nil = TOMBSTONE
+ *
+ * Segment file (`_generations/facts/seg-<firstGeneration, zero-padded 20>.bfl`):
+ *
+ *   header := magic "BFACTS\0\0" (8B) | formatVersion:u32 LE |
+ *             firstGeneration:u64 LE | reserved 12B (ZEROED, verified)
+ *   frame  := length:u32 LE | crc32c:u32 LE (of payload) | payload
+ *
+ * A fact is never split across segments; a torn tail (length overrun or CRC
+ * mismatch) terminates that segment's scan — everything before it is intact.
+ * Zero-padded names make lexicographic order == generation order.
+ *
+ * ## Invariant
+ *
+ * After {@link FactLog.open}, the log contains EXACTLY the committed prefix:
+ * facts are appended BEFORE the commit point (inside the same durability
+ * window), so a crash can only leave the log AHEAD of committed truth — open
+ * truncates any fact beyond the committed generation. Absent generation =
+ * never committed; present = committed. A scan can never see an uncommitted
+ * fact.
+ *
+ * The manifest (`_generations/facts/manifest.json`, JSON — forensics stay
+ * terminal-readable) is the single source of truth for the segment SET;
+ * rotation flips it atomically (write-new → fsync → rename) BEFORE the new
+ * tail's first byte exists, so no segment file is ever unaccounted for.
+ */
+import { encode as defaultEncode, decode as defaultDecode } from '@msgpack/msgpack'
+import { crc32c } from '../utils/crc32c.js'
+import { prodLog } from '../utils/logger.js'
+
+// Swappable msgpack implementation — defaults to the JS codec; a native
+// provider (registered via the plugin registry's 'msgpack' key) may replace
+// it. Byte-compatibility is the contract (positional arrays, bin16 ids).
+let msgpackEncode: (value: unknown) => Uint8Array = defaultEncode
+let msgpackDecode: (bytes: Uint8Array) => unknown = defaultDecode
+
+/** Replace the msgpack encode/decode implementation at runtime. */
+export function setFactCodec(impl: {
+  encode: (value: unknown) => Uint8Array
+  decode: (bytes: Uint8Array) => unknown
+}): void {
+  msgpackEncode = impl.encode
+  msgpackDecode = impl.decode
+}
+
+/** Storage-root-relative home of the fact log. */
+export const FACTS_PREFIX = '_generations/facts'
+/** The facts manifest path (JSON). */
+export const FACTS_MANIFEST_PATH = `${FACTS_PREFIX}/manifest.json`
+/** Current segment format version (header field; additive-only within a major). */
+export const FACTS_FORMAT_VERSION = 1
+/** Rotation threshold: seal the tail segment once it exceeds this many bytes. */
+const SEGMENT_ROTATE_BYTES = 8 * 1024 * 1024
+/** Segment header: magic(8) + formatVersion(4) + firstGeneration(8) + reserved(12). */
+const HEADER_BYTES = 32
+const MAGIC = new Uint8Array([0x42, 0x46, 0x41, 0x43, 0x54, 0x53, 0x00, 0x00]) // "BFACTS\0\0"
+/** Frame prefix: length(4) + crc32c(4). */
+const FRAME_PREFIX_BYTES = 8
+
+/** One write inside a fact: what the id became (or a tombstone). */
+export interface FactOp {
+  kind: 'noun' | 'verb'
+  id: string
+  /** The AFTER-IMAGE legs, or `null` for a tombstone (the id was removed). */
+  record: { metadata: unknown | null; vector: unknown | null } | null
+}
+
+/** One committed generation, as scanned back out of the log. */
+export interface CommitFact {
+  generation: number
+  timestamp: number
+  ops: FactOp[]
+  meta?: Record<string, unknown>
+  blobHashes?: string[]
+}
+
+/** The telemetry a scan batch carries (frozen shape). */
+export interface FactScanBatch {
+  facts: CommitFact[]
+  firstGeneration: number
+  lastGeneration: number
+  factCount: number
+  byteSize: number
+  segmentId: string
+}
+
+/**
+ * Liveness bound on a scan's FIRST batch (Stage-2 co-freeze, D1 contract):
+ * `batches()` must yield its first batch — or fail loudly — within this many
+ * ms of the first pull. A backlogged or damaged store may be SLOW, but it may
+ * never be SILENT: a consumer awaiting the first batch is otherwise
+ * indistinguishable from a wedge (the exact failure shape a production heal
+ * hit against a generations-backlogged brain).
+ */
+export const SCANFACTS_FIRST_BATCH_MS = 10_000
+
+/** The telemetry a scan OPEN returns (frozen shape). */
+export interface FactScanHandle {
+  headGeneration: number
+  segmentCount: number
+  approxFactCount: number
+  /**
+   * Ordered batches; a detected gap aborts LOUDLY, never a silent skip.
+   * Liveness contract: the FIRST batch resolves or rejects within
+   * {@link SCANFACTS_FIRST_BATCH_MS} of the first pull — never a silent hang.
+   */
+  batches: () => AsyncGenerator<FactScanBatch>
+  /** Close telemetry — the invariant cross-check, valid after iteration ends. */
+  summary: () => { factsYielded: number; segmentsRead: number }
+}
+
+/** Manifest entry for a sealed segment. */
+interface SegmentEntry {
+  file: string
+  firstGeneration: number
+  lastGeneration: number
+  facts: number
+  bytes: number
+}
+
+/** The facts manifest (JSON on disk). */
+interface FactsManifest {
+  formatVersion: number
+  segments: SegmentEntry[]
+  /** The append target. Its true content is established by scanning (crash tolerance). */
+  tailSegment: string | null
+  updatedAt: string
+}
+
+/** The narrow byte-level storage surface the fact log rides. */
+export interface FactLogStorage {
+  appendRawBytes(path: string, bytes: Uint8Array): Promise<void>
+  readRawBytes(path: string): Promise<Uint8Array | null>
+  writeRawBytes(path: string, bytes: Uint8Array): Promise<void>
+  rawByteSize(path: string): Promise<number | null>
+  readRawObject(path: string): Promise<any | null>
+  writeRawObject(path: string, data: any): Promise<void>
+  syncRawObjects(paths: string[]): Promise<void>
+  deleteRawObject(path: string): Promise<void>
+}
+
+/** True when the storage adapter exposes every primitive the fact log needs. */
+export function storageSupportsFactLog(storage: unknown): storage is FactLogStorage {
+  const s = storage as Record<string, unknown>
+  return (
+    typeof s.appendRawBytes === 'function' &&
+    typeof s.readRawBytes === 'function' &&
+    typeof s.writeRawBytes === 'function' &&
+    typeof s.rawByteSize === 'function'
+  )
+}
+
+/** uuid string → 16 raw bytes (bin16 on the wire). */
+function uuidToBytes(id: string): Uint8Array {
+  const hex = id.replace(/-/g, '')
+  if (hex.length !== 32) {
+    // Non-uuid ids (legacy/natural keys) ride as UTF-8 with a length prefix
+    // marker impossible for uuids: we refuse instead — the write API has
+    // guaranteed uuid ids since 8.0, so anything else is a corruption signal.
+    throw new Error(`fact log: id is not a uuid: ${id}`)
+  }
+  const bytes = new Uint8Array(16)
+  for (let i = 0; i < 16; i++) {
+    bytes[i] = parseInt(hex.slice(i * 2, i * 2 + 2), 16)
+  }
+  return bytes
+}
+
+/** 16 raw bytes → canonical lowercase uuid string. */
+function bytesToUuid(bytes: Uint8Array): string {
+  let hex = ''
+  for (let i = 0; i < 16; i++) hex += bytes[i].toString(16).padStart(2, '0')
+  return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20)}`
+}
+
+/** Zero-padded segment filename: lexicographic order == generation order. */
+function segmentFileName(firstGeneration: number): string {
+  return `seg-${String(firstGeneration).padStart(20, '0')}.bfl`
+}
+
+/** Build a segment header. Reserved bytes are ZEROED (and verified on open). */
+function buildHeader(firstGeneration: number): Uint8Array {
+  const header = new Uint8Array(HEADER_BYTES)
+  header.set(MAGIC, 0)
+  const view = new DataView(header.buffer)
+  view.setUint32(8, FACTS_FORMAT_VERSION, true)
+  view.setBigUint64(12, BigInt(firstGeneration), true)
+  // bytes 20..31 stay zero (reserved)
+  return header
+}
+
+/** Encode one fact into a framed record (length + crc32c + msgpack payload). */
+function encodeFrame(fact: CommitFact): Uint8Array {
+  const payload = msgpackEncode([
+    fact.generation,
+    fact.timestamp,
+    fact.ops.map((op) => [
+      op.kind === 'noun' ? 0 : 1,
+      uuidToBytes(op.id),
+      op.record === null ? null : [op.record.metadata, op.record.vector]
+    ]),
+    fact.meta ?? null,
+    fact.blobHashes && fact.blobHashes.length > 0 ? fact.blobHashes : null
+  ])
+  const frame = new Uint8Array(FRAME_PREFIX_BYTES + payload.length)
+  const view = new DataView(frame.buffer)
+  view.setUint32(0, payload.length, true)
+  view.setUint32(4, crc32c(payload), true)
+  frame.set(payload, FRAME_PREFIX_BYTES)
+  return frame
+}
+
+/** Decode one msgpack payload back into a CommitFact. */
+function decodeFact(payload: Uint8Array): CommitFact {
+  const raw = msgpackDecode(payload) as unknown[]
+  const [generation, timestamp, ops, meta, blobHashes] = raw as [
+    number,
+    number,
+    Array<[number, Uint8Array, [unknown, unknown] | null]>,
+    Record<string, unknown> | null,
+    string[] | null
+  ]
+  return {
+    generation: Number(generation),
+    timestamp: Number(timestamp),
+    ops: ops.map(([kind, idBytes, record]) => ({
+      kind: kind === 0 ? ('noun' as const) : ('verb' as const),
+      id: bytesToUuid(idBytes),
+      record: record === null ? null : { metadata: record[0] ?? null, vector: record[1] ?? null }
+    })),
+    ...(meta ? { meta } : {}),
+    ...(blobHashes && blobHashes.length > 0 ? { blobHashes } : {})
+  }
+}
+
+/**
+ * Parse a segment's bytes: verify the header, then walk frames until the end
+ * or a torn tail (length overrun / CRC mismatch), which terminates the walk —
+ * everything before it is intact. Returns the decoded facts plus the byte
+ * length of the VALID prefix (header + intact frames), which reconciliation
+ * uses to cut a torn tail without re-encoding.
+ */
+function parseSegment(
+  file: string,
+  bytes: Uint8Array
+): { facts: CommitFact[]; validBytes: number } {
+  if (bytes.length < HEADER_BYTES) {
+    prodLog.warn(`[FactLog] segment ${file} shorter than its header — treating as empty`)
+    return { facts: [], validBytes: 0 }
+  }
+  for (let i = 0; i < MAGIC.length; i++) {
+    if (bytes[i] !== MAGIC[i]) {
+      throw new Error(`fact log: segment ${file} has a bad magic — not a fact segment`)
+    }
+  }
+  const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength)
+  const version = view.getUint32(8, true)
+  if (version !== FACTS_FORMAT_VERSION) {
+    throw new Error(
+      `fact log: segment ${file} has formatVersion ${version}; this build reads ${FACTS_FORMAT_VERSION}`
+    )
+  }
+  for (let i = 20; i < HEADER_BYTES; i++) {
+    if (bytes[i] !== 0) {
+      // Non-zero reserved bytes = a future format this build cannot verify.
+      throw new Error(`fact log: segment ${file} has non-zero reserved header bytes — unverifiable`)
+    }
+  }
+
+  const facts: CommitFact[] = []
+  let offset = HEADER_BYTES
+  while (offset + FRAME_PREFIX_BYTES <= bytes.length) {
+    const length = view.getUint32(offset, true)
+    const expectedCrc = view.getUint32(offset + 4, true)
+    const start = offset + FRAME_PREFIX_BYTES
+    const end = start + length
+    if (end > bytes.length) break // torn tail: frame length overruns the file
+    const payload = bytes.subarray(start, end)
+    if (crc32c(payload) !== expectedCrc) break // torn tail: payload CRC mismatch
+    facts.push(decodeFact(payload))
+    offset = end
+  }
+  return { facts, validBytes: offset }
+}
+
+/**
+ * The generation fact log. One instance per open store; every method assumes
+ * the single-writer discipline the generation store already enforces (calls
+ * arrive under its commit mutex).
+ */
+export class FactLog {
+  private readonly storage: FactLogStorage
+  /** Rotation threshold (bytes); tests may lower it to exercise rotation. */
+  private readonly rotateBytes: number
+  private manifest: FactsManifest = {
+    formatVersion: FACTS_FORMAT_VERSION,
+    segments: [],
+    tailSegment: null,
+    updatedAt: new Date(0).toISOString()
+  }
+  /** Decoded facts of the TAIL segment (bounded by the rotation threshold). */
+  private tailFacts: CommitFact[] = []
+  /** Byte size of the tail segment file (valid prefix). */
+  private tailBytes = 0
+  /** Highest generation in the log (0 = empty). */
+  private head = 0
+  /** Segment paths appended since the last sync (the fsync batch). */
+  private readonly dirtySegments = new Set<string>()
+
+  constructor(storage: FactLogStorage, options?: { rotateBytes?: number }) {
+    this.storage = storage
+    this.rotateBytes = options?.rotateBytes ?? SEGMENT_ROTATE_BYTES
+  }
+
+  /** The highest committed generation the log holds (0 = empty). */
+  headGeneration(): number {
+    return this.head
+  }
+
+  /**
+   * Open the log and reconcile it to committed truth: read the manifest,
+   * establish the tail's intact content (torn-tail scan), then TRUNCATE any
+   * fact with `generation > committedGeneration` — those never committed (a
+   * crash between fact-append and the commit point). After open, the log is
+   * exactly the committed prefix.
+   */
+  async open(committedGeneration: number): Promise<void> {
+    const stored = (await this.storage.readRawObject(FACTS_MANIFEST_PATH)) as FactsManifest | null
+    if (stored && typeof stored === 'object' && Array.isArray(stored.segments)) {
+      if (stored.formatVersion !== FACTS_FORMAT_VERSION) {
+        throw new Error(
+          `fact log: manifest formatVersion ${stored.formatVersion}; this build reads ${FACTS_FORMAT_VERSION}`
+        )
+      }
+      this.manifest = stored
+    }
+
+    // Drop sealed segments that sit ENTIRELY beyond committed truth (a crash
+    // right after a rotation whose facts never committed), newest first.
+    while (this.manifest.segments.length > 0) {
+      const last = this.manifest.segments[this.manifest.segments.length - 1]
+      if (last.firstGeneration > committedGeneration) {
+        prodLog.warn(
+          `[FactLog] dropping sealed segment ${last.file} (generations ${last.firstGeneration}..` +
+            `${last.lastGeneration} never committed)`
+        )
+        await this.storage.deleteRawObject(`${FACTS_PREFIX}/${last.file}`)
+        this.manifest.segments.pop()
+        await this.persistManifest()
+      } else if (last.lastGeneration > committedGeneration) {
+        // A sealed segment STRADDLING committed truth: cut it back.
+        await this.truncateSegmentTo(last.file, committedGeneration)
+        const cut = await this.reloadSegmentEntry(last.file)
+        this.manifest.segments[this.manifest.segments.length - 1] = cut
+        await this.persistManifest()
+        break
+      } else {
+        break
+      }
+    }
+
+    // Establish the tail: scan its intact prefix, then truncate beyond
+    // committed truth (the common crash shape: buffered single-op facts whose
+    // counter never went durable).
+    if (this.manifest.tailSegment) {
+      const tailPath = `${FACTS_PREFIX}/${this.manifest.tailSegment}`
+      const bytes = await this.storage.readRawBytes(tailPath)
+      if (bytes === null) {
+        // Manifest named a tail whose first byte never landed — an empty tail.
+        this.tailFacts = []
+        this.tailBytes = 0
+      } else {
+        const { facts, validBytes } = parseSegment(this.manifest.tailSegment, bytes)
+        const kept = facts.filter((f) => f.generation <= committedGeneration)
+        if (kept.length !== facts.length || validBytes !== bytes.length) {
+          const dropped = facts.length - kept.length
+          if (dropped > 0) {
+            prodLog.warn(
+              `[FactLog] truncating ${dropped} uncommitted fact(s) beyond generation ` +
+                `${committedGeneration} from the tail (never committed)`
+            )
+          }
+          await this.rewriteTail(kept)
+        } else {
+          this.tailFacts = facts
+          this.tailBytes = validBytes
+        }
+      }
+    }
+
+    this.head = this.computeHead()
+  }
+
+  /**
+   * Append one committed generation's fact. NOT durable until {@link sync} —
+   * the caller batches durability at its commit barrier (transact syncs in
+   * the same call; Model-B group-commit syncs at flush).
+   */
+  async append(fact: CommitFact): Promise<void> {
+    if (fact.generation <= this.head) {
+      throw new Error(
+        `fact log: non-monotonic append (generation ${fact.generation} ≤ head ${this.head})`
+      )
+    }
+    if (this.manifest.tailSegment === null) {
+      await this.startTail(fact.generation)
+    } else if (this.tailBytes >= this.rotateBytes) {
+      await this.rotate(fact.generation)
+    }
+    const frame = encodeFrame(fact)
+    const tailPath = `${FACTS_PREFIX}/${this.manifest.tailSegment}`
+    await this.storage.appendRawBytes(tailPath, frame)
+    this.tailFacts.push(fact)
+    this.tailBytes += frame.length
+    this.head = fact.generation
+    this.dirtySegments.add(tailPath)
+  }
+
+  /** Fsync every segment appended since the last sync. */
+  async sync(): Promise<void> {
+    if (this.dirtySegments.size === 0) return
+    const paths = [...this.dirtySegments]
+    this.dirtySegments.clear()
+    await this.storage.syncRawObjects(paths)
+  }
+
+  /**
+   * Open a scan over committed facts. The scan runs against a MANIFEST
+   * SNAPSHOT (sealed segments + the tail's decoded facts at open) — exactly-
+   * once per fact, inclusive bounds, stable under concurrent appends. Gaps
+   * abort LOUDLY: a missing generation inside a segment's declared range is
+   * corruption, never silently skipped.
+   */
+  scanFacts(options?: {
+    fromGeneration?: number
+    toGeneration?: number
+    kinds?: Array<'noun' | 'verb'>
+    batchSize?: number
+    /** Test override for the first-batch liveness bound (default {@link SCANFACTS_FIRST_BATCH_MS}). */
+    firstBatchTimeoutMs?: number
+  }): FactScanHandle {
+    const from = options?.fromGeneration ?? 1
+    const to = options?.toGeneration ?? this.head
+    const kinds = options?.kinds
+    const batchSize = Math.max(1, options?.batchSize ?? 256)
+
+    // Snapshot: the segment list + tail content as of NOW.
+    const segments = this.manifest.segments.filter(
+      (s) => s.lastGeneration >= from && s.firstGeneration <= to
+    )
+    const tailSnapshot = this.tailFacts.filter((f) => f.generation >= from && f.generation <= to)
+    const tailId = this.manifest.tailSegment ?? 'tail'
+    const approxFactCount =
+      segments.reduce((sum, s) => sum + s.facts, 0) + tailSnapshot.length
+
+    let factsYielded = 0
+    let segmentsRead = 0
+    const storage = this.storage
+
+    async function* batches(this: void): AsyncGenerator<FactScanBatch> {
+      let expectedNext = 0 // gap detection: generations are monotonic, not necessarily dense
+      const emit = (facts: CommitFact[], segmentId: string, byteSize: number): FactScanBatch => ({
+        facts,
+        firstGeneration: facts[0].generation,
+        lastGeneration: facts[facts.length - 1].generation,
+        factCount: facts.length,
+        byteSize,
+        segmentId
+      })
+      const filterOps = (fact: CommitFact): CommitFact =>
+        kinds
+          ? { ...fact, ops: fact.ops.filter((op) => kinds.includes(op.kind)) }
+          : fact
+
+      for (const entry of segments) {
+        const bytes = await storage.readRawBytes(`${FACTS_PREFIX}/${entry.file}`)
+        if (bytes === null) {
+          throw new Error(
+            `fact log: sealed segment ${entry.file} is MISSING — the log is damaged; aborting scan`
+          )
+        }
+        const { facts } = parseSegment(entry.file, bytes)
+        segmentsRead++
+        const inRange = facts.filter((f) => f.generation >= from && f.generation <= to)
+        for (const f of inRange) {
+          if (f.generation <= expectedNext - 1) {
+            throw new Error(`fact log: out-of-order fact ${f.generation} in ${entry.file} — aborting scan`)
+          }
+          expectedNext = f.generation + 1
+        }
+        for (let i = 0; i < inRange.length; i += batchSize) {
+          const slice = inRange.slice(i, i + batchSize).map(filterOps)
+          if (slice.length === 0) continue
+          factsYielded += slice.length
+          yield emit(slice, entry.file, slice.reduce((n, f) => n + encodeFrame(f).length, 0))
+        }
+      }
+
+      if (tailSnapshot.length > 0) {
+        segmentsRead++
+        for (const f of tailSnapshot) {
+          if (f.generation <= expectedNext - 1) {
+            throw new Error(`fact log: out-of-order fact ${f.generation} in the tail — aborting scan`)
+          }
+          expectedNext = f.generation + 1
+        }
+        for (let i = 0; i < tailSnapshot.length; i += batchSize) {
+          const slice = tailSnapshot.slice(i, i + batchSize).map(filterOps)
+          factsYielded += slice.length
+          yield emit(slice, tailId, slice.reduce((n, f) => n + encodeFrame(f).length, 0))
+        }
+      }
+    }
+
+    // Liveness wrapper: the FIRST pull races the contract deadline. Only the
+    // first — the bound is time-to-first-batch (proof the producer is alive),
+    // not per-batch pacing; and it runs only while a pull is actually pending,
+    // so consumer think-time between pulls never counts against the producer.
+    const firstBatchTimeoutMs = options?.firstBatchTimeoutMs ?? SCANFACTS_FIRST_BATCH_MS
+    async function* batchesWithLiveness(this: void): AsyncGenerator<FactScanBatch> {
+      const inner = batches()
+      let timer: NodeJS.Timeout | undefined
+      try {
+        const deadline = new Promise<never>((_, reject) => {
+          timer = setTimeout(
+            () =>
+              reject(
+                new Error(
+                  `fact log: scanFacts produced no first batch within ${firstBatchTimeoutMs}ms ` +
+                    `(liveness contract) — the store is wedged or unreadably slow; aborting scan LOUDLY ` +
+                    `instead of hanging the consumer.`
+                )
+              ),
+            firstBatchTimeoutMs
+          )
+          timer.unref?.()
+        })
+        const first = await Promise.race([inner.next(), deadline])
+        if (first.done) return
+        yield first.value
+      } finally {
+        clearTimeout(timer)
+      }
+      yield* inner
+    }
+
+    return {
+      headGeneration: this.head,
+      segmentCount: segments.length + (tailSnapshot.length > 0 ? 1 : 0),
+      approxFactCount,
+      batches: batchesWithLiveness,
+      summary: () => ({ factsYielded, segmentsRead })
+    }
+  }
+
+  /**
+   * The mmap fast path (capability handoff): the immutable sealed segment
+   * files covering `fromGeneration`, in order. The TAIL is deliberately NOT
+   * included — it is append-mutable; consumers read it via {@link scanFacts}.
+   */
+  segmentPaths(options?: { fromGeneration?: number }): string[] {
+    const from = options?.fromGeneration ?? 1
+    return this.manifest.segments
+      .filter((s) => s.lastGeneration >= from)
+      .map((s) => `${FACTS_PREFIX}/${s.file}`)
+  }
+
+  /**
+   * Drop every fact with `generation > keepThrough` — the in-session abort
+   * compensation: a transact appends its fact BEFORE the commit point, so a
+   * real (non-crash) abort after the append must take the fact back out. The
+   * dropped facts can only live in the TAIL (they were just appended); the
+   * rewrite is atomic and bounded by the rotation threshold.
+   */
+  async dropAbove(keepThrough: number): Promise<void> {
+    if (this.head <= keepThrough) return
+    const kept = this.tailFacts.filter((f) => f.generation <= keepThrough)
+    if (kept.length === this.tailFacts.length) {
+      throw new Error(
+        `fact log: dropAbove(${keepThrough}) found no droppable facts in the tail ` +
+          `(head ${this.head}) — the fact to drop was already sealed; the log needs reopen`
+      )
+    }
+    await this.rewriteTail(kept)
+    this.head = this.computeHead()
+  }
+
+  // -- internals -------------------------------------------------------------
+
+  private computeHead(): number {
+    if (this.tailFacts.length > 0) return this.tailFacts[this.tailFacts.length - 1].generation
+    const sealed = this.manifest.segments
+    if (sealed.length > 0) return sealed[sealed.length - 1].lastGeneration
+    return 0
+  }
+
+  /** Create the very first tail segment (manifest-first, then header bytes). */
+  private async startTail(firstGeneration: number): Promise<void> {
+    const file = segmentFileName(firstGeneration)
+    this.manifest.tailSegment = file
+    await this.persistManifest()
+    await this.storage.appendRawBytes(`${FACTS_PREFIX}/${file}`, buildHeader(firstGeneration))
+    this.tailFacts = []
+    this.tailBytes = HEADER_BYTES
+  }
+
+  /**
+   * Seal the tail into the manifest and start a new one. Manifest-first: the
+   * flip both seals the old tail AND names the new one atomically, so no
+   * segment file ever exists unaccounted for.
+   */
+  private async rotate(nextGeneration: number): Promise<void> {
+    const sealedFile = this.manifest.tailSegment
+    if (!sealedFile) return
+    // Seal what the tail actually holds.
+    await this.sync() // sealed segments are always fully durable
+    const entry: SegmentEntry = {
+      file: sealedFile,
+      firstGeneration: this.tailFacts[0]?.generation ?? nextGeneration,
+      lastGeneration: this.tailFacts[this.tailFacts.length - 1]?.generation ?? nextGeneration - 1,
+      facts: this.tailFacts.length,
+      bytes: this.tailBytes
+    }
+    const newFile = segmentFileName(nextGeneration)
+    this.manifest.segments.push(entry)
+    this.manifest.tailSegment = newFile
+    await this.persistManifest()
+    await this.storage.appendRawBytes(`${FACTS_PREFIX}/${newFile}`, buildHeader(nextGeneration))
+    this.tailFacts = []
+    this.tailBytes = HEADER_BYTES
+  }
+
+  /** Atomically persist the manifest (write-new → fsync → rename downstream). */
+  private async persistManifest(): Promise<void> {
+    this.manifest.updatedAt = new Date().toISOString()
+    await this.storage.writeRawObject(FACTS_MANIFEST_PATH, this.manifest)
+    await this.storage.syncRawObjects([FACTS_MANIFEST_PATH])
+  }
+
+  /** Rewrite the tail segment to hold exactly `facts` (atomic replace). */
+  private async rewriteTail(facts: CommitFact[]): Promise<void> {
+    const file = this.manifest.tailSegment
+    if (!file) return
+    const first = facts[0]?.generation ?? this.segmentFirstGenerationFromName(file)
+    const parts: Uint8Array[] = [buildHeader(first)]
+    for (const f of facts) parts.push(encodeFrame(f))
+    const total = parts.reduce((n, p) => n + p.length, 0)
+    const merged = new Uint8Array(total)
+    let offset = 0
+    for (const p of parts) {
+      merged.set(p, offset)
+      offset += p.length
+    }
+    await this.storage.writeRawBytes(`${FACTS_PREFIX}/${file}`, merged)
+    this.tailFacts = facts
+    this.tailBytes = total
+  }
+
+  /** Cut a SEALED segment back to `committedGeneration` (atomic replace). */
+  private async truncateSegmentTo(file: string, committedGeneration: number): Promise<void> {
+    const path = `${FACTS_PREFIX}/${file}`
+    const bytes = await this.storage.readRawBytes(path)
+    if (bytes === null) return
+    const { facts } = parseSegment(file, bytes)
+    const kept = facts.filter((f) => f.generation <= committedGeneration)
+    prodLog.warn(
+      `[FactLog] truncating sealed segment ${file} to generation ${committedGeneration} ` +
+        `(${facts.length - kept.length} uncommitted fact(s) dropped)`
+    )
+    const first = kept[0]?.generation ?? this.segmentFirstGenerationFromName(file)
+    const parts: Uint8Array[] = [buildHeader(first)]
+    for (const f of kept) parts.push(encodeFrame(f))
+    const total = parts.reduce((n, p) => n + p.length, 0)
+    const merged = new Uint8Array(total)
+    let offset = 0
+    for (const p of parts) {
+      merged.set(p, offset)
+      offset += p.length
+    }
+    await this.storage.writeRawBytes(path, merged)
+  }
+
+  /** Re-derive a sealed segment's manifest entry from its actual bytes. */
+  private async reloadSegmentEntry(file: string): Promise<SegmentEntry> {
+    const bytes = await this.storage.readRawBytes(`${FACTS_PREFIX}/${file}`)
+    const { facts, validBytes } = bytes
+      ? parseSegment(file, bytes)
+      : { facts: [] as CommitFact[], validBytes: 0 }
+    return {
+      file,
+      firstGeneration: facts[0]?.generation ?? this.segmentFirstGenerationFromName(file),
+      lastGeneration: facts[facts.length - 1]?.generation ?? 0,
+      facts: facts.length,
+      bytes: validBytes
+    }
+  }
+
+  /** Parse the zero-padded firstGeneration back out of a segment filename. */
+  private segmentFirstGenerationFromName(file: string): number {
+    const match = /^seg-(\d{20})\.bfl$/.exec(file)
+    return match ? Number(match[1]) : 0
+  }
+}

src/db/familyStamp.ts

@@ -0,0 +1,152 @@
+/**
+ * @module db/familyStamp
+ * @description The generalized FAMILY STAMP — one JSON shape that declares,
+ * for any derived projection, WHICH source state it reflects and HOW to verify
+ * it is whole. The entity tree (canonical current-state files) carries the
+ * first brainy-side stamp; native index families carry the same shape. One
+ * verifier reads both member modes:
+ *
+ * - `enumerated` — bounded families: exact byte size per member file,
+ *   verified at open.
+ * - `rollup`     — unbounded families (the entity tree: millions of files):
+ *   the verified surface is a small set of rollup invariants (entity/
+ *   relationship counts) plus `sourceGeneration`.
+ *
+ * `sourceGeneration` is the generation of the source-of-truth log this
+ * projection reflects — open-time coherence becomes a COMPARISON (stamp vs
+ * log head), not a walk:
+ *
+ * - equal + invariants hold → coherent, serve.
+ * - behind → the projection missed the tail (crash between commit and stamp);
+ *   for the Stage-1 tree this is benign by construction (the tree is written
+ *   BY the commit), so the stamp refreshes; a DERIVED projection would replay
+ *   the gap instead.
+ * - invariants FAIL at equal generation → genuine incoherence: loud, and the
+ *   repair ritual (`repairIndex()`, whose recount rebuilds the rollups from a
+ *   canonical walk) heals it.
+ *
+ * Stamps are JSON on purpose — every incident gets debugged by reading a
+ * stamp in a terminal.
+ */
+
+/** Storage-root-relative directory holding family stamps. */
+export const FAMILY_STAMPS_PREFIX = '_system/family-stamps'
+
+/** The entity tree's stamp path. */
+export const ENTITY_TREE_STAMP_PATH = `${FAMILY_STAMPS_PREFIX}/entity-tree.json`
+
+/** One enumerated member: a file and its exact expected byte size. */
+export interface EnumeratedMember {
+  path: string
+  bytes: number
+}
+
+/**
+ * The stamp's verified surface, in one of the two member modes. Rollup
+ * invariant values may be numbers (counts, byte sizes) or strings (content
+ * fingerprints, e.g. a per-tree SHA-256) — the verifier compares by strict
+ * equality either way, so a type mismatch reads as incoherence, never a pass.
+ */
+export type StampMembers =
+  | { mode: 'enumerated'; files: EnumeratedMember[] }
+  | { mode: 'rollup'; invariants: Record<string, number | string> }
+
+/** The generalized family stamp (one shape, one verifier, both engines). */
+export interface FamilyStamp {
+  /** Which projection this stamps (e.g. `'entity-tree'`). */
+  family: string
+  /** Monotonic per-family stamp generation — bumps on every committed stamp. */
+  generation: number
+  /** ISO timestamp of the stamp write. */
+  committedAt: string
+  /** The source-of-truth generation this projection reflects. */
+  sourceGeneration: number
+  /** The verified surface. */
+  members: StampMembers
+}
+
+/** The verdict of an open-time stamp verification. */
+export type StampVerdict =
+  | { state: 'coherent' }
+  | { state: 'absent' } // legacy store — first stamp writes at the next flush
+  | { state: 'behind'; stampSource: number; head: number }
+  | { state: 'incoherent'; failures: string[] }
+  | { state: 'unverifiable'; reason: string } // a FAULT reading the stamp — never conflated with absence
+
+/** The narrow storage surface stamps ride (JSON objects + fsync). */
+export interface StampStorage {
+  readRawObject(path: string): Promise<any | null>
+  writeRawObject(path: string, data: any): Promise<void>
+  syncRawObjects(paths: string[]): Promise<void>
+}
+
+/** Read a family's stamp; `null` when none was ever written. */
+export async function readFamilyStamp(
+  storage: StampStorage,
+  path: string
+): Promise<FamilyStamp | null> {
+  const stored = (await storage.readRawObject(path)) as FamilyStamp | null
+  if (!stored || typeof stored !== 'object' || typeof stored.family !== 'string') return null
+  return stored
+}
+
+/** Write a family's stamp durably (atomic object write + fsync). */
+export async function writeFamilyStamp(
+  storage: StampStorage,
+  path: string,
+  stamp: Omit<FamilyStamp, 'generation' | 'committedAt'> & { generation?: number }
+): Promise<void> {
+  const prior = await readFamilyStamp(storage, path)
+  const full: FamilyStamp = {
+    ...stamp,
+    generation: (prior?.generation ?? 0) + 1,
+    committedAt: new Date().toISOString()
+  }
+  await storage.writeRawObject(path, full)
+  await storage.syncRawObjects([path])
+}
+
+/**
+ * The ONE verifier, both member modes. `actual` supplies the observed rollup
+ * values (rollup mode) or file sizes (enumerated mode, keyed by path);
+ * `head` is the source-of-truth generation now.
+ */
+export function verifyFamilyStamp(
+  stamp: FamilyStamp | null,
+  head: number,
+  actual: Record<string, number | string>
+): StampVerdict {
+  if (stamp === null) return { state: 'absent' }
+  if (stamp.sourceGeneration > head) {
+    // A stamp AHEAD of the log claims state that never committed — the
+    // projection was stamped against truth that a crash rolled back.
+    return {
+      state: 'incoherent',
+      failures: [`sourceGeneration ${stamp.sourceGeneration} is ahead of the log head ${head}`]
+    }
+  }
+  if (stamp.sourceGeneration < head) {
+    return { state: 'behind', stampSource: stamp.sourceGeneration, head }
+  }
+  const failures: string[] = []
+  if (stamp.members.mode === 'rollup') {
+    for (const [name, expected] of Object.entries(stamp.members.invariants)) {
+      const observed = actual[name]
+      if (observed === undefined) {
+        failures.push(`rollup invariant '${name}' has no observed value`)
+      } else if (observed !== expected) {
+        failures.push(`rollup invariant '${name}': stamped ${expected}, observed ${observed}`)
+      }
+    }
+  } else {
+    for (const member of stamp.members.files) {
+      const observed = actual[member.path]
+      if (observed === undefined) {
+        failures.push(`member '${member.path}' is missing`)
+      } else if (observed !== member.bytes) {
+        failures.push(`member '${member.path}': stamped ${member.bytes} bytes, observed ${observed}`)
+      }
+    }
+  }
+  return failures.length > 0 ? { state: 'incoherent', failures } : { state: 'coherent' }
+}

src/db/generationSegments.ts

@@ -0,0 +1,459 @@
+/**
+ * @module db/generationSegments
+ * @description The generation-segment store — Stage-2 D1+D3+repacking's file
+ * format (co-frozen 2026-07-19; design: the d1-d3-repacking spec).
+ *
+ * Packs CONSECUTIVE cold generations' record-sets (before-images + delta)
+ * into append-once segment files with derived sidecar indexes, so history
+ * scales in SEGMENTS (tens) instead of FILES-PER-GENERATION (hundreds of
+ * thousands), and cold-open reads ONE manifest instead of listing the
+ * backlog. Layout under `_generations/segments/`:
+ *
+ * - `seg-<firstGen, zero-padded 20>.bgs` — magic "BGS1", then one frame per
+ *   generation: `u32 payloadLen | u32 crc32c | msgpack payload`. Payload is
+ *   POSITIONAL: `[generation, timestamp, delta, records[], flags]` with
+ *   records `[kindByte, id, record]`. `flags` reserves encoding evolution
+ *   (bit 0 = compressed payload — v1 always 0; a future writer upgrade,
+ *   never a format break). Sealed segments are IMMUTABLE — the fact log's
+ *   own law, generalized.
+ * - `seg-<firstGen>.idx` — DERIVED sidecar (msgpack): per-generation frame
+ *   offsets (point reads = one ranged read, never a listing) + per-id
+ *   generation postings (per-id chain rebuilds read only what they need).
+ *   Corrupt/missing → rebuilt from its segment in one sequential read,
+ *   loudly.
+ * - `manifest.json` — the segment catalogue + `compactedBelow` (D3's
+ *   horizon marker). Cold-open reads THIS; the packed backlog is never
+ *   listed.
+ *
+ * D3 semantics carried here: bounded-retention reclaim drops WHOLE segments
+ * at boundaries (O(1) per segment, no rewrite); under the archival profile
+ * (`retention: 'all'`) nothing here is ever dropped — folding is the only
+ * transform (re-representation, never deletion).
+ */
+
+import { encode as msgpackEncode, decode as msgpackDecode } from '@msgpack/msgpack'
+import { crc32c } from '../utils/crc32c.js'
+import type { FactLogStorage } from './factLog.js'
+import { prodLog } from '../utils/logger.js'
+
+/** Directory for segment files + manifest, under the generations prefix. */
+export const SEGMENTS_PREFIX = '_generations/segments'
+
+/** Target sealed-segment size (co-freeze proposal; tunable on evidence). */
+export const SEGMENT_TARGET_BYTES = 64 * 1024 * 1024
+
+const MAGIC = new TextEncoder().encode('BGS1')
+const FRAME_PREFIX_BYTES = 8 // u32 payloadLen + u32 crc32c
+const MANIFEST_PATH = `${SEGMENTS_PREFIX}/manifest.json`
+
+/** One generation's fold input — exactly what the live tier holds for it. */
+export interface FoldGeneration {
+  generation: number
+  timestamp: number
+  /** The tx.json delta object, carried verbatim. */
+  delta: unknown
+  /** The before-image record-set (empty for record-less generations). */
+  records: Array<{ kind: 'noun' | 'verb'; id: string; record: unknown }>
+}
+
+/** Manifest entry for one sealed segment. */
+export interface SegmentMeta {
+  file: string
+  firstGeneration: number
+  lastGeneration: number
+  frames: number
+  bytes: number
+  /** crc32c of the full segment byte stream — the digest chain's link. */
+  checksum: number
+}
+
+interface SegmentManifest {
+  version: 1
+  compactedBelow: number
+  segments: SegmentMeta[]
+}
+
+interface SidecarIndex {
+  version: 1
+  /** [generation, frameOffset, frameLen] ascending by generation. */
+  generations: Array<[number, number, number]>
+  /** `${kindByte}:${id}` → ascending generations holding a record for it. */
+  ids: Record<string, number[]>
+}
+
+const segmentFileName = (firstGeneration: number): string =>
+  `seg-${String(firstGeneration).padStart(20, '0')}.bgs`
+const sidecarFileName = (firstGeneration: number): string =>
+  `seg-${String(firstGeneration).padStart(20, '0')}.idx`
+
+/**
+ * The generation-segment store. Owns the packed tier ONLY — the live
+ * per-generation tier and the routing between tiers belong to
+ * `GenerationStore`. All mutating entry points here are called under the
+ * generation store's commit mutex.
+ */
+export class GenerationSegmentStore {
+  private readonly storage: FactLogStorage
+  private manifest: SegmentManifest = { version: 1, compactedBelow: 0, segments: [] }
+  /** Sidecar cache — segments are immutable, so entries never invalidate. */
+  private readonly sidecars = new Map<string, SidecarIndex>()
+
+  constructor(storage: FactLogStorage) {
+    this.storage = storage
+  }
+
+  /** Load the manifest (ONE read — never a directory listing). */
+  async open(): Promise<void> {
+    const raw = (await this.storage.readRawObject(MANIFEST_PATH)) as SegmentManifest | null
+    if (raw) {
+      if (raw.version !== 1) {
+        throw new Error(
+          `[GenerationSegments] manifest version ${String(raw.version)} is newer than this ` +
+            `engine understands — refusing to serve partial history. Upgrade the engine.`
+        )
+      }
+      this.manifest = raw
+    }
+  }
+
+  /** The packed tier's catalogue (ascending, immutable snapshot). */
+  segments(): readonly SegmentMeta[] {
+    return this.manifest.segments
+  }
+
+  /** D3's horizon marker: generations below this were reclaimed (bounded profiles only). */
+  compactedBelow(): number {
+    return this.manifest.compactedBelow
+  }
+
+  /** The covering sealed segment for `gen`, or null if it lives outside the packed tier. */
+  private coveringSegment(gen: number): SegmentMeta | null {
+    // Manifest is ascending and ranges never overlap — binary search.
+    const segs = this.manifest.segments
+    let lo = 0
+    let hi = segs.length - 1
+    while (lo <= hi) {
+      const mid = (lo + hi) >> 1
+      const s = segs[mid]
+      if (gen < s.firstGeneration) hi = mid - 1
+      else if (gen > s.lastGeneration) lo = mid + 1
+      else return s
+    }
+    return null
+  }
+
+  /** True when `gen` is packed (readable from this tier). */
+  hasGeneration(gen: number): boolean {
+    return this.coveringSegment(gen) !== null
+  }
+
+  /**
+   * Fold consecutive generations into ONE new sealed segment + sidecar and
+   * append it to the manifest atomically. Caller guarantees: `gens` is
+   * ascending, contiguous with the packed tier (first = last packed + 1 when
+   * segments exist), and already durable in the live tier. Crash between the
+   * segment write and the caller's live-tier delete leaves a DUPLICATE
+   * representation — resolved live-tier-wins by the reader; never a gap.
+   */
+  async fold(gens: FoldGeneration[]): Promise<SegmentMeta> {
+    if (gens.length === 0) {
+      throw new Error('[GenerationSegments] fold() requires at least one generation')
+    }
+    for (let i = 1; i < gens.length; i++) {
+      if (gens[i].generation <= gens[i - 1].generation) {
+        throw new Error('[GenerationSegments] fold() input must be strictly ascending')
+      }
+    }
+    const last = this.manifest.segments[this.manifest.segments.length - 1]
+    if (last && gens[0].generation <= last.lastGeneration) {
+      throw new Error(
+        `[GenerationSegments] fold() overlaps the packed tier: ${gens[0].generation} ≤ ` +
+          `sealed ${last.lastGeneration} — segments are immutable, never rewritten`
+      )
+    }
+
+    const first = gens[0].generation
+    const file = segmentFileName(first)
+    const sidecar: SidecarIndex = { version: 1, generations: [], ids: {} }
+
+    // Encode all frames, tracking offsets for the sidecar.
+    const parts: Uint8Array[] = [MAGIC]
+    let offset = MAGIC.length
+    for (const g of gens) {
+      const payload = msgpackEncode([
+        g.generation,
+        g.timestamp,
+        g.delta,
+        g.records.map((r) => [r.kind === 'noun' ? 0 : 1, r.id, r.record]),
+        0 // flags: v1 = uncompressed
+      ])
+      const frame = new Uint8Array(FRAME_PREFIX_BYTES + payload.length)
+      const view = new DataView(frame.buffer)
+      view.setUint32(0, payload.length, true)
+      view.setUint32(4, crc32c(payload), true)
+      frame.set(payload, FRAME_PREFIX_BYTES)
+      sidecar.generations.push([g.generation, offset, frame.length])
+      for (const r of g.records) {
+        const key = `${r.kind === 'noun' ? 0 : 1}:${r.id}`
+        ;(sidecar.ids[key] ??= []).push(g.generation)
+      }
+      parts.push(frame)
+      offset += frame.length
+    }
+    const total = parts.reduce((n, p) => n + p.length, 0)
+    const bytes = new Uint8Array(total)
+    let at = 0
+    for (const p of parts) {
+      bytes.set(p, at)
+      at += p.length
+    }
+
+    const meta: SegmentMeta = {
+      file,
+      firstGeneration: first,
+      lastGeneration: gens[gens.length - 1].generation,
+      frames: gens.length,
+      bytes: total,
+      checksum: crc32c(bytes)
+    }
+
+    // Durability order: segment + sidecar fsync'd BEFORE the manifest names
+    // them (a crash before the manifest = invisible orphan files, harmless);
+    // manifest last, atomically.
+    const segPath = `${SEGMENTS_PREFIX}/${file}`
+    const idxPath = `${SEGMENTS_PREFIX}/${sidecarFileName(first)}`
+    await this.storage.writeRawBytes(segPath, bytes)
+    await this.storage.writeRawBytes(idxPath, msgpackEncode(sidecar))
+    await this.storage.syncRawObjects([segPath, idxPath])
+    const next: SegmentManifest = {
+      ...this.manifest,
+      segments: [...this.manifest.segments, meta]
+    }
+    await this.storage.writeRawObject(MANIFEST_PATH, next)
+    await this.storage.syncRawObjects([MANIFEST_PATH])
+    this.manifest = next
+    this.sidecars.set(file, sidecar)
+    return meta
+  }
+
+  /** Load (or rebuild, loudly) a segment's sidecar. */
+  private async sidecarFor(meta: SegmentMeta): Promise<SidecarIndex> {
+    const cached = this.sidecars.get(meta.file)
+    if (cached) return cached
+    const idxPath = `${SEGMENTS_PREFIX}/${sidecarFileName(meta.firstGeneration)}`
+    const raw = await this.storage.readRawBytes(idxPath)
+    if (raw) {
+      try {
+        const idx = msgpackDecode(raw) as SidecarIndex
+        if (idx.version === 1) {
+          this.sidecars.set(meta.file, idx)
+          return idx
+        }
+      } catch {
+        // fall through to rebuild
+      }
+    }
+    // Sidecars are DERIVED: rebuild from the segment, loudly — never serve
+    // wrong offsets silently.
+    prodLog.warn(
+      `[GenerationSegments] sidecar for ${meta.file} missing or unreadable — rebuilding from the segment`
+    )
+    const rebuilt = await this.rebuildSidecar(meta)
+    await this.storage.writeRawBytes(idxPath, msgpackEncode(rebuilt))
+    this.sidecars.set(meta.file, rebuilt)
+    return rebuilt
+  }
+
+  /** One sequential read of the segment → a fresh sidecar. Verifies every frame CRC. */
+  private async rebuildSidecar(meta: SegmentMeta): Promise<SidecarIndex> {
+    const frames = await this.readAllFrames(meta)
+    const idx: SidecarIndex = { version: 1, generations: [], ids: {} }
+    for (const f of frames) {
+      idx.generations.push([f.generation, f.offset, f.frameLen])
+      for (const r of f.records) {
+        const key = `${r.kind === 'noun' ? 0 : 1}:${r.id}`
+        ;(idx.ids[key] ??= []).push(f.generation)
+      }
+    }
+    return idx
+  }
+
+  private decodeFrame(
+    payload: Uint8Array
+  ): { generation: number; timestamp: number; delta: unknown; records: FoldGeneration['records'] } {
+    const [generation, timestamp, delta, rawRecords] = msgpackDecode(payload) as [
+      number,
+      number,
+      unknown,
+      Array<[number, string, unknown]>,
+      number
+    ]
+    return {
+      generation,
+      timestamp,
+      delta,
+      records: rawRecords.map(([kindByte, id, record]) => ({
+        kind: kindByte === 0 ? ('noun' as const) : ('verb' as const),
+        id,
+        record
+      }))
+    }
+  }
+
+  private async readAllFrames(meta: SegmentMeta): Promise<
+    Array<ReturnType<GenerationSegmentStore['decodeFrame']> & { offset: number; frameLen: number }>
+  > {
+    const bytes = await this.storage.readRawBytes(`${SEGMENTS_PREFIX}/${meta.file}`)
+    if (!bytes) {
+      throw new Error(
+        `[GenerationSegments] sealed segment ${meta.file} is MISSING — packed history is damaged; ` +
+          `refusing to continue silently`
+      )
+    }
+    const out: Array<ReturnType<GenerationSegmentStore['decodeFrame']> & { offset: number; frameLen: number }> = []
+    let at = MAGIC.length
+    const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength)
+    while (at + FRAME_PREFIX_BYTES <= bytes.length) {
+      const payloadLen = view.getUint32(at, true)
+      const crc = view.getUint32(at + 4, true)
+      const payload = bytes.subarray(at + FRAME_PREFIX_BYTES, at + FRAME_PREFIX_BYTES + payloadLen)
+      if (payload.length !== payloadLen || crc32c(payload) !== crc) {
+        throw new Error(
+          `[GenerationSegments] frame CRC mismatch in ${meta.file} at offset ${at} — ` +
+            `packed history is damaged; refusing to serve it`
+        )
+      }
+      out.push({ ...this.decodeFrame(payload), offset: at, frameLen: FRAME_PREFIX_BYTES + payloadLen })
+      at += FRAME_PREFIX_BYTES + payloadLen
+    }
+    return out
+  }
+
+  /** Read one packed generation's frame via its sidecar offset (one ranged read). */
+  private async readFrame(
+    gen: number
+  ): Promise<ReturnType<GenerationSegmentStore['decodeFrame']> | null> {
+    const meta = this.coveringSegment(gen)
+    if (!meta) return null
+    const idx = await this.sidecarFor(meta)
+    // generations ascending → binary search.
+    const gens = idx.generations
+    let lo = 0
+    let hi = gens.length - 1
+    while (lo <= hi) {
+      const mid = (lo + hi) >> 1
+      if (gens[mid][0] < gen) lo = mid + 1
+      else if (gens[mid][0] > gen) hi = mid - 1
+      else {
+        const [, offset, frameLen] = gens[mid]
+        const bytes = await this.storage.readRawBytes(`${SEGMENTS_PREFIX}/${meta.file}`)
+        if (!bytes) {
+          throw new Error(`[GenerationSegments] sealed segment ${meta.file} is MISSING`)
+        }
+        const frame = bytes.subarray(offset, offset + frameLen)
+        const view = new DataView(frame.buffer, frame.byteOffset, frame.byteLength)
+        const payloadLen = view.getUint32(0, true)
+        const crc = view.getUint32(4, true)
+        const payload = frame.subarray(FRAME_PREFIX_BYTES, FRAME_PREFIX_BYTES + payloadLen)
+        if (payload.length !== payloadLen || crc32c(payload) !== crc) {
+          throw new Error(
+            `[GenerationSegments] frame CRC mismatch for generation ${gen} in ${meta.file} — ` +
+              `packed history is damaged; refusing to serve it`
+          )
+        }
+        return this.decodeFrame(payload)
+      }
+    }
+    // In the covering range but not present: the packed tier is dense by
+    // construction (fold packs every generation it is handed, including
+    // record-less ones) — absence inside a sealed range is damage.
+    throw new Error(
+      `[GenerationSegments] generation ${gen} is inside sealed segment ${meta.file}'s declared ` +
+        `range but has no frame — packed history is damaged`
+    )
+  }
+
+  /** The packed tier's delta for `gen` (null = not packed). */
+  async readDelta(gen: number): Promise<{ delta: unknown; timestamp: number } | null> {
+    const frame = await this.readFrame(gen)
+    return frame ? { delta: frame.delta, timestamp: frame.timestamp } : null
+  }
+
+  /** The packed tier's full record-set for `gen` (null = not packed). */
+  async readRecords(gen: number): Promise<FoldGeneration['records'] | null> {
+    const frame = await this.readFrame(gen)
+    return frame ? frame.records : null
+  }
+
+  /** One packed before-image (null = not packed OR no record for the id in that generation). */
+  async readRecord(gen: number, kind: 'noun' | 'verb', id: string): Promise<unknown | null> {
+    const frame = await this.readFrame(gen)
+    if (!frame) return null
+    const hit = frame.records.find((r) => r.kind === kind && r.id === id)
+    return hit ? hit.record : null
+  }
+
+  /**
+   * D3 reclaim: drop WHOLE segments whose lastGeneration < `belowGeneration`
+   * and bump `compactedBelow`. Partial segments are never dropped — the
+   * boundary waits. NEVER called under the archival profile (the caller
+   * enforces retention semantics; this method only executes boundary drops).
+   */
+  async dropSegmentsBelow(belowGeneration: number): Promise<{ dropped: number; compactedBelow: number }> {
+    const keep: SegmentMeta[] = []
+    const drop: SegmentMeta[] = []
+    for (const s of this.manifest.segments) {
+      ;(s.lastGeneration < belowGeneration ? drop : keep).push(s)
+    }
+    if (drop.length === 0) {
+      return { dropped: 0, compactedBelow: this.manifest.compactedBelow }
+    }
+    const compactedBelow = Math.max(
+      this.manifest.compactedBelow,
+      drop[drop.length - 1].lastGeneration + 1
+    )
+    // Manifest first (the drop is authoritative once named), then bytes —
+    // a crash between leaves orphan segment files invisible to the manifest,
+    // harmless and re-collectable.
+    const next: SegmentManifest = { ...this.manifest, compactedBelow, segments: keep }
+    await this.storage.writeRawObject(MANIFEST_PATH, next)
+    await this.storage.syncRawObjects([MANIFEST_PATH])
+    this.manifest = next
+    for (const s of drop) {
+      await this.storage.deleteRawObject(`${SEGMENTS_PREFIX}/${s.file}`)
+      await this.storage.deleteRawObject(`${SEGMENTS_PREFIX}/${sidecarFileName(s.firstGeneration)}`)
+      this.sidecars.delete(s.file)
+    }
+    return { dropped: drop.length, compactedBelow }
+  }
+
+  /**
+   * D8 rider — the packed portion of `generationDigest(g)`: a deterministic
+   * crc32c chain over sealed-segment checksums fully below `g`, plus the
+   * frame CRC of `g`'s own frame when `g` is mid-segment. O(segments), not
+   * O(generations); identical history ⇒ identical digest on any machine.
+   * The live-tier portion is composed by the caller.
+   */
+  async digestThroughPacked(g: number): Promise<number | null> {
+    let digest = 0
+    let covered = false
+    for (const s of this.manifest.segments) {
+      if (s.lastGeneration <= g) {
+        digest = crc32c(new TextEncoder().encode(`${digest}:${s.checksum}`))
+        if (s.lastGeneration === g) covered = true
+      } else if (s.firstGeneration <= g) {
+        // g is mid-segment: chain the partial prefix via g's frame CRC.
+        const frame = await this.readFrame(g)
+        if (frame === null) return null
+        const idx = await this.sidecarFor(s)
+        const upTo = idx.generations.filter(([gen]) => gen <= g)
+        for (const [gen, offset, frameLen] of upTo) {
+          digest = crc32c(new TextEncoder().encode(`${digest}:${gen}:${offset}:${frameLen}`))
+        }
+        covered = true
+        break
+      }
+    }
+    return covered || this.manifest.segments.length > 0 ? digest : null
+  }
+}

src/db/types.ts

@@ -116,6 +116,16 @@ export interface TransactOptions {
    * record is staged.
    */
   ifAtGeneration?: number
+  /**
+   * Budget (ms) for the atomic apply phase. When omitted, the budget SCALES
+   * with the batch: `max(30 000, opCount × 2 000)` — production imports on
+   * network-attached disks measure ~2 s per operation, so a flat 30 s budget
+   * silently capped honest bulk work at ~15 operations. A tripped budget
+   * rolls the whole batch back and throws a retryable
+   * `TransactionTimeoutError` naming the operation it stopped at, the batch
+   * size, and the elapsed/budget times.
+   */
+  timeoutMs?: number
 }
 
 /**
@@ -138,21 +148,43 @@ export interface TransactReceipt {
 // ============================================================================
 
 /**
- * @description Options for `brain.compactHistory()`. When both options are
- * supplied a generation must satisfy *both* retention rules to be reclaimed
- * (the stricter retention wins). With no options, every generation not
+ * @description Options for `brain.compactHistory()` — explicit retention
+ * **caps** (Model-B `retention` knob). Each supplied field is an upper bound:
+ * the oldest unpinned generations are reclaimed while **ANY** supplied cap is
+ * exceeded (predictable ops — the more you constrain, the more is reclaimed).
+ * Live `Db` pins are ALWAYS exempt. With no options, every generation not
  * protected by a live pin is eligible.
+ *
+ * (8.0 rename: the pre-release `retainGenerations`/`retainMs` *floors* became
+ * `maxGenerations`/`maxAge` *caps* — the `max*` naming signals the semantics.)
  */
 export interface CompactHistoryOptions {
   /**
-   * Keep at least this many of the most recent committed generations
-   * (`0` = keep none beyond what live pins protect).
+   * Keep at most this many of the most recent committed generations — reclaim
+   * the oldest unpinned generations while the count exceeds it (`0` = reclaim
+   * everything not protected by a live pin).
    */
-  retainGenerations?: number
+  maxGenerations?: number
   /**
-   * Keep every generation committed within the last `retainMs` milliseconds.
+   * Keep only generations committed within the last `maxAge` milliseconds —
+   * reclaim unpinned generations older than the window.
    */
-  retainMs?: number
+  maxAge?: number
+  /**
+   * Keep total generational-history bytes at or below this — reclaim the oldest
+   * unpinned generations while the total exceeds it. Byte accounting is the sum
+   * of each surviving generation's serialized record set (`GenerationDelta.bytes`).
+   */
+  maxBytes?: number
+  /**
+   * Stop reclaiming after this many milliseconds even if caps are still
+   * exceeded (8.9.0). Compaction is maintenance — a bounded pass keeps
+   * `close()` (and any explicit maintenance window) from stalling on a large
+   * backlog; the next pass resumes where this one stopped (reclamation is
+   * oldest-first, so an early stop is always a consistent prefix). Unset =
+   * run to completion.
+   */
+  timeBudgetMs?: number
 }
 
 /**
@@ -170,17 +202,46 @@ export interface CompactHistoryResult {
   horizon: number
 }
 
+/**
+ * @description Result of `brain.historyStats()` — the read-only generational
+ * history footprint, for fleet audits and ops doors. A pool operator runs this
+ * per brain to size retention exposure (how much MVCC history each brain
+ * carries and under which policy) without touching any data.
+ */
+export interface HistoryStats {
+  /** Committed generation record-sets currently on disk. */
+  generations: number
+  /** Total on-disk history bytes across those record-sets. */
+  bytes: number
+  /** Oldest committed generation still on disk (null when history is empty). */
+  oldestGeneration: number | null
+  /** Newest committed generation (null when history is empty). */
+  newestGeneration: number | null
+  /** Commit timestamp (ms) of the oldest on-disk generation. */
+  oldestTimestamp: number | null
+  /** Commit timestamp (ms) of the newest on-disk generation. */
+  newestTimestamp: number | null
+  /** Compaction horizon — generations below it were reclaimed. */
+  horizon: number
+  /** The effective retention mode this brain runs under. */
+  retentionMode: 'all' | 'adaptive' | 'explicit'
+  /**
+   * The adaptive byte budget in force (coordinator-driven or the local
+   * free-memory probe); null under 'all' or explicit caps.
+   */
+  effectiveBudgetBytes: number | null
+}
+
 // ============================================================================
 // Db surfaces
 // ============================================================================
 
 /**
  * @description Result of `db.since(priorDb)` — the ids touched by committed
- * transactions in the generation interval `(priorDb.generation,
- * db.generation]`. Granularity note: single-operation writes performed
- * outside `transact()` bump the generation counter but do not produce
- * generation records, so they are not reflected here (documented 8.0
- * history granularity — see `docs/ADR-001-generational-mvcc.md`).
+ * generations in the interval `(priorDb.generation, db.generation]`. Under
+ * Model-B EVERY write is its own generation, so single-operation writes
+ * (`add`/`update`/`remove`/`relate` outside `transact()`) ARE reflected here,
+ * exactly like `transact()` commits.
  */
 export interface ChangedIds {
   /** Entity (noun) ids touched in the interval, sorted. */
@@ -219,7 +280,8 @@ export interface DiffResult {
  * @description One version of a single entity or relationship in its
  * {@link EntityHistory} — the materialized state at a committed generation that
  * changed it. `value` is `null` when the id did not exist at that version
- * (created-after / removed). Granularity is `transact()` commits.
+ * (created-after / removed). Granularity is per-write (every `transact()` AND
+ * single-op write is its own generation — Model-B).
  */
 export interface HistoryVersion<T = any> {
   /** The committed generation that produced this version. */
@@ -271,9 +333,11 @@ export interface GenerationRecord {
 
 /**
  * @description The per-generation delta persisted at
- * `_generations/<N>/tx.json` — written and fsynced *before* any operation of
- * the transaction executes, so crash recovery always knows exactly which ids
- * to restore from before-images.
+ * `_generations/<N>/tx.json`. For a `transact()` generation it is written and
+ * fsynced *before* any operation of the transaction executes, so crash
+ * recovery always knows exactly which ids to restore from before-images. For a
+ * single-operation (Model-B) generation it is written at group-commit flush
+ * time with `groupCommit: true` — see that flag.
  */
 export interface GenerationDelta {
   /** The generation this delta belongs to. */
@@ -286,6 +350,36 @@ export interface GenerationDelta {
   nouns: string[]
   /** Relationship ids touched by this generation. */
   verbs: string[]
+  /**
+   * Content-blob hashes referenced by this generation's before-image records
+   * (a MULTISET — one entry per referencing record occurrence), captured at
+   * persist time so compaction can release the exact history references it
+   * reclaims without re-reading the records. Always present (possibly empty)
+   * on deltas written under the temporal-blob contract; absent only on
+   * pre-contract generations, for which compaction falls back to reading the
+   * record-set itself.
+   */
+  blobHashes?: string[]
+  /**
+   * `true` for a Model-B single-operation generation persisted by the async
+   * group-commit flush (`GenerationStore.flushPendingSingleOps`). It marks the
+   * **drop-without-restore** crash-recovery contract: the live write already
+   * mutated canonical storage and was acknowledged to the caller BEFORE the
+   * flush, so an uncommitted (crashed-mid-flush) generation with this flag set
+   * must be DISCARDED — its before-images must NEVER be restored, or recovery
+   * would silently revert acknowledged user writes. Absent/`false` on a
+   * `transact()` generation, whose before-images ARE the durable undo log and
+   * are restored on rollback (the before-image + execute are one atomic unit).
+   */
+  groupCommit?: boolean
+  /**
+   * Serialized byte size of this generation's record set (the `prev/<id>.json`
+   * before-images plus this delta), recorded at write time so `maxBytes` /
+   * adaptive retention can bound total history without a storage size API. Read
+   * back lazily by `GenerationStore.historyBytes()`. Absent on generations
+   * written before byte accounting existed (treated as 0).
+   */
+  bytes?: number
 }
 
 /**
@@ -346,6 +440,24 @@ export interface GenerationStorage {
   /** Durability barrier: fsync the given object paths (no-op in memory). */
   syncRawObjects(paths: string[]): Promise<void>
 
+  /**
+   * OPTIONAL transaction durability barrier. `commitTransaction` calls
+   * {@link beginWriteBarrier} immediately before running the planned operations
+   * and {@link flushWriteBarrier} immediately after — BEFORE the generation
+   * counter and manifest are advanced. An adapter whose canonical writes are
+   * not synchronously durable (the filesystem adapter's tmp+rename lands in the
+   * page cache) MUST implement these so a transaction reported "committed" is
+   * durable before its generation stamp: otherwise a hard kill can leave the
+   * fsync'd counter ahead of the still-buffered entity bytes (phantom progress
+   * for any generation-based consumer). Adapters whose writes are already
+   * durable per-call (cloud object PUT) may leave these undefined — the store
+   * treats them as no-ops. `beginWriteBarrier` also resets any tracking left by
+   * an aborted prior transaction.
+   */
+  beginWriteBarrier?(): void
+  /** @see beginWriteBarrier — fsync every canonical write since begin. */
+  flushWriteBarrier?(): Promise<void>
+
   /** Read an entity's raw stored metadata+vector objects. */
   readNounRaw(id: string): Promise<{ metadata: any | null; vector: any | null }>
   /** Restore an entity's raw stored objects (`null` part ⇒ delete that file). */
@@ -360,6 +472,44 @@ export interface GenerationStorage {
   /** Read all lines of `_system/tx-log.jsonl` (empty array if absent). */
   readTxLogLines(): Promise<string[]>
 
+  /**
+   * OPTIONAL binary raw-byte primitives — the substrate for the generation
+   * fact log's append-only CRC-framed segments. Feature-detected: a storage
+   * layer that omits them hosts no fact log (dual-write is skipped; readers
+   * fall back to canonical enumeration). Paths are used VERBATIM (no
+   * suffixing). Append durability rides `syncRawObjects` at the commit
+   * barrier, exactly like the staged history files.
+   */
+  appendRawBytes?(path: string, bytes: Uint8Array): Promise<void>
+  /** Read a raw binary file whole; absent → null; a real fault throws. */
+  readRawBytes?(path: string): Promise<Uint8Array | null>
+  /** Replace a raw binary file atomically (tmp → fsync → rename). */
+  writeRawBytes?(path: string, bytes: Uint8Array): Promise<void>
+  /** Byte size of a raw binary file, or null when absent. */
+  rawByteSize?(path: string): Promise<number | null>
+
+  /**
+   * OPTIONAL temporal-blob contract (implemented by blob-aware storage; the
+   * generation store treats the hashes as opaque strings). Extract the
+   * content-blob hashes a record-set references — a pure multiset extraction,
+   * no side effects.
+   */
+  extractBlobHashesFromRecords?(records: GenerationRecord[]): string[]
+  /**
+   * OPTIONAL: record history references for the given hashes (one increment
+   * per occurrence). Called BEFORE the referencing record-set is persisted —
+   * a crash between the two can only over-count (a leak the scrub repairs),
+   * never under-count (which would risk reclaiming bytes history still needs).
+   */
+  recordHistoryBlobReferences?(hashes: string[]): Promise<void>
+  /**
+   * OPTIONAL: release history references for the given hashes (one decrement
+   * per occurrence) and physically reclaim any hash left with zero live AND
+   * zero history references. Called AFTER the referencing record-set is
+   * deleted by compaction — same over-count-only crash ordering.
+   */
+  releaseHistoryBlobReferences?(hashes: string[]): Promise<void>
+
   /**
    * Register the generation-bump hook invoked on every entity-visible
    * single-operation write (see `BaseStorage.setGenerationBumpHook`).

src/db/whereMatcher.ts

@@ -39,9 +39,9 @@ export class UnsupportedWhereOperatorError extends Error {
   constructor(operator: string) {
     super(
       `The where-operator '${operator}' is not supported for historical/speculative ` +
-        `in-memory evaluation. Supported: eq/equals/is, ne/notEquals/isNot, in/oneOf, ` +
-        `gt/greaterThan, gte/greaterThanOrEqual/greaterEqual, lt/lessThan, ` +
-        `lte/lessThanOrEqual/lessEqual, between, contains, exists, missing, ` +
+        `in-memory evaluation. Supported: eq/equals, ne/notEquals, in/oneOf, ` +
+        `gt/greaterThan, gte/greaterThanOrEqual, lt/lessThan, ` +
+        `lte/lessThanOrEqual, between, contains, exists, missing, ` +
         `plus allOf/anyOf/not.`
     )
     this.name = 'UnsupportedWhereOperatorError'
@@ -150,12 +150,10 @@ function fieldConditionMatches(value: unknown, condition: unknown): boolean {
   for (const [op, operand] of Object.entries(condition as Record<string, unknown>)) {
     let matches: boolean
     switch (op) {
-      case 'is':
       case 'equals':
       case 'eq':
         matches = eqMatches(value, operand)
         break
-      case 'isNot':
       case 'notEquals':
       case 'ne':
         matches = !eqMatches(value, operand)
@@ -170,7 +168,6 @@ function fieldConditionMatches(value: unknown, condition: unknown): boolean {
         matches = cmp !== null && cmp > 0
         break
       }
-      case 'greaterEqual':
       case 'greaterThanOrEqual':
       case 'gte': {
         const cmp = compare(value, operand)
@@ -183,7 +180,6 @@ function fieldConditionMatches(value: unknown, condition: unknown): boolean {
         matches = cmp !== null && cmp < 0
         break
       }
-      case 'lessEqual':
       case 'lessThanOrEqual':
       case 'lte': {
         const cmp = compare(value, operand)

src/embeddings/CachedEmbeddings.ts

@@ -1,163 +0,0 @@
-/**
- * Cached Embeddings - Performance Optimization Layer
- * 
- * Provides pre-computed embeddings for common terms to avoid
- * unnecessary model calls. Falls back to EmbeddingManager for
- * unknown terms.
- * 
- * This is purely a performance optimization - it doesn't affect
- * the consistency or accuracy of embeddings.
- */
-
-import { Vector } from '../coreTypes.js'
-import { embeddingManager } from './EmbeddingManager.js'
-
-// Pre-computed embeddings for top common terms
-// In production, this could be loaded from a file or expanded significantly
-const PRECOMPUTED_EMBEDDINGS: Record<string, Vector> = {
-  // Programming languages
-  'javascript': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.1)),
-  'python': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.1)),
-  'typescript': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.15)),
-  'java': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.15)),
-  'rust': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.2)),
-  'go': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.2)),
-  'c++': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.22)),
-  'c#': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.22)),
-  
-  // Web frameworks
-  'react': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.25)),
-  'vue': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.25)),
-  'angular': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.3)),
-  'svelte': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.3)),
-  'nextjs': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.32)),
-  'nuxt': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.32)),
-  
-  // Databases
-  'postgresql': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.35)),
-  'mysql': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.35)),
-  'mongodb': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.4)),
-  'redis': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.4)),
-  'elasticsearch': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.42)),
-  
-  // Common tech terms
-  'database': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.45)),
-  'api': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.45)),
-  'server': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.5)),
-  'client': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.5)),
-  'frontend': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.55)),
-  'backend': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.55)),
-  'fullstack': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.57)),
-  'devops': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.57)),
-  'cloud': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.6)),
-  'docker': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.6)),
-  'kubernetes': new Array(384).fill(0).map((_, i) => Math.sin(i * 0.62)),
-  'microservices': new Array(384).fill(0).map((_, i) => Math.cos(i * 0.62)),
-}
-
-/**
- * Simple character n-gram based embedding for short text
- * This is much faster than using the model for simple terms
- */
-function computeSimpleEmbedding(text: string): Vector {
-  const normalized = text.toLowerCase().trim()
-  const vector = new Array(384).fill(0)
-  
-  // Character trigrams for simple semantic similarity
-  for (let i = 0; i < normalized.length - 2; i++) {
-    const trigram = normalized.slice(i, i + 3)
-    const hash = trigram.charCodeAt(0) * 31 + 
-                 trigram.charCodeAt(1) * 7 + 
-                 trigram.charCodeAt(2)
-    const index = Math.abs(hash) % 384
-    vector[index] += 1 / (normalized.length - 2)
-  }
-  
-  // Normalize vector
-  const magnitude = Math.sqrt(vector.reduce((sum, val) => sum + val * val, 0))
-  if (magnitude > 0) {
-    for (let i = 0; i < vector.length; i++) {
-      vector[i] /= magnitude
-    }
-  }
-  
-  return vector
-}
-
-/**
- * Cached Embeddings with fallback to EmbeddingManager
- */
-export class CachedEmbeddings {
-  private stats = {
-    cacheHits: 0,
-    simpleComputes: 0,
-    modelCalls: 0
-  }
-  
-  /**
-   * Generate embedding with caching
-   */
-  async embed(text: string | string[]): Promise<Vector | Vector[]> {
-    if (Array.isArray(text)) {
-      return Promise.all(text.map(t => this.embedSingle(t)))
-    }
-    return this.embedSingle(text)
-  }
-  
-  /**
-   * Embed single text with cache lookup
-   */
-  private async embedSingle(text: string): Promise<Vector> {
-    const normalized = text.toLowerCase().trim()
-    
-    // 1. Check pre-computed cache (instant, zero cost)
-    if (PRECOMPUTED_EMBEDDINGS[normalized]) {
-      this.stats.cacheHits++
-      return PRECOMPUTED_EMBEDDINGS[normalized]
-    }
-    
-    // 2. Check for partial matches in cache
-    for (const [term, embedding] of Object.entries(PRECOMPUTED_EMBEDDINGS)) {
-      if (normalized.includes(term) || term.includes(normalized)) {
-        this.stats.cacheHits++
-        // Return slightly modified version to maintain uniqueness
-        return embedding.map(v => v * 0.95)
-      }
-    }
-    
-    // 3. For short text, use simple embedding (fast, low cost)
-    if (normalized.length < 50 && normalized.split(' ').length < 5) {
-      this.stats.simpleComputes++
-      return computeSimpleEmbedding(normalized)
-    }
-    
-    // 4. Fall back to EmbeddingManager for complex text
-    this.stats.modelCalls++
-    return await embeddingManager.embed(text)
-  }
-  
-  /**
-   * Get cache statistics
-   */
-  getStats() {
-    return {
-      ...this.stats,
-      totalEmbeddings: this.stats.cacheHits + this.stats.simpleComputes + this.stats.modelCalls,
-      cacheHitRate: this.stats.cacheHits / 
-        (this.stats.cacheHits + this.stats.simpleComputes + this.stats.modelCalls) || 0
-    }
-  }
-  
-  /**
-   * Add custom pre-computed embeddings
-   */
-  addPrecomputed(term: string, embedding: Vector) {
-    if (embedding.length !== 384) {
-      throw new Error('Embedding must have 384 dimensions')
-    }
-    PRECOMPUTED_EMBEDDINGS[term.toLowerCase()] = embedding
-  }
-}
-
-// Export singleton instance
-export const cachedEmbeddings = new CachedEmbeddings()

src/embeddings/EmbeddingManager.ts

@@ -60,7 +60,7 @@ export class EmbeddingManager {
   private constructor() {
     this.engine = WASMEmbeddingEngine.getInstance()
     // Log deferred to init() — at construction time we don't know if a plugin
-    // (like Cortex) will replace the WASM embedder with a native one.
+    // (like Cor) will replace the WASM embedder with a native one.
   }
 
   /**

src/embeddings/index.ts

@@ -1,28 +0,0 @@
-/**
- * Embeddings Module - Clean, Unified Architecture
- * 
- * This module provides all embedding functionality for Brainy.
- * 
- * Main Components:
- * - EmbeddingManager: Core embedding generation with Q8/FP32 support
- * - CachedEmbeddings: Performance optimization layer with pre-computed embeddings
- */
-
-// Core embedding functionality
-export { 
-  EmbeddingManager, 
-  embeddingManager, 
-  embed, 
-  getEmbeddingFunction,
-  getEmbeddingStats,
-  type ModelPrecision
-} from './EmbeddingManager.js'
-
-// Cached embeddings for performance
-export { 
-  CachedEmbeddings, 
-  cachedEmbeddings 
-} from './CachedEmbeddings.js'
-
-// Default export is the singleton manager
-export { embeddingManager as default } from './EmbeddingManager.js'

src/embeddings/wasm/pkg/candle_embeddings.d.ts

@@ -0,0 +1,100 @@
+/* tslint:disable */
+/* eslint-disable */
+
+/**
+ * WASM-compatible embedding engine
+ */
+export class EmbeddingEngine {
+    free(): void;
+    [Symbol.dispose](): void;
+    /**
+     * Get the embedding dimension (384 for all-MiniLM-L6-v2)
+     */
+    dimension(): number;
+    /**
+     * Generate embedding for a single text
+     *
+     * Returns a Float32Array of 384 dimensions
+     */
+    embed(text: string): Float32Array;
+    /**
+     * Generate embeddings for multiple texts
+     *
+     * Takes a JavaScript Array of strings
+     * Returns a JavaScript Array of Float32Array
+     */
+    embed_batch(texts: Array<any>): Array<any>;
+    /**
+     * Check if the engine is ready for inference
+     */
+    is_ready(): boolean;
+    /**
+     * Load the model and tokenizer from bytes
+     *
+     * This is now the ONLY way to initialize the engine.
+     * Model weights are no longer embedded in WASM for faster initialization.
+     *
+     * # Arguments
+     * * `model_bytes` - SafeTensors format model weights
+     * * `tokenizer_bytes` - tokenizer.json contents
+     * * `config_bytes` - config.json contents
+     */
+    load(model_bytes: Uint8Array, tokenizer_bytes: Uint8Array, config_bytes: Uint8Array): void;
+    /**
+     * Get the maximum sequence length
+     */
+    max_sequence_length(): number;
+    /**
+     * Create a new embedding engine instance (not loaded)
+     */
+    constructor();
+}
+
+/**
+ * Calculate cosine similarity between two embeddings
+ */
+export function cosine_similarity(a: Float32Array, b: Float32Array): number;
+
+export type InitInput = RequestInfo | URL | Response | BufferSource | WebAssembly.Module;
+
+export interface InitOutput {
+    readonly memory: WebAssembly.Memory;
+    readonly __wbg_embeddingengine_free: (a: number, b: number) => void;
+    readonly cosine_similarity: (a: number, b: number, c: number, d: number) => number;
+    readonly embeddingengine_dimension: (a: number) => number;
+    readonly embeddingengine_embed: (a: number, b: number, c: number) => [number, number, number];
+    readonly embeddingengine_embed_batch: (a: number, b: any) => [number, number, number];
+    readonly embeddingengine_is_ready: (a: number) => number;
+    readonly embeddingengine_load: (a: number, b: number, c: number, d: number, e: number, f: number, g: number) => [number, number];
+    readonly embeddingengine_max_sequence_length: (a: number) => number;
+    readonly embeddingengine_new: () => number;
+    readonly __wbindgen_malloc: (a: number, b: number) => number;
+    readonly __wbindgen_realloc: (a: number, b: number, c: number, d: number) => number;
+    readonly __wbindgen_exn_store: (a: number) => void;
+    readonly __externref_table_alloc: () => number;
+    readonly __wbindgen_externrefs: WebAssembly.Table;
+    readonly __externref_table_dealloc: (a: number) => void;
+    readonly __wbindgen_start: () => void;
+}
+
+export type SyncInitInput = BufferSource | WebAssembly.Module;
+
+/**
+ * Instantiates the given `module`, which can either be bytes or
+ * a precompiled `WebAssembly.Module`.
+ *
+ * @param {{ module: SyncInitInput }} module - Passing `SyncInitInput` directly is deprecated.
+ *
+ * @returns {InitOutput}
+ */
+export function initSync(module: { module: SyncInitInput } | SyncInitInput): InitOutput;
+
+/**
+ * If `module_or_path` is {RequestInfo} or {URL}, makes a request and
+ * for everything else, calls `WebAssembly.instantiate` directly.
+ *
+ * @param {{ module_or_path: InitInput | Promise<InitInput> }} module_or_path - Passing `InitInput` directly is deprecated.
+ *
+ * @returns {Promise<InitOutput>}
+ */
+export default function __wbg_init (module_or_path?: { module_or_path: InitInput | Promise<InitInput> } | InitInput | Promise<InitInput>): Promise<InitOutput>;

src/embeddings/wasm/pkg/candle_embeddings.js

@@ -0,0 +1,525 @@
+/* @ts-self-types="./candle_embeddings.d.ts" */
+
+/**
+ * WASM-compatible embedding engine
+ */
+export class EmbeddingEngine {
+    __destroy_into_raw() {
+        const ptr = this.__wbg_ptr;
+        this.__wbg_ptr = 0;
+        EmbeddingEngineFinalization.unregister(this);
+        return ptr;
+    }
+    free() {
+        const ptr = this.__destroy_into_raw();
+        wasm.__wbg_embeddingengine_free(ptr, 0);
+    }
+    /**
+     * Get the embedding dimension (384 for all-MiniLM-L6-v2)
+     * @returns {number}
+     */
+    dimension() {
+        const ret = wasm.embeddingengine_dimension(this.__wbg_ptr);
+        return ret >>> 0;
+    }
+    /**
+     * Generate embedding for a single text
+     *
+     * Returns a Float32Array of 384 dimensions
+     * @param {string} text
+     * @returns {Float32Array}
+     */
+    embed(text) {
+        const ptr0 = passStringToWasm0(text, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
+        const len0 = WASM_VECTOR_LEN;
+        const ret = wasm.embeddingengine_embed(this.__wbg_ptr, ptr0, len0);
+        if (ret[2]) {
+            throw takeFromExternrefTable0(ret[1]);
+        }
+        return takeFromExternrefTable0(ret[0]);
+    }
+    /**
+     * Generate embeddings for multiple texts
+     *
+     * Takes a JavaScript Array of strings
+     * Returns a JavaScript Array of Float32Array
+     * @param {Array<any>} texts
+     * @returns {Array<any>}
+     */
+    embed_batch(texts) {
+        const ret = wasm.embeddingengine_embed_batch(this.__wbg_ptr, texts);
+        if (ret[2]) {
+            throw takeFromExternrefTable0(ret[1]);
+        }
+        return takeFromExternrefTable0(ret[0]);
+    }
+    /**
+     * Check if the engine is ready for inference
+     * @returns {boolean}
+     */
+    is_ready() {
+        const ret = wasm.embeddingengine_is_ready(this.__wbg_ptr);
+        return ret !== 0;
+    }
+    /**
+     * Load the model and tokenizer from bytes
+     *
+     * This is now the ONLY way to initialize the engine.
+     * Model weights are no longer embedded in WASM for faster initialization.
+     *
+     * # Arguments
+     * * `model_bytes` - SafeTensors format model weights
+     * * `tokenizer_bytes` - tokenizer.json contents
+     * * `config_bytes` - config.json contents
+     * @param {Uint8Array} model_bytes
+     * @param {Uint8Array} tokenizer_bytes
+     * @param {Uint8Array} config_bytes
+     */
+    load(model_bytes, tokenizer_bytes, config_bytes) {
+        const ptr0 = passArray8ToWasm0(model_bytes, wasm.__wbindgen_malloc);
+        const len0 = WASM_VECTOR_LEN;
+        const ptr1 = passArray8ToWasm0(tokenizer_bytes, wasm.__wbindgen_malloc);
+        const len1 = WASM_VECTOR_LEN;
+        const ptr2 = passArray8ToWasm0(config_bytes, wasm.__wbindgen_malloc);
+        const len2 = WASM_VECTOR_LEN;
+        const ret = wasm.embeddingengine_load(this.__wbg_ptr, ptr0, len0, ptr1, len1, ptr2, len2);
+        if (ret[1]) {
+            throw takeFromExternrefTable0(ret[0]);
+        }
+    }
+    /**
+     * Get the maximum sequence length
+     * @returns {number}
+     */
+    max_sequence_length() {
+        const ret = wasm.embeddingengine_max_sequence_length(this.__wbg_ptr);
+        return ret >>> 0;
+    }
+    /**
+     * Create a new embedding engine instance (not loaded)
+     */
+    constructor() {
+        const ret = wasm.embeddingengine_new();
+        this.__wbg_ptr = ret >>> 0;
+        EmbeddingEngineFinalization.register(this, this.__wbg_ptr, this);
+        return this;
+    }
+}
+if (Symbol.dispose) EmbeddingEngine.prototype[Symbol.dispose] = EmbeddingEngine.prototype.free;
+
+/**
+ * Calculate cosine similarity between two embeddings
+ * @param {Float32Array} a
+ * @param {Float32Array} b
+ * @returns {number}
+ */
+export function cosine_similarity(a, b) {
+    const ptr0 = passArrayF32ToWasm0(a, wasm.__wbindgen_malloc);
+    const len0 = WASM_VECTOR_LEN;
+    const ptr1 = passArrayF32ToWasm0(b, wasm.__wbindgen_malloc);
+    const len1 = WASM_VECTOR_LEN;
+    const ret = wasm.cosine_similarity(ptr0, len0, ptr1, len1);
+    return ret;
+}
+
+function __wbg_get_imports() {
+    const import0 = {
+        __proto__: null,
+        __wbg___wbindgen_is_function_0095a73b8b156f76: function(arg0) {
+            const ret = typeof(arg0) === 'function';
+            return ret;
+        },
+        __wbg___wbindgen_is_object_5ae8e5880f2c1fbd: function(arg0) {
+            const val = arg0;
+            const ret = typeof(val) === 'object' && val !== null;
+            return ret;
+        },
+        __wbg___wbindgen_is_string_cd444516edc5b180: function(arg0) {
+            const ret = typeof(arg0) === 'string';
+            return ret;
+        },
+        __wbg___wbindgen_is_undefined_9e4d92534c42d778: function(arg0) {
+            const ret = arg0 === undefined;
+            return ret;
+        },
+        __wbg___wbindgen_string_get_72fb696202c56729: function(arg0, arg1) {
+            const obj = arg1;
+            const ret = typeof(obj) === 'string' ? obj : undefined;
+            var ptr1 = isLikeNone(ret) ? 0 : passStringToWasm0(ret, wasm.__wbindgen_malloc, wasm.__wbindgen_realloc);
+            var len1 = WASM_VECTOR_LEN;
+            getDataViewMemory0().setInt32(arg0 + 4 * 1, len1, true);
+            getDataViewMemory0().setInt32(arg0 + 4 * 0, ptr1, true);
+        },
+        __wbg___wbindgen_throw_be289d5034ed271b: function(arg0, arg1) {
+            throw new Error(getStringFromWasm0(arg0, arg1));
+        },
+        __wbg_call_389efe28435a9388: function() { return handleError(function (arg0, arg1) {
+            const ret = arg0.call(arg1);
+            return ret;
+        }, arguments); },
+        __wbg_call_4708e0c13bdc8e95: function() { return handleError(function (arg0, arg1, arg2) {
+            const ret = arg0.call(arg1, arg2);
+            return ret;
+        }, arguments); },
+        __wbg_crypto_86f2631e91b51511: function(arg0) {
+            const ret = arg0.crypto;
+            return ret;
+        },
+        __wbg_getRandomValues_b3f15fcbfabb0f8b: function() { return handleError(function (arg0, arg1) {
+            arg0.getRandomValues(arg1);
+        }, arguments); },
+        __wbg_get_9b94d73e6221f75c: function(arg0, arg1) {
+            const ret = arg0[arg1 >>> 0];
+            return ret;
+        },
+        __wbg_length_32ed9a279acd054c: function(arg0) {
+            const ret = arg0.length;
+            return ret;
+        },
+        __wbg_length_35a7bace40f36eac: function(arg0) {
+            const ret = arg0.length;
+            return ret;
+        },
+        __wbg_length_9a7876c9728a0979: function(arg0) {
+            const ret = arg0.length;
+            return ret;
+        },
+        __wbg_msCrypto_d562bbe83e0d4b91: function(arg0) {
+            const ret = arg0.msCrypto;
+            return ret;
+        },
+        __wbg_new_3eb36ae241fe6f44: function() {
+            const ret = new Array();
+            return ret;
+        },
+        __wbg_new_no_args_1c7c842f08d00ebb: function(arg0, arg1) {
+            const ret = new Function(getStringFromWasm0(arg0, arg1));
+            return ret;
+        },
+        __wbg_new_with_length_1763c527b2923202: function(arg0) {
+            const ret = new Array(arg0 >>> 0);
+            return ret;
+        },
+        __wbg_new_with_length_63f2683cc2521026: function(arg0) {
+            const ret = new Float32Array(arg0 >>> 0);
+            return ret;
+        },
+        __wbg_new_with_length_a2c39cbe88fd8ff1: function(arg0) {
+            const ret = new Uint8Array(arg0 >>> 0);
+            return ret;
+        },
+        __wbg_node_e1f24f89a7336c2e: function(arg0) {
+            const ret = arg0.node;
+            return ret;
+        },
+        __wbg_process_3975fd6c72f520aa: function(arg0) {
+            const ret = arg0.process;
+            return ret;
+        },
+        __wbg_prototypesetcall_bdcdcc5842e4d77d: function(arg0, arg1, arg2) {
+            Uint8Array.prototype.set.call(getArrayU8FromWasm0(arg0, arg1), arg2);
+        },
+        __wbg_randomFillSync_f8c153b79f285817: function() { return handleError(function (arg0, arg1) {
+            arg0.randomFillSync(arg1);
+        }, arguments); },
+        __wbg_require_b74f47fc2d022fd6: function() { return handleError(function () {
+            const ret = module.require;
+            return ret;
+        }, arguments); },
+        __wbg_set_f43e577aea94465b: function(arg0, arg1, arg2) {
+            arg0[arg1 >>> 0] = arg2;
+        },
+        __wbg_set_f8edeec46569cc70: function(arg0, arg1, arg2) {
+            arg0.set(getArrayF32FromWasm0(arg1, arg2));
+        },
+        __wbg_static_accessor_GLOBAL_12837167ad935116: function() {
+            const ret = typeof global === 'undefined' ? null : global;
+            return isLikeNone(ret) ? 0 : addToExternrefTable0(ret);
+        },
+        __wbg_static_accessor_GLOBAL_THIS_e628e89ab3b1c95f: function() {
+            const ret = typeof globalThis === 'undefined' ? null : globalThis;
+            return isLikeNone(ret) ? 0 : addToExternrefTable0(ret);
+        },
+        __wbg_static_accessor_SELF_a621d3dfbb60d0ce: function() {
+            const ret = typeof self === 'undefined' ? null : self;
+            return isLikeNone(ret) ? 0 : addToExternrefTable0(ret);
+        },
+        __wbg_static_accessor_WINDOW_f8727f0cf888e0bd: function() {
+            const ret = typeof window === 'undefined' ? null : window;
+            return isLikeNone(ret) ? 0 : addToExternrefTable0(ret);
+        },
+        __wbg_subarray_a96e1fef17ed23cb: function(arg0, arg1, arg2) {
+            const ret = arg0.subarray(arg1 >>> 0, arg2 >>> 0);
+            return ret;
+        },
+        __wbg_versions_4e31226f5e8dc909: function(arg0) {
+            const ret = arg0.versions;
+            return ret;
+        },
+        __wbindgen_cast_0000000000000001: function(arg0, arg1) {
+            // Cast intrinsic for `Ref(Slice(U8)) -> NamedExternref("Uint8Array")`.
+            const ret = getArrayU8FromWasm0(arg0, arg1);
+            return ret;
+        },
+        __wbindgen_cast_0000000000000002: function(arg0, arg1) {
+            // Cast intrinsic for `Ref(String) -> Externref`.
+            const ret = getStringFromWasm0(arg0, arg1);
+            return ret;
+        },
+        __wbindgen_init_externref_table: function() {
+            const table = wasm.__wbindgen_externrefs;
+            const offset = table.grow(4);
+            table.set(0, undefined);
+            table.set(offset + 0, undefined);
+            table.set(offset + 1, null);
+            table.set(offset + 2, true);
+            table.set(offset + 3, false);
+        },
+    };
+    return {
+        __proto__: null,
+        "./candle_embeddings_bg.js": import0,
+    };
+}
+
+const EmbeddingEngineFinalization = (typeof FinalizationRegistry === 'undefined')
+    ? { register: () => {}, unregister: () => {} }
+    : new FinalizationRegistry(ptr => wasm.__wbg_embeddingengine_free(ptr >>> 0, 1));
+
+function addToExternrefTable0(obj) {
+    const idx = wasm.__externref_table_alloc();
+    wasm.__wbindgen_externrefs.set(idx, obj);
+    return idx;
+}
+
+function getArrayF32FromWasm0(ptr, len) {
+    ptr = ptr >>> 0;
+    return getFloat32ArrayMemory0().subarray(ptr / 4, ptr / 4 + len);
+}
+
+function getArrayU8FromWasm0(ptr, len) {
+    ptr = ptr >>> 0;
+    return getUint8ArrayMemory0().subarray(ptr / 1, ptr / 1 + len);
+}
+
+let cachedDataViewMemory0 = null;
+function getDataViewMemory0() {
+    if (cachedDataViewMemory0 === null || cachedDataViewMemory0.buffer.detached === true || (cachedDataViewMemory0.buffer.detached === undefined && cachedDataViewMemory0.buffer !== wasm.memory.buffer)) {
+        cachedDataViewMemory0 = new DataView(wasm.memory.buffer);
+    }
+    return cachedDataViewMemory0;
+}
+
+let cachedFloat32ArrayMemory0 = null;
+function getFloat32ArrayMemory0() {
+    if (cachedFloat32ArrayMemory0 === null || cachedFloat32ArrayMemory0.byteLength === 0) {
+        cachedFloat32ArrayMemory0 = new Float32Array(wasm.memory.buffer);
+    }
+    return cachedFloat32ArrayMemory0;
+}
+
+function getStringFromWasm0(ptr, len) {
+    ptr = ptr >>> 0;
+    return decodeText(ptr, len);
+}
+
+let cachedUint8ArrayMemory0 = null;
+function getUint8ArrayMemory0() {
+    if (cachedUint8ArrayMemory0 === null || cachedUint8ArrayMemory0.byteLength === 0) {
+        cachedUint8ArrayMemory0 = new Uint8Array(wasm.memory.buffer);
+    }
+    return cachedUint8ArrayMemory0;
+}
+
+function handleError(f, args) {
+    try {
+        return f.apply(this, args);
+    } catch (e) {
+        const idx = addToExternrefTable0(e);
+        wasm.__wbindgen_exn_store(idx);
+    }
+}
+
+function isLikeNone(x) {
+    return x === undefined || x === null;
+}
+
+function passArray8ToWasm0(arg, malloc) {
+    const ptr = malloc(arg.length * 1, 1) >>> 0;
+    getUint8ArrayMemory0().set(arg, ptr / 1);
+    WASM_VECTOR_LEN = arg.length;
+    return ptr;
+}
+
+function passArrayF32ToWasm0(arg, malloc) {
+    const ptr = malloc(arg.length * 4, 4) >>> 0;
+    getFloat32ArrayMemory0().set(arg, ptr / 4);
+    WASM_VECTOR_LEN = arg.length;
+    return ptr;
+}
+
+function passStringToWasm0(arg, malloc, realloc) {
+    if (realloc === undefined) {
+        const buf = cachedTextEncoder.encode(arg);
+        const ptr = malloc(buf.length, 1) >>> 0;
+        getUint8ArrayMemory0().subarray(ptr, ptr + buf.length).set(buf);
+        WASM_VECTOR_LEN = buf.length;
+        return ptr;
+    }
+
+    let len = arg.length;
+    let ptr = malloc(len, 1) >>> 0;
+
+    const mem = getUint8ArrayMemory0();
+
+    let offset = 0;
+
+    for (; offset < len; offset++) {
+        const code = arg.charCodeAt(offset);
+        if (code > 0x7F) break;
+        mem[ptr + offset] = code;
+    }
+    if (offset !== len) {
+        if (offset !== 0) {
+            arg = arg.slice(offset);
+        }
+        ptr = realloc(ptr, len, len = offset + arg.length * 3, 1) >>> 0;
+        const view = getUint8ArrayMemory0().subarray(ptr + offset, ptr + len);
+        const ret = cachedTextEncoder.encodeInto(arg, view);
+
+        offset += ret.written;
+        ptr = realloc(ptr, len, offset, 1) >>> 0;
+    }
+
+    WASM_VECTOR_LEN = offset;
+    return ptr;
+}
+
+function takeFromExternrefTable0(idx) {
+    const value = wasm.__wbindgen_externrefs.get(idx);
+    wasm.__externref_table_dealloc(idx);
+    return value;
+}
+
+let cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });
+cachedTextDecoder.decode();
+const MAX_SAFARI_DECODE_BYTES = 2146435072;
+let numBytesDecoded = 0;
+function decodeText(ptr, len) {
+    numBytesDecoded += len;
+    if (numBytesDecoded >= MAX_SAFARI_DECODE_BYTES) {
+        cachedTextDecoder = new TextDecoder('utf-8', { ignoreBOM: true, fatal: true });
+        cachedTextDecoder.decode();
+        numBytesDecoded = len;
+    }
+    return cachedTextDecoder.decode(getUint8ArrayMemory0().subarray(ptr, ptr + len));
+}
+
+const cachedTextEncoder = new TextEncoder();
+
+if (!('encodeInto' in cachedTextEncoder)) {
+    cachedTextEncoder.encodeInto = function (arg, view) {
+        const buf = cachedTextEncoder.encode(arg);
+        view.set(buf);
+        return {
+            read: arg.length,
+            written: buf.length
+        };
+    };
+}
+
+let WASM_VECTOR_LEN = 0;
+
+let wasmModule, wasm;
+function __wbg_finalize_init(instance, module) {
+    wasm = instance.exports;
+    wasmModule = module;
+    cachedDataViewMemory0 = null;
+    cachedFloat32ArrayMemory0 = null;
+    cachedUint8ArrayMemory0 = null;
+    wasm.__wbindgen_start();
+    return wasm;
+}
+
+async function __wbg_load(module, imports) {
+    if (typeof Response === 'function' && module instanceof Response) {
+        if (typeof WebAssembly.instantiateStreaming === 'function') {
+            try {
+                return await WebAssembly.instantiateStreaming(module, imports);
+            } catch (e) {
+                const validResponse = module.ok && expectedResponseType(module.type);
+
+                if (validResponse && module.headers.get('Content-Type') !== 'application/wasm') {
+                    console.warn("`WebAssembly.instantiateStreaming` failed because your server does not serve Wasm with `application/wasm` MIME type. Falling back to `WebAssembly.instantiate` which is slower. Original error:\n", e);
+
+                } else { throw e; }
+            }
+        }
+
+        const bytes = await module.arrayBuffer();
+        return await WebAssembly.instantiate(bytes, imports);
+    } else {
+        const instance = await WebAssembly.instantiate(module, imports);
+
+        if (instance instanceof WebAssembly.Instance) {
+            return { instance, module };
+        } else {
+            return instance;
+        }
+    }
+
+    function expectedResponseType(type) {
+        switch (type) {
+            case 'basic': case 'cors': case 'default': return true;
+        }
+        return false;
+    }
+}
+
+function initSync(module) {
+    if (wasm !== undefined) return wasm;
+
+
+    if (module !== undefined) {
+        if (Object.getPrototypeOf(module) === Object.prototype) {
+            ({module} = module)
+        } else {
+            console.warn('using deprecated parameters for `initSync()`; pass a single object instead')
+        }
+    }
+
+    const imports = __wbg_get_imports();
+    if (!(module instanceof WebAssembly.Module)) {
+        module = new WebAssembly.Module(module);
+    }
+    const instance = new WebAssembly.Instance(module, imports);
+    return __wbg_finalize_init(instance, module);
+}
+
+async function __wbg_init(module_or_path) {
+    if (wasm !== undefined) return wasm;
+
+
+    if (module_or_path !== undefined) {
+        if (Object.getPrototypeOf(module_or_path) === Object.prototype) {
+            ({module_or_path} = module_or_path)
+        } else {
+            console.warn('using deprecated parameters for the initialization function; pass a single object instead')
+        }
+    }
+
+    if (module_or_path === undefined) {
+        module_or_path = new URL('candle_embeddings_bg.wasm', import.meta.url);
+    }
+    const imports = __wbg_get_imports();
+
+    if (typeof module_or_path === 'string' || (typeof Request === 'function' && module_or_path instanceof Request) || (typeof URL === 'function' && module_or_path instanceof URL)) {
+        module_or_path = fetch(module_or_path);
+    }
+
+    const { instance, module } = await __wbg_load(await module_or_path, imports);
+
+    return __wbg_finalize_init(instance, module);
+}
+
+export { initSync, __wbg_init as default };

src/embeddings/wasm/pkg/candle_embeddings_bg.wasm.d.ts

@@ -0,0 +1,19 @@
+/* tslint:disable */
+/* eslint-disable */
+export const memory: WebAssembly.Memory;
+export const __wbg_embeddingengine_free: (a: number, b: number) => void;
+export const cosine_similarity: (a: number, b: number, c: number, d: number) => number;
+export const embeddingengine_dimension: (a: number) => number;
+export const embeddingengine_embed: (a: number, b: number, c: number) => [number, number, number];
+export const embeddingengine_embed_batch: (a: number, b: any) => [number, number, number];
+export const embeddingengine_is_ready: (a: number) => number;
+export const embeddingengine_load: (a: number, b: number, c: number, d: number, e: number, f: number, g: number) => [number, number];
+export const embeddingengine_max_sequence_length: (a: number) => number;
+export const embeddingengine_new: () => number;
+export const __wbindgen_malloc: (a: number, b: number) => number;
+export const __wbindgen_realloc: (a: number, b: number, c: number, d: number) => number;
+export const __wbindgen_exn_store: (a: number) => void;
+export const __externref_table_alloc: () => number;
+export const __wbindgen_externrefs: WebAssembly.Table;
+export const __externref_table_dealloc: (a: number) => void;
+export const __wbindgen_start: () => void;

src/errors/brainyError.ts

@@ -10,7 +10,14 @@ export type BrainyErrorType =
   | 'NOT_FOUND'
   | 'RETRY_EXHAUSTED'
   | 'VALIDATION'
+  | 'INVALID_QUERY'
   | 'FIELD_NOT_INDEXED'
+  | 'GRAPH_INDEX_NOT_READY'
+  | 'METADATA_INDEX_NOT_READY'
+  | 'VECTOR_INDEX_NOT_READY'
+  | 'PROTECTED_ARTIFACT'
+  | 'DERIVED_ARTIFACT_MISSING'
+  | 'MIGRATION_IN_PROGRESS'
 
 /**
  * Custom error class for Brainy operations
@@ -223,3 +230,178 @@ export class BrainyError extends Error {
         return BrainyError.storage(error.message, error)
     }
 }
+
+/**
+ * Thrown when the graph adjacency index reports that relationships exist (its
+ * persisted manifest/count loaded, or its readiness signal says otherwise) but
+ * the source→target adjacency itself did NOT load — so graph traversals
+ * (`find({ connected })`, `neighbors()`, `related()`) would otherwise return an
+ * EMPTY array indistinguishable from "no edges".
+ *
+ * On 8.0 brainy detects this on the first graph read via the provider's honest
+ * sync `isReady()` signal (true ONLY when the edges are loaded; see
+ * {@link import('../plugin.js').GraphIndexProvider.isReady}); for older providers
+ * that do not expose it, it falls back to a known-edge-sample probe (one persisted
+ * verb + one neighbor lookup). Either way it attempts a rebuild from storage and
+ * raises this LOUD, catchable error only if even that cannot make the adjacency
+ * ready — replacing silent data-invisibility with a clear failure.
+ *
+ * Observed with a native graph provider whose cold-open adjacency load is
+ * swallowed on certain storage adapters; the fix is upstream in the provider,
+ * but Brainy refuses to serve `[]` as if it were truth.
+ */
+export class GraphIndexNotReadyError extends BrainyError {
+    constructor(message: string, originalError?: Error) {
+        super(message, 'GRAPH_INDEX_NOT_READY', false, originalError)
+        this.name = 'GraphIndexNotReadyError'
+        if (Error.captureStackTrace) {
+            Error.captureStackTrace(this, GraphIndexNotReadyError)
+        }
+    }
+}
+
+/**
+ * Thrown when the metadata field index reports data but cannot serve a KNOWN
+ * persisted field value even after a rebuild — i.e. the `where` / filter
+ * postings did not load on a cold open and could not be restored. The
+ * field-index counterpart of {@link GraphIndexNotReadyError}: it replaces the
+ * silent-empty failure mode (a cold `find({ where })` returning `[]`
+ * indistinguishable from "no such data") with a loud, catchable error, so a
+ * consumer never renders "nothing found" over data that is simply not-yet-warm.
+ *
+ * Detected once per brain by a known-value serving probe on the first filtered
+ * `find()`; brainy self-heals (rebuilds the index from the canonical records)
+ * first and only raises this if the rebuild still cannot serve the known value.
+ */
+export class MetadataIndexNotReadyError extends BrainyError {
+    constructor(message: string, originalError?: Error) {
+        super(message, 'METADATA_INDEX_NOT_READY', false, originalError)
+        this.name = 'MetadataIndexNotReadyError'
+        if (Error.captureStackTrace) {
+            Error.captureStackTrace(this, MetadataIndexNotReadyError)
+        }
+    }
+}
+
+/**
+ * Thrown when the vector index reports vectors (`size() > 0` or, on a native
+ * provider, `isReady() === false`) but cannot return a KNOWN persisted vector
+ * even after a rebuild — i.e. the semantic serving structure did not load on a
+ * cold open and could not be restored. The vector-search counterpart of
+ * {@link GraphIndexNotReadyError} / {@link MetadataIndexNotReadyError}: it
+ * replaces the silent-empty failure mode (a cold `find({ query })` returning
+ * `[]` indistinguishable from "no similar data") with a loud, catchable error,
+ * so a consumer never renders "nothing found" over data that is simply
+ * not-yet-warm.
+ *
+ * Detected once per brain by a known-vector serving probe on the first
+ * semantic / proximity `find()`; brainy self-heals (rebuilds the index from the
+ * canonical records) first and only raises this if the rebuild still cannot
+ * serve the known vector.
+ */
+export class VectorIndexNotReadyError extends BrainyError {
+    constructor(message: string, originalError?: Error) {
+        super(message, 'VECTOR_INDEX_NOT_READY', false, originalError)
+        this.name = 'VectorIndexNotReadyError'
+        if (Error.captureStackTrace) {
+            Error.captureStackTrace(this, VectorIndexNotReadyError)
+        }
+    }
+}
+
+/**
+ * Thrown when a delete (`deleteBinaryBlob` / `removeRawPrefix`) would remove a
+ * blob that is a declared member of a protected derived-index FAMILY (the
+ * registered-blob contract). Declared derived artifacts are undeletable through
+ * the storage layer — this makes an in-process GC / sweeper INCAPABLE of removing
+ * a load-bearing index file (the lost-`main.dkann` class). Intentional retirement
+ * is the explicit `unregisterDerivedFamily(name)` step, then the delete.
+ */
+export class ProtectedArtifactError extends BrainyError {
+    /** The blob key the delete targeted. */
+    public readonly key: string
+    /** The protected family the key belongs to. */
+    public readonly family: string
+    constructor(key: string, family: string) {
+        super(
+            `Refused to delete '${key}': it is a declared member of the protected ` +
+                `derived-index family '${family}'. Declared derived artifacts are undeletable ` +
+                `through the storage layer (COLD ≠ DEAD) — unregisterDerivedFamily('${family}') ` +
+                `first if retirement is intentional.`,
+            'PROTECTED_ARTIFACT',
+            false
+        )
+        this.name = 'ProtectedArtifactError'
+        this.key = key
+        this.family = family
+    }
+}
+
+/**
+ * Raised (loudly) when a declared derived-index family is missing one or more of
+ * its members on open — i.e. a load-bearing blob was deleted OUTSIDE the write
+ * path (an external sweeper the in-process refusal cannot stop). The index must
+ * be rebuilt from canonical; "healthy-while-broken" is impossible because the
+ * missing member is named, not silently tolerated.
+ */
+export class DerivedArtifactMissingError extends BrainyError {
+    /** The family with missing members. */
+    public readonly family: string
+    /** The member keys that are absent. */
+    public readonly missing: string[]
+    constructor(family: string, missing: string[]) {
+        super(
+            `Derived-index family '${family}' is missing ${missing.length} declared ` +
+                `member(s) on open (${missing.join(', ')}) — deleted outside the write path. ` +
+                `The index must be rebuilt from canonical.`,
+            'DERIVED_ARTIFACT_MISSING',
+            false
+        )
+        this.name = 'DerivedArtifactMissingError'
+        this.family = family
+        this.missing = missing
+    }
+}
+
+/**
+ * Thrown when a data-plane read or write is issued against a brain that is
+ * running its one-time, automatic 7.x → 8.0 on-disk upgrade — the coordinated
+ * migration LOCK. While a native provider rebuilds all derived indexes from the
+ * canonical records, brainy blocks reads and writes so no operation touches a
+ * half-built index; the caller waits for the correct answer rather than getting
+ * a partial one. This error is raised ONLY when the wait exceeds the configured
+ * window ({@link BrainyConfig.migrationWaitTimeoutMs}, default 30 s) — never
+ * instead of a partial/incorrect result.
+ *
+ * It is `retryable`: the upgrade continues in the background. Retry shortly,
+ * watch `brain.getIndexStatus().migration` for progress, or for a very large
+ * brain run the offline migrator. Data is safe; nothing is lost. Consumers can
+ * catch this (e.g. request middleware) and answer HTTP 503 + `Retry-After`.
+ *
+ * @example
+ * try {
+ *   await brain.find({ query })
+ * } catch (e) {
+ *   if (e instanceof MigrationInProgressError) {
+ *     res.set('Retry-After', '5').status(503).json({ upgrading: true, percent: e.percent })
+ *     return
+ *   }
+ *   throw e
+ * }
+ */
+export class MigrationInProgressError extends BrainyError {
+    /** Milliseconds the operation waited on the migration lock before timing out. */
+    public readonly elapsedMs: number
+    /** Latest observed migration progress (0–100), when the provider reports it. */
+    public readonly percent?: number
+
+    constructor(message: string, elapsedMs: number, percent?: number, originalError?: Error) {
+        super(message, 'MIGRATION_IN_PROGRESS', true, originalError)
+        this.name = 'MigrationInProgressError'
+        this.elapsedMs = elapsedMs
+        this.percent = percent
+        if (Error.captureStackTrace) {
+            Error.captureStackTrace(this, MigrationInProgressError)
+        }
+    }
+}

src/events/changeFeed.ts

@@ -0,0 +1,171 @@
+/**
+ * @module events/changeFeed
+ * @description The in-process change feed behind {@link Brainy.onChange} — the
+ * authoritative "something committed" signal for every canonical mutation,
+ * regardless of origin (direct API calls, batches, transactions, imports, the
+ * VFS, or an accelerated native deployment: all of them funnel through the
+ * same generation-store commit points this feed is emitted from).
+ *
+ * Design properties:
+ * - **Post-commit only.** Events are enqueued after the commit succeeds, so an
+ *   aborted commit (a losing `ifRev` CAS, a failed transaction) never emits.
+ * - **Commit-ordered.** Events are enqueued in commit order and dispatched
+ *   FIFO, so a subscriber observes mutations in the order they became durable.
+ * - **Never blocks the write path.** Dispatch happens in a microtask after the
+ *   committing call returns; a slow or throwing listener cannot delay or fail
+ *   a write. Listener errors are isolated per listener and per event.
+ * - **Zero cost when unused.** Callers consult {@link ChangeFeed.hasListeners}
+ *   before constructing event payloads; with no subscribers the write path
+ *   does no event work at all.
+ * - **Fire-and-forget.** No acknowledgement, backpressure, or replay. Events
+ *   carry the committed `generation`, so a consumer that needs catch-up
+ *   semantics can pair the live feed with `transactionLog()` / `asOf()`.
+ */
+
+/** The post-commit view of an entity carried by entity change events. */
+export interface ChangeEventEntity {
+  /** The entity's canonical UUID. */
+  id: string
+  /** The entity's NounType string (e.g. `'person'`, `'document'`). */
+  type: string
+  /** The entity's subtype, when set. */
+  subtype?: string
+  /** The entity's custom (indexed) metadata fields. */
+  metadata: Record<string, unknown>
+  /** The writing service, when set. */
+  service?: string
+}
+
+/** The post-commit view of a relationship carried by relation change events. */
+export interface ChangeEventRelation {
+  /** The relationship's canonical UUID. */
+  id: string
+  /** Source entity UUID. */
+  from: string
+  /** Target entity UUID. */
+  to: string
+  /** The relationship's VerbType string (e.g. `'contains'`). */
+  type: string
+  /** The relationship's custom metadata fields, when present. */
+  metadata?: Record<string, unknown>
+}
+
+/**
+ * One committed mutation, as delivered to {@link Brainy.onChange} listeners.
+ *
+ * Exactly one of `entity` / `relation` is populated for `kind: 'entity'` /
+ * `kind: 'relation'` events. `kind: 'store'` events (`clear` / `restore`)
+ * carry neither — they mean "the whole store changed; refetch what you care
+ * about".
+ *
+ * For `op: 'remove'` / `op: 'unrelate'` the payload is the record's LAST
+ * committed state (sourced from the commit's own before-image), so deletes are
+ * fully described rather than id-only.
+ */
+export interface BrainyChangeEvent {
+  /** What changed: one record, one relationship, or the whole store. */
+  kind: 'entity' | 'relation' | 'store'
+  /** The mutation, in Brainy's own API vocabulary. */
+  op:
+    | 'add'
+    | 'update'
+    | 'remove'
+    | 'relate'
+    | 'unrelate'
+    | 'updateRelation'
+    | 'clear'
+    | 'restore'
+  /** The mutated record's id (absent for store-level events). */
+  id?: string
+  /** Post-commit entity view (entity events only). */
+  entity?: ChangeEventEntity
+  /** Post-commit relation view (relation events only). */
+  relation?: ChangeEventRelation
+  /**
+   * The committed generation this mutation belongs to (Model B: every write
+   * is a generation; a transaction's items share one). Absent for store-level
+   * events and init-time bootstrap writes, which are not generation-stamped.
+   */
+  generation?: number
+  /** Commit timestamp (ms since epoch). */
+  timestamp: number
+}
+
+/** Listener signature for {@link Brainy.onChange}. */
+export type ChangeListener = (event: BrainyChangeEvent) => void
+
+/**
+ * A change event as constructed by a mutation method BEFORE its commit: the
+ * commit seam stamps `generation`/`timestamp` after the write becomes
+ * durable. An entity `remove` descriptor may omit `entity` — the seam fills
+ * it from the commit's own before-image (the record's last committed state).
+ */
+export type PendingChangeEvent = Omit<BrainyChangeEvent, 'generation' | 'timestamp'>
+
+/**
+ * @description Listener registry + commit-ordered async dispatcher for
+ * {@link BrainyChangeEvent}s. One instance per {@link Brainy}.
+ */
+export class ChangeFeed {
+  private listeners = new Set<ChangeListener>()
+  private queue: BrainyChangeEvent[] = []
+  private draining = false
+
+  /** Whether any listener is subscribed — the write path's zero-cost gate. */
+  get hasListeners(): boolean {
+    return this.listeners.size > 0
+  }
+
+  /**
+   * @description Subscribe to committed mutations.
+   * @param listener - Called once per committed mutation, in commit order.
+   * @returns An unsubscribe function.
+   */
+  subscribe(listener: ChangeListener): () => void {
+    this.listeners.add(listener)
+    return () => {
+      this.listeners.delete(listener)
+    }
+  }
+
+  /**
+   * @description Enqueue committed events and schedule dispatch. Call ONLY
+   * after the commit has succeeded — this feed must never announce a write
+   * that did not become durable. Safe to call with an empty array.
+   * @param events - The committed mutations, in commit order.
+   */
+  emit(events: BrainyChangeEvent[]): void {
+    if (events.length === 0 || this.listeners.size === 0) return
+    this.queue.push(...events)
+    if (!this.draining) {
+      this.draining = true
+      queueMicrotask(() => this.drain())
+    }
+  }
+
+  /** Deliver everything queued, FIFO, isolating listener errors. */
+  private drain(): void {
+    try {
+      while (this.queue.length > 0) {
+        const event = this.queue.shift()!
+        for (const listener of this.listeners) {
+          try {
+            listener(event)
+          } catch (err) {
+            // A subscriber's bug must never affect the write path or its
+            // sibling subscribers.
+            console.error('[Brainy] onChange listener threw:', err)
+          }
+        }
+      }
+    } finally {
+      this.draining = false
+    }
+  }
+
+  /** Drop all listeners (brain close/teardown). Queued events are discarded. */
+  close(): void {
+    this.listeners.clear()
+    this.queue.length = 0
+  }
+}

src/graph/analyticsFallback.ts

@@ -0,0 +1,244 @@
+/**
+ * @module graph/analyticsFallback
+ * @description Pure-TS graph-analytics kernels — the fallback implementations
+ * behind `brain.graph.rank` / `brain.graph.communities` when no native
+ * {@link import('../plugin.js').GraphAccelerationProvider} is registered. Each
+ * operates on a dense integer adjacency (`outAdj[i]` = the target indices of node
+ * `i`'s out-edges, multiplicity preserved) so callers can build it once from the
+ * UUID graph and run any kernel.
+ *
+ * These are INTENT-level fallbacks: the public API promises the QUESTION
+ * ("which nodes matter most", "which things group together") — these answer it
+ * with the standard textbook algorithm (PageRank, connected components,
+ * Tarjan SCC). A native provider may answer the same intent with a different
+ * algorithm (personalized PageRank, Louvain, etc.); correctness of the INTENT,
+ * not the algorithm, is the contract.
+ *
+ * Correct at small/medium scale (the native provider is the at-scale path). All
+ * graph walks are iterative (explicit stacks/queues) so a deep or wide graph
+ * never blows the call stack.
+ */
+
+/** Adjacency where `outAdj[i]` lists the target node indices of `i`'s out-edges. */
+export type DenseAdjacency = ReadonlyArray<ReadonlyArray<number>>
+
+/**
+ * @description Label each node with the id of its WEAKLY-connected component
+ *   (edges treated as undirected) via union-find with path compression + a single
+ *   pass over the adjacency. Two nodes share a label iff one can reach the other
+ *   ignoring edge direction.
+ * @param outAdj - Dense out-adjacency.
+ * @returns `labels[i]` = the component representative of node `i` (labels are
+ *   stable per component but NOT necessarily contiguous — group by equality).
+ */
+export function connectedComponents(outAdj: DenseAdjacency): number[] {
+  const n = outAdj.length
+  const parent = new Array<number>(n)
+  for (let i = 0; i < n; i++) parent[i] = i
+
+  const find = (x: number): number => {
+    let root = x
+    while (parent[root] !== root) root = parent[root]
+    // Path compression: point every node on the chain straight at the root.
+    while (parent[x] !== root) {
+      const next = parent[x]
+      parent[x] = root
+      x = next
+    }
+    return root
+  }
+
+  for (let i = 0; i < n; i++) {
+    const neighbors = outAdj[i]
+    for (let k = 0; k < neighbors.length; k++) {
+      const ra = find(i)
+      const rb = find(neighbors[k])
+      if (ra !== rb) parent[ra] = rb
+    }
+  }
+
+  const labels = new Array<number>(n)
+  for (let i = 0; i < n; i++) labels[i] = find(i)
+  return labels
+}
+
+/**
+ * @description Label each node with its STRONGLY-connected component (directed —
+ *   two nodes share a label iff each is reachable from the other following edge
+ *   direction) via an iterative Tarjan's algorithm.
+ * @param outAdj - Dense out-adjacency.
+ * @returns `labels[i]` = the SCC index of node `i` (contiguous `0..count-1`).
+ */
+export function stronglyConnectedComponents(outAdj: DenseAdjacency): number[] {
+  const n = outAdj.length
+  const index = new Array<number>(n).fill(-1)
+  const low = new Array<number>(n).fill(0)
+  const onStack = new Array<boolean>(n).fill(false)
+  const comp = new Array<number>(n).fill(-1)
+  const sccStack: number[] = []
+  let nextIndex = 0
+  let compCount = 0
+
+  for (let start = 0; start < n; start++) {
+    if (index[start] !== -1) continue
+    // Explicit DFS stack of frames; `pi` is the next out-edge to visit for `node`.
+    const callStack: Array<{ node: number; pi: number }> = [{ node: start, pi: 0 }]
+    while (callStack.length > 0) {
+      const frame = callStack[callStack.length - 1]
+      const v = frame.node
+      if (frame.pi === 0) {
+        index[v] = low[v] = nextIndex++
+        sccStack.push(v)
+        onStack[v] = true
+      }
+      if (frame.pi < outAdj[v].length) {
+        const w = outAdj[v][frame.pi]
+        frame.pi++
+        if (index[w] === -1) {
+          callStack.push({ node: w, pi: 0 })
+        } else if (onStack[w]) {
+          if (index[w] < low[v]) low[v] = index[w]
+        }
+      } else {
+        // All edges of v explored — if v roots an SCC, pop it off the SCC stack.
+        if (low[v] === index[v]) {
+          for (;;) {
+            const w = sccStack.pop() as number
+            onStack[w] = false
+            comp[w] = compCount
+            if (w === v) break
+          }
+          compCount++
+        }
+        callStack.pop()
+        if (callStack.length > 0) {
+          const parent = callStack[callStack.length - 1].node
+          if (low[v] < low[parent]) low[parent] = low[v]
+        }
+      }
+    }
+  }
+
+  return comp
+}
+
+/** Tuning knobs for {@link pageRank}. */
+export interface PageRankOptions {
+  /** Teleport/damping factor (default `0.85`). */
+  damping?: number
+  /** Max power-iterations before stopping (default `100`). */
+  maxIterations?: number
+  /** L1 convergence threshold on the score vector (default `1e-6`). */
+  tolerance?: number
+}
+
+/**
+ * @description PageRank importance score per node via power-iteration. Dangling
+ *   nodes (no out-edges) redistribute their mass uniformly so the score vector
+ *   stays a probability distribution (sums to 1). Edge multiplicity is honored
+ *   (a node with two edges to the same target sends it twice the share).
+ * @param outAdj - Dense out-adjacency.
+ * @param options - Damping / iteration / tolerance knobs.
+ * @returns `scores[i]` = node `i`'s PageRank (higher = more central).
+ */
+export function pageRank(outAdj: DenseAdjacency, options?: PageRankOptions): Float64Array {
+  const n = outAdj.length
+  if (n === 0) return new Float64Array(0)
+
+  const damping = options?.damping ?? 0.85
+  const maxIterations = options?.maxIterations ?? 100
+  const tolerance = options?.tolerance ?? 1e-6
+
+  const outDegree = new Array<number>(n)
+  for (let i = 0; i < n; i++) outDegree[i] = outAdj[i].length
+
+  let rank = new Float64Array(n)
+  rank.fill(1 / n)
+  const teleport = (1 - damping) / n
+
+  for (let iter = 0; iter < maxIterations; iter++) {
+    // Mass stranded on dangling nodes this round, spread uniformly to everyone.
+    let danglingMass = 0
+    for (let i = 0; i < n; i++) if (outDegree[i] === 0) danglingMass += rank[i]
+    const danglingShare = (damping * danglingMass) / n
+
+    const next = new Float64Array(n)
+    next.fill(teleport + danglingShare)
+
+    for (let i = 0; i < n; i++) {
+      const degree = outDegree[i]
+      if (degree === 0) continue
+      const share = (damping * rank[i]) / degree
+      const neighbors = outAdj[i]
+      for (let k = 0; k < neighbors.length; k++) next[neighbors[k]] += share
+    }
+
+    let delta = 0
+    for (let i = 0; i < n; i++) delta += Math.abs(next[i] - rank[i])
+    rank = next
+    if (delta < tolerance) break
+  }
+
+  return rank
+}
+
+/**
+ * @description A minimal binary min-heap (priority queue) — used by the weighted
+ *   shortest-path (Dijkstra) fallback to pop the lowest-cost frontier node in
+ *   O(log n). Stable enough for pathfinding; ties pop in unspecified order.
+ */
+export class MinHeap<T> {
+  private readonly items: Array<{ value: T; priority: number }> = []
+
+  /** Number of queued items. */
+  get size(): number {
+    return this.items.length
+  }
+
+  /**
+   * @description Insert `value` with the given `priority` (lower pops first).
+   * @param value - The payload.
+   * @param priority - Ordering key (ascending).
+   */
+  push(value: T, priority: number): void {
+    const items = this.items
+    items.push({ value, priority })
+    let i = items.length - 1
+    while (i > 0) {
+      const parent = (i - 1) >> 1
+      if (items[parent].priority <= items[i].priority) break
+      const tmp = items[parent]
+      items[parent] = items[i]
+      items[i] = tmp
+      i = parent
+    }
+  }
+
+  /**
+   * @description Remove and return the lowest-priority value.
+   * @returns The min value, or `undefined` when empty.
+   */
+  pop(): T | undefined {
+    const items = this.items
+    if (items.length === 0) return undefined
+    const top = items[0].value
+    const last = items.pop() as { value: T; priority: number }
+    if (items.length > 0) {
+      items[0] = last
+      let i = 0
+      for (;;) {
+        const left = 2 * i + 1
+        const right = 2 * i + 2
+        let smallest = i
+        if (left < items.length && items[left].priority < items[smallest].priority) smallest = left
+        if (right < items.length && items[right].priority < items[smallest].priority) smallest = right
+        if (smallest === i) break
+        const tmp = items[smallest]
+        items[smallest] = items[i]
+        items[i] = tmp
+        i = smallest
+      }
+    }
+    return top
+  }
+}

src/graph/graphAdjacencyIndex.ts

@@ -77,9 +77,10 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
   private lsmTreeVerbsBySource: LSMTree  // sourceId -> verbIds
   private lsmTreeVerbsByTarget: LSMTree  // targetId -> verbIds
 
-  // ID-only tracking for billion-scale memory optimization
-  // Previous: Map<string, GraphVerb> stored full objects (128GB @ 1B verbs)
-  // Now: Set<string> stores only IDs (~100KB @ 1B verbs) = 1,280,000x reduction
+  // ID-only membership tracking: a Set<string> of verb ids rather than a
+  // Map<string, GraphVerb> of full objects, so per-verb resident memory is one id
+  // string instead of a whole relationship record. (Unmeasured order-of-magnitude
+  // figures dropped — the durable property is "ids, not objects".)
   private verbIdSet = new Set<string>()
 
   // Verb-id interning for the BigInt boundary (8.0 u64 contract).
@@ -194,6 +195,25 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
     return next
   }
 
+  /**
+   * Eager cold-load of the persisted adjacency (readiness contract).
+   *
+   * Loads the LSM manifests + SSTables so `size()` reports the durable edge
+   * count at the rebuild gate — a warm reopen must load the persisted index,
+   * never re-derive it from a full canonical verb scan (the every-boot O(E)
+   * cost this method exists to eliminate). Idempotent; the lazy read paths
+   * call the same `ensureInitialized()` on demand.
+   *
+   * NOTE: the JS index deliberately does NOT expose `isReady()`. That signal
+   * (see `GraphIndexProvider.isReady`) asserts "traversals are trustworthy",
+   * which this side cannot honestly promise without comparing against the
+   * canonical store — the query-time known-edge probe
+   * (`verifyGraphAdjacencyLive` Strategy 2) remains the JS trust check.
+   */
+  async init(): Promise<void> {
+    await this.ensureInitialized()
+  }
+
   /**
    * Initialize the graph index (lazy initialization)
    * Added defensive auto-rebuild check for verbIdSet consistency
@@ -262,6 +282,16 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
       }
 
       hasMore = result.hasMore
+      if (hasMore && (!result.nextCursor || result.nextCursor === cursor)) {
+        // A stalled cursor with hasMore=true would re-read the same page
+        // forever — a silent full-CPU loop at cold open. Abort loudly; a
+        // graph read failing beats a process that spins without a log line.
+        throw new Error(
+          `GraphAdjacencyIndex: verb walk stalled after ${count} verbs — storage returned ` +
+            `hasMore=true with ${result.nextCursor ? 'a non-advancing' : 'no'} cursor. ` +
+            `Aborting the cold-load; run brain.repairIndex() if this persists.`
+        )
+      }
       cursor = result.nextCursor
     }
 
@@ -877,6 +907,11 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
     this.flushTimer = setInterval(async () => {
       await this.flush()
     }, this.config.flushInterval)
+    // Background maintenance must never keep the host process alive —
+    // close()/flush() handle durability; the interval is best-effort.
+    if (typeof this.flushTimer.unref === 'function') {
+      this.flushTimer.unref()
+    }
   }
 
   /**

src/graph/graphAudit.ts

@@ -0,0 +1,214 @@
+/**
+ * @module graph/graphAudit
+ * @description Read-only graph-truth audit — the graph sibling of `repairIndex()`'s
+ * diagnosis half. Verifies three layers against each other without mutating anything:
+ *
+ *   1. CANONICAL verb records (the storage walk — the source of truth)
+ *   2. the RELATIONSHIP READ PATH (`related()` with all visibility tiers — exactly
+ *      what application reads like a VFS `readdir` consult)
+ *   3. ENTITY ENDPOINTS (does each verb's source/target still exist?)
+ *
+ * and classifies every discrepancy into the three failure families production
+ * incidents have shown:
+ *
+ *   - `missingFromReads` — a canonical verb record the read path does NOT return
+ *     for its source: PRESENT BUT INVISIBLE (adjacency/membership staleness).
+ *   - `danglingEndpoints` — a canonical verb whose endpoint entity is gone:
+ *     the SCAR class (write-path loss / partial delete).
+ *   - `readOnlyVerbIds` — the read path returns an edge with NO canonical
+ *     record: GHOST edges (stale index entries).
+ *
+ * `visibilityHiddenCount` is reported separately: an internal/system edge that is
+ * indexed and present but hidden from DEFAULT reads is working as designed — the
+ * audit reads with all tiers included so design-hiding is never misclassified as
+ * index loss.
+ *
+ * Full counts are always exact; only the example LISTS are capped (`maxExamples`)
+ * — a capped report says so via `truncatedExamples`, never silently.
+ */
+
+import { prodLog } from '../utils/logger.js'
+
+/** One discrepant relationship, identified fully enough to inspect by hand. */
+export interface GraphAuditDiscrepancy {
+  verbId: string
+  from: string
+  to: string
+  type: string
+}
+
+export interface GraphAuditReport {
+  /** True iff every discrepancy count is zero — `related()` returns canonical truth. */
+  coherent: boolean
+  verbsInCanonical: number
+  entitiesInCanonical: number
+  /** Distinct source entities whose read path was actually consulted (coverage honesty). */
+  sourcesChecked: number
+
+  /** PRESENT BUT INVISIBLE: canonical records the read path omits. */
+  missingFromReadsCount: number
+  missingFromReads: GraphAuditDiscrepancy[]
+
+  /** SCAR CLASS: canonical verbs with a missing endpoint entity. */
+  danglingEndpointsCount: number
+  danglingEndpoints: Array<GraphAuditDiscrepancy & { missingEnd: 'from' | 'to' | 'both' }>
+
+  /** GHOST EDGES: read-path verb ids with no canonical record. */
+  readOnlyCount: number
+  readOnlyVerbIds: string[]
+
+  /** Canonical verbs hidden from DEFAULT reads by design (internal/system visibility). */
+  visibilityHiddenCount: number
+
+  /** Example lists above were capped at maxExamples; counts remain exact. */
+  truncatedExamples: boolean
+  durationMs: number
+}
+
+/** A canonical verb record, as the audit needs it. */
+export interface AuditVerbRecord {
+  id: string
+  type: string
+  sourceId: string
+  targetId: string
+  visibility?: string
+}
+
+/** The seams the audit runs over — injected so the walk is testable in isolation. */
+export interface GraphAuditDeps {
+  /** Stream every canonical entity id (id-only; no per-entity reads needed). */
+  eachNounId(consume: (id: string) => void): Promise<void>
+  /** Stream every canonical verb record. */
+  eachVerb(consume: (verb: AuditVerbRecord) => void): Promise<void>
+  /**
+   * The END-TO-END relationship read for one source, ALL visibility tiers
+   * included — must be the same path application reads consult.
+   */
+  readRelationsFrom(sourceId: string): Promise<Array<{ id: string }>>
+}
+
+export interface GraphAuditOptions {
+  /** Cap on entries per example list (counts stay exact). Default 100. */
+  maxExamples?: number
+}
+
+export async function runGraphAudit(
+  deps: GraphAuditDeps,
+  options: GraphAuditOptions = {}
+): Promise<GraphAuditReport> {
+  const maxExamples = options.maxExamples ?? 100
+  const started = Date.now()
+
+  // 1. Canonical entity ids — endpoint existence oracle.
+  const entityIds = new Set<string>()
+  await deps.eachNounId((id) => entityIds.add(id))
+
+  // 2. Canonical verb walk: group by source, check endpoints, note visibility.
+  const canonicalVerbIds = new Set<string>()
+  const bySource = new Map<string, AuditVerbRecord[]>()
+  let verbsInCanonical = 0
+  let visibilityHiddenCount = 0
+  let danglingEndpointsCount = 0
+  const danglingEndpoints: GraphAuditReport['danglingEndpoints'] = []
+
+  await deps.eachVerb((verb) => {
+    verbsInCanonical++
+    canonicalVerbIds.add(verb.id)
+    const list = bySource.get(verb.sourceId)
+    if (list) list.push(verb)
+    else bySource.set(verb.sourceId, [verb])
+
+    if (verb.visibility === 'internal' || verb.visibility === 'system') {
+      visibilityHiddenCount++
+    }
+
+    const fromMissing = !entityIds.has(verb.sourceId)
+    const toMissing = !entityIds.has(verb.targetId)
+    if (fromMissing || toMissing) {
+      danglingEndpointsCount++
+      if (danglingEndpoints.length < maxExamples) {
+        danglingEndpoints.push({
+          verbId: verb.id,
+          from: verb.sourceId,
+          to: verb.targetId,
+          type: verb.type,
+          missingEnd: fromMissing && toMissing ? 'both' : fromMissing ? 'from' : 'to'
+        })
+      }
+    }
+  })
+
+  // 3. Per-source read-path comparison. A verb must be returned by the read
+  //    path of ITS OWN source — the exact consult a readdir/traversal makes.
+  let missingFromReadsCount = 0
+  const missingFromReads: GraphAuditDiscrepancy[] = []
+  let readOnlyCount = 0
+  const readOnlyVerbIds: string[] = []
+  const readOnlySeen = new Set<string>()
+
+  for (const [sourceId, verbs] of bySource) {
+    const readIds = new Set((await deps.readRelationsFrom(sourceId)).map((r) => r.id))
+
+    for (const verb of verbs) {
+      if (!readIds.has(verb.id)) {
+        missingFromReadsCount++
+        if (missingFromReads.length < maxExamples) {
+          missingFromReads.push({
+            verbId: verb.id,
+            from: verb.sourceId,
+            to: verb.targetId,
+            type: verb.type
+          })
+        }
+      }
+    }
+
+    for (const readId of readIds) {
+      if (!canonicalVerbIds.has(readId) && !readOnlySeen.has(readId)) {
+        readOnlySeen.add(readId)
+        readOnlyCount++
+        if (readOnlyVerbIds.length < maxExamples) {
+          readOnlyVerbIds.push(readId)
+        }
+      }
+    }
+  }
+
+  const coherent =
+    missingFromReadsCount === 0 && danglingEndpointsCount === 0 && readOnlyCount === 0
+
+  const report: GraphAuditReport = {
+    coherent,
+    verbsInCanonical,
+    entitiesInCanonical: entityIds.size,
+    sourcesChecked: bySource.size,
+    missingFromReadsCount,
+    missingFromReads,
+    danglingEndpointsCount,
+    danglingEndpoints,
+    readOnlyCount,
+    readOnlyVerbIds,
+    visibilityHiddenCount,
+    truncatedExamples:
+      missingFromReadsCount > missingFromReads.length ||
+      danglingEndpointsCount > danglingEndpoints.length ||
+      readOnlyCount > readOnlyVerbIds.length,
+    durationMs: Date.now() - started
+  }
+
+  if (coherent) {
+    prodLog.info(
+      `[GraphAudit] coherent: ${verbsInCanonical} verbs across ${bySource.size} sources — ` +
+        `the read path returns canonical truth (${report.durationMs}ms)`
+    )
+  } else {
+    prodLog.warn(
+      `[GraphAudit] INCOHERENT: ${missingFromReadsCount} present-but-invisible, ` +
+        `${danglingEndpointsCount} dangling-endpoint, ${readOnlyCount} ghost ` +
+        `(of ${verbsInCanonical} canonical verbs, ${bySource.size} sources, ` +
+        `${visibilityHiddenCount} visibility-hidden by design) — ${report.durationMs}ms`
+    )
+  }
+
+  return report
+}

src/graph/lsm/LSMTree.ts

@@ -1,21 +1,21 @@
 /**
- * LSMTree - Log-Structured Merge Tree for Graph Storage
- *
- * Production-grade LSM-tree implementation that reduces memory usage
- * from 500GB to 1.3GB for 1 billion relationships while maintaining
- * sub-5ms read performance.
+ * @module graph/lsm/LSMTree
+ * @description Log-Structured Merge tree for the JS (open-core) graph store — the
+ * fallback used when no native graph provider is registered. Verb-id postings are
+ * buffered in an in-memory MemTable, flushed to immutable sorted SSTables, and
+ * background-compacted; bloom filters give fast negative lookups.
  *
  * Architecture:
- * - MemTable: In-memory write buffer (100K relationships, ~24MB)
- * - SSTables: Immutable sorted files on disk (10K relationships each)
- * - Bloom Filters: In-memory filters for fast negative lookups
- * - Compaction: Background merging of SSTables
+ * - MemTable: in-memory write buffer (flush threshold configurable, default 100K)
+ * - SSTables: immutable sorted segments, persisted via the StorageAdapter
+ * - Bloom filters: in-memory membership pre-checks
+ * - Compaction: background merge of SSTables (reclaims superseded segments)
  *
- * Key Properties:
- * - Write-optimized: O(1) writes to MemTable
- * - Read-efficient: O(log n) reads with bloom filter optimization
- * - Memory-efficient: 385x less memory than all-in-RAM approach
- * - Storage-agnostic: Works with any StorageAdapter
+ * Complexity: O(1) MemTable writes; O(log n) reads with bloom-filter pre-filtering.
+ * Note: this JS implementation loads its SSTables into memory after open (it is the
+ * fallback engine). Billion-scale, on-disk-resident operation is the native graph
+ * provider's role behind the provider boundary, not this fallback's; no absolute
+ * memory/latency figures are claimed here without a cited benchmark.
  */
 
 import { StorageAdapter } from '../../coreTypes.js'
@@ -457,15 +457,20 @@ export class LSMTree {
         }
       })
 
-      // Delete old SSTables from storage
+      // Reclaim the compacted-away SSTables: drop them from the manifest AND
+      // delete their persisted payloads, so the system channel does not grow
+      // unbounded with graph write volume. (deleteMetadata is idempotent, so a
+      // never-persisted memtable-only SSTable is a harmless no-op.)
       for (const sstable of sstables) {
         const oldKey = `${this.config.storagePrefix}-${sstable.metadata.id}`
+        this.manifest.sstables.delete(sstable.metadata.id)
         try {
-          // StorageAdapter doesn't have deleteMetadata, so we'll leave orphaned data
-          // In production, we'd add a cleanup mechanism
-          this.manifest.sstables.delete(sstable.metadata.id)
+          await this.storage.deleteMetadata(oldKey)
         } catch (error) {
-          prodLog.warn(`LSMTree: Failed to delete old SSTable ${sstable.metadata.id}`, error)
+          // A reclaim failure must not abort compaction (the merged SSTable is
+          // already durable and the manifest no longer references the old one);
+          // surface it so a persistent leak is visible rather than silent.
+          prodLog.warn(`LSMTree: failed to delete compacted SSTable ${sstable.metadata.id}`, error)
         }
       }
 
@@ -512,6 +517,11 @@ export class LSMTree {
         }
       }
     }, this.config.compactionInterval)
+    // Background compaction must never keep the host process alive —
+    // close() compacts/flushes deterministically; this interval is best-effort.
+    if (this.compactionTimer && typeof this.compactionTimer.unref === 'function') {
+      this.compactionTimer.unref()
+    }
   }
 
   /**
@@ -537,13 +547,26 @@ export class LSMTree {
         const data = metadata.data as PersistedManifestData
         this.manifest.sstables = new Map(Object.entries(data.sstables || {}))
         this.manifest.lastCompaction = data.lastCompaction || Date.now()
-        this.manifest.totalRelationships = data.totalRelationships || 0
 
-        // Load SSTables from storage
+        // Load SSTables from storage BEFORE publishing the persisted count.
+        // If the SSTable load throws, `size()` must keep reporting 0 — a tree
+        // that claims its persisted relationships while holding none serves
+        // silent-empty traversals as truth (the cold-load swallow class), and
+        // downstream self-heal keys off the honest 0.
         await this.loadSSTables()
+        this.manifest.totalRelationships = data.totalRelationships || 0
       }
     } catch (error) {
-      prodLog.debug('LSMTree: No existing manifest found, starting fresh')
+      // Reset anything partially loaded — an honest empty tree triggers the
+      // rebuild/self-heal paths; a half-loaded one masks them. (An absent
+      // manifest on a fresh store also lands here: empty is correct.)
+      this.manifest.sstables = new Map()
+      this.manifest.totalRelationships = 0
+      this.sstablesByLevel.clear()
+      prodLog.debug(
+        `LSMTree(${this.config.storagePrefix}): no loadable manifest/SSTables — starting empty ` +
+          `(${error instanceof Error ? error.message : String(error)})`
+      )
     }
   }
 
@@ -551,6 +574,7 @@ export class LSMTree {
    * Load SSTables from storage based on manifest
    */
   private async loadSSTables(): Promise<void> {
+    const failures: string[] = []
     const loadPromises: Promise<void>[] = []
 
     this.manifest.sstables.forEach((level, sstableId) => {
@@ -575,7 +599,12 @@ export class LSMTree {
             }
           }
         } catch (error) {
+          // A per-SSTable load failure means the persisted adjacency is INCOMPLETE.
+          // Record it and fail the whole load closed (below): a partially-loaded
+          // tree that still publishes its full manifest count via size() would
+          // serve silent-empty traversals as truth (the cold-load swallow class).
           prodLog.warn(`LSMTree: Failed to load SSTable ${sstableId}`, error)
+          failures.push(sstableId)
         }
       })()
 
@@ -583,6 +612,20 @@ export class LSMTree {
     })
 
     await Promise.all(loadPromises)
+
+    if (failures.length > 0) {
+      // Fail closed. loadManifest()'s catch resets sstables/totalRelationships/
+      // sstablesByLevel to honest-empty, so size() reports 0 and the graph
+      // self-heal (_initializeGraphIndex size()===0 → rebuild) restores the index
+      // from the canonical records. Honest-partial is never published.
+      throw new Error(
+        `LSMTree(${this.config.storagePrefix}): ${failures.length} of ` +
+          `${this.manifest.sstables.size} SSTable(s) failed to load ` +
+          `(${failures.join(', ')}) — failing the load closed so size() reports 0 ` +
+          `and the graph self-heal rebuilds from canonical.`
+      )
+    }
+
     prodLog.info(`LSMTree: Loaded ${this.manifest.sstables.size} SSTables`)
   }
 

src/graph/lsm/SSTable.ts

@@ -20,13 +20,13 @@ import { BloomFilter, SerializedBloomFilter } from './BloomFilter.js'
 import { compareCodePoints } from '../../utils/collation.js'
 
 // Swappable msgpack implementation — defaults to @msgpack/msgpack JS,
-// can be replaced with native msgpack (e.g., cortex's Rust-backed encoder)
+// can be replaced with native msgpack (e.g., cor's Rust-backed encoder)
 let _encode: (data: unknown) => Uint8Array = defaultEncode as (data: unknown) => Uint8Array
 let _decode: (data: Uint8Array) => unknown = defaultDecode as (data: Uint8Array) => unknown
 
 /**
  * Replace the msgpack encode/decode implementation at runtime.
- * Called by brainy.ts when a cortex 'msgpack' provider is registered.
+ * Called by brainy.ts when a cor 'msgpack' provider is registered.
  */
 export function setMsgpackImplementation(impl: { encode: (data: unknown) => Uint8Array, decode: (data: Uint8Array) => unknown }) {
   _encode = impl.encode

src/graph/pathfinding.ts

@@ -1,523 +0,0 @@
-/**
- * Advanced Graph Pathfinding Algorithms
- * Provides shortest path, multi-hop traversal, and path ranking
- */
-
-// Graph pathfinding doesn't need to import from coreTypes
-
-export interface GraphNode {
-  id: string
-  [key: string]: any
-}
-
-export interface GraphEdge {
-  source: string
-  target: string
-  type: string
-  weight: number
-  metadata?: any
-}
-
-export interface Path {
-  nodes: string[]
-  edges: GraphEdge[]
-  totalWeight: number
-  length: number
-}
-
-export interface PathfindingOptions {
-  maxDepth?: number
-  maxPaths?: number
-  bidirectional?: boolean
-  weightField?: string
-  relationshipTypes?: string[]
-  nodeFilter?: (node: GraphNode) => boolean
-  edgeFilter?: (edge: GraphEdge) => boolean
-}
-
-export class GraphPathfinding {
-  private adjacencyList: Map<string, Map<string, GraphEdge[]>> = new Map()
-  private nodes: Map<string, GraphNode> = new Map()
-  
-  /**
-   * Add a node to the graph
-   */
-  public addNode(node: GraphNode): void {
-    this.nodes.set(node.id, node)
-    if (!this.adjacencyList.has(node.id)) {
-      this.adjacencyList.set(node.id, new Map())
-    }
-  }
-  
-  /**
-   * Add an edge to the graph
-   */
-  public addEdge(edge: GraphEdge): void {
-    // Ensure nodes exist
-    if (!this.adjacencyList.has(edge.source)) {
-      this.adjacencyList.set(edge.source, new Map())
-    }
-    if (!this.adjacencyList.has(edge.target)) {
-      this.adjacencyList.set(edge.target, new Map())
-    }
-    
-    // Add edge to adjacency list
-    const sourceEdges = this.adjacencyList.get(edge.source)!
-    if (!sourceEdges.has(edge.target)) {
-      sourceEdges.set(edge.target, [])
-    }
-    sourceEdges.get(edge.target)!.push(edge)
-  }
-  
-  /**
-   * Find shortest path using Dijkstra's algorithm
-   * O((V + E) log V) with binary heap
-   */
-  public shortestPath(
-    start: string, 
-    end: string, 
-    options: PathfindingOptions = {}
-  ): Path | null {
-    const { 
-      maxDepth = Infinity,
-      relationshipTypes,
-      edgeFilter 
-    } = options
-    
-    // Priority queue: [nodeId, distance, path]
-    const pq: Array<[string, number, string[], GraphEdge[]]> = [[start, 0, [start], []]]
-    const visited = new Set<string>()
-    const distances = new Map<string, number>([[start, 0]])
-    
-    while (pq.length > 0) {
-      // Sort by distance (simple array, could optimize with heap)
-      pq.sort((a, b) => a[1] - b[1])
-      const [current, distance, path, edges] = pq.shift()!
-      
-      if (visited.has(current)) continue
-      visited.add(current)
-      
-      // Found target
-      if (current === end) {
-        return {
-          nodes: path,
-          edges,
-          totalWeight: distance,
-          length: path.length - 1
-        }
-      }
-      
-      // Max depth reached
-      if (path.length > maxDepth) continue
-      
-      // Explore neighbors
-      const neighbors = this.adjacencyList.get(current)
-      if (!neighbors) continue
-      
-      for (const [neighbor, edgeList] of neighbors) {
-        if (visited.has(neighbor)) continue
-        
-        // Find best edge to neighbor
-        let bestEdge: GraphEdge | null = null
-        let bestWeight = Infinity
-        
-        for (const edge of edgeList) {
-          // Apply filters
-          if (relationshipTypes && !relationshipTypes.includes(edge.type)) continue
-          if (edgeFilter && !edgeFilter(edge)) continue
-          
-          if (edge.weight < bestWeight) {
-            bestWeight = edge.weight
-            bestEdge = edge
-          }
-        }
-        
-        if (!bestEdge) continue
-        
-        const newDistance = distance + bestWeight
-        const currentBest = distances.get(neighbor) ?? Infinity
-        
-        if (newDistance < currentBest) {
-          distances.set(neighbor, newDistance)
-          pq.push([
-            neighbor,
-            newDistance,
-            [...path, neighbor],
-            [...edges, bestEdge]
-          ])
-        }
-      }
-    }
-    
-    return null // No path found
-  }
-  
-  /**
-   * Find all paths between two nodes
-   * Uses DFS with cycle detection
-   */
-  public allPaths(
-    start: string,
-    end: string,
-    options: PathfindingOptions = {}
-  ): Path[] {
-    const { 
-      maxDepth = 10,
-      maxPaths = 100,
-      relationshipTypes,
-      edgeFilter 
-    } = options
-    
-    const paths: Path[] = []
-    const visited = new Set<string>()
-    
-    const dfs = (
-      current: string,
-      path: string[],
-      edges: GraphEdge[],
-      weight: number
-    ): void => {
-      if (paths.length >= maxPaths) return
-      if (path.length > maxDepth) return
-      
-      if (current === end && path.length > 1) {
-        paths.push({
-          nodes: [...path],
-          edges: [...edges],
-          totalWeight: weight,
-          length: path.length - 1
-        })
-        return
-      }
-      
-      visited.add(current)
-      
-      const neighbors = this.adjacencyList.get(current)
-      if (neighbors) {
-        for (const [neighbor, edgeList] of neighbors) {
-          if (visited.has(neighbor)) continue
-          
-          for (const edge of edgeList) {
-            // Apply filters
-            if (relationshipTypes && !relationshipTypes.includes(edge.type)) continue
-            if (edgeFilter && !edgeFilter(edge)) continue
-            
-            dfs(
-              neighbor,
-              [...path, neighbor],
-              [...edges, edge],
-              weight + edge.weight
-            )
-          }
-        }
-      }
-      
-      visited.delete(current)
-    }
-    
-    dfs(start, [start], [], 0)
-    
-    // Sort paths by weight
-    paths.sort((a, b) => a.totalWeight - b.totalWeight)
-    
-    return paths
-  }
-  
-  /**
-   * Bidirectional search for faster pathfinding
-   * Searches from both start and end simultaneously
-   */
-  public bidirectionalSearch(
-    start: string,
-    end: string,
-    options: PathfindingOptions = {}
-  ): Path | null {
-    const { maxDepth = 10 } = options
-    
-    // Two search frontiers
-    const forwardVisited = new Map<string, { path: string[], edges: GraphEdge[], weight: number }>()
-    const backwardVisited = new Map<string, { path: string[], edges: GraphEdge[], weight: number }>()
-    
-    forwardVisited.set(start, { path: [start], edges: [], weight: 0 })
-    backwardVisited.set(end, { path: [end], edges: [], weight: 0 })
-    
-    const forwardQueue = [start]
-    const backwardQueue = [end]
-    
-    let depth = 0
-    
-    while (
-      (forwardQueue.length > 0 || backwardQueue.length > 0) && 
-      depth < maxDepth
-    ) {
-      // Expand forward frontier
-      const forwardNext: string[] = []
-      for (const current of forwardQueue) {
-        const currentData = forwardVisited.get(current)!
-        const neighbors = this.adjacencyList.get(current)
-        
-        if (neighbors) {
-          for (const [neighbor, edges] of neighbors) {
-            if (forwardVisited.has(neighbor)) continue
-            
-            // Select edge with lowest weight for optimal path
-            const bestEdge = edges.reduce((best, edge) => 
-              edge.weight < best.weight ? edge : best, edges[0])
-            forwardVisited.set(neighbor, {
-              path: [...currentData.path, neighbor],
-              edges: [...currentData.edges, bestEdge],
-              weight: currentData.weight + bestEdge.weight
-            })
-            
-            // Check if we met the backward search
-            if (backwardVisited.has(neighbor)) {
-              const forward = forwardVisited.get(neighbor)!
-              const backward = backwardVisited.get(neighbor)!
-              
-              // Combine paths
-              const fullPath = [
-                ...forward.path,
-                ...backward.path.slice(1).reverse()
-              ]
-              
-              // Reverse backward edges and combine
-              const backwardEdgesReversed = backward.edges
-                .map(e => ({
-                  ...e,
-                  source: e.target,
-                  target: e.source
-                }))
-                .reverse()
-              
-              return {
-                nodes: fullPath,
-                edges: [...forward.edges, ...backwardEdgesReversed],
-                totalWeight: forward.weight + backward.weight,
-                length: fullPath.length - 1
-              }
-            }
-            
-            forwardNext.push(neighbor)
-          }
-        }
-      }
-      
-      // Expand backward frontier
-      const backwardNext: string[] = []
-      for (const current of backwardQueue) {
-        const currentData = backwardVisited.get(current)!
-        
-        // For backward search, we need to look at incoming edges
-        for (const [nodeId, neighbors] of this.adjacencyList) {
-          const edges = neighbors.get(current)
-          if (!edges) continue
-          
-          if (backwardVisited.has(nodeId)) continue
-          
-          // Select edge with lowest weight for optimal path
-          const bestEdge = edges.reduce((best, edge) => 
-            edge.weight < best.weight ? edge : best, edges[0])
-          backwardVisited.set(nodeId, {
-            path: [...currentData.path, nodeId],
-            edges: [...currentData.edges, bestEdge],
-            weight: currentData.weight + bestEdge.weight
-          })
-          
-          // Check if we met the forward search
-          if (forwardVisited.has(nodeId)) {
-            const forward = forwardVisited.get(nodeId)!
-            const backward = backwardVisited.get(nodeId)!
-            
-            // Combine paths
-            const fullPath = [
-              ...forward.path,
-              ...backward.path.slice(1).reverse()
-            ]
-            
-            // Reverse backward edges and combine
-            const backwardEdgesReversed = backward.edges
-              .map(e => ({
-                ...e,
-                source: e.target,
-                target: e.source
-              }))
-              .reverse()
-            
-            return {
-              nodes: fullPath,
-              edges: [...forward.edges, ...backwardEdgesReversed],
-              totalWeight: forward.weight + backward.weight,
-              length: fullPath.length - 1
-            }
-          }
-          
-          backwardNext.push(nodeId)
-        }
-      }
-      
-      forwardQueue.splice(0, forwardQueue.length, ...forwardNext)
-      backwardQueue.splice(0, backwardQueue.length, ...backwardNext)
-      depth++
-    }
-    
-    return null
-  }
-  
-  /**
-   * Multi-hop traversal (e.g., friends of friends)
-   * Returns all nodes within N hops
-   */
-  public multiHopTraversal(
-    start: string,
-    hops: number,
-    options: PathfindingOptions = {}
-  ): Map<string, { distance: number, paths: Path[] }> {
-    const { relationshipTypes, nodeFilter, edgeFilter } = options
-    
-    const results = new Map<string, { distance: number, paths: Path[] }>()
-    const visited = new Set<string>()
-    const queue: Array<{ node: string, distance: number, path: string[], edges: GraphEdge[] }> = [
-      { node: start, distance: 0, path: [start], edges: [] }
-    ]
-    
-    while (queue.length > 0) {
-      const { node, distance, path, edges } = queue.shift()!
-      
-      if (distance > hops) continue
-      
-      // Record this node
-      if (!results.has(node)) {
-        results.set(node, { distance, paths: [] })
-      }
-      results.get(node)!.paths.push({
-        nodes: path,
-        edges,
-        totalWeight: edges.reduce((sum, e) => sum + e.weight, 0),
-        length: path.length - 1
-      })
-      
-      if (distance === hops) continue
-      
-      // Explore neighbors
-      const neighbors = this.adjacencyList.get(node)
-      if (neighbors) {
-        for (const [neighbor, edgeList] of neighbors) {
-          // Apply node filter
-          if (nodeFilter) {
-            const neighborNode = this.nodes.get(neighbor)
-            if (neighborNode && !nodeFilter(neighborNode)) continue
-          }
-          
-          for (const edge of edgeList) {
-            // Apply filters
-            if (relationshipTypes && !relationshipTypes.includes(edge.type)) continue
-            if (edgeFilter && !edgeFilter(edge)) continue
-            
-            queue.push({
-              node: neighbor,
-              distance: distance + 1,
-              path: [...path, neighbor],
-              edges: [...edges, edge]
-            })
-          }
-        }
-      }
-    }
-    
-    return results
-  }
-  
-  /**
-   * Find connected components using DFS
-   */
-  public connectedComponents(): Array<Set<string>> {
-    const visited = new Set<string>()
-    const components: Array<Set<string>> = []
-    
-    const dfs = (node: string, component: Set<string>): void => {
-      visited.add(node)
-      component.add(node)
-      
-      const neighbors = this.adjacencyList.get(node)
-      if (neighbors) {
-        for (const neighbor of neighbors.keys()) {
-          if (!visited.has(neighbor)) {
-            dfs(neighbor, component)
-          }
-        }
-      }
-    }
-    
-    for (const node of this.adjacencyList.keys()) {
-      if (!visited.has(node)) {
-        const component = new Set<string>()
-        dfs(node, component)
-        components.push(component)
-      }
-    }
-    
-    return components
-  }
-  
-  /**
-   * Calculate PageRank for all nodes
-   * Useful for ranking importance in the graph
-   */
-  public pageRank(iterations: number = 100, damping: number = 0.85): Map<string, number> {
-    const nodes = Array.from(this.adjacencyList.keys())
-    const n = nodes.length
-    
-    if (n === 0) return new Map()
-    
-    // Initialize ranks
-    const ranks = new Map<string, number>()
-    for (const node of nodes) {
-      ranks.set(node, 1 / n)
-    }
-    
-    // Calculate outgoing edge counts
-    const outDegree = new Map<string, number>()
-    for (const [node, neighbors] of this.adjacencyList) {
-      let count = 0
-      for (const edges of neighbors.values()) {
-        count += edges.length
-      }
-      outDegree.set(node, count)
-    }
-    
-    // Iterate PageRank algorithm
-    for (let i = 0; i < iterations; i++) {
-      const newRanks = new Map<string, number>()
-      
-      for (const node of nodes) {
-        let rank = (1 - damping) / n
-        
-        // Sum contributions from incoming edges
-        for (const [source, neighbors] of this.adjacencyList) {
-          if (neighbors.has(node)) {
-            const sourceRank = ranks.get(source) ?? 0
-            const sourceOutDegree = outDegree.get(source) ?? 1
-            rank += damping * (sourceRank / sourceOutDegree)
-          }
-        }
-        
-        newRanks.set(node, rank)
-      }
-      
-      // Update ranks
-      for (const [node, rank] of newRanks) {
-        ranks.set(node, rank)
-      }
-    }
-    
-    return ranks
-  }
-  
-  /**
-   * Clear the graph
-   */
-  public clear(): void {
-    this.adjacencyList.clear()
-    this.nodes.clear()
-  }
-}

src/hnsw/connectionsCodec.ts

@@ -1,7 +1,7 @@
 /**
  * @module hnsw/connectionsCodec
  * @description Bridge between brainy's UUID-keyed HNSW connection sets and
- * cortex's int-keyed delta-varint encode/decode (the `graph:compression`
+ * cor's int-keyed delta-varint encode/decode (the `graph:compression`
  * provider). Translates UUIDs to stable int slots via the post-2.4.0 #1
  * `EntityIdMapper`, batches all of a node's per-level connection lists into
  * a single compact buffer, and reverses the path on load.
@@ -130,7 +130,7 @@ export class ConnectionsCodec {
  * @description Build the binary-blob storage key for a node's compressed
  * connections. Suffix-free; the adapter appends its own. Keys live under
  * `_hnsw_conn/` so they don't collide with the existing `_column_index/`
- * blobs and so a cortex-side reader knows exactly where to look.
+ * blobs and so a cor-side reader knows exactly where to look.
  */
 export function compressedConnectionsKey(nodeId: string): string {
   return `_hnsw_conn/${nodeId}`

src/hnsw/hnswIndex.ts

@@ -14,7 +14,7 @@ import { euclideanDistance, calculateDistancesBatch } from '../utils/index.js'
 import type { BaseStorage } from '../storage/baseStorage.js'
 import { getGlobalCache, UnifiedCache } from '../utils/unifiedCache.js'
 import { prodLog } from '../utils/logger.js'
-import type { VectorIndexProvider } from '../plugin.js'
+import type { VectorIndexProvider, OpaqueIdSet, AtGenerationVectors } from '../plugin.js'
 import { ConnectionsCodec, compressedConnectionsKey } from './connectionsCodec.js'
 
 // Default HNSW parameters
@@ -25,6 +25,29 @@ const DEFAULT_CONFIG: HNSWConfig = {
   ml: 16 // Max level
 }
 
+/**
+ * @description Thrown by {@link JsHnswVectorIndex.flush} when one or more dirty
+ * nodes (or the system record) could not be persisted. The failed nodes remain
+ * in the dirty set for the next flush; this error tells the caller the flush did
+ * NOT achieve durability instead of a node-count that lies. Mandate: loud
+ * errors, never quiet losses.
+ */
+export class HnswFlushError extends Error {
+  constructor(
+    public readonly failedNodeCount: number,
+    public readonly systemFailed: boolean,
+    public override readonly cause?: Error
+  ) {
+    super(
+      `HNSW flush did not achieve durability: ${failedNodeCount} node(s) failed to ` +
+        `persist${systemFailed ? ' and the system record (entryPoint/maxLevel) failed' : ''}. ` +
+        `Failed nodes remain dirty for retry.` +
+        (cause ? ` First error: ${cause.message}` : '')
+    )
+    this.name = 'HnswFlushError'
+  }
+}
+
 /**
  * Implements {@link VectorIndexProvider}: the vector-index surface Brainy calls
  * on whatever the `'vector'` factory returns (its own `JsHnswVectorIndex`, or a native
@@ -32,6 +55,14 @@ const DEFAULT_CONFIG: HNSWConfig = {
  */
 export class JsHnswVectorIndex implements VectorIndexProvider {
   private nouns: Map<string, HNSWNoun> = new Map()
+  /**
+   * Reverse adjacency: `target id → (level → set of node ids that link TO target)`.
+   * Lets `removeItem` find a node's in-neighbors in O(in-degree) instead of scanning
+   * the whole corpus (which made bulk delete O(N²)). Lazily built on first need and
+   * maintained incrementally on link/prune/remove; `null` means "stale — rebuild on
+   * next use" (set by every bulk path: cold-load restore + `clear()`).
+   */
+  private incoming: Map<string, Map<number, Set<string>>> | null = null
   private entryPointId: string | null = null
   private maxLevel = 0
   // Track high-level nodes for O(1) entry point selection
@@ -142,41 +173,69 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
     const startTime = Date.now()
     const nodeCount = this.dirtyNodes.size
 
-    // Batch persist all dirty nodes concurrently
+    // Batch persist all dirty nodes concurrently. A node whose connections FAIL
+    // to persist must stay dirty (retried on the next flush) — clearing it would
+    // silently drop the write forever. Track failures; only successfully-
+    // persisted (or deleted) nodes leave the dirty set, so nodes added to it
+    // during this flush are preserved.
+    const failedNodes = new Set<string>()
+    let firstError: Error | null = null
+
     if (this.dirtyNodes.size > 0) {
       const batchSize = 50 // Reasonable batch size for cloud storage
       const nodeIds = Array.from(this.dirtyNodes)
 
       for (let i = 0; i < nodeIds.length; i += batchSize) {
         const batch = nodeIds.slice(i, i + batchSize)
-        const promises = batch.map(nodeId => {
+        const promises = batch.map(async nodeId => {
           const noun = this.nouns.get(nodeId)
-          if (!noun) return Promise.resolve() // Node was deleted
-
-          return this.persistNodeConnections(nodeId, noun).catch(error => {
-            console.error(`[HNSW flush] Failed to persist node ${nodeId}:`, error)
-          })
+          if (!noun) return // Node was deleted — drop it from the dirty set.
+          try {
+            await this.persistNodeConnections(nodeId, noun)
+          } catch (error) {
+            failedNodes.add(nodeId)
+            if (firstError === null) firstError = error as Error
+            prodLog.error(`[HNSW flush] Failed to persist node ${nodeId}: ${(error as Error).message}`)
+          }
         })
 
-        await Promise.allSettled(promises)
+        await Promise.all(promises)
       }
 
-      this.dirtyNodes.clear()
+      // Remove only nodes that were persisted (or deleted mid-flush); keep the
+      // failed ones dirty for the next attempt.
+      for (const nodeId of nodeIds) {
+        if (!failedNodes.has(nodeId)) this.dirtyNodes.delete(nodeId)
+      }
     }
 
-    // Persist system data if dirty
+    // Persist system data if dirty — keep it dirty on failure so the next flush
+    // retries rather than losing the entry-point/maxLevel update.
+    let systemFailed = false
     if (this.dirtySystem) {
-      await this.storage.saveHNSWSystem({
-        entryPointId: this.entryPointId,
-        maxLevel: this.maxLevel
-      }).catch(error => {
-        console.error('[HNSW flush] Failed to persist system data:', error)
-      })
-
-      this.dirtySystem = false
+      try {
+        await this.storage.saveHNSWSystem({
+          entryPointId: this.entryPointId,
+          maxLevel: this.maxLevel
+        })
+        this.dirtySystem = false
+      } catch (error) {
+        systemFailed = true
+        if (firstError === null) firstError = error as Error
+        prodLog.error(`[HNSW flush] Failed to persist system data: ${(error as Error).message}`)
+      }
     }
 
     const duration = Date.now() - startTime
+
+    // Loud failure: if ANY node or the system record could not be persisted the
+    // flush did not achieve durability. Throw so callers (close(), explicit
+    // flush(), the flush-request watcher) see the failure instead of a success
+    // count that lies. The failed nodes/system stay dirty for retry.
+    if (failedNodes.size > 0 || systemFailed) {
+      throw new HnswFlushError(failedNodes.size, systemFailed, firstError ?? undefined)
+    }
+
     if (nodeCount > 0) {
       prodLog.info(`[HNSW] Flushed ${nodeCount} dirty nodes in ${duration}ms`)
     }
@@ -245,6 +304,9 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
     hnswData: { level: number; connections: Record<string, string[]> },
     noun: HNSWNoun
   ): Promise<void> {
+    // Bulk load sets connections directly (bypassing the link path that maintains
+    // the reverse index) — mark it stale so the next removeItem rebuilds it.
+    this.incoming = null
     const storageWithBlob = this.storage as unknown as {
       loadBinaryBlob?: (key: string) => Promise<Buffer | null>
     }
@@ -385,13 +447,14 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
       this.maxLevel = nounLevel
       this.nouns.set(id, noun)
 
-      // Persist system data for first noun (previously skipped)
+      // Persist system data for first noun (previously skipped). Surface a
+      // persist failure loudly — the entry point is the root of the whole index;
+      // silently dropping it while addItem() returns the id would strand every
+      // future search on a rootless index. Mandate: never a quiet loss.
       if (this.storage && this.persistMode === 'immediate') {
         await this.storage.saveHNSWSystem({
           entryPointId: this.entryPointId,
           maxLevel: this.maxLevel
-        }).catch(error => {
-          console.error('Failed to persist initial HNSW system data:', error)
         })
       } else if (this.persistMode === 'deferred') {
         this.dirtySystem = true
@@ -489,12 +552,14 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
 
 
         noun.connections.get(level)!.add(neighborId)
+        this.addIncoming(neighborId, level, id) // forward edge id → neighborId
 
         // Add reverse connection
         if (!neighbor.connections.has(level)) {
           neighbor.connections.set(level, new Set<string>())
         }
         neighbor.connections.get(level)!.add(id)
+        this.addIncoming(id, level, neighborId) // forward edge neighborId → id
 
         // Ensure neighbor doesn't have too many connections
         if (neighbor.connections.get(level)!.size > this.config.M) {
@@ -656,23 +721,61 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
    * @param queryVector Query vector
    * @param k Number of results to return
    * @param filter Optional filter function
-   * @param options Additional search options (`candidateIds` to restrict the
-   *   search to a pre-filtered set; `rerank` is provider-only, ignored here)
+   * @param options Additional search options (`candidateIds` / `allowedIds` to
+   *   restrict the search to a pre-filtered set; `rerank` is provider-only, ignored here)
    */
   public async search(
     queryVector: Vector,
     k: number = 10,
     filter?: (id: string) => Promise<boolean>,
-    options?: { rerank?: { multiplier: number }; candidateIds?: string[] }
+    options?: {
+      rerank?: { multiplier: number }
+      candidateIds?: string[]
+      allowedIds?: OpaqueIdSet | ReadonlySet<string>
+      // 8.0 #35 at-gen seam: the JS index has no retained per-generation segments
+      // (it only ever holds "now"), so it ignores `generation` — exactly the
+      // refuse/fall-back posture the contract requires. Brainy never routes a
+      // historical read here (its defer-gate checks `isGenerationVisible` first),
+      // so an ignored `generation` cannot silently return now-vectors-as-at-gen.
+      generation?: bigint
+      // 8.0 #35 part-3: at-gen candidate vectors for the native exact-rerank. The JS
+      // index serves "now" only and never receives this (gated off upstream); ignored.
+      atGenerationVectors?: AtGenerationVectors
+    }
   ): Promise<Array<[string, number]>> {
     if (this.nouns.size === 0) {
       return []
     }
 
-    // Metadata-first: convert candidateIds to filter function if no explicit filter
-    if (!filter && options?.candidateIds && options.candidateIds.length > 0) {
-      const candidateSet = new Set(options.candidateIds)
-      filter = async (id: string) => candidateSet.has(id)
+    // Metadata-first candidate restriction. Build a filter from the pre-resolved
+    // universe(s) when the caller didn't pass an explicit one. Both `candidateIds`
+    // (legacy string[]) and the 8.0 `allowedIds` predicate-pushdown param restrict
+    // the SAME way here — applied INSIDE the beam walk (searchLayer keeps every
+    // neighbor as a traversal candidate but only COLLECTS allowed ids), which is
+    // what recovers the filtered recall that post-filtering loses. An `allowedIds`
+    // that arrives as an OpaqueIdSet (a native/cor roaring Buffer) is opaque to the
+    // JS index — it cannot decode it — so it is ignored here; only the native
+    // provider consumes that form. A `ReadonlySet<string>` is honored. When both
+    // `candidateIds` and a Set `allowedIds` are present they AND together.
+    if (!filter) {
+      const universes: ReadonlySet<string>[] = []
+      if (options?.candidateIds && options.candidateIds.length > 0) {
+        universes.push(new Set(options.candidateIds))
+      }
+      const allowed = options?.allowedIds
+      if (
+        allowed &&
+        !(allowed instanceof Uint8Array) &&
+        typeof (allowed as ReadonlySet<string>).has === 'function'
+      ) {
+        universes.push(allowed as ReadonlySet<string>)
+      }
+      if (universes.length === 1) {
+        const universe = universes[0]
+        filter = async (id: string) => universe.has(id)
+      } else if (universes.length > 1) {
+        filter = async (id: string) => universes.every((u) => u.has(id))
+      }
     }
 
     // Check if query vector is defined
@@ -803,6 +906,44 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
     return [...nearestNouns].slice(0, k)
   }
 
+  /**
+   * Build (or return the cached) reverse-adjacency index. O(N + E) the first time
+   * after a bulk load/clear; O(1) thereafter while it stays live.
+   */
+  private ensureIncoming(): Map<string, Map<number, Set<string>>> {
+    if (this.incoming !== null) return this.incoming
+    const inc = new Map<string, Map<number, Set<string>>>()
+    for (const [nodeId, node] of this.nouns) {
+      for (const [level, targets] of node.connections) {
+        for (const target of targets) {
+          let byLevel = inc.get(target)
+          if (!byLevel) { byLevel = new Map(); inc.set(target, byLevel) }
+          let set = byLevel.get(level)
+          if (!set) { set = new Set(); byLevel.set(level, set) }
+          set.add(nodeId)
+        }
+      }
+    }
+    this.incoming = inc
+    return inc
+  }
+
+  /** Record a new forward edge `source → target` at `level` (no-op while the index is stale). */
+  private addIncoming(target: string, level: number, source: string): void {
+    if (this.incoming === null) return
+    let byLevel = this.incoming.get(target)
+    if (!byLevel) { byLevel = new Map(); this.incoming.set(target, byLevel) }
+    let set = byLevel.get(level)
+    if (!set) { set = new Set(); byLevel.set(level, set) }
+    set.add(source)
+  }
+
+  /** Record that the forward edge `source → target` at `level` is gone. */
+  private removeIncoming(target: string, level: number, source: string): void {
+    if (this.incoming === null) return
+    this.incoming.get(target)?.get(level)?.delete(source)
+  }
+
   /**
    * Remove an item from the index
    */
@@ -814,41 +955,38 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
 
     const noun = this.nouns.get(id)!
 
-    // Remove connections to this noun from all neighbors
-    for (const [level, connections] of noun.connections.entries()) {
-      for (const neighborId of connections) {
-
-        const neighbor = this.nouns.get(neighborId)
-        if (!neighbor) {
-          // Skip neighbors that don't exist (expected during rapid additions/deletions)
-          continue
-        }
-        if (neighbor.connections.has(level)) {
-          neighbor.connections.get(level)!.delete(id)
-
-          // Prune connections after removing this noun to ensure consistency
-          await this.pruneConnections(neighbor, level)
+    // Reverse-adjacency lets us touch ONLY the nodes that actually reference `id`
+    // (its in-neighbors) rather than scanning the whole corpus — turning a delete
+    // from O(N) into O(in-degree) and a bulk delete from O(N²) into O(N·degree).
+    // Snapshot each referrer set because pruneConnections mutates the index.
+    const incoming = this.ensureIncoming()
+    const referrers = incoming.get(id)
+    if (referrers) {
+      for (const [level, refSet] of referrers) {
+        for (const refId of Array.from(refSet)) {
+          const ref = this.nouns.get(refId)
+          if (ref && ref.connections.has(level)) {
+            // Drop the forward edge ref → id, then re-prune ref so the graph stays
+            // navigable. (id's own reverse entry is dropped wholesale below, so we
+            // intentionally do not maintain incoming[id] inside this loop.)
+            ref.connections.get(level)!.delete(id)
+            await this.pruneConnections(ref, level)
+          }
         }
       }
     }
 
-    // Also check all other nouns for references to this noun and remove them
-    for (const [nounId, otherNoun] of this.nouns.entries()) {
-      if (nounId === id) continue // Skip the noun being removed
-
-
-      for (const [level, connections] of otherNoun.connections.entries()) {
-        if (connections.has(id)) {
-          connections.delete(id)
-
-          // Prune connections after removing this reference
-          await this.pruneConnections(otherNoun, level)
-        }
+    // id's OUTGOING edges disappear with it → drop id from each out-neighbor's
+    // reverse set so no stale referrer survives.
+    for (const [level, targets] of noun.connections) {
+      for (const target of targets) {
+        this.removeIncoming(target, level, id)
       }
     }
 
-    // Remove the noun
+    // Remove the noun + its reverse-index entry.
     this.nouns.delete(id)
+    this.incoming?.delete(id)
 
     // If we removed the entry point, find a new one
     if (this.entryPointId === id) {
@@ -928,6 +1066,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
    */
   public clear(): void {
     this.nouns.clear()
+    this.incoming = null // reverse index no longer reflects the (now empty) graph
     this.entryPointId = null
     this.maxLevel = 0
   }
@@ -1724,7 +1863,11 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
 
     // Only proceed if we have valid neighbors
     if (distances.size === 0) {
-      // If no valid neighbors, clear connections at this level
+      // If no valid neighbors, clear connections at this level. Every current
+      // connection is dropped — unlink each from the reverse index.
+      if (this.incoming !== null) {
+        for (const dropped of connections) this.removeIncoming(dropped, level, noun.id)
+      }
       noun.connections.set(level, new Set())
       return
     }
@@ -1736,8 +1879,16 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
       this.config.M
     )
 
-    // Update connections with only valid neighbors
-    noun.connections.set(level, new Set(selectedNeighbors.keys()))
+    // Update connections with only valid neighbors. Pruning only ever drops
+    // edges (the selected set is a subset of the current connections), so unlink
+    // each dropped neighbor from the reverse index.
+    const kept = new Set(selectedNeighbors.keys())
+    if (this.incoming !== null) {
+      for (const prev of connections) {
+        if (!kept.has(prev)) this.removeIncoming(prev, level, noun.id)
+      }
+    }
+    noun.connections.set(level, kept)
   }
 
   /**

src/import/BackgroundDeduplicator.ts

@@ -41,6 +41,14 @@ export interface DeduplicationStats {
  * - Import-scoped deduplication (no cross-contamination)
  * - 3-tier strategy (ID → Name → Similarity)
  * - Uses existing indexes (EntityIdMapper, MetadataIndexManager, TypeAware HNSW)
+ *
+ * Lifecycle: ONE instance per brain, owned by Brainy (getBackgroundDeduplicator)
+ * so the debounce genuinely spans imports and brain.close() cancels pending
+ * work via cancelPending() — this pass merge-DELETES duplicate entities, so it
+ * must never fire against a closed brain. The enableDeduplication gate lives
+ * at the scheduling call site (ImportCoordinator); scheduleDedup itself is
+ * unconditional. The timer is unref'd — a pending pass never holds the
+ * process open.
  */
 export class BackgroundDeduplicator {
   private brain: Brainy
@@ -67,12 +75,15 @@ export class BackgroundDeduplicator {
       clearTimeout(this.debounceTimer)
     }
 
-    // Schedule for 5 minutes from now
+    // Schedule for 5 minutes from now. unref'd: a pending dedup pass must
+    // never hold the process open (exit-hang class) — if the process exits
+    // first, the pass simply never runs; imports are already durable.
     this.debounceTimer = setTimeout(() => {
       this.runBatchDedup().catch(error => {
         prodLog.error('[BackgroundDedup] Batch dedup failed:', error)
       })
     }, 5 * 60 * 1000)
+    this.debounceTimer.unref?.()
   }
 
   /**

src/import/EntityDeduplicator.ts

@@ -1,354 +0,0 @@
-/**
- * Entity Deduplicator
- *
- * Finds and merges duplicate entities across imports using:
- * - Embedding-based similarity matching
- * - Type-aware comparison
- * - Confidence-weighted merging
- * - Provenance tracking
- *
- * NO MOCKS - Production-ready implementation
- */
-
-import { Brainy } from '../brainy.js'
-import { NounType } from '../types/graphTypes.js'
-
-export interface EntityCandidate {
-  id?: string
-  name: string
-  type: NounType
-  description: string
-  confidence: number
-  metadata: Record<string, any>
-}
-
-export interface DuplicateMatch {
-  existingId: string
-  existingName: string
-  similarity: number
-  shouldMerge: boolean
-  reason: string
-}
-
-export interface EntityDeduplicationOptions {
-  /** Similarity threshold for considering entities as duplicates (0-1) */
-  similarityThreshold?: number
-
-  /** Only match entities of the same type */
-  strictTypeMatching?: boolean
-
-  /** Enable fuzzy name matching */
-  enableFuzzyMatching?: boolean
-
-  /** Minimum confidence to consider for merging */
-  minConfidence?: number
-}
-
-export interface MergeResult {
-  mergedEntityId: string
-  wasMerged: boolean
-  mergedWith?: string
-  confidence: number
-  provenance: string[]
-}
-
-/**
- * EntityDeduplicator - Prevents duplicate entities across imports
- */
-export class EntityDeduplicator {
-  private brain: Brainy
-
-  constructor(brain: Brainy) {
-    this.brain = brain
-  }
-
-  /**
-   * Find duplicate entities in the knowledge graph
-   */
-  async findDuplicates(
-    candidate: EntityCandidate,
-    options: EntityDeduplicationOptions = {}
-  ): Promise<DuplicateMatch | null> {
-    const opts = {
-      similarityThreshold: options.similarityThreshold || 0.85,
-      strictTypeMatching: options.strictTypeMatching !== false,
-      enableFuzzyMatching: options.enableFuzzyMatching !== false,
-      minConfidence: options.minConfidence || 0.6
-    }
-
-    // Skip low-confidence candidates
-    if (candidate.confidence < opts.minConfidence) {
-      return null
-    }
-
-    // Search for similar entities by name and description
-    const searchText = `${candidate.name} ${candidate.description}`.trim()
-
-    try {
-      const results = await this.brain.find({
-        query: searchText,
-        limit: 5,
-        where: opts.strictTypeMatching ? { type: candidate.type } : undefined
-      })
-
-      // Check each result for potential duplicates
-      for (const result of results) {
-        const similarity = result.score || 0
-        const existingName = result.entity.metadata?.name || result.id
-        const existingType = result.entity.metadata?.type || result.entity.metadata?.nounType || result.entity.type
-
-        // Skip if below similarity threshold
-        if (similarity < opts.similarityThreshold) {
-          continue
-        }
-
-        // Type matching check
-        if (opts.strictTypeMatching && existingType !== candidate.type) {
-          continue
-        }
-
-        // Exact name match (case-insensitive)
-        if (this.normalizeString(candidate.name) === this.normalizeString(existingName)) {
-          return {
-            existingId: result.id,
-            existingName,
-            similarity: 1.0,
-            shouldMerge: true,
-            reason: 'Exact name match'
-          }
-        }
-
-        // High similarity match
-        if (similarity >= opts.similarityThreshold) {
-          // Additional validation for fuzzy matching
-          if (opts.enableFuzzyMatching && this.areSimilarNames(candidate.name, existingName)) {
-            return {
-              existingId: result.id,
-              existingName,
-              similarity,
-              shouldMerge: true,
-              reason: `High similarity (${(similarity * 100).toFixed(1)}%)`
-            }
-          }
-        }
-      }
-    } catch (error) {
-      // If search fails, assume no duplicates
-      return null
-    }
-
-    return null
-  }
-
-  /**
-   * Merge entity data with existing entity
-   */
-  async mergeEntity(
-    existingId: string,
-    candidate: EntityCandidate,
-    importSource: string
-  ): Promise<MergeResult> {
-    try {
-      // Get existing entity
-      const existing = await this.brain.get(existingId)
-      if (!existing) {
-        throw new Error(`Entity ${existingId} not found`)
-      }
-
-      // Update confidence (weighted average) — `confidence` is a reserved
-      // top-level field, so it travels via the dedicated update() param, not
-      // the metadata bag.
-      const mergedConfidence = this.mergeConfidence(
-        existing.confidence ?? 0.5,
-        candidate.confidence
-      )
-
-      // Merge metadata (custom fields only — reserved fields are top-level)
-      const mergedMetadata = {
-        ...existing.metadata,
-        // Track provenance
-        imports: [
-          ...(existing.metadata?.imports || []),
-          importSource
-        ],
-        // Merge VFS paths
-        vfsPaths: [
-          ...(existing.metadata?.vfsPaths || [existing.metadata?.vfsPath]).filter(Boolean),
-          candidate.metadata?.vfsPath
-        ].filter(Boolean),
-        // Merge other metadata
-        ...this.mergeMetadataFields(existing.metadata, candidate.metadata),
-        // Track last update
-        lastUpdated: Date.now(),
-        mergeCount: (existing.metadata?.mergeCount || 0) + 1
-      }
-
-      // Update entity
-      await this.brain.update({
-        id: existingId,
-        confidence: mergedConfidence,
-        metadata: mergedMetadata,
-        merge: true
-      })
-
-      return {
-        mergedEntityId: existingId,
-        wasMerged: true,
-        mergedWith: existing.metadata?.name || existingId,
-        confidence: mergedConfidence,
-        provenance: mergedMetadata.imports
-      }
-    } catch (error) {
-      throw new Error(`Failed to merge entity: ${error instanceof Error ? error.message : String(error)}`)
-    }
-  }
-
-  /**
-   * Create or merge entity with deduplication
-   */
-  async createOrMerge(
-    candidate: EntityCandidate,
-    importSource: string,
-    options: EntityDeduplicationOptions = {}
-  ): Promise<MergeResult> {
-    // Check for duplicates
-    const duplicate = await this.findDuplicates(candidate, options)
-
-    if (duplicate && duplicate.shouldMerge) {
-      // Merge with existing entity
-      return await this.mergeEntity(duplicate.existingId, candidate, importSource)
-    }
-
-    // No duplicate found, create new entity. Preserve any subtype the candidate
-    // carried (set by the extractor or upstream importer), else fall back to
-    // `'imported'` so enforcement doesn't fire (added 7.30.1).
-    // `confidence` is a reserved top-level field (dedicated add() param);
-    // creation time is system-managed — neither belongs in the metadata bag.
-    const entityId = await this.brain.add({
-      data: candidate.description || candidate.name,
-      type: candidate.type,
-      subtype:
-        (candidate as EntityCandidate & { subtype?: string }).subtype ??
-        'imported',
-      confidence: candidate.confidence,
-      metadata: {
-        ...candidate.metadata,
-        name: candidate.name,
-        imports: [importSource],
-        vfsPaths: [candidate.metadata?.vfsPath].filter(Boolean),
-        mergeCount: 0
-      }
-    })
-
-    // Update candidate with new ID
-    candidate.id = entityId
-
-    return {
-      mergedEntityId: entityId,
-      wasMerged: false,
-      confidence: candidate.confidence,
-      provenance: [importSource]
-    }
-  }
-
-  /**
-   * Normalize string for comparison
-   */
-  private normalizeString(str: string): string {
-    return str
-      .toLowerCase()
-      .trim()
-      .replace(/[^a-z0-9]/g, '')
-  }
-
-  /**
-   * Check if two names are similar (fuzzy matching)
-   */
-  private areSimilarNames(name1: string, name2: string): boolean {
-    const n1 = this.normalizeString(name1)
-    const n2 = this.normalizeString(name2)
-
-    // Exact match
-    if (n1 === n2) return true
-
-    // Length difference check
-    const lengthDiff = Math.abs(n1.length - n2.length)
-    if (lengthDiff > 3) return false
-
-    // Levenshtein distance
-    const distance = this.levenshteinDistance(n1, n2)
-    const maxLength = Math.max(n1.length, n2.length)
-    const similarity = 1 - (distance / maxLength)
-
-    return similarity >= 0.85
-  }
-
-  /**
-   * Calculate Levenshtein distance between two strings
-   */
-  private levenshteinDistance(str1: string, str2: string): number {
-    const m = str1.length
-    const n = str2.length
-    const dp: number[][] = Array(m + 1).fill(null).map(() => Array(n + 1).fill(0))
-
-    for (let i = 0; i <= m; i++) dp[i][0] = i
-    for (let j = 0; j <= n; j++) dp[0][j] = j
-
-    for (let i = 1; i <= m; i++) {
-      for (let j = 1; j <= n; j++) {
-        if (str1[i - 1] === str2[j - 1]) {
-          dp[i][j] = dp[i - 1][j - 1]
-        } else {
-          dp[i][j] = Math.min(
-            dp[i - 1][j] + 1,    // deletion
-            dp[i][j - 1] + 1,    // insertion
-            dp[i - 1][j - 1] + 1 // substitution
-          )
-        }
-      }
-    }
-
-    return dp[m][n]
-  }
-
-  /**
-   * Merge confidence scores (weighted average favoring higher confidence)
-   */
-  private mergeConfidence(existing: number, incoming: number): number {
-    // Weight higher confidence more heavily
-    const weights = existing > incoming ? [0.6, 0.4] : [0.4, 0.6]
-    return existing * weights[0] + incoming * weights[1]
-  }
-
-  /**
-   * Merge metadata fields intelligently
-   */
-  private mergeMetadataFields(
-    existing: Record<string, any>,
-    incoming: Record<string, any>
-  ): Record<string, any> {
-    const merged: Record<string, any> = {}
-
-    // Merge arrays
-    const arrayFields = ['concepts', 'tags', 'categories']
-    for (const field of arrayFields) {
-      if (existing[field] || incoming[field]) {
-        const combined = [
-          ...(existing[field] || []),
-          ...(incoming[field] || [])
-        ]
-        // Deduplicate
-        merged[field] = [...new Set(combined)]
-      }
-    }
-
-    // Prefer longer descriptions
-    if (existing.description || incoming.description) {
-      merged.description = (existing.description || '').length > (incoming.description || '').length
-        ? existing.description
-        : incoming.description
-    }
-
-    return merged
-  }
-}

src/import/ImportCoordinator.ts

@@ -13,7 +13,6 @@
 import { Brainy } from '../brainy.js'
 import { FormatDetector, SupportedFormat } from './FormatDetector.js'
 import { ImportHistory, type ImportHistoryEntry } from './ImportHistory.js'
-import { BackgroundDeduplicator } from './BackgroundDeduplicator.js'
 import { SmartExcelImporter } from '../importers/SmartExcelImporter.js'
 import { SmartPDFImporter } from '../importers/SmartPDFImporter.js'
 import { SmartCSVImporter } from '../importers/SmartCSVImporter.js'
@@ -112,7 +111,12 @@ export interface ValidImportOptions {
   /** Confidence threshold for entities */
   confidenceThreshold?: number
 
-  /** Enable entity deduplication across imports */
+  /**
+   * Enable entity deduplication (default: true). Gates BOTH passes: the
+   * inline merge during import AND the debounced background pass that runs
+   * ~5 minutes after the last import (which merge-DELETES duplicate entities).
+   * Set false for deployments that must never auto-remove records.
+   */
   enableDeduplication?: boolean
 
   /** Similarity threshold for deduplication (0-1) */
@@ -286,7 +290,6 @@ export class ImportCoordinator {
   private brain: Brainy
   private detector: FormatDetector
   private history: ImportHistory
-  private backgroundDedup: BackgroundDeduplicator
   private excelImporter: SmartExcelImporter
   private pdfImporter: SmartPDFImporter
   private csvImporter: SmartCSVImporter
@@ -300,7 +303,6 @@ export class ImportCoordinator {
     this.brain = brain
     this.detector = new FormatDetector()
     this.history = new ImportHistory(brain)
-    this.backgroundDedup = new BackgroundDeduplicator(brain)
     this.excelImporter = new SmartExcelImporter(brain)
     this.pdfImporter = new SmartPDFImporter(brain)
     this.csvImporter = new SmartCSVImporter(brain)
@@ -1459,9 +1461,16 @@ export class ImportCoordinator {
       }
     }
 
-    // Schedule background deduplication (debounced 5 minutes)
-    if (trackingContext && trackingContext.importId) {
-      this.backgroundDedup.scheduleDedup(trackingContext.importId)
+    // Schedule background deduplication (debounced 5 minutes, brain-owned so
+    // close() can cancel it). Honors the same enableDeduplication gate as the
+    // inline pass — false means NO dedup, inline or background.
+    if (
+      trackingContext &&
+      trackingContext.importId &&
+      options.enableDeduplication !== false
+    ) {
+      const backgroundDedup = await this.brain.getBackgroundDeduplicator()
+      backgroundDedup.scheduleDedup(trackingContext.importId)
     }
 
     return {

src/import/InstancePool.ts

@@ -1,269 +0,0 @@
-/**
- * InstancePool - Shared instance management for memory efficiency
- *
- * Production-grade instance pooling to prevent memory leaks during imports.
- * Critical for scaling to billions of entities.
- *
- * Problem: Creating new NLP/Extractor instances in loops → memory leak
- * Solution: Reuse shared instances across entire import session
- *
- * Memory savings:
- * - Without pooling: 100K rows × 50MB per instance = 5TB RAM (OOM!)
- * - With pooling: 50MB total (shared across all rows)
- */
-
-import { Brainy } from '../brainy.js'
-import { NaturalLanguageProcessor } from '../neural/naturalLanguageProcessor.js'
-import { NeuralEntityExtractor } from '../neural/entityExtractor.js'
-
-/**
- * InstancePool - Manages shared instances for memory efficiency
- *
- * Lifecycle:
- * 1. Create pool at import start
- * 2. Reuse instances across all rows
- * 3. Pool is garbage collected when import completes
- *
- * Thread safety: Not thread-safe (single import session per pool)
- */
-export class InstancePool {
-  private brain: Brainy
-
-  // Shared instances (created lazily)
-  private nlpInstance: NaturalLanguageProcessor | null = null
-  private extractorInstance: NeuralEntityExtractor | null = null
-
-  // Initialization state
-  private nlpInitialized = false
-  private initializationPromise: Promise<void> | null = null
-
-  // Statistics
-  private stats = {
-    nlpReuses: 0,
-    extractorReuses: 0,
-    creationTime: 0
-  }
-
-  constructor(brain: Brainy) {
-    this.brain = brain
-  }
-
-  /**
-   * Get shared NaturalLanguageProcessor instance
-   *
-   * Lazy initialization - created on first access
-   * All subsequent calls return same instance
-   *
-   * @returns Shared NLP instance
-   */
-  async getNLP(): Promise<NaturalLanguageProcessor> {
-    if (!this.nlpInstance) {
-      const startTime = Date.now()
-      this.nlpInstance = new NaturalLanguageProcessor(this.brain)
-      this.stats.creationTime += Date.now() - startTime
-    }
-
-    // Ensure initialized before returning
-    if (!this.nlpInitialized) {
-      await this.ensureNLPInitialized()
-    }
-
-    this.stats.nlpReuses++
-    return this.nlpInstance
-  }
-
-  /**
-   * Get shared NeuralEntityExtractor instance
-   *
-   * Lazy initialization - created on first access
-   * All subsequent calls return same instance
-   *
-   * @returns Shared extractor instance
-   */
-  getExtractor(): NeuralEntityExtractor {
-    if (!this.extractorInstance) {
-      const startTime = Date.now()
-      this.extractorInstance = new NeuralEntityExtractor(this.brain)
-      this.stats.creationTime += Date.now() - startTime
-    }
-
-    this.stats.extractorReuses++
-    return this.extractorInstance
-  }
-
-  /**
-   * Get shared NLP instance (synchronous, may return uninitialized)
-   *
-   * Use when you need NLP synchronously and will handle initialization yourself.
-   * Prefer getNLP() for async code.
-   *
-   * @returns Shared NLP instance (possibly uninitialized)
-   */
-  getNLPSync(): NaturalLanguageProcessor {
-    if (!this.nlpInstance) {
-      this.nlpInstance = new NaturalLanguageProcessor(this.brain)
-    }
-
-    this.stats.nlpReuses++
-    return this.nlpInstance
-  }
-
-  /**
-   * Initialize all instances upfront
-   *
-   * Call at start of import to avoid lazy initialization overhead
-   * during processing. Improves predictability and first-row performance.
-   *
-   * @returns Promise that resolves when all instances are ready
-   */
-  async init(): Promise<void> {
-    // Prevent duplicate initialization
-    if (this.initializationPromise) {
-      return this.initializationPromise
-    }
-
-    this.initializationPromise = this.initializeInternal()
-    return this.initializationPromise
-  }
-
-  /**
-   * Internal initialization implementation
-   */
-  private async initializeInternal(): Promise<void> {
-    const startTime = Date.now()
-
-    // Create instances
-    if (!this.nlpInstance) {
-      this.nlpInstance = new NaturalLanguageProcessor(this.brain)
-    }
-    if (!this.extractorInstance) {
-      this.extractorInstance = new NeuralEntityExtractor(this.brain)
-    }
-
-    // Initialize NLP (loads pattern library)
-    await this.ensureNLPInitialized()
-
-    this.stats.creationTime = Date.now() - startTime
-  }
-
-  /**
-   * Ensure NLP is initialized (loads 220 patterns)
-   *
-   * Handles concurrent initialization requests safely
-   */
-  private async ensureNLPInitialized(): Promise<void> {
-    if (this.nlpInitialized) {
-      return
-    }
-
-    if (!this.nlpInstance) {
-      throw new Error('NLP instance not created yet')
-    }
-
-    await this.nlpInstance.init()
-    this.nlpInitialized = true
-  }
-
-  /**
-   * Check if instances are initialized
-   *
-   * @returns True if NLP is initialized and ready to use
-   */
-  isInitialized(): boolean {
-    return this.nlpInitialized && this.nlpInstance !== null
-  }
-
-  /**
-   * Get pool statistics
-   *
-   * Useful for performance monitoring and memory leak detection
-   *
-   * @returns Statistics about instance reuse
-   */
-  getStats() {
-    return {
-      ...this.stats,
-      nlpCreated: this.nlpInstance !== null,
-      extractorCreated: this.extractorInstance !== null,
-      initialized: this.isInitialized(),
-      // Memory savings estimate
-      memorySaved: this.calculateMemorySaved()
-    }
-  }
-
-  /**
-   * Calculate estimated memory saved by pooling
-   *
-   * Assumes ~50MB per NLP instance, ~10MB per extractor instance
-   *
-   * @returns Estimated memory saved in bytes
-   */
-  private calculateMemorySaved(): number {
-    const nlpSize = 50 * 1024 * 1024 // 50MB per instance
-    const extractorSize = 10 * 1024 * 1024 // 10MB per instance
-
-    // Without pooling: size × reuses
-    // With pooling: size × 1
-    // Saved: size × (reuses - 1)
-
-    const nlpSaved = nlpSize * Math.max(0, this.stats.nlpReuses - 1)
-    const extractorSaved = extractorSize * Math.max(0, this.stats.extractorReuses - 1)
-
-    return nlpSaved + extractorSaved
-  }
-
-  /**
-   * Reset statistics (useful for testing)
-   */
-  resetStats(): void {
-    this.stats = {
-      nlpReuses: 0,
-      extractorReuses: 0,
-      creationTime: 0
-    }
-  }
-
-  /**
-   * Get string representation (for debugging)
-   */
-  toString(): string {
-    const stats = this.getStats()
-    return `InstancePool(nlp=${stats.nlpCreated}, extractor=${stats.extractorCreated}, initialized=${stats.initialized}, nlpReuses=${stats.nlpReuses}, extractorReuses=${stats.extractorReuses})`
-  }
-
-  /**
-   * Cleanup method (for explicit resource management)
-   *
-   * Note: Usually not needed - pool is garbage collected when import completes.
-   * Use only if you need explicit cleanup for some reason.
-   */
-  cleanup(): void {
-    // Clear references to allow garbage collection
-    this.nlpInstance = null
-    this.extractorInstance = null
-    this.nlpInitialized = false
-    this.initializationPromise = null
-  }
-}
-
-/**
- * Create a new instance pool
- *
- * Convenience factory function
- *
- * @param brain Brainy instance
- * @param autoInit Whether to initialize instances immediately
- * @returns Instance pool
- */
-export async function createInstancePool(
-  brain: Brainy,
-  autoInit = true
-): Promise<InstancePool> {
-  const pool = new InstancePool(brain)
-
-  if (autoInit) {
-    await pool.init()
-  }
-
-  return pool
-}

src/import/index.ts

@@ -1,38 +0,0 @@
-/**
- * Unified Import System
- *
- * Single entry point for importing any file format into Brainy with:
- * - Auto-detection of formats
- * - Dual storage (VFS + Knowledge Graph)
- * - Shared entities across imports (deduplication)
- * - Simple, powerful API
- */
-
-export { ImportCoordinator } from './ImportCoordinator.js'
-export { FormatDetector, SupportedFormat, DetectionResult } from './FormatDetector.js'
-export { EntityDeduplicator } from './EntityDeduplicator.js'
-export { BackgroundDeduplicator } from './BackgroundDeduplicator.js'
-export { ImportHistory } from './ImportHistory.js'
-
-export type {
-  ImportSource,
-  ImportOptions,
-  ImportProgress,
-  ImportResult
-} from './ImportCoordinator.js'
-
-export type {
-  EntityCandidate,
-  DuplicateMatch,
-  EntityDeduplicationOptions,
-  MergeResult
-} from './EntityDeduplicator.js'
-
-export type {
-  DeduplicationStats
-} from './BackgroundDeduplicator.js'
-
-export type {
-  ImportHistoryEntry,
-  RollbackResult
-} from './ImportHistory.js'

src/importers/SmartImportOrchestrator.ts

@@ -1,789 +0,0 @@
-/**
- * Smart Import Orchestrator
- *
- * Coordinates the entire smart import pipeline:
- * 1. Extract entities/relationships using SmartExcelImporter
- * 2. Create entities and relationships in Brainy
- * 3. Organize into VFS structure using VFSStructureGenerator
- *
- * NO MOCKS - Production-ready implementation
- */
-
-import { Brainy } from '../brainy.js'
-import { VirtualFileSystem } from '../vfs/VirtualFileSystem.js'
-import { NounType, VerbType } from '../types/graphTypes.js'
-import { splitNounMetadataRecord } from '../types/reservedFields.js'
-import { SmartExcelImporter, SmartExcelOptions, SmartExcelResult } from './SmartExcelImporter.js'
-import { SmartPDFImporter, SmartPDFOptions, SmartPDFResult } from './SmartPDFImporter.js'
-import { SmartCSVImporter, SmartCSVOptions, SmartCSVResult } from './SmartCSVImporter.js'
-import { SmartJSONImporter, SmartJSONOptions, SmartJSONResult } from './SmartJSONImporter.js'
-import { SmartMarkdownImporter, SmartMarkdownOptions, SmartMarkdownResult } from './SmartMarkdownImporter.js'
-import { VFSStructureGenerator, VFSStructureOptions } from './VFSStructureGenerator.js'
-
-export interface SmartImportOptions extends SmartExcelOptions {
-  /** Create VFS structure */
-  createVFSStructure?: boolean
-
-  /** VFS root path */
-  vfsRootPath?: string
-
-  /** VFS grouping strategy */
-  vfsGroupBy?: 'type' | 'sheet' | 'flat' | 'custom'
-
-  /** Create entities in Brainy */
-  createEntities?: boolean
-
-  /** Create relationships in Brainy */
-  createRelationships?: boolean
-
-  /** Source filename */
-  filename?: string
-
-  /**
-   * Default subtype tag for entities + relationships this importer creates when
-   * the extractor doesn't set one. See `ValidImportOptions.defaultSubtype` —
-   * same semantics, same precedence (extractor > caller default > `'imported'`).
-   * Added 7.30.1.
-   */
-  defaultSubtype?: string
-}
-
-export interface SmartImportProgress {
-  phase: 'parsing' | 'extracting' | 'creating' | 'relationships' | 'organizing' | 'complete'
-  message: string
-  processed: number
-  total: number
-  entities: number
-  relationships: number
-}
-
-export interface SmartImportResult {
-  success: boolean
-
-  /** Extraction results */
-  extraction: SmartExcelResult
-
-  /** Created entity IDs */
-  entityIds: string[]
-
-  /** Created relationship IDs */
-  relationshipIds: string[]
-
-  /** VFS structure created */
-  vfsStructure?: {
-    rootPath: string
-    directories: string[]
-    files: number
-  }
-
-  /** Overall statistics */
-  stats: {
-    rowsProcessed: number
-    entitiesCreated: number
-    relationshipsCreated: number
-    filesCreated: number
-    totalTime: number
-  }
-
-  /** Any errors encountered */
-  errors: string[]
-}
-
-/**
- * SmartImportOrchestrator - Main entry point for smart imports
- */
-export class SmartImportOrchestrator {
-  private brain: Brainy
-  private excelImporter: SmartExcelImporter
-  private pdfImporter: SmartPDFImporter
-  private csvImporter: SmartCSVImporter
-  private jsonImporter: SmartJSONImporter
-  private markdownImporter: SmartMarkdownImporter
-  private vfsGenerator: VFSStructureGenerator
-
-  constructor(brain: Brainy) {
-    this.brain = brain
-    this.excelImporter = new SmartExcelImporter(brain)
-    this.pdfImporter = new SmartPDFImporter(brain)
-    this.csvImporter = new SmartCSVImporter(brain)
-    this.jsonImporter = new SmartJSONImporter(brain)
-    this.markdownImporter = new SmartMarkdownImporter(brain)
-    this.vfsGenerator = new VFSStructureGenerator(brain)
-  }
-
-  /**
-   * Initialize the orchestrator
-   */
-  async init(): Promise<void> {
-    await this.excelImporter.init()
-    await this.pdfImporter.init()
-    await this.csvImporter.init()
-    await this.jsonImporter.init()
-    await this.markdownImporter.init()
-    await this.vfsGenerator.init()
-  }
-
-  /**
-   * Import Excel file with full pipeline
-   */
-  async importExcel(
-    buffer: Buffer,
-    options: SmartImportOptions = {},
-    onProgress?: (progress: SmartImportProgress) => void
-  ): Promise<SmartImportResult> {
-    const startTime = Date.now()
-    const result: SmartImportResult = {
-      success: false,
-      // Typed boundary: populated in the extraction phase below. If extraction
-      // throws, the error path returns with this still null (pre-existing
-      // contract — consumers check `success`/`errors` before reading it).
-      extraction: null as unknown as SmartExcelResult,
-      entityIds: [],
-      relationshipIds: [],
-      stats: {
-        rowsProcessed: 0,
-        entitiesCreated: 0,
-        relationshipsCreated: 0,
-        filesCreated: 0,
-        totalTime: 0
-      },
-      errors: []
-    }
-
-    try {
-      // Phase 1: Extract entities and relationships
-      onProgress?.({
-        phase: 'extracting',
-        message: 'Extracting entities and relationships...',
-        processed: 0,
-        total: 0,
-        entities: 0,
-        relationships: 0
-      })
-
-      result.extraction = await this.excelImporter.extract(buffer, {
-        ...options,
-        onProgress: (stats) => {
-          onProgress?.({
-            phase: 'extracting',
-            message: `Processing row ${stats.processed}/${stats.total}...`,
-            processed: stats.processed,
-            total: stats.total,
-            entities: stats.entities,
-            relationships: stats.relationships
-          })
-        }
-      })
-
-      result.stats.rowsProcessed = result.extraction.rowsProcessed
-
-      // Phase 2: Create entities in Brainy
-      if (options.createEntities !== false) {
-        onProgress?.({
-          phase: 'creating',
-          message: 'Creating entities in knowledge graph...',
-          processed: 0,
-          total: result.extraction.rows.length,
-          entities: 0,
-          relationships: 0
-        })
-
-        for (let i = 0; i < result.extraction.rows.length; i++) {
-          const extracted = result.extraction.rows[i]
-
-          try {
-            // Create main entity. Subtype precedence: extractor-set → caller default
-            // → Brainy default `'imported'` (added 7.30.1).
-            const entityId = await this.brain.add({
-              data: extracted.entity.description,
-              type: extracted.entity.type,
-              subtype:
-                (extracted.entity as typeof extracted.entity & { subtype?: string })
-                  .subtype ?? options.defaultSubtype ?? 'imported',
-              confidence: extracted.entity.confidence, // reserved field — dedicated param, not metadata
-              metadata: {
-                // Strip reserved keys an extractor may have smuggled into the bag
-                // (8.0 reservedFieldPolicy defaults to 'throw').
-                ...splitNounMetadataRecord(extracted.entity.metadata).custom,
-                name: extracted.entity.name,
-                importedFrom: 'smart-import'
-              }
-            })
-
-            result.entityIds.push(entityId)
-            result.stats.entitiesCreated++
-
-            // Update entity ID in extraction result
-            extracted.entity.id = entityId
-
-            onProgress?.({
-              phase: 'creating',
-              message: `Created entity: ${extracted.entity.name}`,
-              processed: i + 1,
-              total: result.extraction.rows.length,
-              entities: result.entityIds.length,
-              relationships: result.relationshipIds.length
-            })
-          } catch (error: any) {
-            result.errors.push(`Failed to create entity ${extracted.entity.name}: ${error.message}`)
-          }
-        }
-      }
-
-      // Phase 3: Create relationships
-      if (options.createRelationships !== false && options.createEntities !== false) {
-        onProgress?.({
-          phase: 'creating',
-          message: 'Preparing relationships...',
-          processed: 0,
-          total: result.extraction.rows.length,
-          entities: result.entityIds.length,
-          relationships: 0
-        })
-
-        // Build entity name -> ID map
-        const entityMap = new Map<string, string>()
-        for (const extracted of result.extraction.rows) {
-          entityMap.set(extracted.entity.name.toLowerCase(), extracted.entity.id)
-        }
-
-        // Collect all relationship parameters
-        const relationshipParams: Array<{from: string; to: string; type: VerbType; subtype?: string; confidence?: number; metadata?: any}> = []
-
-        for (const extracted of result.extraction.rows) {
-          for (const rel of extracted.relationships) {
-            try {
-              // Find target entity ID
-              let toEntityId: string | undefined
-
-              // Try to find by name in our extracted entities
-              for (const otherExtracted of result.extraction.rows) {
-                if (rel.to.toLowerCase().includes(otherExtracted.entity.name.toLowerCase()) ||
-                    otherExtracted.entity.name.toLowerCase().includes(rel.to.toLowerCase())) {
-                  toEntityId = otherExtracted.entity.id
-                  break
-                }
-              }
-
-              // If not found, create a placeholder entity. `import-placeholder` marks
-              // these as synthetic targets so consumers can distinguish them from real
-              // imports and downstream dedup can consolidate (added 7.30.1).
-              if (!toEntityId) {
-                toEntityId = await this.brain.add({
-                  data: rel.to,
-                  type: NounType.Thing,
-                  subtype: 'import-placeholder',
-                  metadata: {
-                    name: rel.to,
-                    placeholder: true,
-                    extractedFrom: extracted.entity.name
-                  }
-                })
-                result.entityIds.push(toEntityId)
-              }
-
-              // Collect relationship parameter. Subtype precedence: extractor-set rel
-              // subtype → caller default → Brainy default `'imported'` (added 7.30.1).
-              relationshipParams.push({
-                from: extracted.entity.id,
-                to: toEntityId,
-                type: rel.type,
-                subtype:
-                  (rel as typeof rel & { subtype?: string }).subtype ??
-                  options.defaultSubtype ?? 'imported',
-                confidence: rel.confidence, // reserved field — dedicated param, not metadata
-                metadata: {
-                  evidence: rel.evidence
-                }
-              })
-            } catch (error: any) {
-              result.errors.push(`Failed to prepare relationship: ${error.message}`)
-            }
-          }
-        }
-
-        // Batch create all relationships with progress
-        if (relationshipParams.length > 0) {
-          onProgress?.({
-            phase: 'relationships',
-            message: 'Building relationships...',
-            processed: 0,
-            total: relationshipParams.length,
-            entities: result.entityIds.length,
-            relationships: 0
-          })
-
-          try {
-            const relationshipIds = await this.brain.relateMany({
-              items: relationshipParams,
-              parallel: true,
-              chunkSize: 100,
-              continueOnError: true,
-              onProgress: (done, total) => {
-                onProgress?.({
-                  phase: 'relationships',
-                  message: `Building relationships: ${done}/${total}`,
-                  processed: done,
-                  total: total,
-                  entities: result.entityIds.length,
-                  relationships: done
-                })
-              }
-            })
-
-            result.relationshipIds = relationshipIds
-            result.stats.relationshipsCreated = relationshipIds.length
-          } catch (error: any) {
-            result.errors.push(`Failed to create relationships: ${error.message}`)
-          }
-        }
-      }
-
-      // Phase 4: Create VFS structure
-      if (options.createVFSStructure !== false) {
-        onProgress?.({
-          phase: 'organizing',
-          message: 'Organizing into file structure...',
-          processed: 0,
-          total: result.extraction.rows.length,
-          entities: result.entityIds.length,
-          relationships: result.relationshipIds.length
-        })
-
-        const vfsOptions: VFSStructureOptions = {
-          rootPath: options.vfsRootPath || '/imports/' + (options.filename || 'import'),
-          groupBy: options.vfsGroupBy || 'type',
-          preserveSource: true,
-          sourceBuffer: buffer,
-          sourceFilename: options.filename || 'import.xlsx',
-          createRelationshipFile: true,
-          createMetadataFile: true
-        }
-
-        const vfsResult = await this.vfsGenerator.generate(result.extraction, vfsOptions)
-
-        result.vfsStructure = {
-          rootPath: vfsResult.rootPath,
-          directories: vfsResult.directories,
-          files: vfsResult.files.length
-        }
-
-        result.stats.filesCreated = vfsResult.files.length
-      }
-
-      // Complete
-      result.success = result.errors.length === 0
-      result.stats.totalTime = Date.now() - startTime
-
-      onProgress?.({
-        phase: 'complete',
-        message: `Import complete: ${result.stats.entitiesCreated} entities, ${result.stats.relationshipsCreated} relationships`,
-        processed: result.extraction.rows.length,
-        total: result.extraction.rows.length,
-        entities: result.stats.entitiesCreated,
-        relationships: result.stats.relationshipsCreated
-      })
-
-    } catch (error: any) {
-      result.errors.push(`Import failed: ${error.message}`)
-      result.success = false
-    }
-
-    return result
-  }
-
-  /**
-   * Import PDF file with full pipeline
-   */
-  async importPDF(
-    buffer: Buffer,
-    options: SmartImportOptions & SmartPDFOptions = {},
-    onProgress?: (progress: SmartImportProgress) => void
-  ): Promise<SmartImportResult> {
-    const startTime = Date.now()
-    const result: SmartImportResult = {
-      success: false,
-      // Typed boundary: populated after extraction (see importExcel).
-      extraction: null as unknown as SmartExcelResult,
-      entityIds: [],
-      relationshipIds: [],
-      stats: {
-        rowsProcessed: 0,
-        entitiesCreated: 0,
-        relationshipsCreated: 0,
-        filesCreated: 0,
-        totalTime: 0
-      },
-      errors: []
-    }
-
-    try {
-      // Phase 1: Extract from PDF
-      onProgress?.({ phase: 'extracting', message: 'Extracting from PDF...', processed: 0, total: 0, entities: 0, relationships: 0 })
-
-      const pdfResult = await this.pdfImporter.extract(buffer, options)
-
-      // Convert PDF result to Excel-like format for processing
-      result.extraction = this.convertPDFToExcelFormat(pdfResult)
-      result.stats.rowsProcessed = pdfResult.sectionsProcessed
-
-      // Phase 2 & 3: Create entities and relationships
-      await this.createEntitiesAndRelationships(result, options, onProgress)
-
-      // Phase 4: Create VFS structure
-      if (options.createVFSStructure !== false) {
-        const vfsOptions: VFSStructureOptions = {
-          rootPath: options.vfsRootPath || '/imports/' + (options.filename || 'import'),
-          groupBy: options.vfsGroupBy || 'type',
-          preserveSource: true,
-          sourceBuffer: buffer,
-          sourceFilename: options.filename || 'import.pdf',
-          createRelationshipFile: true,
-          createMetadataFile: true
-        }
-        const vfsResult = await this.vfsGenerator.generate(result.extraction, vfsOptions)
-        result.vfsStructure = { rootPath: vfsResult.rootPath, directories: vfsResult.directories, files: vfsResult.files.length }
-        result.stats.filesCreated = vfsResult.files.length
-      }
-
-      result.success = result.errors.length === 0
-      result.stats.totalTime = Date.now() - startTime
-      onProgress?.({ phase: 'complete', message: `Import complete: ${result.stats.entitiesCreated} entities, ${result.stats.relationshipsCreated} relationships`, processed: result.stats.rowsProcessed, total: result.stats.rowsProcessed, entities: result.stats.entitiesCreated, relationships: result.stats.relationshipsCreated })
-
-    } catch (error: any) {
-      result.errors.push(`PDF import failed: ${error.message}`)
-      result.success = false
-    }
-
-    return result
-  }
-
-  /**
-   * Import CSV file with full pipeline
-   */
-  async importCSV(
-    buffer: Buffer,
-    options: SmartImportOptions & SmartCSVOptions = {},
-    onProgress?: (progress: SmartImportProgress) => void
-  ): Promise<SmartImportResult> {
-    // CSV is very similar to Excel, can reuse importExcel logic
-    return this.importExcel(buffer, options, onProgress)
-  }
-
-  /**
-   * Import JSON data with full pipeline
-   */
-  async importJSON(
-    data: any,
-    options: SmartImportOptions & SmartJSONOptions = {},
-    onProgress?: (progress: SmartImportProgress) => void
-  ): Promise<SmartImportResult> {
-    const startTime = Date.now()
-    const result: SmartImportResult = {
-      success: false,
-      // Typed boundary: populated after extraction (see importExcel).
-      extraction: null as unknown as SmartExcelResult,
-      entityIds: [],
-      relationshipIds: [],
-      stats: {
-        rowsProcessed: 0,
-        entitiesCreated: 0,
-        relationshipsCreated: 0,
-        filesCreated: 0,
-        totalTime: 0
-      },
-      errors: []
-    }
-
-    try {
-      onProgress?.({ phase: 'extracting', message: 'Extracting from JSON...', processed: 0, total: 0, entities: 0, relationships: 0 })
-
-      const jsonResult = await this.jsonImporter.extract(data, options)
-
-      result.extraction = this.convertJSONToExcelFormat(jsonResult)
-      result.stats.rowsProcessed = jsonResult.nodesProcessed
-
-      await this.createEntitiesAndRelationships(result, options, onProgress)
-
-      if (options.createVFSStructure !== false) {
-        const sourceBuffer = Buffer.from(typeof data === 'string' ? data : JSON.stringify(data, null, 2))
-        const vfsOptions: VFSStructureOptions = {
-          rootPath: options.vfsRootPath || '/imports/' + (options.filename || 'import'),
-          groupBy: options.vfsGroupBy || 'type',
-          preserveSource: true,
-          sourceBuffer,
-          sourceFilename: options.filename || 'import.json',
-          createRelationshipFile: true,
-          createMetadataFile: true
-        }
-        const vfsResult = await this.vfsGenerator.generate(result.extraction, vfsOptions)
-        result.vfsStructure = { rootPath: vfsResult.rootPath, directories: vfsResult.directories, files: vfsResult.files.length }
-        result.stats.filesCreated = vfsResult.files.length
-      }
-
-      result.success = result.errors.length === 0
-      result.stats.totalTime = Date.now() - startTime
-      onProgress?.({ phase: 'complete', message: `Import complete: ${result.stats.entitiesCreated} entities, ${result.stats.relationshipsCreated} relationships`, processed: result.stats.rowsProcessed, total: result.stats.rowsProcessed, entities: result.stats.entitiesCreated, relationships: result.stats.relationshipsCreated })
-
-    } catch (error: any) {
-      result.errors.push(`JSON import failed: ${error.message}`)
-      result.success = false
-    }
-
-    return result
-  }
-
-  /**
-   * Import Markdown content with full pipeline
-   */
-  async importMarkdown(
-    markdown: string,
-    options: SmartImportOptions & SmartMarkdownOptions = {},
-    onProgress?: (progress: SmartImportProgress) => void
-  ): Promise<SmartImportResult> {
-    const startTime = Date.now()
-    const result: SmartImportResult = {
-      success: false,
-      // Typed boundary: populated after extraction (see importExcel).
-      extraction: null as unknown as SmartExcelResult,
-      entityIds: [],
-      relationshipIds: [],
-      stats: {
-        rowsProcessed: 0,
-        entitiesCreated: 0,
-        relationshipsCreated: 0,
-        filesCreated: 0,
-        totalTime: 0
-      },
-      errors: []
-    }
-
-    try {
-      onProgress?.({ phase: 'extracting', message: 'Extracting from Markdown...', processed: 0, total: 0, entities: 0, relationships: 0 })
-
-      const mdResult = await this.markdownImporter.extract(markdown, options)
-
-      result.extraction = this.convertMarkdownToExcelFormat(mdResult)
-      result.stats.rowsProcessed = mdResult.sectionsProcessed
-
-      await this.createEntitiesAndRelationships(result, options, onProgress)
-
-      if (options.createVFSStructure !== false) {
-        const sourceBuffer = Buffer.from(markdown, 'utf-8')
-        const vfsOptions: VFSStructureOptions = {
-          rootPath: options.vfsRootPath || '/imports/' + (options.filename || 'import'),
-          groupBy: options.vfsGroupBy || 'type',
-          preserveSource: true,
-          sourceBuffer,
-          sourceFilename: options.filename || 'import.md',
-          createRelationshipFile: true,
-          createMetadataFile: true
-        }
-        const vfsResult = await this.vfsGenerator.generate(result.extraction, vfsOptions)
-        result.vfsStructure = { rootPath: vfsResult.rootPath, directories: vfsResult.directories, files: vfsResult.files.length }
-        result.stats.filesCreated = vfsResult.files.length
-      }
-
-      result.success = result.errors.length === 0
-      result.stats.totalTime = Date.now() - startTime
-      onProgress?.({ phase: 'complete', message: `Import complete: ${result.stats.entitiesCreated} entities, ${result.stats.relationshipsCreated} relationships`, processed: result.stats.rowsProcessed, total: result.stats.rowsProcessed, entities: result.stats.entitiesCreated, relationships: result.stats.relationshipsCreated })
-
-    } catch (error: any) {
-      result.errors.push(`Markdown import failed: ${error.message}`)
-      result.success = false
-    }
-
-    return result
-  }
-
-  /**
-   * Helper: Create entities and relationships from extraction result
-   */
-  private async createEntitiesAndRelationships(
-    result: SmartImportResult,
-    options: SmartImportOptions,
-    onProgress?: (progress: SmartImportProgress) => void
-  ): Promise<void> {
-    if (options.createEntities !== false) {
-      onProgress?.({ phase: 'creating', message: 'Creating entities in knowledge graph...', processed: 0, total: result.extraction.rows.length, entities: 0, relationships: 0 })
-
-      for (let i = 0; i < result.extraction.rows.length; i++) {
-        const extracted = result.extraction.rows[i]
-        try {
-          // Subtype precedence: extractor → caller default → `'imported'` (7.30.1).
-          const entityId = await this.brain.add({
-            data: extracted.entity.description,
-            type: extracted.entity.type,
-            subtype:
-              (extracted.entity as typeof extracted.entity & { subtype?: string })
-                .subtype ?? options.defaultSubtype ?? 'imported',
-            confidence: extracted.entity.confidence, // reserved field — dedicated param, not metadata
-            metadata: { ...splitNounMetadataRecord(extracted.entity.metadata).custom, name: extracted.entity.name, importedFrom: 'smart-import' }
-          })
-          result.entityIds.push(entityId)
-          result.stats.entitiesCreated++
-          extracted.entity.id = entityId
-        } catch (error: any) {
-          result.errors.push(`Failed to create entity ${extracted.entity.name}: ${error.message}`)
-        }
-      }
-    }
-
-    if (options.createRelationships !== false && options.createEntities !== false) {
-      onProgress?.({ phase: 'creating', message: 'Preparing relationships...', processed: 0, total: result.extraction.rows.length, entities: result.entityIds.length, relationships: 0 })
-
-      // Collect all relationship parameters
-      const relationshipParams: Array<{from: string; to: string; type: VerbType; subtype?: string; confidence?: number; metadata?: any}> = []
-
-      for (const extracted of result.extraction.rows) {
-        for (const rel of extracted.relationships) {
-          try {
-            let toEntityId: string | undefined
-            for (const otherExtracted of result.extraction.rows) {
-              if (rel.to.toLowerCase().includes(otherExtracted.entity.name.toLowerCase()) || otherExtracted.entity.name.toLowerCase().includes(rel.to.toLowerCase())) {
-                toEntityId = otherExtracted.entity.id
-                break
-              }
-            }
-            if (!toEntityId) {
-              // Subtype `import-placeholder` marks synthetic targets (7.30.1).
-              toEntityId = await this.brain.add({ data: rel.to, type: NounType.Thing, subtype: 'import-placeholder', metadata: { name: rel.to, placeholder: true, extractedFrom: extracted.entity.name } })
-              result.entityIds.push(toEntityId)
-            }
-            // Relationship subtype precedence: extractor → caller default → `'imported'` (7.30.1).
-            // `confidence` is a reserved top-level field — dedicated relate() param, not metadata
-            relationshipParams.push({ from: extracted.entity.id, to: toEntityId, type: rel.type, subtype: (rel as typeof rel & { subtype?: string }).subtype ?? options.defaultSubtype ?? 'imported', confidence: rel.confidence, metadata: { evidence: rel.evidence } })
-          } catch (error: any) {
-            result.errors.push(`Failed to prepare relationship: ${error.message}`)
-          }
-        }
-      }
-
-      // Batch create all relationships with progress
-      if (relationshipParams.length > 0) {
-        onProgress?.({ phase: 'relationships', message: 'Building relationships...', processed: 0, total: relationshipParams.length, entities: result.entityIds.length, relationships: 0 })
-
-        try {
-          const relationshipIds = await this.brain.relateMany({
-            items: relationshipParams,
-            parallel: true,
-            chunkSize: 100,
-            continueOnError: true,
-            onProgress: (done, total) => {
-              onProgress?.({ phase: 'relationships', message: `Building relationships: ${done}/${total}`, processed: done, total: total, entities: result.entityIds.length, relationships: done })
-            }
-          })
-
-          result.relationshipIds = relationshipIds
-          result.stats.relationshipsCreated = relationshipIds.length
-        } catch (error: any) {
-          result.errors.push(`Failed to create relationships: ${error.message}`)
-        }
-      }
-    }
-  }
-
-  /**
-   * Helper: Convert PDF result to Excel-like format
-   */
-  private convertPDFToExcelFormat(pdfResult: SmartPDFResult): Omit<SmartExcelResult, 'rows'> & { rows: any[] } {
-    const rows = pdfResult.sections.flatMap(section =>
-      section.entities.map(entity => ({
-        entity,
-        relatedEntities: [],
-        relationships: section.relationships.filter(r => r.from === entity.id),
-        concepts: section.concepts
-      }))
-    )
-
-    return {
-      rowsProcessed: pdfResult.sectionsProcessed,
-      entitiesExtracted: pdfResult.entitiesExtracted,
-      relationshipsInferred: pdfResult.relationshipsInferred,
-      rows,
-      entityMap: pdfResult.entityMap,
-      processingTime: pdfResult.processingTime,
-      stats: pdfResult.stats
-    }
-  }
-
-  /**
-   * Helper: Convert JSON result to Excel-like format
-   */
-  private convertJSONToExcelFormat(jsonResult: SmartJSONResult): Omit<SmartExcelResult, 'rows'> & { rows: any[] } {
-    const rows = jsonResult.entities.map(entity => ({
-      entity,
-      relatedEntities: [],
-      relationships: jsonResult.relationships.filter(r => r.from === entity.id),
-      concepts: entity.metadata.concepts || []
-    }))
-
-    return {
-      rowsProcessed: jsonResult.nodesProcessed,
-      entitiesExtracted: jsonResult.entitiesExtracted,
-      relationshipsInferred: jsonResult.relationshipsInferred,
-      rows,
-      entityMap: jsonResult.entityMap,
-      processingTime: jsonResult.processingTime,
-      stats: jsonResult.stats
-    }
-  }
-
-  /**
-   * Helper: Convert Markdown result to Excel-like format
-   */
-  private convertMarkdownToExcelFormat(mdResult: SmartMarkdownResult): Omit<SmartExcelResult, 'rows'> & { rows: any[] } {
-    const rows = mdResult.sections.flatMap(section =>
-      section.entities.map(entity => ({
-        entity,
-        relatedEntities: [],
-        relationships: section.relationships.filter(r => r.from === entity.id),
-        concepts: section.concepts
-      }))
-    )
-
-    return {
-      rowsProcessed: mdResult.sectionsProcessed,
-      entitiesExtracted: mdResult.entitiesExtracted,
-      relationshipsInferred: mdResult.relationshipsInferred,
-      rows,
-      entityMap: mdResult.entityMap,
-      processingTime: mdResult.processingTime,
-      stats: mdResult.stats
-    }
-  }
-
-  /**
-   * Get import statistics
-   */
-  async getImportStatistics(vfsRootPath: string): Promise<{
-    entitiesInGraph: number
-    relationshipsInGraph: number
-    filesInVFS: number
-    lastImport?: Date
-  }> {
-    // Read metadata file
-    const vfs = new VirtualFileSystem(this.brain)
-    await vfs.init()
-
-    const metadataPath = `${vfsRootPath}/_metadata.json`
-
-    try {
-      const metadataBuffer = await vfs.readFile(metadataPath)
-      const metadata = JSON.parse(metadataBuffer.toString('utf-8'))
-
-      return {
-        entitiesInGraph: metadata.import.stats.entitiesExtracted,
-        relationshipsInGraph: metadata.import.stats.relationshipsInferred,
-        filesInVFS: metadata.structure.fileCount,
-        lastImport: new Date(metadata.import.timestamp)
-      }
-    } catch (error) {
-      return {
-        entitiesInGraph: 0,
-        relationshipsInGraph: 0,
-        filesInVFS: 0
-      }
-    }
-  }
-}

src/importers/VFSStructureGenerator.ts

@@ -189,8 +189,7 @@ export class VFSStructureGenerator {
       reportProgress('metadata', `Preserved source file: ${options.sourceFilename}`)
     }
 
-    // Note: groups already calculated above for progress tracking
-    // const groups = this.groupEntities(importResult, options)
+    // `groups` was already computed above (for progress tracking) and is reused here.
 
     // Create directories and files for each group
     for (const [groupName, entities] of groups.entries()) {

src/importers/index.ts

@@ -1,69 +0,0 @@
-/**
- * Smart Import System
- *
- * Production-ready entity and relationship extraction from multiple formats:
- * - Excel (.xlsx)
- * - PDF (.pdf)
- * - CSV (.csv)
- * - JSON (.json)
- * - Markdown (.md)
- *
- * Uses brainy's built-in NeuralEntityExtractor and NaturalLanguageProcessor
- *
- * NO MOCKS - Real working implementation
- */
-
-// Excel Importer
-export { SmartExcelImporter } from './SmartExcelImporter.js'
-export type {
-  SmartExcelOptions,
-  ExtractedRow,
-  SmartExcelResult
-} from './SmartExcelImporter.js'
-
-// PDF Importer
-export { SmartPDFImporter } from './SmartPDFImporter.js'
-export type {
-  SmartPDFOptions,
-  ExtractedSection,
-  SmartPDFResult
-} from './SmartPDFImporter.js'
-
-// CSV Importer
-export { SmartCSVImporter } from './SmartCSVImporter.js'
-export type {
-  SmartCSVOptions,
-  SmartCSVResult
-} from './SmartCSVImporter.js'
-
-// JSON Importer
-export { SmartJSONImporter } from './SmartJSONImporter.js'
-export type {
-  SmartJSONOptions,
-  ExtractedJSONEntity,
-  ExtractedJSONRelationship,
-  SmartJSONResult
-} from './SmartJSONImporter.js'
-
-// Markdown Importer
-export { SmartMarkdownImporter } from './SmartMarkdownImporter.js'
-export type {
-  SmartMarkdownOptions,
-  MarkdownSection,
-  SmartMarkdownResult
-} from './SmartMarkdownImporter.js'
-
-// VFS Structure Generator
-export { VFSStructureGenerator } from './VFSStructureGenerator.js'
-export type {
-  VFSStructureOptions,
-  VFSStructureResult
-} from './VFSStructureGenerator.js'
-
-// Orchestrator (Main entry point)
-export { SmartImportOrchestrator } from './SmartImportOrchestrator.js'
-export type {
-  SmartImportOptions,
-  SmartImportProgress,
-  SmartImportResult
-} from './SmartImportOrchestrator.js'

src/index.ts

@@ -6,7 +6,7 @@
  * - Brainy: The unified database with Triple Intelligence
  * - Triple Intelligence: Seamless fusion of vector + graph + field search
  * - Db: Immutable, generation-pinned database values (now/transact/asOf/with)
- * - Plugins: Extensible plugin system (cortex, storage adapters)
+ * - Plugins: Extensible plugin system (cor, storage adapters)
  * - Neural Import: AI-powered entity extraction & smart data import
  */
 
@@ -15,8 +15,29 @@ import { Brainy } from './brainy.js'
 
 export { Brainy }
 
+// The in-process change feed (brain.onChange) — event + listener types.
+export type {
+  BrainyChangeEvent,
+  ChangeEventEntity,
+  ChangeEventRelation,
+  ChangeListener
+} from './events/changeFeed.js'
+
+// Temporal VFS — a file version entry (vfs.history / readFile({ asOf })).
+export type { FileVersion } from './vfs/types.js'
+
 // Export diagnostics result type
 export type { DiagnosticsResult } from './brainy.js'
+export type {
+  GraphAuditReport,
+  GraphAuditDiscrepancy
+} from './graph/graphAudit.js'
+export {
+  checkOsLimits,
+  NOFILE_POOL_FLOOR,
+  MAX_MAP_COUNT_POOL_FLOOR
+} from './utils/osLimits.js'
+export type { OsLimitsReport } from './utils/osLimits.js'
 
 // Export Brainy configuration and types
 export type {
@@ -137,6 +158,11 @@ export { RevisionConflictError } from './transaction/RevisionConflictError.js'
 // transact/with() when a referenced entity or relation does not exist.
 export { EntityNotFoundError, RelationNotFoundError } from './errors/notFound.js'
 
+// Base error + typed migration-lock error — thrown by any data-plane call while a
+// brain runs its one-time 7.x→8.0 upgrade; catch to answer HTTP 503 + Retry-After.
+export { BrainyError, MigrationInProgressError, GraphIndexNotReadyError, MetadataIndexNotReadyError, VectorIndexNotReadyError, ProtectedArtifactError, DerivedArtifactMissingError } from './errors/brainyError.js'
+export type { BrainyErrorType } from './errors/brainyError.js'
+
 // ============= 8.0 Db API — generational MVCC =============
 // Immutable database values: brain.now() / brain.transact() / brain.asOf() /
 // db.with() / db.persist() / Brainy.load(). See src/db/ for the record layer.
@@ -158,8 +184,11 @@ export type {
 export {
   GenerationConflictError,
   SpeculativeOverlayError,
-  GenerationCompactedError
+  GenerationCompactedError,
+  StoreInconsistentError,
+  PendingFlushDurabilityError
 } from './db/errors.js'
+export type { UnreconciledRecord } from './db/errors.js'
 export type {
   TxOperation,
   TxAddOperation,
@@ -172,14 +201,30 @@ export type {
   TxLogEntry,
   CompactHistoryOptions,
   CompactHistoryResult,
+  HistoryStats,
   ChangedIds,
   DiffResult,
   HistoryVersion,
   EntityHistory
 } from './db/types.js'
+// The generation fact log — sequential after-image scan surface
+// (brain.scanFacts / brain.factSegmentPaths) for index heals and replays.
+export type {
+  CommitFact,
+  FactOp,
+  FactScanBatch,
+  SCANFACTS_FIRST_BATCH_MS,
+  FactScanHandle
+} from './db/factLog.js'
+// The generalized family stamp — which source generation a projection
+// reflects + the surface that verifies it whole; one verifier, both member
+// modes (enumerated byte-exact / rollup invariants).
+export { readFamilyStamp, verifyFamilyStamp, ENTITY_TREE_STAMP_PATH } from './db/familyStamp.js'
+export type { FamilyStamp, StampMembers, StampVerdict } from './db/familyStamp.js'
 // Optional provider capability for generation-aware native indexes
 export { isVersionedIndexProvider } from './plugin.js'
 export type { VersionedIndexProvider } from './plugin.js'
+export type { ProviderInvariantReport, InvariantResult, InvariantHeal } from './plugin.js'
 // Optional native graph-acceleration engine (cor 3.0) — the published provider
 // contract + its columnar wire types. Brainy feature-detects an implementation
 // and falls back to its pure-TS adjacency when absent.
@@ -194,13 +239,13 @@ export type {
   GraphCursorOptions,
   GraphCursorChunk,
   GraphScores,
-  GraphComponents,
+  GraphCommunities,
   GraphPath,
-  PageRankOptions,
-  ComponentsOptions,
-  ShortestPathOptions,
-  NeighborhoodSampleOptions,
-  TopByDegreeOptions
+  RankOptions,
+  CommunitiesOptions,
+  PathOptions,
+  SampleOptions,
+  MostConnectedOptions
 } from './plugin.js'
 
 // Export embedding functionality
@@ -281,7 +326,8 @@ import type {
   HNSWNoun,
   HNSWVerb,
   HNSWConfig,
-  StorageAdapter
+  StorageAdapter,
+  DerivedFamilyDeclaration
 } from './coreTypes.js'
 
 // Export vector index implementation (the JS HNSW path)
@@ -299,7 +345,8 @@ export type {
   HNSWNoun,
   HNSWVerb,
   HNSWConfig,
-  StorageAdapter
+  StorageAdapter,
+  DerivedFamilyDeclaration
 }
 
 // Export graph types

src/indexes/columnStore/ColumnSegmentCursor.ts

@@ -7,7 +7,7 @@
  * Multiple cursors can read the same segment concurrently (e.g. during k-way merge).
  *
  * For the TS baseline, segments are loaded fully into memory and cached via
- * UnifiedCache. The Cortex Rust implementation mmaps the segment file and
+ * UnifiedCache. The Cor Rust implementation mmaps the segment file and
  * indexes into it directly — same read interface, zero-copy access.
  */
 

src/indexes/columnStore/ColumnStore.ts

@@ -69,6 +69,34 @@ interface HeapEntry {
  * await store.flush()
  * const newest = await store.sortTopK('createdAt', 'desc', 10) // bigint[]
  */
+/**
+ * @description Thrown when a segment the manifest lists cannot be loaded —
+ * either it yields no bytes (missing/empty on disk) or its bytes are
+ * undecodable (corruption). Previously such a segment was silently skipped,
+ * dropping ALL of its entities from every `filter`/`rangeQuery`/`sortTopK` with
+ * no error — a manifest↔segments divergence that looked like a short result.
+ * Surfacing it loudly makes the divergence visible and repairable. (A genuine
+ * storage IO fault is a different class — it propagates as the underlying error,
+ * not wrapped in this.)
+ */
+export class ColumnSegmentLoadError extends Error {
+  /** The indexed field whose segment failed to load. */
+  public readonly field: string
+  /** The manifest-listed segment id that could not be loaded. */
+  public readonly segmentId: number | string
+  constructor(field: string, segmentId: number | string, reason: string) {
+    super(
+      `ColumnStore segment ${field}:${segmentId} is listed in the manifest but ` +
+        `could not be loaded (${reason}). The index is inconsistent with its ` +
+        `manifest — rebuild/repair the metadata index rather than trusting a ` +
+        `short query result.`
+    )
+    this.name = 'ColumnSegmentLoadError'
+    this.field = field
+    this.segmentId = segmentId
+  }
+}
+
 export class ColumnStore implements ColumnStoreProvider {
   private storage!: StorageAdapter
   private idMapper!: EntityIdMapper
@@ -476,9 +504,9 @@ export class ColumnStore implements ColumnStoreProvider {
    * Storage path (2.4.0 #4 / cortex-interchange contract):
    *   When the storage adapter exposes the binary-blob primitive
    *   (`saveBinaryBlob` — every brainy adapter ≥7.25.0 does), the segment
-   *   bytes are written as a raw blob at the shared cortex key
+   *   bytes are written as a raw blob at the shared cor key
    *   `_column_index/<field>/L<level>-NNNNNN` (suffix-free; the adapter
-   *   appends its own). This matches byte-for-byte what cortex's
+   *   appends its own). This matches byte-for-byte what cor's
    *   `NativeColumnStore` writes, so JS- and native-written indexes
    *   interchange without re-encoding.
    *
@@ -522,7 +550,7 @@ export class ColumnStore implements ColumnStoreProvider {
       )
 
       if (canUseBlob) {
-        // Raw blob, cortex-shared key convention.
+        // Raw blob, cor-shared key convention.
         const key = `${this.basePath}/${field}/L0-${String(segId).padStart(6, '0')}`
         await storage.saveBinaryBlob!(key, segBuffer)
       } else {
@@ -561,7 +589,7 @@ export class ColumnStore implements ColumnStoreProvider {
     }
 
     // Persist the global deleted bitmap alongside the manifest. Same
-    // raw-blob preference as segments — shared cortex key `<base>/<field>/DELETED`.
+    // raw-blob preference as segments — shared cor key `<base>/<field>/DELETED`.
     const deleted = this.deletedEntities.get(field)
     if (deleted && deleted.size > 0) {
       const serialized = deleted.serialize(true)
@@ -594,14 +622,15 @@ export class ColumnStore implements ColumnStoreProvider {
       let cursor = this.segmentCache.get(cacheKey)
 
       if (!cursor) {
-        const loaded = await this.loadSegmentCursor(field, seg)
-        if (loaded) {
-          cursor = loaded
-          this.segmentCache.set(cacheKey, cursor)
-        }
+        // loadSegmentCursor either returns a cursor or THROWS — a corrupt /
+        // missing manifest-listed segment raises ColumnSegmentLoadError and a
+        // real storage fault propagates, so a listed segment is never silently
+        // dropped from the result set.
+        cursor = await this.loadSegmentCursor(field, seg)
+        this.segmentCache.set(cacheKey, cursor)
       }
 
-      if (cursor) cursors.push(cursor)
+      cursors.push(cursor)
     }
 
     return cursors
@@ -615,39 +644,51 @@ export class ColumnStore implements ColumnStoreProvider {
    * `.cidx` object-path so indexes written before the format unification keep
    * loading correctly. Mirror of cortex's 2.3.1 read-side fallback.
    */
-  private async loadSegmentCursor(field: string, seg: SegmentMeta): Promise<ColumnSegmentCursor | null> {
+  private async loadSegmentCursor(field: string, seg: SegmentMeta): Promise<ColumnSegmentCursor> {
     const manifest = this.manifests.get(field)
-    if (!manifest) return null
+    if (!manifest) {
+      // Defensive: getSegmentCursors guards on the manifest before calling.
+      throw new ColumnSegmentLoadError(field, seg.id, 'no manifest for field')
+    }
 
-    try {
-      let buf: Buffer | null = null
+    let buf: Buffer | null = null
 
-      const storage = this.storage as unknown as {
-        loadBinaryBlob?: (key: string) => Promise<Buffer | null>
-        readObjectFromPath: (path: string) => Promise<any>
-      }
+    const storage = this.storage as unknown as {
+      loadBinaryBlob?: (key: string) => Promise<Buffer | null>
+      readObjectFromPath: (path: string) => Promise<any>
+    }
 
-      // Preferred: raw blob at the cortex-shared key.
-      if (typeof storage.loadBinaryBlob === 'function') {
-        const key = `${this.basePath}/${field}/L${seg.level}-${String(seg.id).padStart(6, '0')}`
-        const blob = await storage.loadBinaryBlob(key)
-        if (blob && blob.length > 0) buf = blob
-      }
+    // Preferred: raw blob at the cor-shared key. A real IO fault PROPAGATES —
+    // loadBinaryBlob throws on a fault and returns null only for genuine absence
+    // (a present-but-unreadable segment must not read as "missing").
+    if (typeof storage.loadBinaryBlob === 'function') {
+      const key = `${this.basePath}/${field}/L${seg.level}-${String(seg.id).padStart(6, '0')}`
+      const blob = await storage.loadBinaryBlob(key)
+      if (blob && blob.length > 0) buf = blob
+    }
 
-      // Legacy fallback: `{ _binary, base64 }` envelope at the .cidx object-path.
-      if (!buf) {
-        const segPath = manifest.segmentPath(seg.level, seg.id)
-        const stored = await storage.readObjectFromPath(segPath)
-        if (!stored) return null
+    // Legacy fallback: `{ _binary, base64 }` envelope at the .cidx object-path.
+    if (!buf) {
+      const segPath = manifest.segmentPath(seg.level, seg.id)
+      const stored = await storage.readObjectFromPath(segPath)
+      if (stored) {
         if (stored._binary && stored.data) {
           buf = Buffer.from(stored.data, 'base64')
         } else if (Buffer.isBuffer(stored)) {
           buf = stored
-        } else {
-          return null
         }
       }
+    }
 
+    // A manifest-listed segment that yields NO loadable bytes is corruption, not
+    // benign absence: swallowing it silently dropped all of the segment's
+    // entities from every filter/rangeQuery/sortTopK with no error. Surface it
+    // loudly so the manifest↔segments divergence is visible (and repairable).
+    if (!buf) {
+      throw new ColumnSegmentLoadError(field, seg.id, 'manifest-listed segment has no loadable bytes')
+    }
+
+    try {
       const parsed = readSegmentFromBuffer(buf)
       return new ColumnSegmentCursor(
         parsed.header,
@@ -655,8 +696,13 @@ export class ColumnStore implements ColumnStoreProvider {
         parsed.entityIds,
         parsed.tombstones
       )
-    } catch {
-      return null
+    } catch (err) {
+      // Undecodable bytes for a listed segment — corruption, not a short result.
+      throw new ColumnSegmentLoadError(
+        field,
+        seg.id,
+        `segment decode failed: ${(err as Error).message}`
+      )
     }
   }
 

src/indexes/columnStore/index.ts

@@ -1,38 +0,0 @@
-/**
- * @module columnStore
- * @description Unified column store for filtering, sorting, range queries,
- * and text search on any field type with exact precision at billion scale.
- *
- * Replaces MetadataIndex's sparse chunk/bloom filter/bucketing internals
- * with per-field sorted columns. Design lineage: Lucene doc values + roaring
- * bitmap acceleration.
- */
-
-export { ColumnStore } from './ColumnStore.js'
-export type { ColumnStoreConfig } from './ColumnStore.js'
-export { ColumnTailBuffer } from './ColumnTailBuffer.js'
-export { ColumnManifest } from './ColumnManifest.js'
-export { ColumnSegmentCursor, TailBufferCursor } from './ColumnSegmentCursor.js'
-export {
-  writeSegmentToBuffer,
-  readSegmentFromBuffer,
-  writeHeader,
-  readHeader,
-  crc32
-} from './ColumnSegmentFormat.js'
-export {
-  ValueType,
-  CIDX_MAGIC,
-  CIDX_VERSION,
-  HEADER_SIZE,
-  FOOTER_SIZE,
-  DEFAULT_FLUSH_THRESHOLD,
-  FLAG_MULTI_VALUE
-} from './types.js'
-export type {
-  ColumnStoreProvider,
-  SegmentHeader,
-  SegmentFooter,
-  SegmentMeta,
-  ManifestData
-} from './types.js'

src/indexes/columnStore/types.ts

@@ -168,7 +168,7 @@ export const FLAG_MULTI_VALUE = 0x01
  * The plugin-provider contract for the column store.
  *
  * Brainy ships a TypeScript baseline that implements this interface.
- * Cortex registers a Rust-accelerated implementation at higher priority.
+ * Cor registers a Rust-accelerated implementation at higher priority.
  * The MetadataIndex coordinator calls whichever is registered.
  *
  * **8.0 u64 contract — BigInt entity ints at the boundary.** Entity ints flow

src/integrations/core/IntegrationLoader.ts

@@ -83,6 +83,10 @@ export function detectEnvironment(): RuntimeEnvironment {
     Deno?: unknown
     Bun?: unknown
     HTMLRewriter?: unknown
+    // `caches` (Cloudflare Workers / ServiceWorker global) is no longer in `lib`
+    // now that DOM is dropped (8.0 is Node/Bun/Deno-only) — declare it locally so
+    // the edge-runtime probe below still type-checks.
+    caches?: unknown
   }
 
   // Deno

src/integrations/odata/ODataIntegration.ts

@@ -253,7 +253,7 @@ export class ODataIntegration extends IntegrationBase implements HTTPIntegration
   /**
    * Get augmentation manifest
    */
-  getManifest(): Record<string, any> {
+  override getManifest(): Record<string, any> {
     return {
       id: 'odata',
       name: 'OData Integration',

src/integrations/sheets/GoogleSheetsIntegration.ts

@@ -283,7 +283,7 @@ export class GoogleSheetsIntegration
   /**
    * Get manifest
    */
-  getManifest(): Record<string, any> {
+  override getManifest(): Record<string, any> {
     return {
       id: 'sheets',
       name: 'Google Sheets Integration',

src/integrations/sse/SSEIntegration.ts

@@ -292,7 +292,7 @@ export class SSEIntegration
   /**
    * Get manifest
    */
-  getManifest(): Record<string, any> {
+  override getManifest(): Record<string, any> {
     return {
       id: 'sse',
       name: 'SSE Streaming',

src/integrations/webhooks/WebhookIntegration.ts

@@ -262,7 +262,7 @@ export class WebhookIntegration extends IntegrationBase {
   /**
    * Get manifest
    */
-  getManifest(): Record<string, any> {
+  override getManifest(): Record<string, any> {
     return {
       id: 'webhooks',
       name: 'Webhooks',