diff --git a/CLAUDE.md b/CLAUDE.md index 03fd5c2c..568c10db 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -12,7 +12,7 @@ Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md` **Brainy's current open actions:** None. MIT open-source — no platform-specific actions. -**Current version:** `@soulcraft/brainy@7.19.10` +**Current version:** `@soulcraft/brainy@7.31.5` (latest published; 8.0.0 release candidate on `feat/8.0-u64-ids`) --- diff --git a/RELEASES.md b/RELEASES.md index 58b20bd2..732feafd 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -10,6 +10,292 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so --- +## v8.0.0 — release candidate (unreleased, branch `feat/8.0-u64-ids`) + +**Affected products:** every consumer — this is a major release with removed +surfaces, hard renames (no aliases, no deprecation period), and one flipped +default. This entry **is** the migration guide: the find/replace pairs, the +sed snippet, and the upgrade checklist below are the complete story — there is +no separate migration doc. Latest published version remains 7.31.x until this +ships. + +### Headline: Database as a Value + +8.0 replaces Brainy's two overlapping version-control subsystems (copy-on-write +branching and per-entity versioning, ~5,100 LOC combined) with **one +mechanism**: generational MVCC over immutable, generation-stamped records, +exposed through a Datomic-style immutable database value — the **`Db`**. + +```ts +const db = brain.now() // pin the current state — O(1), no I/O + +await brain.transact([ + { op: 'update', id: invoiceId, metadata: { status: 'paid' } } +], { meta: { author: 'billing-service', reason: 'PO-7741' } }) + +await db.get(invoiceId) // still 'pending' — pinned, forever +await brain.get(invoiceId) // 'paid' — live +await db.release() // unpin when done +``` + +What you get: + +- **`brain.now()`** — pins the current generation as an immutable `Db` view. + True snapshot isolation: the view reads exactly its pinned state no matter + what commits afterwards, including deletes. Readers never block writers and + writers never block readers. +- **`brain.transact(ops, { meta, ifAtGeneration })`** — atomic multi-write + batches (`add` / `update` / `remove` / `relate` / `unrelate`) committed as + exactly one generation. Either every operation applies or none do. + `ifAtGeneration` is whole-store compare-and-swap (`GenerationConflictError` + on conflict) — the big sibling of the per-entity `ifRev` CAS from 7.31. + `meta` is reified Datomic-style into an append-only transaction log, + readable via `brain.transactionLog()`. +- **`brain.asOf(generation | Date | snapshotPath)`** — time travel with the + **full query surface**: `get()`, `find()` in every mode, semantic search, + graph traversal, cursors, aggregation, all at the pinned past state. +- **`db.with(ops)`** — speculative writes applied in memory on top of a view. + Nothing touches disk, the generation counter, or the indexes. What-if + analysis, then `transact()` the same ops for real. +- **`db.persist(path)`** — instant self-contained snapshots. On filesystem + storage they are built from hard links (no entity data is copied; later + writes to the source can never alter the snapshot because rewrites swap + inodes). `brain.restore(path, { confirm: true })` replaces the whole store + from one; `Brainy.load(path)` opens one read-only with the full query + surface. +- **`db.since(olderDb)`** — exactly the entity and relationship ids that + committed transactions touched between two views. +- **`brain.compactHistory({ retainGenerations, retainMs })`** — explicit + retention. Compaction never breaks a pinned read. + +The precise guarantees (and the honest limits — history granularity is +`transact()` commits; single-operation writes advance the clock but produce no +historical records) are documented in: + +- [docs/concepts/consistency-model.md](docs/concepts/consistency-model.md) — + the guarantees, each proven by a dedicated test in + `tests/integration/db-mvcc.test.ts` +- [docs/guides/snapshots-and-time-travel.md](docs/guides/snapshots-and-time-travel.md) + — the recipes: backup, restore, time-travel debugging, what-if, audit trails +- [docs/ADR-001-generational-mvcc.md](docs/ADR-001-generational-mvcc.md) — the + design record: persisted layout, commit protocol, crash recovery, proof table + +### Removed surfaces and their replacements + +| Removed in 8.0 | Replacement | +|---|---| +| `brain.fork(name)` | Speculation: `db.with(ops)` (in-memory). Long-lived writable copy: `brain.restore()` a persisted snapshot into a fresh data directory. | +| `brain.checkout(branch)` | Open the snapshot you want — `Brainy.load(path)` read-only, or restore into its own directory. No in-place switching: every handle always sees one unambiguous store. | +| `brain.listBranches()` / `brain.getCurrentBranch()` / `brain.deleteBranch()` | A "branch" is now a name → snapshot-path mapping owned by your application. | +| `brain.commit({ message })` | `brain.transact(ops, { meta })` — every batch is an atomic, logged, time-travelable commit; audit fields live in the database, not in commit messages. | +| `brain.getHistory()` / `brain.streamHistory()` | `brain.transactionLog({ limit })` + `db.since(olderDb)`. | +| `brain.versions.*` (`save` / `list` / `compare` / `restore` / `prune`) | A pinned `Db` or persisted snapshot captures *every* entity at that moment; `asOf()` reads any entity's past state; `restore()` for whole-store rollback. | +| `brain.data()` (DataAPI: `clear` / `import` / `export` / `getStats`) | `brain.clear()`, `brain.import()`, `db.persist(path)` (the full-fidelity export format), `brain.restore(path, { confirm: true })`, `brain.stats()`. | +| Cloud + browser storage adapters (`s3` / `gcs` / `r2` / `opfs` storage types, Azure Blob, plus the `storage.branch` option) | `filesystem` and `memory` only. Cloud backup is now an operator concern: `db.persist()` produces a plain directory — sync it with `gsutil` / `aws s3 cp` / `rclone` / `azcopy`. Passing a removed storage type throws at construction with this exact guidance. | +| Browser support | Node.js 22 LTS or Bun ≥ 1.0 (`engines` enforces `node: 22.x`); Deno works through its Node compatibility layer. | +| `brain.migrateToDiskAnn()` / `brain.migrateToHnsw()` | Gone — there is one canonical `'vector'` provider slot. A registered native vector provider selects its own operating mode; there is nothing to call. | +| CLI `fork` / `branch` / `checkout` / `migrate` | CLI `snapshot ` / `restore ` / `history` / `generation`. | + +If you used 7.x COW branches, **materialize every branch you care about while +still on 7.x** (fork → export, or copy each branch's store) — 8.0 does not +read COW branch state. + +### Renames — find/replace pairs + +Hard renames, no aliases. Every pair below is verified against the 8.0 source. + +| Brainy 7.x | Brainy 8.0 | +|---|---| +| `HnswProvider` (from `@soulcraft/brainy/plugin`) | `VectorIndexProvider` | +| `HNSWIndex` (exported class) | `JsHnswVectorIndex` | +| Provider keys `'hnsw'` and `'diskann'` | `'vector'` (the only key Brainy consults for the vector index) | +| `config.hnsw = { quantization, vectorStorage }` | `config.vector = { recall?, quantization?, persistMode? }` (shape change — see below) | +| `hnswPersistMode: 'immediate' \| 'deferred'` (top level) | `config.vector.persistMode` | +| `config.storage.type: 's3' \| 'gcs' \| 'r2' \| 'opfs'` | `'filesystem'` (or `'memory'` / `'auto'`) | +| `config.storage.branch` | Removed (no COW branches) | +| `brain.stats().indexHealth.hnsw` | `brain.stats().indexHealth.vector` | +| Storage adapter contract `saveHNSWData()` / `getHNSWData()` | `saveVectorIndexData()` / `getVectorIndexData()` | +| Cache category `'hnsw'` (`CacheProvider` union) | `'vectors'` | + +Mechanical renames as one command (run from your repo root, review the diff): + +```bash +find . -name '*.ts' -not -path '*/node_modules/*' -print0 | xargs -0 sed -i \ + -e 's/\bHnswProvider\b/VectorIndexProvider/g' \ + -e 's/\bHNSWIndex\b/JsHnswVectorIndex/g' \ + -e 's/\bsaveHNSWData\b/saveVectorIndexData/g' \ + -e 's/\bgetHNSWData\b/getVectorIndexData/g' \ + -e 's/indexHealth\.hnsw\b/indexHealth.vector/g' \ + -e "s/registerProvider('hnsw'/registerProvider('vector'/g" \ + -e "s/registerProvider('diskann'/registerProvider('vector'/g" \ + -e "s/getProvider('hnsw'/getProvider('vector'/g" \ + -e "s/getProvider('diskann'/getProvider('vector'/g" +``` + +The config shape changes need a human (they are not 1:1 textual): + +```ts +// 7.x +new Brainy({ + hnswPersistMode: 'deferred', + hnsw: { quantization: { enabled: true, bits: 8, rerankMultiplier: 3 } } +}) + +// 8.0 +new Brainy({ + vector: { + recall: 'balanced', // 'fast' | 'balanced' | 'accurate' + quantization: { enabled: true, bits: 8 }, // rerankMultiplier is fixed at 3 + persistMode: 'deferred' + } +}) +``` + +`config.vector.recall` replaces the algorithm-internal HNSW knobs: `'balanced'` +(the default) maps to exactly the 7.x default parameters, so an upgrade without +explicit knobs changes nothing about search behaviour. `config.hnsw.vectorStorage` +is removed. The on-disk vector index file names are **unchanged** (e.g. +`hnsw-system.json`) — existing filesystem stores need no data migration for +this rename; only the API surface moved. + +### Behavior changes + +- **`subtype` is required by default.** 7.30's opt-in strict mode is now the + default: every `add()` / `addMany()` / `update()` / `relate()` / + `relateMany()` / `updateRelation()` rejects writes whose type carries no + non-empty `subtype` (`src/brainy.ts:9759` — `requireSubtype ?? true`). + Escape hatches: `requireSubtype: false` (last-resort opt-out for legacy + data) or `requireSubtype: { except: [NounType.Thing, ...] }` (per-type + allowlist). Per-type rules registered via `brain.requireSubtype(type, opts)` + still compose. `brain.audit()` reports entries missing a subtype and + `brain.migrateField()` backfills them. +- **Verb ids are Brainy-generated UUIDs — by contract.** `relate()` now throws + a teaching error if a caller passes an `id` field + (`src/utils/paramValidation.ts:526`). In 7.x a supplied id was silently + ignored (a generated UUID was used anyway); 8.0 says so out loud, because + graph-index providers key verb-int interning on the raw UUID bytes. +- **`update({ vector })` is honored on its own.** 7.x only applied a supplied + pre-computed vector when `data` was also passed (it was silently dropped + otherwise). 8.0 applies an explicit `params.vector` with dimension + validation (`src/brainy.ts:1702-1713`; proven by + `tests/unit/brainy/update.test.ts:256`). +- **`brain.clear()` re-resolves all indexes exactly as `init()` does** — + including plugin-provided vector/metadata/entityIdMapper factories and VFS + root re-creation. In 7.x, clearing a plugin-accelerated brain could leave + the metadata index rebuilt without its native id-mapper wiring. +- **`find({ connected: { …, subtype } })` with `depth > 1` now works.** 7.30 + threw `NOT_SUPPORTED` for multi-hop subtype-filtered traversal; 8.0 + implements the BFS in the JS graph index and routes to a native provider + when one is registered. +- **Every write advances the generation clock; only `transact()` writes + history.** Single-operation writes (`add` / `update` / `delete` / `relate` + outside `transact()`) bump `brain.generation()` so watermarks and CAS stay + sound, but they do not stage historical records — they remain visible + through earlier pins and are not reported by `db.since()`. Writes you want + to travel back through go through `transact()`. This is the documented + contract, stated rather than papered over. +- **Unchanged from 7.31:** per-entity `_rev`, `update({ ifRev })` + (`RevisionConflictError`), and `add({ ifAbsent })` work exactly as before, + and `ifRev` is also accepted on `transact()` update operations (a conflict + rejects the whole batch). + +### For provider and plugin authors + +The provider contracts in `@soulcraft/brainy/plugin` (`src/plugin.ts`) changed +shape: + +- **`GraphIndexProvider` speaks BigInt at the boundary.** Reads take entity + ints and return entity/verb ints as `bigint[]`; the coordinator owns all + UUID ↔ int conversion. `addVerb(verb, sourceInt, targetInt)` receives the + resolved endpoint ints (also mirrored on the new optional + `GraphVerb.sourceInt` / `GraphVerb.targetInt` fields) and returns the + interned verb int. The batch reverse resolver + `verbIntsToIds(verbInts: bigint[]): Promise<(string | null)[]>` is + **required** — the provider owns durable verb-int interning; Brainy keeps + only a bounded in-memory warm cache. Implementations may stay u32 internally + (`Number(bigint)` narrowing is lossless under the `EntityIdSpaceExceeded` + guard) but must speak the bigint contract. +- **`VersionedIndexProvider`** is a new optional 4-method capability — + `generation()`, `isGenerationVisible(g)`, `pin(g)`, `release(g)`, BigInt + generations — feature-detected on every registered index provider. Providers + are *post-commit appliers*: the storage-record commit is the source of + truth; on open, a provider behind the committed watermark replays the gap or + requests a rebuild. Explicit pins override any time-based snapshot retention + the provider has. Speculative `db.with()` overlays never reach providers. A + provider implementing this serves historical reads with **no rebuild** — + the open-core materializer is the correctness baseline, the provider is the + accelerator. +- **The vector index registers under `'vector'`** and implements + `VectorIndexProvider`. The `'hnsw'` and `'diskann'` keys are retired and + never looked up. `CacheProvider`'s category union uses `'vectors'` in place + of `'hnsw'`. + +### Scale and cost — how the mechanisms behave + +Mechanism descriptions, not benchmark numbers (none are published for 8.0 yet): + +- `brain.now()` pins in O(1) and adds **zero read overhead until history + actually moves** — while nothing has committed past the pin, every read + delegates to the live fast paths. +- A `transact()` commit pays O(ids touched) extra writes (before-images + + delta + manifest) — never O(store size). Single-operation writes pay an + in-memory counter bump with coalesced persistence. +- `db.persist()` on filesystem storage is a hard-link farm: snapshot creation + copies no entity data and the snapshot shares disk space with the source. + Cross-device targets fall back to per-file byte copies; persisting an + in-memory brain serializes a real, durable, loadable store. +- Historical **index-accelerated** queries (semantic search, traversal, + cursors, aggregation) on the open-core path pay a one-time O(n at the + pinned generation) in-memory index materialization per `Db`, cached until + `release()`. Record-path reads (`get`, metadata `find`, filter `related`) + at any reachable generation are served directly from the immutable record + layer. A native `VersionedIndexProvider` serves the same reads rebuild-free. + +The full cost model is in +[docs/concepts/consistency-model.md](docs/concepts/consistency-model.md). + +### Upgrade checklist (from 7.x) + +1. **While still on 7.x:** back up your data directory (a plain file copy is + fine). If you used COW branches, materialize each branch you need — 8.0 + does not read branch state. If you ran a cloud storage adapter, export your + data with 7.x tooling (`(await brain.data()).export()`) — 8.0 opens + `filesystem` and `memory` stores only. +2. **Runtime:** Node.js 22 LTS or Bun ≥ 1.0. +3. **Config:** change removed storage types to `'filesystem'`; delete + `storage.branch`; move `config.hnsw` / `hnswPersistMode` to + `config.vector` per the shape example above. +4. **Renames:** run the sed snippet, then fix the manual shape changes. +5. **Removed APIs:** replace `fork` / `checkout` / branch methods / `commit` / + `getHistory` / `streamHistory` / `versions` / `data()` per the table above. +6. **Subtypes:** if your data predates subtype discipline, start with + `requireSubtype: false`, run `brain.audit()`, backfill with + `brain.migrateField()`, then remove the opt-out so the 8.0 default + enforcement protects you going forward. +7. **CLI scripts:** `fork` / `branch` / `checkout` / `migrate` → + `snapshot ` / `restore ` / `history` / `generation`. +8. **Plugin authors:** apply the contract renames and the BigInt graph + contract; optionally implement `VersionedIndexProvider`. +9. **Verify, then take your first snapshot:** run your test suite, then + `const db = brain.now(); await db.persist('/backups/post-8.0-upgrade'); + await db.release()`. + +What did **not** change: the core query surface (`find` / `search` / `get` / +`add` / `update` / `relate` and friends), the aggregation engine, the VFS, +neural extraction, and the on-disk entity/relationship layout — 8.0 creates +its MVCC bookkeeping (`_system/generation.json`, `_system/manifest.json`, +`_system/tx-log.jsonl`, `_generations/`) alongside the existing files on +first use. + +### Cortex compatibility + +Brainy 8.0 pairs with `@soulcraft/cortex` 3.0 (shipping in lockstep). Cortex +2.x cannot accelerate Brainy 8.0: 8.0 never consults the `'hnsw'` / +`'diskann'` provider keys, and the graph/column provider contracts are BigInt +at the boundary. Upgrade both packages together. + +--- + ## v7.31.2 — 2026-06-09 **Affected products:** none in behaviour; affects anyone reading Brainy's TypeScript