From 431cd644062a7b5c089630501c08d4063b594e63 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Wed, 10 Jun 2026 14:14:07 -0700 Subject: [PATCH] feat(8.0): generational MVCC storage + Datomic-style Db API (now/transact/asOf/with/persist) One mechanism replaces the COW and versioning subsystems: immutable generation-stamped records behind a Datomic-style database value. Record layer (src/db/generationStore.ts): - Monotonic generation counter in _system/generation.json; bumped once per transact() commit and once per single-operation write (storage hook), so brain.generation() is always a meaningful watermark. - Commit protocol: stage before-images + tx.json delta -> fsync -> execute batch via TransactionManager -> atomic tmp+rename of _system/manifest.json (the rename IS the commit point) -> append _system/tx-log.jsonl. - Crash recovery on open rolls uncommitted generations back byte-identically and forces an index rebuild; refcounted pins gate compactHistory(), which records a horizon (asOf below it throws GenerationCompactedError). Db API: - Db: get/find/search/related pinned at a generation, with() speculative overlays, since() diffs, persist() hard-link-farm snapshots, timestamp, generation, release() + FinalizationRegistry backstop. - Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch (GenerationConflictError CAS), asOf(generation|Date|path), restore(path, {confirm}), compactHistory(), generation(), static open() + load(). - get()/metadata find()/related() are fully correct at any reachable pinned generation; index-accelerated queries at historical generations throw NotYetSupportedAtHistoricalGenerationError - never silently-wrong results. - VersionedIndexProvider (generation/isGenerationVisible/pin/release) in plugin.ts: feature-detected, balanced pin/release in lockstep with Db lifecycle, post-commit applier + replay-gap model documented. Storage primitives (BaseStorage + filesystem/memory adapters): raw-object read/write/list/remove, fsync barrier, noun/verb raw before-image capture, tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and append-in-place exceptions), restoreFromDirectory + derived-state reload. Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation across 200 mutations, batch atomicity under injected execution failure, ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety under pins, with() overlay isolation, generation monotonicity across reopen, crash consistency through the real recovery path, and versioned provider pin/release balance. Design record in docs/ADR-001-generational-mvcc.md. --- docs/ADR-001-generational-mvcc.md | 262 ++++ src/brainy.ts | 1236 ++++++++++++++++- src/db/db.ts | 806 +++++++++++ src/db/errors.ts | 146 ++ src/db/generationStore.ts | 838 +++++++++++ src/db/types.ts | 321 +++++ src/db/whereMatcher.ts | 292 ++++ src/index.ts | 26 + src/plugin.ts | 102 ++ src/storage/adapters/fileSystemStorage.ts | 248 ++++ .../adapters/historicalStorageAdapter.ts | 43 + src/storage/adapters/memoryStorage.ts | 135 ++ src/storage/baseStorage.ts | 283 ++++ tests/integration/db-mvcc.test.ts | 955 +++++++++++++ tests/unit/db/generationStore.test.ts | 333 +++++ tests/unit/db/whereMatcher.test.ts | 223 +++ 16 files changed, 6244 insertions(+), 5 deletions(-) create mode 100644 docs/ADR-001-generational-mvcc.md create mode 100644 src/db/db.ts create mode 100644 src/db/errors.ts create mode 100644 src/db/generationStore.ts create mode 100644 src/db/types.ts create mode 100644 src/db/whereMatcher.ts create mode 100644 tests/integration/db-mvcc.test.ts create mode 100644 tests/unit/db/generationStore.test.ts create mode 100644 tests/unit/db/whereMatcher.test.ts diff --git a/docs/ADR-001-generational-mvcc.md b/docs/ADR-001-generational-mvcc.md new file mode 100644 index 00000000..8f7d6f36 --- /dev/null +++ b/docs/ADR-001-generational-mvcc.md @@ -0,0 +1,262 @@ +# ADR-001: Generational MVCC storage and the immutable Db API + +**Status:** Accepted (ships in 8.0) +**Date:** 2026-06-10 + +## Context + +Before 8.0, Brainy carried two overlapping version-control subsystems: a +copy-on-write branching layer (`fork`/`checkout`/`commit`/`branches`) and a +separate versioning subsystem (`versions.save/list/compare/restore/prune`), +plus a read-only historical adapter for commit-based time travel. Together +they were ~5,100 LOC of mechanism for one product need: *read a consistent +past state while the store keeps moving, and snapshot/restore cheaply.* + +Neither subsystem gave a precise isolation guarantee. Reads raced in-place +JSON overwrites, so a "snapshot" was only as immutable as the bytes it +happened to share with the live store. + +8.0 replaces both with **one mechanism**: generational MVCC over immutable, +generation-stamped records, exposed through a Datomic-style immutable +database value (`Db`). The same model is implemented natively by versioned +index providers (LSM snapshots), so semantics are identical on the pure-JS +path and the native path. + +## Decision + +### The model + +- A **monotonic u64 generation counter** is the store's logical clock. It + advances once per committed `transact()` batch and once per + single-operation write (`add`/`update`/`delete`/`relate`/…), so + `brain.generation()` is always a meaningful watermark. It is persisted in + `_system/generation.json` and never reissued for anything durable. +- `brain.now()` **pins** the current generation in O(1) and returns a `Db` — + an immutable view. Pins are refcounted; `db.release()` (with a + `FinalizationRegistry` backstop for leaked values) ends the pin. +- `brain.transact(ops, { meta, ifAtGeneration })` commits a declarative + batch atomically as **exactly one generation**, with whole-store + compare-and-swap (`ifAtGeneration` → `GenerationConflictError`) and + reified transaction metadata appended to `_system/tx-log.jsonl`. +- `brain.asOf(generation | Date | snapshotPath)` opens past state; + `db.with(ops)` layers a speculative in-memory overlay (never touching + disk, the counter, or index providers); `db.persist(path)` cuts an + instant snapshot; `brain.restore(path, { confirm: true })` replaces state + from one; `Brainy.load(path)` opens a snapshot read-only with the full + query surface. + +### Persisted layout + +All paths are storage-root-relative: + +``` +_system/generation.json { generation, updatedAt } atomic tmp+rename +_system/manifest.json { version, generation, atomic tmp+rename + committedAt, horizon } (the commit point) +_system/tx-log.jsonl one line per committed append-only + transact: { generation, + timestamp, meta? } +_generations//tx.json the generation-N delta: immutable + touched noun/verb ids + meta +_generations//prev/.json before-image of as of immutable + commit N (raw stored bytes; + null parts = file was absent) +``` + +**Why per-generation deltas instead of a global `id → latest generation` +map in the manifest:** a global map makes every commit O(all ids) — the +whole map must be rewritten to swap it atomically. The delta layout makes a +commit O(ids touched) and keeps the manifest a fixed-size watermark, while +point-in-time resolution stays correct (see "Read resolution" below). The +trade is that resolution at a pinned generation scans the deltas of later +commits — bounded by the number of commits since the pin, which is exactly +the window compaction keeps short. + +Before-images are deliberately the *only* per-id records. They serve both +roles the layer needs — the crash-recovery undo log and the point-in-time +read source. After-images would duplicate state that is already readable +(the canonical entity files hold the latest bytes; earlier states resolve +from later before-images) and would double record I/O per commit. + +### Commit protocol (durability) + +`transact()` commits under a store-wide mutex: + +1. **CAS check.** A stale `ifAtGeneration` throws `GenerationConflictError` + before anything is staged. +2. **Reserve** generation `N` (counter increment). +3. **Stage the undo log:** write the before-image of every touched id plus + `tx.json` under `_generations/N/`, then **fsync** the files and their + directories. From this point, any crash is recoverable to the exact + pre-transaction bytes. +4. **Execute** the planned batch through the TransactionManager (which has + its own operation-level rollback for in-flight failures). +5. **Commit point:** persist the counter, then write `_system/manifest.json` + via atomic tmp+rename and fsync it. The rename *is* the commit: a + generation directory is committed if and only if `N ≤ + manifest.generation`. +6. Append the tx-log line (advisory metadata — a crash between 5 and 6 + keeps the transaction). + +**Crash recovery (on open):** any `_generations/` directory with +`N > manifest.generation` is an uncommitted transaction. Its before-images +are restored to the canonical paths (idempotently — recovery itself can +crash and rerun) and the directory is removed. Because recovery runs before +any index is built, and a recovery that rolled something back forces a full +index rebuild, derived indexes never observe rolled-back state. Reader-mode +instances skip recovery (readers never write; the next writer repairs). + +A failed (non-crash) transaction takes the same staging directory down the +abort path: the TransactionManager rolls back applied operations, the +staging directory is removed, and the generation reservation is returned — +a failed batch leaves the generation counter unchanged. + +### Read resolution at a pinned generation + +The state of id X at pinned generation G is: + +- the before-image stored by the **first committed generation after G that + touched X**, or +- the live canonical bytes, when nothing after G touched X. + +While nothing has committed past G, *every* read on the `Db` delegates to +the live fast paths untouched — `now()` adds no read overhead until history +actually moves. + +**Honesty boundary.** `get()`, metadata-level `find()`, and `related()` are +fully correct at any reachable pinned generation. Index-accelerated +dimensions (semantic/vector search, graph traversal, cursors, aggregation) +are served by live indexes that represent only the current generation, so +at a historical generation they throw +`NotYetSupportedAtHistoricalGenerationError` — never silently-wrong +results. The escape hatch is `persist()` + `Brainy.load()`, which rebuilds +indexes from the snapshot and supports the full query surface. Rebuilding +indexes from a pinned record set in-place (the standard LSM answer) is the +documented follow-up; versioned native providers already expose +`isGenerationVisible()` for provider-accelerated historical reads. + +**History granularity.** Generation *records* are written per `transact()` +batch only. Single-operation writes advance the counter (so watermarks and +CAS stay sound) but do not stage before-images: they remain visible through +earlier pins and are not reported by `db.since()`. Code that needs pinned +isolation across its writes uses `transact()` — that is the documented 8.0 +contract, stated rather than papered over. + +### Pinning, retention, compaction + +- Each live `Db` holds one refcounted pin on its generation (plus a + `pin(generation)` on every registered `VersionedIndexProvider`, whose + explicit pin lifetime overrides any time-based snapshot retention the + provider has). +- `compactHistory({ retainGenerations?, retainMs? })` reclaims record-sets. + A record-set `N` is reclaimed only when `N` is at or below **every** live + pin — deleting `N` can only break readers pinned *below* `N`, because + resolution reads before-images from generations strictly greater than the + pin. With both retention options supplied, both must allow the reclaim. +- The manifest records the **horizon** (highest reclaimed generation). + Generations below the horizon are unreachable; `asOf()` on them throws + `GenerationCompactedError`. The horizon itself stays reachable, resolved + from the record-sets above it. To keep a generation readable forever, + `persist()` it first — snapshots are self-contained. + +### Snapshots and restore + +`db.persist(path)` flushes indexes, then cuts the snapshot under the +store's commit mutex (no commit, compaction, or counter write can +interleave). On filesystem storage it is a **hard-link farm**: every data +file is immutable-by-rename, so linking is safe — later rewrites swap +inodes and the snapshot keeps the old bytes. The two exceptions are handled +explicitly: the append-in-place tx-log is byte-copied, and process-local +lock state is excluded. Cross-device targets (and filesystems that refuse +links) fall back to per-file byte copies. In-memory stores serialize to the +same directory layout, so persisting a memory brain produces a real, +durable, loadable store. + +`persist()` requires the view to still be the store's latest generation +(a snapshot captures current bytes); a view that history has moved past +throws rather than persisting the wrong state. + +`restore(path, { confirm: true })` replaces the store's contents from a +snapshot via byte copy (never links — the snapshot stays independent), +reloads all adapter-internal derived state, rebuilds all indexes, and +floors the generation counter at its pre-restore value so observed +generation numbers are never reissued. Live pins do not survive a restore; +a warning is logged when any exist. + +### Versioned index providers + +Native index providers may implement the optional 4-method +`VersionedIndexProvider` capability (`generation()`, +`isGenerationVisible()`, `pin()`, `release()` — BigInt generations at the +boundary). The locked consistency model: providers are **post-commit +appliers**. The storage-record commit is the source of truth; provider +index state is derived. On open, a provider behind the committed watermark +replays the gap from storage (or requests a rebuild) — there are no +provider rollback hooks, because uncommitted transactions are repaired at +the storage layer before any index opens. Speculative `with()` overlays +never reach providers. + +## Guarantees (and their proofs) + +Each stated guarantee has a test that proves it, not merely exercises it +(`tests/integration/db-mvcc.test.ts`, plus +`tests/unit/db/generationStore.test.ts` for the record layer in isolation): + +| Guarantee | Proof | +|---|---| +| Snapshot isolation: a pinned `Db` reads exactly its pinned state, forever | proof 1 (200 mutations, including deletes, against a pinned view) | +| Atomicity: a failing batch applies nothing; generation unchanged | proofs 2a/2b/2c (plan-time failure, injected execution-phase storage failure, `ifRev` conflict) | +| Whole-store CAS | proof 3 (`ifAtGeneration` success + conflict with exact expected/actual) | +| Snapshot integrity under source mutation (hard-link safety) | proofs 4a/4b/4c | +| Compaction never breaks a pinned read; release enables reclaim | proof 5 | +| `with()` overlays touch nothing durable | proof 6 | +| Generation monotonicity across close/reopen | proof 7 | +| Crash before the manifest rename recovers to exact pre-transaction state through the real recovery path | proof 8 (fault injection that skips abort cleanup, exactly as a dead process would) | +| Balanced provider pin/release lockstep | proof 9 | + +One deliberate softness: single-operation generation bumps persist the +counter coalesced (per write burst), not per write. Durable artifacts — +records, manifests, snapshots — always persist the counter synchronously at +their own commit points, so a crash inside the coalescing window can lose +only counter values that nothing durable ever referenced. + +## Failure modes + +| Failure | Outcome | +|---|---| +| Crash before staging completes | Partial staging directory > manifest watermark → removed on next open; canonical state untouched. | +| Crash after staging, before/during batch execution | Before-images restored on next open; indexes rebuilt; byte-identical pre-transaction state. | +| Crash after execution, before manifest rename | Same as above — the rename is the only commit point. | +| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. | +| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. | +| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. | +| Index-accelerated query at a historical pin | `NotYetSupportedAtHistoricalGenerationError` — explicit, never silently-wrong results. | +| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. | + +## Lineage + +The design is an assembly of well-understood prior art, chosen for being +boring where it counts: + +- **Datomic** — the database-as-a-value: an immutable `Db` you query, with + `with()` for speculation and reified transaction metadata instead of + commit messages. +- **LMDB** — reader pins: readers never block writers; a reader's view + stays valid because nothing overwrites the pages (here: records) it + references; reclamation waits for the last reader. +- **LSM trees / Cassandra** — immutable segments make snapshots hard links + and make compaction a retention policy instead of a locking problem. + +## Consequences + +- One mechanism replaces the COW and versioning subsystems (their removal + is the companion change to this ADR). +- In-place branch switching (`checkout`) is gone by design; the replacement + is opening a persisted snapshot as a separate instance — a name→path + mapping where a product needs named branches. +- Every commit pays O(ids touched) extra writes (before-images + delta + + manifest). Single-operation writes pay only an in-memory counter bump + with coalesced persistence. +- Historical index-accelerated queries are an explicit error until the + asOf-rebuild ships; correctness-critical historical reads (`get`, + metadata `find`, `related`) work today and forever at any reachable pin. diff --git a/src/brainy.ts b/src/brainy.ts index f10eedde..1e229e1a 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -32,11 +32,11 @@ import { CommitBuilder } from './storage/cow/CommitObject.js' import { BlobStorage } from './storage/cow/BlobStorage.js' import { NULL_HASH } from './storage/cow/constants.js' import { createPipeline } from './streaming/pipeline.js' -import { configureLogger, LogLevel } from './utils/logger.js' +import { configureLogger, LogLevel, prodLog } from './utils/logger.js' import { setGlobalCache } from './utils/unifiedCache.js' import type { UnifiedCache } from './utils/unifiedCache.js' import { rankIndicesByScore, reorderByIndices } from './utils/resultRanking.js' -import { PluginRegistry } from './plugin.js' +import { PluginRegistry, isVersionedIndexProvider } from './plugin.js' import type { BrainyPlugin, BrainyPluginContext, @@ -109,6 +109,22 @@ import { AggregationIndex } from './aggregation/AggregationIndex.js' import { AggregateMaterializer } from './aggregation/materializer.js' import type { AggregateDefinition, AggregateQueryParams, AggregateResult } from './types/brainy.types.js' import { resolveJsHnswConfig } from './utils/recallPreset.js' +import * as fs from 'node:fs' +import { Db, type DbHost } from './db/db.js' +import { GenerationStore } from './db/generationStore.js' +import { + GenerationConflictError, + NotYetSupportedAtHistoricalGenerationError +} from './db/errors.js' +import type { + CompactHistoryOptions, + CompactHistoryResult, + TransactOptions, + TransactReceipt, + TxOperation +} from './db/types.js' +import type { VersionedIndexProvider } from './plugin.js' +import type { Operation } from './transaction/types.js' /** * Stopwords for semantic highlighting @@ -143,6 +159,47 @@ export interface DiagnosticsResult { } } +/** + * In-flight planning state for one `brain.transact()` batch (8.0 MVCC). + * + * Operations inside a batch may reference ids written by EARLIER operations + * of the same batch (`add` an entity, then `relate` to it) — but nothing has + * touched storage yet during planning. This state carries the + * would-be-written stored objects so later planners read + * "current state + batch so far", exactly what the executed batch produces. + */ +interface TxPlanState { + /** Stored metadata + vector of entities written earlier in this batch. */ + nouns: Map; vector: Vector }> + /** Entity ids removed earlier in this batch. */ + removedNouns: Set + /** Verbs created earlier in this batch (keyed by relationship id). */ + verbs: Map + /** Relationship ids removed earlier in this batch. */ + removedVerbs: Set +} + +/** + * The fully planned form of one `brain.transact()` batch: the ordered + * `TransactionManager` operations, the resolved id per input operation (the + * receipt), the touched-id sets the generation store stages before-images + * for, and the post-commit hooks (aggregation-index maintenance — derived + * data, applied outside the atomic batch exactly as the single-op methods + * do). + */ +interface PlannedTransact { + /** Ordered operations for one atomic TransactionManager execution. */ + operations: Operation[] + /** Resolved id per input operation, in input order. */ + ids: string[] + /** Entity ids the batch writes or deletes. */ + touchedNouns: string[] + /** Relationship ids the batch writes or deletes. */ + touchedVerbs: string[] + /** Aggregation-index hooks to run after the commit point. */ + postCommit: Array<() => void> +} + /** * The main Brainy class - Clean, Beautiful, Powerful * REAL IMPLEMENTATION - No stubs, no mocks @@ -160,6 +217,27 @@ export class Brainy implements BrainyInterface { private metadataIndex!: MetadataIndexManager private graphIndex!: GraphAdjacencyIndex private transactionManager: TransactionManager + + /** + * 8.0 generational MVCC record layer (created in `init()`, before any index + * is built — crash recovery may rewrite canonical entity files). Owns the + * generation counter, the transact commit protocol, pin refcounts, and + * point-in-time record resolution. Backs `now()`/`transact()`/`asOf()`/ + * `compactHistory()` and the `Db` value type. + */ + private generationStore!: GenerationStore + /** Lazily built host surface shared by every `Db` value of this brain. */ + private _dbHost?: DbHost + /** + * GC backstop for leaked `Db` pins: when an unreleased `Db` is collected, + * its generation pin (store + versioned providers) is released and an + * owned snapshot brain (from `Brainy.load()`) is closed. Explicit + * `db.release()` unregisters first, so pins are never double-released. + */ + private _dbFinalizationRegistry?: FinalizationRegistry<{ + generation: number + closeOnRelease?: () => Promise + }> private embedder: EmbeddingFunction private distance: DistanceFunction private config: Required @@ -513,6 +591,18 @@ export class Brainy implements BrainyInterface { (this.storage as any).enableCOWLightweight((this.config.storage as any)?.branch || 'main') } + // 8.0 generational MVCC: open the record layer BEFORE any index is + // created or loaded. Crash recovery may rewrite canonical entity files + // (restoring before-images of an uncommitted transaction), and every + // index below loads from those files — opening the store first + // guarantees indexes never observe rolled-back state. Reader-mode + // instances skip recovery (readers never write; the next writer + // repairs). + this.generationStore = new GenerationStore(this.storage) + const generationOpenResult = await this.generationStore.open({ + readOnly: this.config.mode === 'reader' + }) + // Provider: embeddings (reassign embedder if plugin provides one) const embeddingProvider = this.pluginRegistry.getProvider('embeddings') if (embeddingProvider) { @@ -625,6 +715,40 @@ export class Brainy implements BrainyInterface { // mmap-vector backend this layer engages even on cloud adapters. this.wireConnectionsCodec() + // 8.0 generational MVCC: if crash recovery rolled back an uncommitted + // transaction, every derived index is suspect — persisted index state + // (flushed before the crash) may reference the rolled-back writes. + // Rebuild all three from the repaired canonical records (the JS-index + // equivalent of the locked design's "manifest-vs-index generation + // comparison + replay on open"). + if (generationOpenResult.rolledBackGenerations > 0) { + prodLog.warn( + `[Brainy] Rebuilding indexes after crash recovery rolled back ` + + `${generationOpenResult.rolledBackGenerations} uncommitted transaction(s)` + ) + await Promise.all([ + this.metadataIndex.rebuild(), + this.index.rebuild(), + this.graphIndex.rebuild() + ]) + } + + // 8.0 versioned-provider replay-gap check: a provider whose persisted + // index generation is behind the storage layer's committed generation + // replays the gap itself (post-commit applier contract) — surface the + // gap for observability. + for (const provider of this.versionedIndexProviders()) { + const providerGen = provider.generation() + const committed = BigInt(this.generationStore.committedGeneration()) + if (providerGen < committed) { + prodLog.info( + `[Brainy] Versioned index provider is at generation ${providerGen} ` + + `(storage committed: ${committed}) — provider replays the gap per ` + + `the post-commit applier contract` + ) + } + } + // Rebuild indexes if needed for existing data await this.rebuildIndexesIfNeeded() @@ -4721,6 +4845,8 @@ export class Brainy implements BrainyInterface { /** * Create a read-only snapshot of the workspace at a specific commit + * (pre-8.0 commit-based time travel — the generational `asOf()` is the + * 8.0 replacement; this method is retired with the COW surface). * * Time-travel API for historical queries. Returns a new Brainy instance that: * - Contains all entities and relationships from that commit @@ -4742,7 +4868,7 @@ export class Brainy implements BrainyInterface { * @example * ```typescript * // Create snapshot at specific commit - * const snapshot = await brain.asOf(commitId) + * const snapshot = await brain.asOfCommit(commitId) * * // Query historical state (full triple intelligence works!) * const files = await snapshot.find({ @@ -4757,7 +4883,7 @@ export class Brainy implements BrainyInterface { * await snapshot.close() * ``` */ - async asOf(commitId: string, options?: { + async asOfCommit(commitId: string, options?: { cacheSize?: number // LRU cache size (default: 10000) }): Promise { await this.ensureInitialized() @@ -5168,6 +5294,1096 @@ export class Brainy implements BrainyInterface { } } + // ============= 8.0 DB API — GENERATIONAL MVCC ============= + // + // Datomic-style immutable database values over the generational record + // layer (src/db/). `now()` pins the current generation in O(1); + // `transact()` commits a declarative batch atomically as exactly one + // generation; `asOf()` opens past generations, timestamps, or persisted + // snapshots; `compactHistory()` reclaims unpinned history. The `Db` value + // type lives in src/db/db.ts; this region is the brain-side host: + // pin/release plumbing (storage + versioned index providers), record + // materialization, the transact planner, and snapshot/restore. + + /** + * @description The store's current generation — a monotonic u64 watermark + * bumped once per committed `transact()` batch and once per + * single-operation write batch (`add`/`update`/`delete`/`relate`/…). + * Persisted in `_system/generation.json`; never reused, including across + * restarts and `restore()`. + * + * @returns The current generation number. + * @throws Error when called before `init()` completes. + * @example + * const before = brain.generation() + * await brain.transact([{ op: 'add', type: NounType.Document, data: 'x', subtype: 'note' }]) + * brain.generation() // before + 1 + */ + generation(): number { + this.assertGenerationStoreReady('generation') + return this.generationStore.generation() + } + + /** + * @description Pin the CURRENT generation and return an immutable `Db` + * view of it — O(1), no I/O. The returned view keeps reading exactly this + * state no matter what commits afterwards: `get()` and metadata-level + * `find()`/`related()` stay correct at the pinned generation forever + * (resolved from immutable generation records), while index-accelerated + * queries throw the documented historical-query error once history moves + * past the pin. + * + * Call `db.release()` when done — pins gate `compactHistory()`. A + * `FinalizationRegistry` backstop releases leaked pins at GC time, but + * explicit release is what makes compaction deterministic. + * + * @returns A `Db` pinned at the current generation. + * @throws Error when called before `init()` completes (pinning is + * synchronous, so it cannot await initialization). + * @example + * const db = brain.now() + * await brain.transact([{ op: 'update', id, metadata: { v: 2 } }]) + * await db.get(id) // still sees v: 1 + * await brain.get(id) // sees v: 2 + * await db.release() + */ + now(): Db { + this.assertGenerationStoreReady('now') + return this.createPinnedDb({ + generation: this.generationStore.generation(), + timestamp: Date.now() + }) + } + + /** + * @description Execute a declarative operation batch atomically: either + * every operation applies and the store advances exactly ONE generation, + * or none apply and the store is byte-identical to its pre-transaction + * state. The semantics of each operation mirror the corresponding + * single-operation method (`add`/`update`/`delete`/`relate`/`unrelate`), + * including validation, subtype enforcement, per-entity `ifRev` CAS, + * relationship deduplication, and delete cascades. Operations may + * reference ids created by earlier operations of the same batch. + * + * Durability protocol (see `src/db/generationStore.ts`): before-images of + * every touched id are staged and fsynced, the batch executes through the + * TransactionManager, and the atomic rename of `_system/manifest.json` is + * the commit point — a crash anywhere before the rename is rolled back to + * the exact pre-transaction bytes on the next open. + * + * Isolation: concurrent `transact()` calls commit serially + * (snapshot-isolated batches). For strict lost-update protection across a + * read-modify-write cycle, pass `ifAtGeneration` (whole-store CAS) or use + * per-entity `ifRev` on update operations. + * + * @param ops - Declarative operations: `{ op: 'add', ... }`, + * `{ op: 'update', ... }`, `{ op: 'remove', id }`, + * `{ op: 'relate', ... }`, `{ op: 'unrelate', id }`. + * @param options - `meta` (reified transaction metadata, recorded in + * `_system/tx-log.jsonl`) and `ifAtGeneration` (whole-store CAS). + * @returns A `Db` pinned at the freshly committed generation, carrying a + * `receipt` with the resolved id per input operation. + * @throws GenerationConflictError when `ifAtGeneration` does not match the + * store's current generation. + * @throws RevisionConflictError when an update operation's `ifRev` does + * not match the entity's persisted revision (whole batch rejected). + * @example + * const db = await brain.transact([ + * { op: 'add', id: aId, type: NounType.Person, subtype: 'employee', data: 'Ada' }, + * { op: 'add', id: bId, type: NounType.Project, subtype: 'milestone', data: 'Apollo' }, + * { op: 'relate', from: aId, to: bId, type: VerbType.WorksOn, subtype: 'assignment' } + * ], { meta: { author: 'import-job-42' } }) + * db.receipt.ids // [aId, bId, relationshipId] + */ + async transact(ops: TxOperation[], options?: TransactOptions): Promise> { + this.assertWritable('transact') + await this.ensureInitialized() + + if (!Array.isArray(ops) || ops.length === 0) { + throw new Error('transact(): provide a non-empty array of transaction operations') + } + + // Early CAS check — avoids expensive planning (embedding) on a stale + // expectation. The generation store re-checks authoritatively under the + // commit mutex; this one is purely a fast-fail. + if ( + options?.ifAtGeneration !== undefined && + options.ifAtGeneration !== this.generationStore.generation() + ) { + throw new GenerationConflictError( + options.ifAtGeneration, + this.generationStore.generation() + ) + } + + const plan = await this.planTransact(ops) + + const { generation, timestamp } = await this.generationStore.commitTransaction({ + touched: { nouns: plan.touchedNouns, verbs: plan.touchedVerbs }, + meta: options?.meta, + ifAtGeneration: options?.ifAtGeneration, + execute: async () => { + await this.transactionManager.executeTransaction(async (tx) => { + for (const operation of plan.operations) { + tx.addOperation(operation) + } + }) + } + }) + + // Aggregation-index maintenance: derived data, applied after the commit + // point — exactly where the single-operation methods apply it. + for (const hook of plan.postCommit) { + hook() + } + + const receipt: TransactReceipt = { generation, timestamp, ids: plan.ids } + return this.createPinnedDb({ generation, timestamp, receipt }) + } + + /** + * @description Open an immutable `Db` view of PAST state: + * + * - **`number`** — a generation: pins it on this store; reads resolve + * through the generational record layer. History granularity is + * `transact()` commits — single-operation writes between commits do not + * produce records and remain visible through earlier pins. + * - **`Date`** — a wall-clock instant: resolved via the transaction log to + * the newest generation committed at or before it, then pinned as above. + * - **`string`** — a snapshot directory previously produced by + * `db.persist(path)`: opened as a self-contained read-only store + * (equivalent to {@link Brainy.load}) with the FULL query surface, + * including vector search. + * + * Release the returned `Db` when done. (For pre-8.0 commit-hash time + * travel, see `asOfCommit()` — retired with the COW surface.) + * + * @param target - Generation number, `Date`, or snapshot directory path. + * @returns A `Db` pinned at the resolved state. + * @throws GenerationCompactedError when the generation's records were + * reclaimed by `compactHistory()`. + * @throws RangeError for negative or future generations. + * @example + * const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000)) + * const atGen42 = await brain.asOf(42) + * const fromSnapshot = await brain.asOf('/backups/2026-06-01') + */ + async asOf(target: number | Date | string): Promise> { + await this.ensureInitialized() + + if (typeof target === 'string') { + const stat = await fs.promises.stat(target).catch(() => null) + if (!stat || !stat.isDirectory()) { + throw new Error( + `asOf(): '${target}' is not a snapshot directory. Pass a generation number, ` + + `a Date, or a directory created by db.persist(path).` + ) + } + return Brainy.load(target) + } + + let generation: number + let timestamp: number + if (target instanceof Date) { + const resolved = await this.generationStore.resolveTimestamp(target.getTime()) + this.generationStore.assertReachable(resolved.generation) + generation = resolved.generation + timestamp = resolved.entry?.timestamp ?? target.getTime() + } else { + this.generationStore.assertReachable(target) + generation = target + timestamp = + (await this.generationStore.commitTimestampAtOrBefore(target)) ?? Date.now() + } + + return this.createPinnedDb({ generation, timestamp }) + } + + /** + * @description Reclaim the immutable generation record-sets that no + * retention rule and no live pin protects. Records are what serve + * historical reads (`asOf()`, pinned `Db` values); compaction trades that + * history for disk space. Generations at or below the resulting horizon + * become unreachable (`asOf()` throws `GenerationCompactedError`). + * + * Safety invariant: a record-set is never removed while any live `Db` pin + * could need it — pinned reads stay correct across compaction, always. + * Note that `transact()` returns a PINNED `Db`: release views you do not + * keep (the GC backstop eventually releases leaked ones, but until then + * compaction retains the history they protect). + * + * @param options - `retainGenerations` (keep the N most recent committed + * generations) and/or `retainMs` (keep everything committed within the + * window). Both supplied = both must allow reclaim. Neither = reclaim + * everything unpinned. + * @returns Count of reclaimed record-sets and the new horizon. + * @example + * await brain.compactHistory({ retainGenerations: 100, retainMs: 7 * 86_400_000 }) + */ + async compactHistory(options?: CompactHistoryOptions): Promise { + this.assertWritable('compactHistory') + await this.ensureInitialized() + return this.generationStore.compact(options) + } + + /** + * @description Replace this store's ENTIRE state from a snapshot directory + * previously produced by `db.persist(path)`. Destructive: current + * entities, relationships, indexes, and history records are all replaced + * by the snapshot's. The generation counter is floored at its pre-restore + * value, so generation numbers observed before the restore are never + * reissued. + * + * Live `Db` values pinned before a restore are NOT remapped — their + * snapshot-isolation guarantee does not survive a wholesale state + * replacement. Release them first; a warning is logged when live pins + * exist. + * + * @param path - Snapshot directory to restore from. + * @param options - Must be `{ confirm: true }` — an explicit acknowledgment + * that current state is destroyed. + * @throws Error when `confirm` is not `true`, or `path` is not a snapshot + * directory. + * @example + * await brain.restore('/backups/2026-06-01', { confirm: true }) + */ + async restore(path: string, options: { confirm: boolean }): Promise { + this.assertWritable('restore') + await this.ensureInitialized() + + if (options?.confirm !== true) { + throw new Error( + `restore() replaces the store's entire current state with the snapshot at ` + + `'${path}'. Pass { confirm: true } to proceed.` + ) + } + + const pinCount = this.generationStore.activePinCount() + if (pinCount > 0) { + prodLog.warn( + `[Brainy] restore() with ${pinCount} live Db pin(s): pinned views do not ` + + `survive a restore — release Db values before restoring.` + ) + } + + const floorGeneration = this.generationStore.generation() + await this.storage.restoreFromDirectory(path) + await this.generationStore.reopenAfterRestore(floorGeneration) + + // Every in-memory index is now stale — rebuild all three from the + // restored canonical records (each rebuild clears its own state first). + await Promise.all([ + this.metadataIndex.rebuild(), + this.index.rebuild(), + this.graphIndex.rebuild() + ]) + } + + /** + * @description Construct AND initialize a `Brainy` instance in one call — + * `new Brainy(config)` + `await init()`. + * + * @param config - Standard `BrainyConfig`. + * @returns The initialized instance. + * @example + * const brain = await Brainy.open({ storage: { type: 'filesystem', rootDirectory: './data' } }) + */ + static async open(config?: BrainyConfig): Promise> { + const brain = new Brainy(config) + await brain.init() + return brain + } + + /** + * @description Open a persisted snapshot directory (from `db.persist()`) + * as a self-contained READ-ONLY store and return it as a `Db`. The + * snapshot's indexes load from its own files, so the full query surface — + * including vector search — works at the snapshot's generation. Releasing + * the returned `Db` closes the underlying read-only instance. + * + * @param path - Snapshot directory produced by `db.persist(path)`. + * @returns A `Db` over the snapshot (release it to free resources). + * @example + * const db = await Brainy.load('/backups/2026-06-01') + * const hits = await db.search('quarterly invoices') + * await db.release() + */ + static async load(path: string): Promise> { + const stat = await fs.promises.stat(path).catch(() => null) + if (!stat || !stat.isDirectory()) { + throw new Error( + `Brainy.load(): '${path}' is not a snapshot directory ` + + `(expected a directory created by db.persist(path))` + ) + } + + const brain = new Brainy({ + storage: { type: 'filesystem', rootDirectory: path }, + mode: 'reader' + }) + await brain.init() + return brain.createPinnedDb({ + generation: brain.generationStore.generation(), + timestamp: Date.now(), + closeOnRelease: () => brain.close() + }) + } + + // --- Db host plumbing ------------------------------------------------------ + + /** + * Guard for the synchronous Db entry points (`now()`, `generation()`): + * they cannot await initialization, so they demand it up front. + */ + private assertGenerationStoreReady(method: string): void { + if (!this.initialized || !this.generationStore) { + throw new Error( + `Cannot call ${method}() before init() completes. ` + + `Await brain.init() (or brain.ready, or use Brainy.open()).` + ) + } + } + + /** + * Pin `generation` and construct the `Db` value over the shared host. The + * `Db` constructor registers itself with the GC-backstop finalization + * registry; explicit `db.release()` unregisters first. + */ + private createPinnedDb(init: { + generation: number + timestamp: number + receipt?: TransactReceipt + closeOnRelease?: () => Promise + }): Db { + this.pinGeneration(init.generation) + return new Db({ host: this.dbHost, ...init }) + } + + /** + * The host surface every `Db` of this brain shares (lazily built once): + * live read fast paths, generation-record materialization, snapshot + * support, and pin lifecycle. See {@link DbHost} in src/db/db.ts. + */ + private get dbHost(): DbHost { + if (!this._dbHost) { + this._dbHost = { + store: this.generationStore, + get: (id, options) => this.get(id, options), + find: (query) => this.find(query), + getRelations: (paramsOrId) => this.getRelations(paramsOrId), + entityFromRecord: (id, record, includeVectors) => + this.entityFromGenerationRecord(id, record, includeVectors), + relationFromRecord: (id, record) => this.relationFromGenerationRecord(id, record), + persistPinned: (targetPath, generation) => + this.persistPinnedGeneration(targetPath, generation), + pinGeneration: (generation) => this.pinGeneration(generation), + releaseGeneration: (generation) => this.releaseGeneration(generation), + registerDbForFinalization: (db, generation, closeOnRelease) => { + this.dbFinalizationRegistry.register(db, { generation, closeOnRelease }, db) + }, + unregisterDbFromFinalization: (db) => { + this._dbFinalizationRegistry?.unregister(db) + } + } + } + return this._dbHost + } + + /** Lazily build the GC backstop for leaked (never-released) `Db` pins. */ + private get dbFinalizationRegistry(): FinalizationRegistry<{ + generation: number + closeOnRelease?: () => Promise + }> { + if (!this._dbFinalizationRegistry) { + this._dbFinalizationRegistry = new FinalizationRegistry((held) => { + this.releaseGeneration(held.generation) + if (held.closeOnRelease) { + void held.closeOnRelease().catch((err: Error) => { + prodLog.warn(`[Brainy] Db finalization close failed: ${err.message}`) + }) + } + }) + } + return this._dbFinalizationRegistry + } + + /** + * Registered index providers that implement the optional + * {@link VersionedIndexProvider} capability (feature-detected on the three + * index surfaces: vector, metadata, graph). Brainy's own JS indexes do not + * implement it. + */ + private versionedIndexProviders(): VersionedIndexProvider[] { + const candidates: unknown[] = [this.index, this.metadataIndex, this.graphIndex] + return candidates.filter(isVersionedIndexProvider) + } + + /** + * Take one refcounted pin on `generation`: storage-record pin (gates + * compaction) plus a `pin()` on every versioned index provider — the + * explicit pin lifetime overrides any time-based snapshot retention the + * provider has. Provider visibility (`isGenerationVisible`) is evaluated + * at pin time per the locked read-routing rule; provider-accelerated + * at-generation reads engage with the asOf-rebuild follow-up — until then + * historical reads always resolve from canonical generation records + * (strictly correct, provider-independent). + */ + private pinGeneration(generation: number): void { + this.generationStore.pin(generation) + const big = BigInt(generation) + for (const provider of this.versionedIndexProviders()) { + const visible = provider.isGenerationVisible(big) + provider.pin(big) + if (!visible) { + prodLog.debug( + `[Brainy] generation ${generation} pinned but not provider-visible — ` + + `historical reads at this pin resolve from canonical generation records` + ) + } + } + } + + /** Release one refcounted pin on `generation` (mirror of {@link pinGeneration}). */ + private releaseGeneration(generation: number): void { + this.generationStore.release(generation) + const big = BigInt(generation) + for (const provider of this.versionedIndexProviders()) { + provider.release(big) + } + } + + /** + * Materialize an `Entity` from a generation record's raw stored objects — + * the historical-read counterpart of the live `get()` paths. + * `record.metadata` is the exact stored metadata object; + * `record.vector` is the stored HNSW noun object (whose `.vector` carries + * the embedding) or `null` when the vector file was absent. + */ + private async entityFromGenerationRecord( + id: string, + record: { metadata: any; vector: any | null }, + includeVectors: boolean + ): Promise> { + const entity = await this.convertMetadataToEntity(id, record.metadata) + if (includeVectors && record.vector && Array.isArray(record.vector.vector)) { + entity.vector = record.vector.vector + } + return entity + } + + /** + * Materialize a `Relation` from a generation record's raw stored objects. + * Field split mirrors `storage.getVerb()` + `verbsToRelations()`: the + * stored vector object carries the structural core (`sourceId`/`targetId`/ + * `verb`), the stored metadata object carries typed fields plus the custom + * metadata bag. Returns `null` when the metadata part is absent (the + * relationship did not exist as a live edge). + * @throws Error when metadata exists but the structural core is missing — + * that state is unreachable through the API (relate() writes both parts + * in one atomic batch) and indicates external tampering; throwing beats + * silently dropping an edge from historical results. + */ + private relationFromGenerationRecord( + id: string, + record: { metadata: any; vector: any | null } + ): Relation | null { + if (record.metadata === null || record.metadata === undefined) { + return null + } + const core = record.vector as { sourceId?: string; targetId?: string; verb?: string } | null + if (!core || typeof core.sourceId !== 'string' || typeof core.targetId !== 'string') { + throw new Error( + `Generation record for relationship ${id} has metadata but no structural core ` + + `(sourceId/targetId) — store corrupted or records modified outside Brainy` + ) + } + + const { + verb, + subtype, + createdAt, + updatedAt, + confidence, + weight, + service, + data, + ...customMetadata + } = record.metadata as Record + + return { + id, + from: core.sourceId, + to: core.targetId, + type: (verb ?? core.verb) as VerbType, + ...(subtype !== undefined && { subtype: subtype as string }), + weight: (weight as number) ?? 1.0, + data, + metadata: customMetadata as T, + service: service as string, + createdAt: typeof createdAt === 'number' ? createdAt : Date.now(), + ...(typeof updatedAt === 'number' && { updatedAt }), + ...(typeof confidence === 'number' && { confidence }) + } + } + + /** + * Snapshot the store at `generation` into `targetPath` — the brain-side + * half of `db.persist()`. Indexes are flushed first (so the snapshot's + * persisted index files match its records), then the snapshot is cut + * under the generation store's commit lock: no transact commit, no + * compaction, and no counter write can interleave with the hard-link + * walk, and the counter is durably persisted into the snapshot. + * + * @throws NotYetSupportedAtHistoricalGenerationError when `generation` is + * no longer the store's latest — a snapshot captures current bytes, so + * persist the view before further writes (pin with `brain.now()`, + * persist, then mutate). + */ + private async persistPinnedGeneration(targetPath: string, generation: number): Promise { + await this.ensureInitialized() + await this.flush() + await this.generationStore.snapshotWith(async () => { + if (generation !== this.generationStore.generation()) { + throw new NotYetSupportedAtHistoricalGenerationError( + 'persist of a historical generation', + generation, + this.generationStore.generation() + ) + } + await this.storage.snapshotToDirectory(targetPath) + }) + } + + // --- Transact planner ------------------------------------------------------ + + /** + * Plan a full `transact()` batch: validate and resolve every operation + * against "current state + batch so far", producing the ordered + * TransactionManager operations, the receipt ids, the touched-id sets + * (which the generation store stages before-images for), and the + * post-commit aggregation hooks. Planning performs reads and embedding + * only — nothing touches storage until the generation store runs the + * batch. + */ + private async planTransact(ops: TxOperation[]): Promise { + const state: TxPlanState = { + nouns: new Map(), + removedNouns: new Set(), + verbs: new Map(), + removedVerbs: new Set() + } + const plan: PlannedTransact = { + operations: [], + ids: [], + touchedNouns: [], + touchedVerbs: [], + postCommit: [] + } + + for (const op of ops) { + switch (op.op) { + case 'add': + plan.ids.push(await this.planTxAdd(op, state, plan)) + break + case 'update': + plan.ids.push(await this.planTxUpdate(op, state, plan)) + break + case 'remove': + plan.ids.push(await this.planTxRemove(op, state, plan)) + break + case 'relate': + plan.ids.push(await this.planTxRelate(op, state, plan)) + break + case 'unrelate': + plan.ids.push(await this.planTxUnrelate(op, state, plan)) + break + default: { + const exhaustive: never = op + throw new Error(`transact(): unknown operation ${JSON.stringify(exhaustive)}`) + } + } + } + + return plan + } + + /** + * Resolve an entity during planning: batch-pending writes win, then the + * live store. `includeVectors: false` returns the stub vector exactly as + * the live metadata-only `get()` does, so planned relationship vectors + * match `relate()` byte-for-byte. + */ + private async planGetEntity( + state: TxPlanState, + id: string, + options?: GetOptions + ): Promise | null> { + if (state.removedNouns.has(id)) { + return null + } + const pending = state.nouns.get(id) + if (pending) { + const entity = await this.convertMetadataToEntity(id, pending.metadata) + if (options?.includeVectors) { + entity.vector = pending.vector + } + return entity + } + return this.get(id, options) + } + + /** Plan one `{ op: 'add' }` — mirror of `add()`. Returns the entity id. */ + private async planTxAdd( + op: Extract, { op: 'add' }>, + state: TxPlanState, + plan: PlannedTransact + ): Promise { + const { op: _discriminator, ...params } = op + validateAddParams(params as AddParams) + this.enforceTrackedFieldValues(params.metadata as Record | undefined, 'metadata') + this.enforceTrackedFieldValues({ subtype: params.subtype } as Record, 'top-level') + this.enforceSubtypeOnAdd('add', params.type, params.subtype, params.metadata) + + // ifAbsent — idempotent by-id insert, resolved against batch + store. + if (params.id && params.ifAbsent) { + const existing = await this.planGetEntity(state, params.id) + if (existing) { + return params.id + } + } + + const id = params.id || uuidv4() + const vector = params.vector || (await this.embed(params.data)) + if (!this.dimensions) { + this.dimensions = vector.length + } else if (vector.length !== this.dimensions) { + throw new Error( + `Vector dimension mismatch: expected ${this.dimensions}, got ${vector.length}` + ) + } + + // isNew controls the operation's rollback strategy: a custom id may + // collide with an existing entity (add() overwrite semantics), and a + // failed batch must restore the overwritten bytes. + const isNew = params.id + ? !state.nouns.has(id) && (await this.storage.getNounMetadata(id)) === null + : true + + const now = Date.now() + const storageMetadata = { + ...params.metadata, + data: params.data, + noun: params.type, + ...(params.subtype !== undefined && { subtype: params.subtype }), + service: params.service, + createdAt: now, + updatedAt: now, + _rev: 1, + ...(params.confidence !== undefined && { confidence: params.confidence }), + ...(params.weight !== undefined && { weight: params.weight }), + ...(params.createdBy && { createdBy: params.createdBy }) + } + const entityForIndexing = { + id, + vector, + connections: new Map(), + level: 0, + type: params.type, + ...(params.subtype !== undefined && { subtype: params.subtype }), + ...(params.confidence !== undefined && { confidence: params.confidence }), + ...(params.weight !== undefined && { weight: params.weight }), + createdAt: now, + updatedAt: now, + service: params.service, + data: params.data, + ...(params.createdBy && { createdBy: params.createdBy }), + metadata: params.metadata || {} + } + + plan.operations.push( + new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew), + new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew), + new AddToHNSWOperation(this.index as any, id, vector), + new AddToMetadataIndexOperation(this.metadataIndex, id, entityForIndexing) + ) + plan.touchedNouns.push(id) + plan.postCommit.push(() => { + if (this._aggregationIndex) { + this._aggregationIndex.onEntityAdded(id, entityForIndexing) + } + }) + + state.nouns.set(id, { metadata: storageMetadata, vector }) + state.removedNouns.delete(id) + return id + } + + /** Plan one `{ op: 'update' }` — mirror of `update()`. Returns the entity id. */ + private async planTxUpdate( + op: Extract, { op: 'update' }>, + state: TxPlanState, + plan: PlannedTransact + ): Promise { + const { op: _discriminator, ...params } = op + validateUpdateParams(params as UpdateParams) + this.enforceTrackedFieldValues(params.metadata as Record | undefined, 'metadata') + if (params.subtype !== undefined) { + this.enforceTrackedFieldValues({ subtype: params.subtype } as Record, 'top-level') + } + + const existing = await this.planGetEntity(state, params.id, { includeVectors: true }) + if (!existing) { + throw new Error(`Entity ${params.id} not found`) + } + + if (params.subtype !== undefined || params.type !== undefined) { + const effectiveType = params.type ?? existing.type + const effectiveSubtype = params.subtype !== undefined ? params.subtype : existing.subtype + const effectiveMetadata = params.metadata ?? existing.metadata + this.enforceSubtypeOnAdd('update', effectiveType, effectiveSubtype, effectiveMetadata) + } + + // ifRev CAS — identical resolution to update(); a conflict rejects the + // WHOLE batch before anything is staged or applied. + const currentRev = + typeof (existing.metadata as any)?._rev === 'number' + ? (existing.metadata as any)._rev + : typeof existing._rev === 'number' + ? existing._rev + : 1 + if (typeof params.ifRev === 'number' && params.ifRev !== currentRev) { + throw new RevisionConflictError(params.id, params.ifRev, currentRev) + } + + let vector = existing.vector + const needsReindexing = params.data || params.type + if (params.data) { + vector = params.vector || (await this.embed(params.data)) + } + + const newMetadata = + params.merge !== false + ? { ...existing.metadata, ...params.metadata } + : params.metadata || existing.metadata + const now = Date.now() + const updatedMetadata = { + ...newMetadata, + data: params.data !== undefined ? params.data : existing.data, + noun: params.type || existing.type, + service: existing.service, + createdAt: existing.createdAt, + updatedAt: now, + _rev: currentRev + 1, + ...(params.confidence !== undefined && { confidence: params.confidence }), + ...(params.weight !== undefined && { weight: params.weight }), + ...(params.confidence === undefined && + existing.confidence !== undefined && { confidence: existing.confidence }), + ...(params.weight === undefined && + existing.weight !== undefined && { weight: existing.weight }), + ...(params.subtype !== undefined && { subtype: params.subtype }), + ...(params.subtype === undefined && + existing.subtype !== undefined && { subtype: existing.subtype }) + } + const entityForIndexing = { + id: params.id, + vector, + connections: new Map(), + level: 0, + type: params.type || existing.type, + subtype: params.subtype !== undefined ? params.subtype : existing.subtype, + confidence: params.confidence !== undefined ? params.confidence : existing.confidence, + weight: params.weight !== undefined ? params.weight : existing.weight, + createdAt: existing.createdAt, + updatedAt: now, + service: existing.service, + data: params.data !== undefined ? params.data : existing.data, + createdBy: existing.createdBy, + metadata: newMetadata + } + const removalMetadata = { + type: existing.type, + confidence: existing.confidence, + weight: existing.weight, + createdAt: existing.createdAt, + updatedAt: existing.updatedAt, + service: existing.service, + data: existing.data, + createdBy: existing.createdBy, + metadata: existing.metadata + } + + plan.operations.push( + new UpdateNounMetadataOperation(this.storage, params.id, updatedMetadata), + new SaveNounOperation(this.storage, { + id: params.id, + vector, + connections: new Map(), + level: 0 + }) + ) + if (needsReindexing) { + plan.operations.push( + new RemoveFromHNSWOperation(this.index as any, params.id, existing.vector), + new AddToHNSWOperation(this.index as any, params.id, vector) + ) + } + plan.operations.push( + new RemoveFromMetadataIndexOperation(this.metadataIndex, params.id, removalMetadata), + new AddToMetadataIndexOperation(this.metadataIndex, params.id, entityForIndexing) + ) + plan.touchedNouns.push(params.id) + + const oldEntityForAgg = { + type: existing.type, + service: existing.service, + data: existing.data, + metadata: existing.metadata + } + plan.postCommit.push(() => { + if (this._aggregationIndex) { + this._aggregationIndex.onEntityUpdated(params.id, entityForIndexing, oldEntityForAgg) + } + }) + + state.nouns.set(params.id, { metadata: updatedMetadata, vector }) + return params.id + } + + /** + * Plan one `{ op: 'remove' }` — mirror of `delete()`, including the + * relationship cascade (every edge where the entity is source or target, + * plus edges created earlier in this batch). Returns the entity id. + */ + private async planTxRemove( + op: Extract, { op: 'remove' }>, + state: TxPlanState, + plan: PlannedTransact + ): Promise { + if (!op.id || typeof op.id !== 'string') { + throw new Error(`transact(): remove operation requires an entity id (got ${JSON.stringify(op.id)})`) + } + const id = op.id + + const pending = state.nouns.get(id) + const metadata = pending ? pending.metadata : await this.storage.getNounMetadata(id) + const noun = pending + ? { id, vector: pending.vector, connections: new Map(), level: 0 } + : await this.storage.getNoun(id) + + // Cascade set: stored edges touching the entity, plus batch-pending + // edges, minus edges already removed in this batch. + const storedVerbs = [ + ...(await this.storage.getVerbsBySource(id)), + ...(await this.storage.getVerbsByTarget(id)) + ] + const cascade = new Map() + for (const verb of storedVerbs) { + if (!state.removedVerbs.has(verb.id)) { + cascade.set(verb.id, verb) + } + } + for (const [verbId, verb] of state.verbs) { + if ( + !state.removedVerbs.has(verbId) && + (verb.sourceId === id || verb.targetId === id) + ) { + cascade.set(verbId, verb) + } + } + + if (noun) { + plan.operations.push(new RemoveFromHNSWOperation(this.index as any, id, noun.vector)) + } + if (metadata) { + plan.operations.push(new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata)) + } + plan.operations.push(new DeleteNounMetadataOperation(this.storage, id)) + for (const verb of cascade.values()) { + const { sourceInt, targetInt } = this.resolveVerbEndpointInts(verb) + plan.operations.push( + new RemoveFromGraphIndexOperation(this.graphIndex, verb, sourceInt, targetInt), + new DeleteVerbMetadataOperation(this.storage, verb.id) + ) + plan.touchedVerbs.push(verb.id) + state.verbs.delete(verb.id) + state.removedVerbs.add(verb.id) + } + plan.touchedNouns.push(id) + + if (metadata) { + const { noun: nounType, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata + const entityForAgg = { type: nounType, service, data, metadata: customMetadata } + plan.postCommit.push(() => { + if (this._aggregationIndex) { + this._aggregationIndex.onEntityDeleted(id, entityForAgg) + } + }) + } + + state.nouns.delete(id) + state.removedNouns.add(id) + return id + } + + /** + * Plan one `{ op: 'relate' }` — mirror of `relate()`, including duplicate + * deduplication (against the store AND earlier batch operations) and + * `bidirectional`. Returns the (primary) relationship id — the existing id + * when deduplicated. + */ + private async planTxRelate( + op: Extract, { op: 'relate' }>, + state: TxPlanState, + plan: PlannedTransact + ): Promise { + const { op: _discriminator, ...params } = op + validateRelateParams(params as RelateParams) + this.enforceSubtypeOnRelate('relate', params.type, params.subtype, params.metadata) + + const fromEntity = await this.planGetEntity(state, params.from) + const toEntity = await this.planGetEntity(state, params.to) + if (!fromEntity) { + throw new Error(`Source entity ${params.from} not found`) + } + if (!toEntity) { + throw new Error(`Target entity ${params.to} not found`) + } + + // Dedupe against earlier operations of this batch first… + for (const [verbId, verb] of state.verbs) { + if ( + !state.removedVerbs.has(verbId) && + verb.sourceId === params.from && + verb.targetId === params.to && + verb.verb === params.type + ) { + return verbId + } + } + // …then against the committed graph (same lookup relate() performs). + const dupSourceInt = this.graphEntityInt(params.from) + const verbIds = + dupSourceInt === undefined + ? [] + : await this.resolveVerbIntsToIds(await this.graphIndex.getVerbIdsBySource(dupSourceInt)) + if (verbIds.length > 0) { + const verbsMap = await this.graphIndex.getVerbsBatchCached(verbIds) + for (const verb of verbsMap.values()) { + if ( + !state.removedVerbs.has(verb.id) && + verb.targetId === params.to && + verb.verb === params.type + ) { + return verb.id + } + } + } + + const id = uuidv4() + const relationVector = fromEntity.vector.map((v, i) => (v + toEntity.vector[i]) / 2) + const now = Date.now() + const verbMetadata = { + ...(params.metadata || {}), + verb: params.type, + ...(params.subtype !== undefined && { subtype: params.subtype }), + weight: params.weight ?? 1.0, + createdAt: now, + ...(params.data !== undefined && { data: params.data }) + } + const verb: GraphVerb = { + id, + vector: relationVector, + sourceId: params.from, + targetId: params.to, + verb: params.type, + type: params.type, + ...(params.subtype !== undefined && { subtype: params.subtype }), + weight: params.weight ?? 1.0, + metadata: params.metadata, + data: params.data, + createdAt: now + } + const { sourceInt, targetInt } = this.resolveVerbEndpointInts(verb) + + plan.operations.push( + new SaveVerbOperation(this.storage, { + id, + vector: relationVector, + connections: new Map(), + verb: params.type, + sourceId: params.from, + targetId: params.to + }), + new SaveVerbMetadataOperation(this.storage, id, verbMetadata), + new AddToGraphIndexOperation(this.graphIndex, verb, sourceInt, targetInt, (verbInt) => + this.cacheVerbInt(verbInt, id) + ) + ) + plan.touchedVerbs.push(id) + state.verbs.set(id, verb) + state.removedVerbs.delete(id) + + if (params.bidirectional) { + const reverseId = uuidv4() + const reverseVerb: GraphVerb = { + ...verb, + id: reverseId, + sourceId: params.to, + targetId: params.from, + sourceInt: targetInt, + targetInt: sourceInt + } + plan.operations.push( + new SaveVerbOperation(this.storage, { + id: reverseId, + vector: relationVector, + connections: new Map(), + verb: params.type, + sourceId: params.to, + targetId: params.from + }), + new SaveVerbMetadataOperation(this.storage, reverseId, verbMetadata), + new AddToGraphIndexOperation(this.graphIndex, reverseVerb, targetInt, sourceInt, (verbInt) => + this.cacheVerbInt(verbInt, reverseId) + ) + ) + plan.touchedVerbs.push(reverseId) + state.verbs.set(reverseId, reverseVerb) + state.removedVerbs.delete(reverseId) + } + + return id + } + + /** Plan one `{ op: 'unrelate' }` — mirror of `unrelate()`. Returns the relationship id. */ + private async planTxUnrelate( + op: Extract, { op: 'unrelate' }>, + state: TxPlanState, + plan: PlannedTransact + ): Promise { + if (!op.id || typeof op.id !== 'string') { + throw new Error(`transact(): unrelate operation requires a relationship id (got ${JSON.stringify(op.id)})`) + } + const id = op.id + + const verb = state.removedVerbs.has(id) + ? null + : (state.verbs.get(id) ?? (await this.storage.getVerb(id))) + + if (verb) { + const { sourceInt, targetInt } = this.resolveVerbEndpointInts(verb) + plan.operations.push( + new RemoveFromGraphIndexOperation(this.graphIndex, verb, sourceInt, targetInt) + ) + } + plan.operations.push(new DeleteVerbMetadataOperation(this.storage, id)) + plan.touchedVerbs.push(id) + + state.verbs.delete(id) + state.removedVerbs.add(id) + return id + } + /** * Get total count of nouns - O(1) operation * @returns Promise that resolves to the total number of nouns @@ -5905,7 +7121,11 @@ export class Brainy implements BrainyInterface { if (this.index && typeof (this.index as any).flush === 'function') { await (this.index as any).flush() } - })() + })(), + + // 5. Persist the generation counter (8.0 MVCC — coalesced single-op + // bumps become durable on every explicit flush) + this.generationStore.persistCounterNow() ]) const elapsed = Date.now() - startTime @@ -9796,6 +11016,12 @@ export class Brainy implements BrainyInterface { if (this._aggregationIndex) { await this._aggregationIndex.flush() } + })(), + // 8.0 MVCC: detach the generation-bump hook and persist the counter + (async () => { + if (this.generationStore) { + await this.generationStore.close() + } })() ]) diff --git a/src/db/db.ts b/src/db/db.ts new file mode 100644 index 00000000..48d1aa53 --- /dev/null +++ b/src/db/db.ts @@ -0,0 +1,806 @@ +/** + * @module db/db + * @description `Db` — Brainy 8.0's immutable, Datomic-style database value. + * + * A `Db` is a **readonly view pinned at one generation** of a Brainy store. + * It is produced by `brain.now()` (O(1) pin of the current generation), + * `brain.transact()` (pinned at the freshly committed generation), + * `brain.asOf()` (a past generation, a timestamp, or a persisted snapshot + * path), `Brainy.load(path)` (a snapshot opened read-only), and `db.with()` + * (a speculative in-memory overlay). + * + * **Read semantics.** While no transaction has committed past the pinned + * generation, every read delegates straight to the live brain — the pinned + * view IS the current state, and the existing fast paths serve it untouched. + * Once later transactions commit, the `Db` resolves reads through the + * generational record layer: `get()` and metadata-level `find()`/`related()` + * remain fully correct at the pinned generation (changed ids are resolved + * from immutable before-images; unchanged ids still ride the live fast + * path), while index-accelerated queries (semantic/vector search) throw the + * documented {@link NotYetSupportedAtHistoricalGenerationError} — an honest + * boundary, never silently-wrong results. + * + * **History granularity.** Generation records are written per `transact()` + * batch. Single-operation writes (`add`/`update`/`relate`/… outside + * `transact()`) advance the generation counter but do not produce historical + * records, so they remain visible through earlier pins — the documented 8.0 + * granularity (see `docs/ADR-001-generational-mvcc.md`, "History + * granularity"). + * + * **Lifecycle.** Each `Db` holds one refcounted pin on its generation (and + * on every registered {@link ../plugin.js VersionedIndexProvider}). Call + * {@link Db.release} when done — a `FinalizationRegistry` backstop releases + * leaked pins on garbage collection, but explicit release is what makes + * `compactHistory()` deterministic. + */ + +import type { + Entity, + FindParams, + GetOptions, + GetRelationsParams, + Relation, + Result +} from '../types/brainy.types.js' +import { v4 as uuidv4 } from '../universal/uuid.js' +import { NotYetSupportedAtHistoricalGenerationError } from './errors.js' +import type { GenerationStore } from './generationStore.js' +import type { ChangedIds, TransactReceipt, TxOperation } from './types.js' +import { entityMatchesFind, resolveEntityField, UnsupportedWhereOperatorError } from './whereMatcher.js' + +/** + * @description The speculative overlay carried by a `db.with()` value: + * per-id replacement entities/relations (`null` = tombstone). Reads on the + * overlay `Db` consult these maps first, then fall through to the pinned + * base view. Overlays are pure in-memory values — they never touch disk or + * index providers. + */ +export interface SpeculativeOverlay { + /** Entity overlays: replacement entity, or `null` for a speculative delete. */ + nouns: Map | null> + /** Relation overlays: replacement relation, or `null` for a speculative delete. */ + verbs: Map | null> +} + +/** + * @description The host surface a `Db` needs from its `Brainy` instance — + * constructed once per brain (see `brainy.ts`). Internal: consumers never + * build one; `Db` values are only created by `Brainy` and `db.with()`. + */ +export interface DbHost { + /** The generational record layer of the host brain. */ + readonly store: GenerationStore + /** Live `brain.get()` (current-generation fast path). */ + get(id: string, options?: GetOptions): Promise | null> + /** Live `brain.find()` (current-generation fast path). */ + find(query: string | FindParams): Promise[]> + /** Live `brain.getRelations()` (current-generation fast path). */ + getRelations(paramsOrId?: string | GetRelationsParams): Promise[]> + /** Materialize an entity from a generation record's raw stored objects. */ + entityFromRecord( + id: string, + record: { metadata: any; vector: any | null }, + includeVectors: boolean + ): Promise> + /** Materialize a relation from a generation record's raw stored objects. */ + relationFromRecord(id: string, record: { metadata: any; vector: any | null }): Relation | null + /** Snapshot the store at `generation` into `targetPath` (throws if the pin is stale). */ + persistPinned(targetPath: string, generation: number): Promise + /** Add one refcounted pin (store + versioned providers) on `generation`. */ + pinGeneration(generation: number): void + /** Release one refcounted pin (store + versioned providers) on `generation`. */ + releaseGeneration(generation: number): void + /** Register a `Db` with the GC-backstop finalization registry. */ + registerDbForFinalization(db: Db, generation: number, closeOnRelease?: () => Promise): void + /** Unregister a `Db` after an explicit release (no double-release on GC). */ + unregisterDbFromFinalization(db: Db): void +} + +/** + * @description Internal constructor arguments for {@link Db}. The pin on + * `generation` is taken by the creator (`Brainy` or `db.with()`) **before** + * construction; the constructor only registers the GC backstop. + */ +export interface DbInit { + /** The host surface (one per brain). */ + host: DbHost + /** The pinned generation. */ + generation: number + /** The view's timestamp (ms since epoch) — see {@link Db.timestamp}. */ + timestamp: number + /** Per-operation receipt, present only on `transact()`-produced values. */ + receipt?: TransactReceipt + /** Speculative overlay, present only on `with()`-produced values. */ + overlay?: SpeculativeOverlay + /** Extra teardown on release (e.g. closing an owned snapshot brain). */ + closeOnRelease?: () => Promise +} + +/** + * @description An immutable, generation-pinned view of a Brainy store. See + * the module documentation for the full read-semantics contract. + * + * @example + * const db = brain.now() // O(1) pin + * await brain.transact([{ op: 'update', id, metadata: { v: 2 } }]) + * await db.get(id) // still sees v: 1 — pinned + * await brain.get(id) // sees v: 2 — live + * await db.release() // unpin (enables compaction) + */ +export class Db { + private readonly host: DbHost + private readonly gen: number + private readonly ts: number + private readonly overlay?: SpeculativeOverlay + private readonly closeOnRelease?: () => Promise + private isReleased = false + + /** + * Per-operation receipt (committed generation, timestamp, resolved ids in + * input order). Present only on values returned by `brain.transact()`. + */ + public readonly receipt?: TransactReceipt + + /** + * @param init - Internal construction arguments. `Db` values are created + * by `Brainy` (`now`/`transact`/`asOf`/`load`) and `db.with()` — never + * directly by consumers. + */ + constructor(init: DbInit) { + this.host = init.host + this.gen = init.generation + this.ts = init.timestamp + this.receipt = init.receipt + this.overlay = init.overlay + this.closeOnRelease = init.closeOnRelease + this.host.registerDbForFinalization(this, this.gen, this.closeOnRelease) + } + + /** The generation this view is pinned at. */ + get generation(): number { + return this.gen + } + + /** + * The view's wall-clock timestamp (ms since epoch): pin time for `now()`, + * commit time for `transact()`, and the resolved commit time for `asOf()`. + */ + get timestamp(): number { + return this.ts + } + + /** Whether {@link Db.release} has been called (released views throw on use). */ + get released(): boolean { + return this.isReleased + } + + /** Whether this view carries a speculative `with()` overlay. */ + get speculative(): boolean { + return this.overlay !== undefined + } + + // ========================================================================== + // Reads + // ========================================================================== + + /** + * @description Get an entity by id **as of this view's generation**. + * Overlay entries win (speculative views); ids untouched since the pin ride + * the live fast path; ids changed by later transactions resolve from + * immutable generation records. Always correct at the pinned generation. + * + * @param id - The entity id. + * @param options - Same options as `brain.get()` (`includeVectors`). + * @returns The entity as of this generation, or `null`. + */ + async get(id: string, options?: GetOptions): Promise | null> { + this.assertUsable('get') + + if (this.overlay && this.overlay.nouns.has(id)) { + return this.overlay.nouns.get(id) ?? null + } + + if (!this.isHistorical()) { + return this.host.get(id, options) + } + + const resolved = await this.host.store.resolveAt('noun', id, this.gen) + if (resolved.source === 'current') { + return this.host.get(id, options) + } + if (resolved.source === 'absent' || resolved.metadata === null) { + // Live semantics: an entity without metadata is not a live entity. + return null + } + return this.host.entityFromRecord( + id, + { metadata: resolved.metadata, vector: resolved.vector }, + options?.includeVectors ?? false + ) + } + + /** + * @description Metadata-level `find()` **as of this view's generation**. + * + * At the current generation (nothing committed past the pin, no overlay) + * this delegates to `brain.find()` untouched — the full query surface, + * including semantic search. At a historical generation, or on a + * speculative view, only the metadata dimensions are supported (`type`, + * `subtype`, `where`, `service`, `excludeVFS`, `orderBy`/`order`, + * `limit`/`offset`): results are computed by combining the live index for + * unchanged ids with per-entity evaluation of the same filters against + * generation records / overlay entries — set-correct at the pinned + * generation. Index-only dimensions (`query`, `vector`, `near`, + * `connected`, `cursor`, `aggregate`, …) throw + * {@link NotYetSupportedAtHistoricalGenerationError}. + * + * Result ordering is deterministic when `orderBy` is supplied; without it, + * overlay/record-resolved matches follow the live-index matches. + * + * @param query - A `FindParams` object, or a string (semantic search — + * current-generation views only). + * @returns Matching results as of this generation (score `1.0` for + * record-resolved metadata matches, mirroring live metadata-only finds). + */ + async find(query: string | FindParams): Promise[]> { + this.assertUsable('find') + + const params: FindParams = typeof query === 'string' ? { query } : query + const historical = this.isHistorical() + + if (!historical && !this.overlay) { + return this.host.find(query) + } + + this.assertMetadataOnlyFind(params) + + const limit = params.limit ?? 10 + const offset = params.offset ?? 0 + + // Ids whose live-index answer is NOT valid at this generation. + const changedNouns = historical + ? (await this.host.store.changedBetween(this.gen, this.host.store.committedGeneration())).nouns + : [] + const overlayNouns = this.overlay?.nouns ?? new Map | null>() + const excluded = new Set([...changedNouns, ...overlayNouns.keys()]) + + try { + // Over-fetch from the live index so dropping superseded ids cannot + // starve the requested page, then merge and re-window. + const base = await this.host.find({ + ...params, + limit: limit + offset + excluded.size, + offset: 0 + }) + const merged = base.filter((result) => !excluded.has(result.id)) + + for (const id of changedNouns) { + if (overlayNouns.has(id)) continue // The overlay supersedes history. + const resolved = await this.host.store.resolveAt('noun', id, this.gen) + let entity: Entity | null = null + if (resolved.source === 'record' && resolved.metadata !== null) { + entity = await this.host.entityFromRecord( + id, + { metadata: resolved.metadata, vector: resolved.vector }, + false + ) + } else if (resolved.source === 'current') { + // Defensive: changed ids always resolve to a record or absent, but + // a 'current' answer is still served correctly from the live path. + entity = await this.host.get(id) + } + if (entity && entityMatchesFind(entity as Entity, params as FindParams)) { + merged.push(resultFromEntity(entity)) + } + } + + for (const [, entity] of overlayNouns) { + if (entity === null) continue // Speculative delete. + if (entityMatchesFind(entity as Entity, params as FindParams)) { + merged.push(resultFromEntity(entity)) + } + } + + if (params.orderBy) { + sortResultsBy(merged, params.orderBy, params.order ?? 'asc') + } + return merged.slice(offset, offset + limit) + } catch (err) { + if (err instanceof UnsupportedWhereOperatorError) { + throw new NotYetSupportedAtHistoricalGenerationError( + `metadata find with where-operator '${err.operator}'`, + this.gen, + this.host.store.generation() + ) + } + throw err + } + } + + /** + * @description Semantic (vector) search **at the current generation only**. + * Convenience for `find({ query, ...options })`. Vector indexes serve the + * live state; at a historical generation, or on a speculative overlay + * (whose entities carry no embeddings), this throws the documented + * {@link NotYetSupportedAtHistoricalGenerationError} — persist the + * generation you need and `Brainy.load()` it for the full query surface. + * + * @param query - Natural-language search query. + * @param options - Additional `FindParams` (limit, filters, …). + * @returns Semantic search results (current-generation views only). + * @throws NotYetSupportedAtHistoricalGenerationError at historical + * generations and on speculative overlays. + */ + async search(query: string, options?: Omit, 'query'>): Promise[]> { + this.assertUsable('search') + + if (!this.isHistorical() && !this.overlay) { + return this.host.find({ ...(options ?? {}), query }) + } + throw new NotYetSupportedAtHistoricalGenerationError( + this.overlay ? 'vector search (speculative overlay)' : 'vector search', + this.gen, + this.host.store.generation() + ) + } + + /** + * @description Relationships **as of this view's generation** — the `Db` + * counterpart of `brain.getRelations()` (same string-id shorthand and + * filter surface). Relations whose edges changed after the pin are + * resolved from generation records; overlay relations (speculative + * `relate`/`unrelate`) and cascade tombstones (relations touching a + * speculatively deleted entity) are applied on top. Cursor pagination is + * index-only and throws the documented historical error on + * non-current/speculative views. + * + * @param paramsOrId - An entity id (shorthand for `{ from: id }`) or a + * `GetRelationsParams` filter object. + * @returns Relations as of this generation. + */ + async related(paramsOrId?: string | GetRelationsParams): Promise[]> { + this.assertUsable('related') + + const params: GetRelationsParams = + typeof paramsOrId === 'string' ? { from: paramsOrId } : (paramsOrId ?? {}) + const historical = this.isHistorical() + + if (!historical && !this.overlay) { + return this.host.getRelations(paramsOrId) + } + + if (params.cursor !== undefined) { + throw new NotYetSupportedAtHistoricalGenerationError( + 'cursor pagination on related()', + this.gen, + this.host.store.generation() + ) + } + + const limit = params.limit ?? 100 + const offset = params.offset ?? 0 + + const changedVerbs = historical + ? (await this.host.store.changedBetween(this.gen, this.host.store.committedGeneration())).verbs + : [] + const overlayVerbs = this.overlay?.verbs ?? new Map | null>() + const excluded = new Set([...changedVerbs, ...overlayVerbs.keys()]) + + const base = await this.host.getRelations({ + ...params, + limit: limit + offset + excluded.size, + offset: 0 + }) + let merged = base.filter((relation) => !excluded.has(relation.id)) + + for (const id of changedVerbs) { + if (overlayVerbs.has(id)) continue + const resolved = await this.host.store.resolveAt('verb', id, this.gen) + if (resolved.source !== 'record') continue + const relation = this.host.relationFromRecord(id, { + metadata: resolved.metadata, + vector: resolved.vector + }) + if (relation && relationMatchesParams(relation, params)) { + merged.push(relation) + } + } + + for (const [, relation] of overlayVerbs) { + if (relation === null) continue // Speculative unrelate. + if (relationMatchesParams(relation, params)) { + merged.push(relation) + } + } + + // Cascade semantics for speculative deletes: a relation whose endpoint + // is tombstoned in the overlay is gone in this view, exactly as + // brain.delete() cascades on the durable path. + if (this.overlay) { + const tombstoned = new Set() + for (const [id, entity] of this.overlay.nouns) { + if (entity === null) tombstoned.add(id) + } + if (tombstoned.size > 0) { + merged = merged.filter( + (relation) => !tombstoned.has(relation.from) && !tombstoned.has(relation.to) + ) + } + } + + return merged.slice(offset, offset + limit) + } + + // ========================================================================== + // Derived views + // ========================================================================== + + /** + * @description Speculative write: returns a NEW `Db` whose reads see the + * given transaction data applied **in memory, on top of this view** — + * Datomic's `with`. Nothing touches disk, the generation counter, or index + * providers; the underlying brain and this view are untouched. Use it to + * answer "what would the store look like if…" questions, then call + * `brain.transact()` with the same operations to make it real. + * + * Speculative semantics (documented deltas from the durable paths): + * - `add` without an explicit `vector` produces an entity with an empty + * stub vector — speculative entities are not semantically searchable + * (`with()` never invokes the embedder). + * - `remove` cascades on the read side: relations touching the removed + * entity disappear from `related()` without being enumerated. + * - `relate` deduplicates against edges visible in this view when verb + * history allows the check, mirroring `brain.relate()`. + * - Brain-level vocabulary/subtype enforcement does not run — overlays + * never persist, so there is nothing to protect. + * + * The returned view takes its own pin on this generation; release both + * views independently. + * + * @param ops - The same declarative operations `brain.transact()` accepts. + * @returns A new speculative `Db` over this view. + * @throws Error when an operation references a missing entity/relation, + * mirroring the durable path's planning errors. + */ + async with(ops: TxOperation[]): Promise> { + this.assertUsable('with') + if (!Array.isArray(ops) || ops.length === 0) { + throw new Error('with(): provide a non-empty array of transaction operations') + } + + // Child overlay starts as a copy of this view's overlay (chained with()). + const overlay: SpeculativeOverlay = { + nouns: new Map(this.overlay?.nouns ?? []), + verbs: new Map(this.overlay?.verbs ?? []) + } + + // Reads during planning must see "this view + overlay so far". + const speculativeGet = async (id: string): Promise | null> => { + if (overlay.nouns.has(id)) return overlay.nouns.get(id) ?? null + return this.get(id) + } + + for (const op of ops) { + switch (op.op) { + case 'add': { + const id = op.id ?? uuidv4() + const now = Date.now() + overlay.nouns.set(id, { + id, + vector: op.vector ?? [], + type: op.type, + ...(op.subtype !== undefined && { subtype: op.subtype }), + data: op.data, + metadata: (op.metadata ?? {}) as T, + ...(op.service !== undefined && { service: op.service }), + createdAt: now, + updatedAt: now, + ...(op.confidence !== undefined && { confidence: op.confidence }), + ...(op.weight !== undefined && { weight: op.weight }), + _rev: 1 + }) + break + } + case 'update': { + const base = await speculativeGet(op.id) + if (!base) { + throw new Error(`with(): entity ${op.id} not found at generation ${this.gen}`) + } + const mergedMetadata = + op.merge !== false + ? ({ ...(base.metadata as object), ...(op.metadata as object) } as T) + : ((op.metadata ?? base.metadata) as T) + overlay.nouns.set(op.id, { + ...base, + ...(op.type !== undefined && { type: op.type }), + ...(op.subtype !== undefined && { subtype: op.subtype }), + ...(op.data !== undefined && { data: op.data }), + ...(op.vector !== undefined && { vector: op.vector }), + ...(op.confidence !== undefined && { confidence: op.confidence }), + ...(op.weight !== undefined && { weight: op.weight }), + metadata: mergedMetadata, + updatedAt: Date.now(), + _rev: (base._rev ?? 1) + 1 + }) + break + } + case 'remove': { + overlay.nouns.set(op.id, null) + // Tombstone overlay relations touching the removed entity; base + // relations are cascade-filtered at read time in related(). + for (const [verbId, relation] of overlay.verbs) { + if (relation && (relation.from === op.id || relation.to === op.id)) { + overlay.verbs.set(verbId, null) + } + } + break + } + case 'relate': { + const from = await speculativeGet(op.from) + const to = await speculativeGet(op.to) + if (!from) throw new Error(`with(): source entity ${op.from} not found`) + if (!to) throw new Error(`with(): target entity ${op.to} not found`) + + // Dedupe against the view (overlay first, then committed edges + // when verb history allows the read) — mirror of relate(). + let duplicate: Relation | undefined + for (const relation of overlay.verbs.values()) { + if (relation && relation.from === op.from && relation.to === op.to && relation.type === op.type) { + duplicate = relation + break + } + } + if (!duplicate) { + try { + const existing = await this.related({ from: op.from, type: op.type }) + duplicate = existing.find((relation) => relation.to === op.to) + } catch (err) { + if (!(err instanceof NotYetSupportedAtHistoricalGenerationError)) throw err + // Verb history can't serve the check at this generation — + // proceed with overlay-only dedupe (documented). + } + } + if (duplicate) break + + const id = uuidv4() + overlay.verbs.set(id, { + id, + from: op.from, + to: op.to, + type: op.type, + ...(op.subtype !== undefined && { subtype: op.subtype }), + weight: op.weight ?? 1.0, + ...(op.data !== undefined && { data: op.data }), + metadata: op.metadata, + ...(op.service !== undefined && { service: op.service }), + createdAt: Date.now() + }) + if (op.bidirectional) { + const reverseId = uuidv4() + overlay.verbs.set(reverseId, { + id: reverseId, + from: op.to, + to: op.from, + type: op.type, + ...(op.subtype !== undefined && { subtype: op.subtype }), + weight: op.weight ?? 1.0, + ...(op.data !== undefined && { data: op.data }), + metadata: op.metadata, + ...(op.service !== undefined && { service: op.service }), + createdAt: Date.now() + }) + } + break + } + case 'unrelate': { + overlay.verbs.set(op.id, null) + break + } + default: { + const exhaustive: never = op + throw new Error(`with(): unknown operation ${JSON.stringify(exhaustive)}`) + } + } + } + + this.host.pinGeneration(this.gen) + return new Db({ + host: this.host, + generation: this.gen, + timestamp: Date.now(), + overlay + }) + } + + /** + * @description The ids changed by committed transactions in the interval + * `(priorDb.generation, this.generation]` — the diff between two pinned + * views of the same store. Single-operation writes between commits are not + * reflected (documented 8.0 history granularity). + * + * @param priorDb - An earlier view of the SAME store. + * @returns Changed entity and relation ids, sorted. + * @throws RangeError when `priorDb` is newer than this view, or Error when + * the views belong to different stores. + */ + async since(priorDb: Db): Promise { + this.assertUsable('since') + if (priorDb.host.store !== this.host.store) { + throw new Error('since(): both Db values must come from the same Brainy instance') + } + if (priorDb.gen > this.gen) { + throw new RangeError( + `since(): priorDb is at generation ${priorDb.gen}, which is newer than this view ` + + `(generation ${this.gen}). Call newerDb.since(olderDb).` + ) + } + return this.host.store.changedBetween(priorDb.gen, this.gen) + } + + // ========================================================================== + // Durability + // ========================================================================== + + /** + * @description Snapshot this view to `path` as a self-contained store: + * flush, then hard-link every immutable file into the target + * (Cassandra-style — instant and space-shared; cross-device targets fall + * back to byte copies, and the small append-in-place file list is always + * byte-copied). The result opens with `Brainy.load(path)` with the full + * query surface, and later mutations of the source can never alter it — + * rewrites swap inodes, the snapshot keeps the old bytes. + * + * `persist()` requires this view to still be the store's **latest** + * generation (snapshotting captures current bytes): pin with `brain.now()` + * and persist before further writes. A view that history has moved past + * throws {@link NotYetSupportedAtHistoricalGenerationError}; speculative + * overlays cannot be persisted (commit them with `brain.transact()` first). + * + * @param path - Absolute directory for the snapshot (created; must be + * empty or absent). + * @throws NotYetSupportedAtHistoricalGenerationError when this view is no + * longer the latest generation, or is speculative. + */ + async persist(path: string): Promise { + this.assertUsable('persist') + if (this.overlay) { + throw new NotYetSupportedAtHistoricalGenerationError( + 'persist of a speculative overlay', + this.gen, + this.host.store.generation() + ) + } + await this.host.persistPinned(path, this.gen) + } + + // ========================================================================== + // Lifecycle + // ========================================================================== + + /** + * @description Release this view's pin (store + versioned index + * providers). Idempotent. After release, every read throws. Views from + * `Brainy.load()` / `asOf(path)` also close their underlying read-only + * brain here. A `FinalizationRegistry` backstop releases leaked pins when + * a `Db` is garbage-collected, but explicit release is what makes + * `compactHistory()` deterministic — prefer it. + */ + async release(): Promise { + if (this.isReleased) return + this.isReleased = true + this.host.unregisterDbFromFinalization(this) + this.host.releaseGeneration(this.gen) + if (this.closeOnRelease) { + await this.closeOnRelease() + } + } + + // ========================================================================== + // Internals + // ========================================================================== + + /** Whether any transaction committed past the pinned generation. */ + private isHistorical(): boolean { + return this.host.store.hasCommittedAfter(this.gen) + } + + /** Throw on use-after-release. */ + private assertUsable(method: string): void { + if (this.isReleased) { + throw new Error( + `Cannot call ${method}() on a released Db (generation ${this.gen}). ` + + `Pin a fresh view with brain.now() or brain.asOf().` + ) + } + } + + /** + * Reject find() dimensions that require index machinery the record layer + * cannot answer at a historical generation / for a speculative overlay. + */ + private assertMetadataOnlyFind(params: FindParams): void { + const unsupported: Array<[string, boolean]> = [ + ['semantic query', params.query !== undefined], + ['vector search', params.vector !== undefined], + ['proximity search (near)', params.near !== undefined], + ['graph traversal (connected)', params.connected !== undefined], + ['cursor pagination', params.cursor !== undefined], + ['aggregation', params.aggregate !== undefined], + ['relation expansion (includeRelations)', params.includeRelations === true], + [ + `search mode '${params.mode ?? params.searchMode}'`, + (params.mode !== undefined && params.mode !== 'metadata' && params.mode !== 'auto') || + (params.searchMode !== undefined && params.searchMode !== 'auto') + ] + ] + for (const [capability, present] of unsupported) { + if (present) { + throw new NotYetSupportedAtHistoricalGenerationError( + capability, + this.gen, + this.host.store.generation() + ) + } + } + } +} + +/** + * @description Build a `Result` row from a record/overlay-resolved entity, + * mirroring the live metadata-only find path (flattened fields, score `1.0`). + */ +function resultFromEntity(entity: Entity): Result { + return { + id: entity.id, + score: 1.0, + type: entity.type, + subtype: entity.subtype, + metadata: entity.metadata, + data: entity.data, + confidence: entity.confidence, + weight: entity.weight, + _rev: entity._rev, + entity + } +} + +/** + * @description In-place stable sort of merged results by a find() `orderBy` + * field, mirroring live ordering semantics (`asc` default, undefined values + * last in both directions). + */ +function sortResultsBy(results: Result[], orderBy: string, order: 'asc' | 'desc'): void { + const direction = order === 'desc' ? -1 : 1 + results.sort((a, b) => { + const va = resolveEntityField(a.entity as Entity, orderBy) + const vb = resolveEntityField(b.entity as Entity, orderBy) + if (va === undefined && vb === undefined) return 0 + if (va === undefined) return 1 + if (vb === undefined) return -1 + if (typeof va === 'number' && typeof vb === 'number') return (va - vb) * direction + const sa = String(va) + const sb = String(vb) + return (sa < sb ? -1 : sa > sb ? 1 : 0) * direction + }) +} + +/** + * @description Filter one relation against `GetRelationsParams`, mirroring + * the storage-level filter `brain.getRelations()` builds (`from` → sourceId, + * `to` → targetId, type/subtype set membership, service equality). + */ +function relationMatchesParams(relation: Relation, params: GetRelationsParams): boolean { + if (params.from !== undefined && relation.from !== params.from) return false + if (params.to !== undefined && relation.to !== params.to) return false + if (params.type !== undefined) { + const types = Array.isArray(params.type) ? params.type : [params.type] + if (!types.includes(relation.type)) return false + } + if (params.subtype !== undefined) { + const subtypes = Array.isArray(params.subtype) ? params.subtype : [params.subtype] + if (relation.subtype === undefined || !subtypes.includes(relation.subtype)) return false + } + if (params.service !== undefined && relation.service !== params.service) return false + return true +} + diff --git a/src/db/errors.ts b/src/db/errors.ts new file mode 100644 index 00000000..f369c1e3 --- /dev/null +++ b/src/db/errors.ts @@ -0,0 +1,146 @@ +/** + * @module db/errors + * @description Error types for the 8.0 generational-MVCC Db API. + * + * Three failure modes have dedicated classes so callers can branch on them + * with `instanceof` instead of string matching: + * + * - {@link GenerationConflictError} — a `transact()` compare-and-swap + * (`ifAtGeneration`) observed a different current generation than the + * caller expected. The standard retry pattern is: re-read via + * `brain.now()`, re-derive the transaction, and re-submit. + * - {@link NotYetSupportedAtHistoricalGenerationError} — an index-accelerated + * query (vector / hybrid / graph-traversal search) was issued against a + * `Db` pinned at a generation the live indexes no longer represent. + * `get()` and metadata-filter `find()` remain fully supported at pinned + * generations; index-accelerated queries at historical generations require + * an index rebuild from the pinned record set, which ships as a follow-up + * (`asOf` rebuild). The error message names the unsupported capability and + * points at the supported alternatives. + * - {@link GenerationCompactedError} — `asOf()` asked for a generation whose + * immutable records were reclaimed by `compactHistory()`. + * + * All three are exported from the package root (`@soulcraft/brainy`). + */ + +/** + * @description Thrown by `brain.transact(ops, { ifAtGeneration })` when the + * store's current generation does not equal the caller-supplied expectation. + * This is the whole-store compare-and-swap counterpart to the per-entity + * `ifRev` / `RevisionConflictError` pair: it guarantees that *nothing* was + * committed between the caller's read (`brain.now()`) and this transaction. + * + * The transaction is rejected before any record is staged — the store is + * untouched and the generation counter is unchanged. + * + * @example + * const db = brain.now() + * try { + * await brain.transact(ops, { ifAtGeneration: db.generation }) + * } catch (err) { + * if (err instanceof GenerationConflictError) { + * // Someone committed since we pinned — re-read and retry. + * console.log(`expected ${err.expected}, store is at ${err.actual}`) + * } + * } + */ +export class GenerationConflictError extends Error { + /** The generation the caller expected the store to be at. */ + public readonly expected: number + /** The generation the store was actually at when the transaction arrived. */ + public readonly actual: number + + /** + * @param expected - The generation supplied via `ifAtGeneration`. + * @param actual - The store's current generation at submission time. + */ + constructor(expected: number, actual: number) { + super( + `Generation conflict: transact() was submitted with ifAtGeneration: ${expected}, ` + + `but the store is at generation ${actual}. Another write committed in between. ` + + `Re-read with brain.now(), rebuild the transaction, and retry.` + ) + this.name = 'GenerationConflictError' + this.expected = expected + this.actual = actual + } +} + +/** + * @description Thrown when an operation on a pinned `Db` requires + * index-accelerated machinery (vector search, hybrid search, graph + * traversal, cursor pagination, aggregation) that the live indexes cannot + * answer for a *historical* generation. Storage-record reads (`get()`) and + * metadata-filter `find()` are always supported at pinned generations. + * + * Rebuilding indexes from a pinned record set (the standard LSM answer for + * historical index queries) is the documented `asOf`-rebuild follow-up; this + * error is the honest boundary until it ships. + * + * @example + * const db = brain.now() + * await brain.transact([{ op: 'update', id, metadata: { v: 2 } }]) + * await db.get(id) // ✅ pinned read — supported + * await db.find({ where: { v: 1 } }) // ✅ metadata find — supported + * await db.search('semantic query') // ❌ throws this error + */ +export class NotYetSupportedAtHistoricalGenerationError extends Error { + /** The pinned generation the operation was attempted against. */ + public readonly generation: number + /** The capability that is not supported at historical generations. */ + public readonly capability: string + + /** + * @param capability - Human-readable name of the unsupported capability + * (e.g. `'vector search'`, `'graph traversal (connected)'`). + * @param generation - The Db's pinned generation. + * @param currentGeneration - The store's current generation, for context. + */ + constructor(capability: string, generation: number, currentGeneration: number) { + super( + `${capability} is not yet supported at a historical generation ` + + `(Db pinned at generation ${generation}; store is at ${currentGeneration}). ` + + `Supported at pinned generations: get() and metadata-filter find(). ` + + `For index-accelerated queries over historical state, persist() the generation ` + + `you need and open it with Brainy.load(path) — the loaded snapshot rebuilds ` + + `its indexes and supports the full query surface.` + ) + this.name = 'NotYetSupportedAtHistoricalGenerationError' + this.capability = capability + this.generation = generation + } +} + +/** + * @description Thrown by `brain.asOf(generationOrTimestamp)` when the + * requested generation's history was reclaimed by `brain.compactHistory()`. + * The store records the compaction horizon (the highest reclaimed + * generation) in its manifest; any generation BELOW the horizon is + * unreachable — its reads would need the reclaimed before-images. The + * horizon itself stays reachable, resolved from the record-sets above it. + * + * To keep a generation readable forever, `persist()` it to a snapshot + * directory before compacting — snapshots are self-contained and unaffected + * by compaction of the source store. + */ +export class GenerationCompactedError extends Error { + /** The generation that was requested. */ + public readonly requested: number + /** The compaction horizon — generations < this value are unreachable. */ + public readonly horizon: number + + /** + * @param requested - The generation the caller asked for. + * @param horizon - The store's current compaction horizon. + */ + constructor(requested: number, horizon: number) { + super( + `Generation ${requested} has been compacted away (compaction horizon: ${horizon}). ` + + `Only generations at or above the horizon are reachable via asOf(). ` + + `Use db.persist(path) before compactHistory() to keep a generation readable.` + ) + this.name = 'GenerationCompactedError' + this.requested = requested + this.horizon = horizon + } +} diff --git a/src/db/generationStore.ts b/src/db/generationStore.ts new file mode 100644 index 00000000..e56d43c7 --- /dev/null +++ b/src/db/generationStore.ts @@ -0,0 +1,838 @@ +/** + * @module db/generationStore + * @description The generational record layer behind Brainy 8.0's MVCC Db API. + * + * One `GenerationStore` is attached to every writable `Brainy` instance. It + * owns: + * + * - the **monotonic generation counter** (`_system/generation.json`), + * reserved per `transact()` commit and bumped per single-operation write + * batch via the storage-layer hook; + * - the **commit protocol** for `brain.transact()`: capture before-images → + * write the generation delta → fsync → execute the operation batch → + * atomically rename `_system/manifest.json` (the rename IS the commit + * point) → append the tx-log line; + * - **crash recovery**: on open, any `_generations/` directory with + * `N > manifest.generation` is an uncommitted transaction — its + * before-images are restored to the canonical entity paths and the + * directory is removed, so a crash anywhere before the manifest rename + * leaves the store byte-identical to its pre-transaction state; + * - **pin/release refcounting** for live `Db` values, which is what makes + * `compactHistory()` safe: a generation record-set is reclaimed only when + * no live pin could ever need it; + * - **point-in-time resolution**: the state of id X at pinned generation G + * is the before-image stored by the *first* committed generation after G + * that touched X — or the live storage state when nothing after G touched + * it (so current-generation reads stay on the existing fast paths, + * untouched). + * + * Durability and atomicity guarantees, failure modes, and the intellectual + * lineage (Datomic database-as-value, LMDB reader pins, LSM immutable + * segments) are specified in `docs/ADR-001-generational-mvcc.md`. + */ + +import { prodLog } from '../utils/logger.js' +import { GenerationCompactedError, GenerationConflictError } from './errors.js' +import type { + ChangedIds, + CompactHistoryOptions, + CompactHistoryResult, + GenerationDelta, + GenerationManifest, + GenerationRecord, + GenerationStorage, + TxLogEntry +} from './types.js' + +/** Storage-root-relative path of the persisted generation counter. */ +export const GENERATION_COUNTER_PATH = '_system/generation.json' +/** Storage-root-relative path of the commit manifest. */ +export const MANIFEST_PATH = '_system/manifest.json' +/** Storage-root-relative prefix of the per-generation record directories. */ +export const GENERATIONS_PREFIX = '_generations' + +/** + * @description Phases of the {@link GenerationStore.commitTransaction} commit + * protocol at which a test-only fault injector can simulate a process crash. + * The phases map 1:1 onto the protocol steps documented on + * `commitTransaction`: + * + * - `'after-staging'` — before-images + the `tx.json` delta are written and + * fsynced; the operation batch has NOT executed yet. + * - `'after-execute'` — the operation batch has been applied to canonical + * storage and indexes; the manifest still points at the prior generation. + * - `'before-manifest-rename'` — the batch has executed and the counter is + * persisted; the atomic manifest rename — the commit point — has NOT + * happened. A crash here is the canonical "fully staged, never committed" + * case that recovery must roll back byte-identically. + * - `'after-manifest-rename'` — the manifest rename landed (the transaction + * IS committed); the tx-log append has NOT happened yet. A crash here must + * keep the transaction (the tx-log is advisory metadata, not the source of + * commit truth). + */ +export type CommitFaultPhase = + | 'after-staging' + | 'after-execute' + | 'before-manifest-rename' + | 'after-manifest-rename' + +/** + * @description Identifies which ids a transaction touches, split by kind. + */ +export interface TouchedIds { + /** Entity ids the transaction writes or deletes. */ + nouns: string[] + /** Relationship ids the transaction writes or deletes. */ + verbs: string[] +} + +/** + * @description Result of {@link GenerationStore.open} — how many uncommitted + * generations crash recovery rolled back (a non-zero value tells `Brainy` to + * force an index reconciliation, since derived index state may reference the + * rolled-back writes). + */ +export interface GenerationStoreOpenResult { + /** Count of uncommitted generation directories rolled back and removed. */ + rolledBackGenerations: number +} + +/** + * @description The generational MVCC record layer. See the module + * documentation for the full protocol; every public method documents its + * own contract. + */ +export class GenerationStore { + private readonly storage: GenerationStorage + + /** Latest reserved/observed generation (≥ {@link committed}). */ + private counter = 0 + /** Committed-transaction watermark (manifest generation). */ + private committed = 0 + /** Compaction horizon — record-sets ≤ this are reclaimed. */ + private horizonGen = 0 + + /** Sorted list of committed generations whose record dirs exist. */ + private committedGens: number[] = [] + /** Delta cache, keyed by generation (lazily loaded from `tx.json`). */ + private readonly deltaCache = new Map< + number, + { nouns: Set; verbs: Set; timestamp: number } + >() + + /** Live pin refcounts, keyed by pinned generation. */ + private readonly pins = new Map() + + /** Serializes transact commits, compaction, and counter persistence. */ + private mutexTail: Promise = Promise.resolve() + + /** True while a transact batch is executing (suppresses the bump hook). */ + private inTransact = false + + /** Coalescing state for lazy persistence of single-op counter bumps. */ + private counterPersistScheduled = false + + /** + * Test-only fault injector for the commit protocol (see + * {@link GenerationStore.setCommitFaultInjector}). `undefined` in production. + */ + private commitFaultInjector?: (phase: CommitFaultPhase) => void + + private opened = false + + /** + * @param storage - The storage adapter, narrowed to the + * {@link GenerationStorage} contract (`BaseStorage` implements it). + */ + constructor(storage: GenerationStorage) { + this.storage = storage + } + + // ========================================================================== + // Lifecycle + // ========================================================================== + + /** + * @description Load the persisted counter + manifest and run crash + * recovery. Must be called once, before indexes are built, because + * recovery may rewrite canonical entity files (restoring before-images of + * an uncommitted transaction). + * + * @param options.readOnly - When true (reader-mode brains), recovery is + * skipped: readers never write. An un-repaired crashed directory is + * repaired by the next *writer* open. + * @returns How many uncommitted generations were rolled back. + */ + async open(options?: { readOnly?: boolean }): Promise { + const counterFile = (await this.storage.readRawObject(GENERATION_COUNTER_PATH)) as + | { generation?: number } + | null + const manifest = (await this.storage.readRawObject(MANIFEST_PATH)) as GenerationManifest | null + + this.committed = manifest?.generation ?? 0 + this.horizonGen = manifest?.horizon ?? 0 + this.counter = Math.max(counterFile?.generation ?? 0, this.committed) + + // Discover existing generation record directories. + const recordPaths = await this.storage.listRawObjects(GENERATIONS_PREFIX) + const seenGens = new Set() + for (const p of recordPaths) { + const gen = parseGenerationFromPath(p) + if (gen !== null) seenGens.add(gen) + } + + let rolledBack = 0 + const committedGens: number[] = [] + for (const gen of [...seenGens].sort((a, b) => a - b)) { + if (gen <= this.committed) { + committedGens.push(gen) + } else if (!options?.readOnly) { + // Uncommitted (crashed) transaction: restore before-images, drop the dir. + await this.rollBackUncommittedGeneration(gen) + rolledBack++ + } + // Reader mode: leave orphan dirs alone; resolution ignores them because + // committedGens only includes generations ≤ the manifest watermark. + this.counter = Math.max(this.counter, gen) + } + this.committedGens = committedGens + + if (rolledBack > 0) { + prodLog.warn( + `[GenerationStore] Crash recovery rolled back ${rolledBack} uncommitted ` + + `transaction(s); store restored to generation ${this.committed}.` + ) + } + + this.opened = true + + // Hook single-op write batches so generation() is always meaningful. + // Suppressed while a transact batch executes (the batch is ONE generation). + if (!options?.readOnly) { + this.storage.setGenerationBumpHook(() => this.noteSingleOpWrite()) + } + + return { rolledBackGenerations: rolledBack } + } + + /** + * @description Detach the storage hook and persist the counter. Called + * from `brain.close()`. + */ + async close(): Promise { + this.storage.setGenerationBumpHook(undefined) + await this.persistCounterNow() + } + + /** + * @description TEST-ONLY: install (or clear, with `undefined`) a fault + * injector that is invoked at each {@link CommitFaultPhase} of the commit + * protocol. A **throwing** injector simulates a process crash at that + * phase: the abort cleanup (staging-directory removal, generation-counter + * reservation return) is deliberately skipped — exactly as if the process + * had died — so crash-consistency tests exercise the REAL recovery path + * ({@link GenerationStore.open} rolling back the uncommitted generation on + * the next open). Never installed in production code; the injector exists + * solely so the durability protocol can be proven, not trusted. + * + * @param injector - Callback invoked synchronously at each phase, or + * `undefined` to detach. + */ + setCommitFaultInjector(injector: ((phase: CommitFaultPhase) => void) | undefined): void { + this.commitFaultInjector = injector + } + + // ========================================================================== + // Generation counter + // ========================================================================== + + /** @returns The store's current generation (latest write watermark). */ + generation(): number { + return this.counter + } + + /** @returns The committed-transaction watermark (manifest generation). */ + committedGeneration(): number { + return this.committed + } + + /** @returns The compaction horizon (generations ≤ horizon are reclaimed). */ + horizon(): number { + return this.horizonGen + } + + /** + * @description Single-operation write hook (registered with the storage + * layer in {@link open}). Bumps the in-memory counter and schedules a + * coalesced persist of `_system/generation.json` — durable artifacts + * (records, manifests, snapshots) always persist the counter + * synchronously at their own commit points, so a crash inside the + * coalescing window can never move the counter backwards relative to + * anything durable. + */ + private noteSingleOpWrite(): void { + if (this.inTransact) return + this.counter++ + this.schedulePersistCounter() + } + + /** Schedule one coalesced counter persist for the current write burst. */ + private schedulePersistCounter(): void { + if (this.counterPersistScheduled) return + this.counterPersistScheduled = true + setImmediate(() => { + this.counterPersistScheduled = false + void this.withMutex(() => this.persistCounterUnlocked()).catch((err) => { + prodLog.warn(`[GenerationStore] generation counter persist failed: ${(err as Error).message}`) + }) + }) + } + + /** + * @description Persist the generation counter now (atomic tmp+rename). + * Called from `flush()`, `close()`, snapshotting, and the commit protocol. + */ + async persistCounterNow(): Promise { + if (!this.opened) return + await this.withMutex(() => this.persistCounterUnlocked()) + } + + private async persistCounterUnlocked(): Promise { + await this.storage.writeRawObject(GENERATION_COUNTER_PATH, { + generation: this.counter, + updatedAt: new Date().toISOString() + }) + } + + // ========================================================================== + // Pins + // ========================================================================== + + /** + * @description Pin a generation (refcounted). Compaction never reclaims a + * record-set a live pin could need (i.e. any generation directory `N` + * with `N > G` for some pinned `G`). + * @param gen - The generation to pin. + */ + pin(gen: number): void { + this.pins.set(gen, (this.pins.get(gen) ?? 0) + 1) + } + + /** + * @description Release one pin on a generation. Safe to call for a + * generation with no live pins (no-op with a warning). + * @param gen - The generation to release. + */ + release(gen: number): void { + const count = this.pins.get(gen) + if (count === undefined) { + prodLog.warn(`[GenerationStore] release(${gen}) called with no live pin at that generation`) + return + } + if (count <= 1) this.pins.delete(gen) + else this.pins.set(gen, count - 1) + } + + /** @returns Total number of live pins across all generations. */ + activePinCount(): number { + let total = 0 + for (const count of this.pins.values()) total += count + return total + } + + /** @returns The smallest pinned generation, or `Infinity` with no pins. */ + private minPinnedGeneration(): number { + let min = Infinity + for (const gen of this.pins.keys()) min = Math.min(min, gen) + return min + } + + // ========================================================================== + // Commit protocol + // ========================================================================== + + /** + * @description Commit one transaction. The caller (Brainy.transact) plans + * the batch up front — resolving every touched id, including generated + * entity/relationship ids and delete cascades — and hands this method the + * touched-id sets plus an `execute` closure that runs the already-built + * operation batch through the TransactionManager. + * + * Protocol (under the commit mutex): + * 1. `ifAtGeneration` CAS check (rejects before anything is staged). + * 2. Reserve generation `N` (counter increment). + * 3. Capture before-images of every touched id and write them, plus the + * `tx.json` delta, under `_generations/N/`; fsync. From this point a + * crash anywhere is recoverable to the exact pre-transaction bytes. + * 4. Run `execute()`. On failure the TransactionManager has already + * rolled back its applied operations; the staging directory is removed + * and the reservation is returned (when no concurrent bump consumed a + * later number), so a failed transaction leaves the generation + * unchanged. + * 5. Persist the counter, then atomically rename the manifest to + * generation `N` — **the rename is the commit point** — and fsync it. + * 6. Append the tx-log line and update in-memory bookkeeping. + * + * @param args.touched - Every id the batch writes or deletes. + * @param args.meta - Transaction metadata for the tx-log. + * @param args.ifAtGeneration - Whole-store CAS expectation. + * @param args.execute - Runs the planned operation batch atomically. + * @returns The committed generation and its commit timestamp. + * @throws GenerationConflictError when the CAS expectation fails. + */ + async commitTransaction(args: { + touched: TouchedIds + meta?: Record + ifAtGeneration?: number + execute: () => Promise + }): Promise<{ generation: number; timestamp: number }> { + return this.withMutex(async () => { + if (args.ifAtGeneration !== undefined && args.ifAtGeneration !== this.counter) { + throw new GenerationConflictError(args.ifAtGeneration, this.counter) + } + + const nouns = [...new Set(args.touched.nouns)] + const verbs = [...new Set(args.touched.verbs)] + const gen = ++this.counter + const dir = `${GENERATIONS_PREFIX}/${gen}` + const timestamp = Date.now() + + // Test-only crash simulation (see setCommitFaultInjector): a throwing + // injector must bypass the abort cleanup below, exactly as a real + // process death would, so recovery-on-open is what restores the store. + let crashSimulated = false + const faultPoint = (phase: CommitFaultPhase): void => { + if (!this.commitFaultInjector) return + try { + this.commitFaultInjector(phase) + } catch (err) { + crashSimulated = true + throw err + } + } + + try { + // -- 3. Before-images + delta (the durable undo log) ------------------ + const stagedPaths: string[] = [] + for (const id of nouns) { + const prev = await this.storage.readNounRaw(id) + const recordPath = `${dir}/prev/${id}.json` + const record: GenerationRecord = { kind: 'noun', metadata: prev.metadata, vector: prev.vector } + await this.storage.writeRawObject(recordPath, record) + stagedPaths.push(recordPath) + } + for (const id of verbs) { + const prev = await this.storage.readVerbRaw(id) + const recordPath = `${dir}/prev/${id}.json` + const record: GenerationRecord = { kind: 'verb', metadata: prev.metadata, vector: prev.vector } + await this.storage.writeRawObject(recordPath, record) + stagedPaths.push(recordPath) + } + const delta: GenerationDelta = { + generation: gen, + timestamp, + ...(args.meta && { meta: args.meta }), + nouns, + verbs + } + const deltaPath = `${dir}/tx.json` + await this.storage.writeRawObject(deltaPath, delta) + stagedPaths.push(deltaPath) + await this.storage.syncRawObjects(stagedPaths) + faultPoint('after-staging') + + // -- 4. Execute the planned batch ------------------------------------- + this.inTransact = true + try { + await args.execute() + } finally { + this.inTransact = false + } + faultPoint('after-execute') + + // -- 5. Counter + manifest rename (COMMIT POINT) ---------------------- + await this.persistCounterUnlocked() + faultPoint('before-manifest-rename') + const manifest: GenerationManifest = { + version: 1, + generation: gen, + committedAt: new Date(timestamp).toISOString(), + horizon: this.horizonGen + } + await this.storage.writeRawObject(MANIFEST_PATH, manifest) + await this.storage.syncRawObjects([MANIFEST_PATH]) + faultPoint('after-manifest-rename') + + // -- 6. Post-commit bookkeeping --------------------------------------- + this.committed = gen + this.committedGens.push(gen) + this.deltaCache.set(gen, { nouns: new Set(nouns), verbs: new Set(verbs), timestamp }) + const logEntry: TxLogEntry = { generation: gen, timestamp, ...(args.meta && { meta: args.meta }) } + await this.storage.appendTxLogLine(JSON.stringify(logEntry)) + + return { generation: gen, timestamp } + } catch (err) { + // Simulated crash (test-only fault injector): skip the abort cleanup + // entirely — a dead process cannot clean up. Recovery on the next + // open() is what must (and does) restore the store. + if (crashSimulated) { + throw err + } + // Failed before the manifest rename: nothing is committed. Remove the + // staging directory; the TransactionManager already restored any + // applied operation byte-identically. + try { + await this.storage.removeRawPrefix(dir) + } catch (cleanupErr) { + prodLog.warn( + `[GenerationStore] failed to remove staging dir ${dir} after aborted transaction: ` + + `${(cleanupErr as Error).message} (recovery will remove it on next open)` + ) + } + // Return the reservation when no concurrent bump consumed a later + // number, so a failed transaction leaves generation() unchanged. + if (this.counter === gen) this.counter = gen - 1 + throw err + } + }) + } + + // ========================================================================== + // Point-in-time resolution + // ========================================================================== + + /** + * @description Whether any transaction committed after generation `gen`. + * O(1) — this is the fast-path check that lets a `Db` pinned at the + * current generation delegate every read to the live brain untouched. + * @param gen - The pinned generation. + */ + hasCommittedAfter(gen: number): boolean { + const last = this.committedGens[this.committedGens.length - 1] + return last !== undefined && last > gen + } + + /** + * @description Resolve the state of an entity or relationship at pinned + * generation `gen`: the before-image stored by the first committed + * generation after `gen` that touched the id — or `{ source: 'current' }` + * when nothing after `gen` touched it (the live storage state *is* the + * state at `gen`). + * + * @param kind - `'noun'` for entities, `'verb'` for relationships. + * @param id - The id to resolve. + * @param gen - The pinned generation. + * @returns `{ source: 'current' }`, `{ source: 'absent' }` (did not exist + * at `gen`), or `{ source: 'record', metadata, vector }` with the raw + * stored objects as of `gen`. + */ + async resolveAt( + kind: 'noun' | 'verb', + id: string, + gen: number + ): Promise< + | { source: 'current' } + | { source: 'absent' } + | { source: 'record'; metadata: any; vector: any | null } + > { + for (const candidate of this.committedGens) { + if (candidate <= gen) continue + const delta = await this.getDelta(candidate) + const touched = kind === 'noun' ? delta.nouns : delta.verbs + if (!touched.has(id)) continue + const record = (await this.storage.readRawObject( + `${GENERATIONS_PREFIX}/${candidate}/prev/${id}.json` + )) as GenerationRecord | null + if (record === null) { + // The record-set exists (candidate is committed) but the id's + // before-image is missing — only possible through external tampering. + throw new Error( + `Generation record missing: ${GENERATIONS_PREFIX}/${candidate}/prev/${id}.json ` + + `(store corrupted or records removed outside compactHistory())` + ) + } + if (record.metadata === null && record.vector === null) { + return { source: 'absent' } + } + return { source: 'record', metadata: record.metadata, vector: record.vector } + } + return { source: 'current' } + } + + /** + * @description The ids touched by committed transactions in the interval + * `(fromGen, toGen]`. Used by `db.since()` and by historical `find()` to + * build its overlay of changed entities. + * @param fromGen - Exclusive lower bound. + * @param toGen - Inclusive upper bound. + */ + async changedBetween(fromGen: number, toGen: number): Promise { + const nouns = new Set() + const verbs = new Set() + for (const gen of this.committedGens) { + if (gen <= fromGen || gen > toGen) continue + const delta = await this.getDelta(gen) + for (const id of delta.nouns) nouns.add(id) + for (const id of delta.verbs) verbs.add(id) + } + return { + nouns: [...nouns].sort(), + verbs: [...verbs].sort(), + fromGeneration: fromGen, + toGeneration: toGen + } + } + + /** + * @description Assert that generation `gen` is reachable (not compacted + * away) and within the committed range, then return it. Used by `asOf()`. + * Reachability boundary: reclaiming record-set `N` removes the + * before-images that reads at generations BELOW `N` depend on, so + * generations `< horizon` are unreachable while the horizon itself remains + * servable from the record-sets above it. + * @param gen - The requested generation. + * @throws GenerationCompactedError when `gen` is below the horizon. + * @throws RangeError when `gen` is negative or beyond the current counter. + */ + assertReachable(gen: number): number { + if (!Number.isInteger(gen) || gen < 0) { + throw new RangeError(`asOf(): generation must be a non-negative integer (got ${gen})`) + } + if (gen > this.counter) { + throw new RangeError( + `asOf(): generation ${gen} is in the future (store is at generation ${this.counter})` + ) + } + if (gen < this.horizonGen) { + throw new GenerationCompactedError(gen, this.horizonGen) + } + return gen + } + + /** + * @description Resolve a wall-clock timestamp to the generation that was + * committed at-or-before it, via the tx-log. Resolution granularity is + * transact commits — single-operation writes between commits are not + * individually addressable (documented 8.0 history granularity). + * @param timestampMs - Milliseconds since epoch. + * @returns The resolved generation (`0` when `timestampMs` predates the + * first commit) and the matched tx-log entry when one exists. + */ + async resolveTimestamp( + timestampMs: number + ): Promise<{ generation: number; entry: TxLogEntry | null }> { + const lines = await this.storage.readTxLogLines() + let best: TxLogEntry | null = null + for (const line of lines) { + let entry: TxLogEntry + try { + entry = JSON.parse(line) as TxLogEntry + } catch { + continue // tolerate a torn trailing line from a crashed append + } + if (entry.timestamp <= timestampMs && entry.generation <= this.committed) { + if (best === null || entry.generation > best.generation) best = entry + } + } + return { generation: best?.generation ?? 0, entry: best } + } + + /** + * @description The commit timestamp of the newest committed generation at + * or below `gen`, when its record-set is still resident. Used to populate + * `Db.timestamp` for `asOf()` values. + * @param gen - The pinned generation. + */ + async commitTimestampAtOrBefore(gen: number): Promise { + for (let i = this.committedGens.length - 1; i >= 0; i--) { + const candidate = this.committedGens[i] + if (candidate > gen) continue + const delta = await this.getDelta(candidate) + return delta.timestamp + } + return null + } + + private async getDelta( + gen: number + ): Promise<{ nouns: Set; verbs: Set; timestamp: number }> { + const cached = this.deltaCache.get(gen) + if (cached) return cached + const delta = (await this.storage.readRawObject( + `${GENERATIONS_PREFIX}/${gen}/tx.json` + )) as GenerationDelta | null + if (delta === null) { + throw new Error( + `Generation delta missing: ${GENERATIONS_PREFIX}/${gen}/tx.json ` + + `(store corrupted or records removed outside compactHistory())` + ) + } + const entry = { + nouns: new Set(delta.nouns), + verbs: new Set(delta.verbs), + timestamp: delta.timestamp + } + this.deltaCache.set(gen, entry) + return entry + } + + // ========================================================================== + // Compaction + // ========================================================================== + + /** + * @description Reclaim generation record-sets past the retention policy. + * A generation directory `N` may be removed only when `N` is at or below + * **every** live pin (deleting `N` can only break readers pinned *below* + * `N`, because resolution reads before-images from generations strictly + * greater than the pin). With both retention options supplied, a + * generation must satisfy both to be eligible. + * + * @param options - Retention policy (see {@link CompactHistoryOptions}). + * @returns Count of removed record-sets and the new horizon. + */ + async compact(options?: CompactHistoryOptions): Promise { + return this.withMutex(async () => { + const minPinned = this.minPinnedGeneration() + const retainGenerations = options?.retainGenerations ?? 0 + const countWatermark = this.committed - retainGenerations + const timeCutoff = + options?.retainMs !== undefined ? Date.now() - options.retainMs : Infinity + + const removed: number[] = [] + for (const gen of [...this.committedGens]) { + if (gen > countWatermark) continue + if (gen > minPinned) continue + if (timeCutoff !== Infinity) { + const delta = await this.getDelta(gen) + if (delta.timestamp >= timeCutoff) continue + } + await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${gen}`) + this.deltaCache.delete(gen) + removed.push(gen) + } + + if (removed.length > 0) { + const removedSet = new Set(removed) + this.committedGens = this.committedGens.filter((gen) => !removedSet.has(gen)) + this.horizonGen = Math.max(this.horizonGen, ...removed) + const manifest: GenerationManifest = { + version: 1, + generation: this.committed, + committedAt: new Date().toISOString(), + horizon: this.horizonGen + } + await this.storage.writeRawObject(MANIFEST_PATH, manifest) + await this.storage.syncRawObjects([MANIFEST_PATH]) + } + + return { removedGenerations: removed.length, horizon: this.horizonGen } + }) + } + + // ========================================================================== + // Restore support + // ========================================================================== + + /** + * @description Re-read all persisted state after `brain.restore()` + * replaced the store's contents from a snapshot, enforcing counter + * monotonicity: the counter never moves below `floorGeneration` (the + * pre-restore counter), so generation numbers observed before a restore + * are never reissued. + * @param floorGeneration - The counter value before the restore. + */ + async reopenAfterRestore(floorGeneration: number): Promise { + await this.withMutex(async () => { + this.deltaCache.clear() + this.opened = false + // open() re-reads counter/manifest and re-registers the bump hook. + await this.open() + if (this.counter < floorGeneration) { + this.counter = floorGeneration + await this.persistCounterUnlocked() + } + }) + } + + // ========================================================================== + // Internals + // ========================================================================== + + /** + * Restore the before-images of an uncommitted generation to the canonical + * entity paths, then remove its record directory. Restores are idempotent + * (writing identical bytes / deleting already-absent files), so recovery + * is safe at every crash point of the commit protocol. + */ + private async rollBackUncommittedGeneration(gen: number): Promise { + const dir = `${GENERATIONS_PREFIX}/${gen}` + const prevPaths = await this.storage.listRawObjects(`${dir}/prev`) + for (const recordPath of prevPaths) { + const id = recordIdFromPath(recordPath) + if (id === null) continue + const record = (await this.storage.readRawObject(recordPath)) as GenerationRecord | null + if (record === null) continue + const image = { metadata: record.metadata, vector: record.vector } + if (record.kind === 'verb') { + await this.storage.writeVerbRaw(id, image) + } else { + await this.storage.writeNounRaw(id, image) + } + } + await this.storage.removeRawPrefix(dir) + prodLog.warn( + `[GenerationStore] rolled back uncommitted generation ${gen} ` + + `(${prevPaths.length} record(s) restored)` + ) + } + + /** + * @description Run a snapshot section serialized against transact commits, + * compaction, and counter persistence (the store's single commit mutex), + * with the generation counter durably persisted first. Used by + * `db.persist()`: the lock guarantees a snapshot can never interleave with + * a commit's record-write/manifest-rename sequence, and the up-front + * counter persist guarantees the snapshot's `_system/generation.json` + * reflects every single-operation bump (whose persistence is otherwise + * coalesced and may still be scheduled). + * @param fn - The exclusive snapshot section. + * @returns `fn`'s result. + */ + snapshotWith(fn: () => Promise): Promise { + return this.withMutex(async () => { + await this.persistCounterUnlocked() + return fn() + }) + } + + /** Run `fn` exclusively, chained behind every prior exclusive section. */ + private withMutex(fn: () => Promise): Promise { + const run = this.mutexTail.then(fn, fn) + // Keep the chain alive regardless of fn's outcome. + this.mutexTail = run.catch(() => {}) + return run + } +} + +/** + * @description Extract the generation number from a record path of the form + * `_generations//...`. Returns `null` for paths that don't match. + * @param path - A storage-root-relative object path. + */ +function parseGenerationFromPath(path: string): number | null { + const match = /^_generations[/\\](\d+)[/\\]/.exec(path) + if (!match) return null + const gen = Number(match[1]) + return Number.isSafeInteger(gen) ? gen : null +} + +/** + * @description Extract the record id (file basename without `.json`) from a + * `prev/` before-image record path. Returns `null` for non-record paths. + * @param path - A storage-root-relative object path. + */ +function recordIdFromPath(path: string): string | null { + const match = /[/\\]prev[/\\]([^/\\]+)\.json$/.exec(path) + return match ? match[1] : null +} diff --git a/src/db/types.ts b/src/db/types.ts new file mode 100644 index 00000000..26d19a27 --- /dev/null +++ b/src/db/types.ts @@ -0,0 +1,321 @@ +/** + * @module db/types + * @description Shared types for the 8.0 generational-MVCC Db API: the + * declarative transaction-operation union consumed by `brain.transact()`, + * the options bags for `transact()` / `compactHistory()`, the persisted + * record shapes of the generational record layer, and the narrow storage + * contract (`GenerationStorage`) the record layer needs from an adapter. + * + * Persisted layout (all paths relative to the storage root): + * + * ``` + * _system/generation.json — { generation, updatedAt } (atomic tmp+rename) + * _system/manifest.json — { version, generation, committedAt, horizon } + * (atomic tmp+rename — the rename IS the commit point) + * _system/tx-log.jsonl — one JSON line per committed transact: + * { generation, timestamp, meta? } (append-only) + * _generations//tx.json — the generation-N delta: touched noun/verb ids + meta + * _generations//prev/.json — immutable before-image of as of commit N + * ``` + * + * Design note: the manifest holds the committed watermark + compaction + * horizon rather than a full `id → latest record generation` map. The id + * mapping is distributed across the per-generation `tx.json` deltas, which + * makes each commit O(ids touched) instead of O(all ids) — see + * `docs/ADR-001-generational-mvcc.md` for the full justification. + */ + +import type { AddParams, UpdateParams, RelateParams } from '../types/brainy.types.js' + +// ============================================================================ +// Transaction operations (brain.transact input) +// ============================================================================ + +/** + * @description Create an entity. Carries the same parameters as + * `brain.add()`; supply `id` to choose the entity id (otherwise a UUID v4 is + * generated and returned in the operation's planning, exactly as `add()` + * does). + */ +export interface TxAddOperation extends AddParams { + /** Discriminator. */ + op: 'add' +} + +/** + * @description Update an entity. Carries the same parameters as + * `brain.update()` including per-entity CAS via `ifRev` — an `ifRev` + * conflict rejects the whole transaction before anything is applied. + */ +export interface TxUpdateOperation extends UpdateParams { + /** Discriminator. */ + op: 'update' +} + +/** + * @description Delete an entity (and, exactly like `brain.delete()`, every + * relationship where it is source or target — the cascade is part of the + * same atomic batch). + */ +export interface TxRemoveOperation { + /** Discriminator. */ + op: 'remove' + /** Id of the entity to delete. */ + id: string +} + +/** + * @description Create a relationship. Carries the same parameters as + * `brain.relate()` (including `bidirectional`). Duplicate relationships + * (same from/to/type) are deduplicated exactly as `relate()` does — the + * operation becomes a no-op that resolves to the existing relationship id. + */ +export interface TxRelateOperation extends RelateParams { + /** Discriminator. */ + op: 'relate' +} + +/** + * @description Delete a relationship by id (mirror of `brain.unrelate()`). + */ +export interface TxUnrelateOperation { + /** Discriminator. */ + op: 'unrelate' + /** Id of the relationship to delete. */ + id: string +} + +/** + * @description One declarative operation inside a `brain.transact()` batch. + * The batch executes atomically: either every operation applies and the + * store advances exactly one generation, or none apply and the store is + * byte-identical to its pre-transaction state. + */ +export type TxOperation = + | TxAddOperation + | TxUpdateOperation + | TxRemoveOperation + | TxRelateOperation + | TxUnrelateOperation + +/** + * @description Options for `brain.transact()`. + */ +export interface TransactOptions { + /** + * Transaction metadata, reified Datomic-style: recorded in + * `_system/tx-log.jsonl` alongside the generation number and commit + * timestamp. Use it for audit fields (author, reason, request id) — + * it replaces commit messages from the pre-8.0 versioning API. + */ + meta?: Record + /** + * Whole-store compare-and-swap. When provided, the transaction commits + * only if the store's current generation equals this value; otherwise it + * throws `GenerationConflictError` (with `expected`/`actual`) before any + * record is staged. + */ + ifAtGeneration?: number +} + +/** + * @description Per-operation results of a committed transaction, in input + * order. `add` resolves to the created entity id, `relate` to the created + * (or deduplicated existing) relationship id; `update`/`remove`/`unrelate` + * resolve to the id they acted on. + */ +export interface TransactReceipt { + /** The generation this transaction committed as. */ + generation: number + /** Commit timestamp (ms since epoch) recorded in the tx-log. */ + timestamp: number + /** Resolved id per input operation, in input order. */ + ids: string[] +} + +// ============================================================================ +// Compaction +// ============================================================================ + +/** + * @description Options for `brain.compactHistory()`. When both options are + * supplied a generation must satisfy *both* retention rules to be reclaimed + * (the stricter retention wins). With no options, every generation not + * protected by a live pin is eligible. + */ +export interface CompactHistoryOptions { + /** + * Keep at least this many of the most recent committed generations + * (`0` = keep none beyond what live pins protect). + */ + retainGenerations?: number + /** + * Keep every generation committed within the last `retainMs` milliseconds. + */ + retainMs?: number +} + +/** + * @description Result of `brain.compactHistory()`. + */ +export interface CompactHistoryResult { + /** Number of generation record-sets reclaimed by this call. */ + removedGenerations: number + /** + * The new compaction horizon — the highest generation whose record-set has + * been reclaimed. Generations BELOW the horizon are no longer reachable via + * `asOf()` (their reads would need the reclaimed before-images); the + * horizon itself stays reachable, resolved from the record-sets above it. + */ + horizon: number +} + +// ============================================================================ +// Db surfaces +// ============================================================================ + +/** + * @description Result of `db.since(priorDb)` — the ids touched by committed + * transactions in the generation interval `(priorDb.generation, + * db.generation]`. Granularity note: single-operation writes performed + * outside `transact()` bump the generation counter but do not produce + * generation records, so they are not reflected here (documented 8.0 + * history granularity — see `docs/ADR-001-generational-mvcc.md`). + */ +export interface ChangedIds { + /** Entity (noun) ids touched in the interval, sorted. */ + nouns: string[] + /** Relationship (verb) ids touched in the interval, sorted. */ + verbs: string[] + /** Exclusive lower bound of the interval. */ + fromGeneration: number + /** Inclusive upper bound of the interval. */ + toGeneration: number +} + +// ============================================================================ +// Persisted record shapes +// ============================================================================ + +/** + * @description An immutable before-image of one entity or relationship, + * persisted under `_generations//prev/.json` — the state of the id + * immediately before commit `N` touched it. `metadata`/`vector` hold the + * *raw stored objects* (exact bytes that live at the entity's canonical + * storage paths); `null` means the corresponding file did not exist (for + * before-images of created ids, both are `null`). Before-images are both + * the crash-recovery undo log and the point-in-time read source: the state + * of id X at pinned generation G is the before-image of the first committed + * generation after G that touched X. + */ +export interface GenerationRecord { + /** Whether this record images an entity (noun) or a relationship (verb). */ + kind: 'noun' | 'verb' + /** Raw stored metadata object, or `null` if the metadata file was absent. */ + metadata: unknown | null + /** Raw stored vector object, or `null` if the vector file was absent. */ + vector: unknown | null +} + +/** + * @description The per-generation delta persisted at + * `_generations//tx.json` — written and fsynced *before* any operation of + * the transaction executes, so crash recovery always knows exactly which ids + * to restore from before-images. + */ +export interface GenerationDelta { + /** The generation this delta belongs to. */ + generation: number + /** Commit timestamp (ms since epoch). */ + timestamp: number + /** Transaction metadata (mirrored into the tx-log on commit). */ + meta?: Record + /** Entity ids touched by this generation. */ + nouns: string[] + /** Relationship ids touched by this generation. */ + verbs: string[] +} + +/** + * @description Shape of `_system/manifest.json`. Replaced via atomic + * tmp+rename on every commit — the rename is the commit point: a generation + * directory is committed if and only if its number is ≤ `generation`. + */ +export interface GenerationManifest { + /** Schema version of the manifest file. */ + version: 1 + /** Committed-transaction watermark. */ + generation: number + /** ISO timestamp of the most recent commit (or compaction). */ + committedAt: string + /** Compaction horizon — record-sets for generations ≤ this are reclaimed. */ + horizon: number +} + +/** + * @description One line of `_system/tx-log.jsonl`. + */ +export interface TxLogEntry { + /** The committed generation. */ + generation: number + /** Commit timestamp (ms since epoch). */ + timestamp: number + /** Transaction metadata, when supplied to `transact()`. */ + meta?: Record +} + +// ============================================================================ +// Storage contract +// ============================================================================ + +/** + * @description The narrow storage contract the generational record layer + * needs. `BaseStorage` implements it structurally (both shipped adapters — + * filesystem and memory — inherit the implementation), so any + * `StorageAdapter` produced by the 8.0 storage factory satisfies it. + * + * Raw-object methods bypass branch scoping and operate on storage-root- + * relative paths (the record layer owns `_system/` + `_generations/`); + * entity-raw methods read/write the canonical entity files (branch-scoped, + * write-cache coherent) so before/after images capture exactly the bytes + * the live read paths see. + */ +export interface GenerationStorage { + /** Read a raw object at a storage-root-relative path (`null` if absent). */ + readRawObject(path: string): Promise + /** Write a raw object at a storage-root-relative path (atomic tmp+rename on disk). */ + writeRawObject(path: string, data: any): Promise + /** Delete a raw object (no-op if absent). */ + deleteRawObject(path: string): Promise + /** List raw object paths under a prefix (normalized, `.gz`-stripped). */ + listRawObjects(prefix: string): Promise + /** Remove every object under a prefix (and the directory itself on disk). */ + removeRawPrefix(prefix: string): Promise + /** Durability barrier: fsync the given object paths (no-op in memory). */ + syncRawObjects(paths: string[]): Promise + + /** Read an entity's raw stored metadata+vector objects. */ + readNounRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> + /** Restore an entity's raw stored objects (`null` part ⇒ delete that file). */ + writeNounRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise + /** Read a relationship's raw stored metadata+vector objects. */ + readVerbRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> + /** Restore a relationship's raw stored objects (`null` part ⇒ delete that file). */ + writeVerbRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise + + /** Append one line to `_system/tx-log.jsonl`. */ + appendTxLogLine(line: string): Promise + /** Read all lines of `_system/tx-log.jsonl` (empty array if absent). */ + readTxLogLines(): Promise + + /** + * Register the generation-bump hook invoked on every entity-visible + * single-operation write (see `BaseStorage.setGenerationBumpHook`). + */ + setGenerationBumpHook(hook: (() => void) | undefined): void + + /** Snapshot the entire store to a directory (hard-link farm on disk). */ + snapshotToDirectory(targetPath: string): Promise + /** Replace the entire store's contents from a snapshot directory. */ + restoreFromDirectory(sourcePath: string): Promise +} diff --git a/src/db/whereMatcher.ts b/src/db/whereMatcher.ts new file mode 100644 index 00000000..c4b6ed4d --- /dev/null +++ b/src/db/whereMatcher.ts @@ -0,0 +1,292 @@ +/** + * @module db/whereMatcher + * @description In-memory evaluation of `find()` metadata filters against a + * single resolved entity — the overlay half of historical and speculative + * reads on a `Db` value. + * + * A `Db` pinned at a past generation answers metadata-level `find()` by + * combining the live index's results (for entities untouched since the pin) + * with per-entity evaluation of the same filter against generation records + * (for entities that DID change). The same evaluator backs `db.with()` + * speculative overlays. The semantics here deliberately mirror the index + * evaluator in `src/utils/metadataIndex.ts` (`getIdsForFilter`) — operator + * aliases, array-membership equality, `exists`/`missing`, and the + * `allOf`/`anyOf`/`not` logical forms — so an entity matches in-memory if and + * only if it would have matched through the index. + * + * **Honesty contract:** an operator this module does not recognize throws + * {@link UnsupportedWhereOperatorError} instead of guessing — a historical + * read must never silently return wrong results. + */ + +import type { Entity, FindParams } from '../types/brainy.types.js' + +/** + * @description Thrown when a `where` clause uses an operator this in-memory + * evaluator does not implement. Callers (historical `find()` on a `Db`) + * convert this into the documented historical-query error rather than + * returning silently-wrong results. + */ +export class UnsupportedWhereOperatorError extends Error { + /** The unrecognized operator name. */ + public readonly operator: string + + /** + * @param operator - The operator name that could not be evaluated. + */ + constructor(operator: string) { + super( + `The where-operator '${operator}' is not supported for historical/speculative ` + + `in-memory evaluation. Supported: eq/equals/is, ne/notEquals/isNot, in/oneOf, ` + + `gt/greaterThan, gte/greaterThanOrEqual/greaterEqual, lt/lessThan, ` + + `lte/lessThanOrEqual/lessEqual, between, contains, exists, missing, ` + + `plus allOf/anyOf/not.` + ) + this.name = 'UnsupportedWhereOperatorError' + this.operator = operator + } +} + +/** + * @description Resolve a filter field name to its value on an entity, + * mirroring how `extractIndexableFields` lays entities out for the metadata + * index: standard fields live at the top level (with the `noun` ↔ `type` + * alias), everything else is a custom field inside the `metadata` bag. + * Dotted paths (`metadata.priority`, `address.city`) traverse nested objects. + * + * @param entity - The resolved entity to read from. + * @param field - The filter field name. + * @returns The field's value, or `undefined` when absent. + */ +export function resolveEntityField(entity: Entity, field: string): unknown { + switch (field) { + case 'noun': + case 'type': + return entity.type + case 'subtype': + return entity.subtype + case 'id': + return entity.id + case 'createdAt': + return entity.createdAt + case 'updatedAt': + return entity.updatedAt + case 'service': + return entity.service + case 'createdBy': + return entity.createdBy + case 'confidence': + return entity.confidence + case 'weight': + return entity.weight + case '_rev': + return entity._rev + case 'data': + return entity.data + } + + if (field.includes('.')) { + // Dotted path: resolve against the whole entity first (`metadata.x`), + // then against the metadata bag (`address.city` on nested metadata). + const fromEntity = resolvePath(entity as unknown as Record, field) + if (fromEntity !== undefined) return fromEntity + return resolvePath((entity.metadata ?? {}) as Record, field) + } + + return ((entity.metadata ?? {}) as Record)[field] +} + +/** Walk a dotted path through nested plain objects. */ +function resolvePath(obj: Record, path: string): unknown { + let current: unknown = obj + for (const segment of path.split('.')) { + if (current === null || typeof current !== 'object') return undefined + current = (current as Record)[segment] + } + return current +} + +/** + * @description Index-equality semantics: scalar strict equality, with + * array-valued fields matching by membership (the index stores one posting + * per array element, so `eq` on an array field means "contains"). + */ +function eqMatches(value: unknown, operand: unknown): boolean { + if (Array.isArray(value)) { + return value.some((element) => element === operand) + } + return value === operand +} + +/** Ordered comparison for range operators (numbers, or both-strings lexicographic). */ +function compare(value: unknown, operand: unknown): number | null { + if (typeof value === 'number' && typeof operand === 'number') { + return value - operand + } + if (typeof value === 'string' && typeof operand === 'string') { + return value < operand ? -1 : value > operand ? 1 : 0 + } + return null +} + +/** + * @description Evaluate one field condition (shorthand value or operator + * object) against a resolved field value, mirroring the operator table in + * `metadataIndex.getIdsForFilter`. + * + * @param value - The entity's field value (possibly `undefined`). + * @param condition - The filter condition for this field. + * @returns Whether the value satisfies the condition. + * @throws UnsupportedWhereOperatorError for unrecognized operators. + */ +function fieldConditionMatches(value: unknown, condition: unknown): boolean { + if (condition === null || typeof condition !== 'object' || Array.isArray(condition)) { + // Shorthand for 'eq'. + return eqMatches(value, condition) + } + + for (const [op, operand] of Object.entries(condition as Record)) { + let matches: boolean + switch (op) { + case 'is': + case 'equals': + case 'eq': + matches = eqMatches(value, operand) + break + case 'isNot': + case 'notEquals': + case 'ne': + matches = !eqMatches(value, operand) + break + case 'oneOf': + case 'in': + matches = Array.isArray(operand) && operand.some((candidate) => eqMatches(value, candidate)) + break + case 'greaterThan': + case 'gt': { + const cmp = compare(value, operand) + matches = cmp !== null && cmp > 0 + break + } + case 'greaterEqual': + case 'greaterThanOrEqual': + case 'gte': { + const cmp = compare(value, operand) + matches = cmp !== null && cmp >= 0 + break + } + case 'lessThan': + case 'lt': { + const cmp = compare(value, operand) + matches = cmp !== null && cmp < 0 + break + } + case 'lessEqual': + case 'lessThanOrEqual': + case 'lte': { + const cmp = compare(value, operand) + matches = cmp !== null && cmp <= 0 + break + } + case 'between': { + if (!Array.isArray(operand) || operand.length !== 2) { + matches = false + break + } + const lower = compare(value, operand[0]) + const upper = compare(value, operand[1]) + matches = lower !== null && upper !== null && lower >= 0 && upper <= 0 + break + } + case 'contains': + // Index semantics: array fields post one entry per element, so + // 'contains' is the same lookup as 'eq' (membership on arrays). + matches = eqMatches(value, operand) + break + case 'exists': + matches = operand ? value !== undefined : value === undefined + break + case 'missing': + matches = operand ? value === undefined : value !== undefined + break + default: + throw new UnsupportedWhereOperatorError(op) + } + if (!matches) return false // Multiple operators on one field AND together. + } + return true +} + +/** + * @description Evaluate a full `where` filter (field conditions plus + * `allOf`/`anyOf`/`not` logical composition) against one entity. + * + * @param entity - The resolved entity (historical record or speculative overlay). + * @param where - The `where` clause from `FindParams`. + * @returns Whether the entity satisfies every clause. + * @throws UnsupportedWhereOperatorError for unrecognized operators. + */ +export function whereMatches(entity: Entity, where: Record): boolean { + for (const [field, condition] of Object.entries(where)) { + if (field === 'allOf') { + if (!Array.isArray(condition)) return false + if (!condition.every((sub) => whereMatches(entity, sub as Record))) return false + continue + } + if (field === 'anyOf') { + if (!Array.isArray(condition)) return false + if (!condition.some((sub) => whereMatches(entity, sub as Record))) return false + continue + } + if (field === 'not') { + if (whereMatches(entity, condition as Record)) return false + continue + } + if (!fieldConditionMatches(resolveEntityField(entity, field), condition)) return false + } + return true +} + +/** + * @description Evaluate the metadata-level portions of a `find()` query + * (`type`, `subtype`, `where`, `service`, `excludeVFS`) against one resolved + * entity. Used by historical and speculative `find()` to decide whether a + * changed/overlaid entity belongs in the result set. The caller guarantees + * the query carries no index-only dimensions (semantic `query`/`vector`, + * `connected` traversal, …) — those are rejected with the documented + * historical-query error before evaluation starts. + * + * @param entity - The resolved entity. + * @param params - The metadata-level find parameters. + * @returns Whether the entity matches every requested filter. + * @throws UnsupportedWhereOperatorError for unrecognized `where` operators. + */ +export function entityMatchesFind(entity: Entity, params: FindParams): boolean { + if (params.type !== undefined) { + const types = Array.isArray(params.type) ? params.type : [params.type] + if (!types.includes(entity.type)) return false + } + + if (params.subtype !== undefined) { + const subtypes = Array.isArray(params.subtype) ? params.subtype : [params.subtype] + if (entity.subtype === undefined || !subtypes.includes(entity.subtype)) return false + } + + if (params.service !== undefined && entity.service !== params.service) { + return false + } + + if (params.excludeVFS === true) { + // Mirror of find()'s exclusion filter (`vfsType: { exists: false }`, + // `isVFSEntity: { ne: true }`) and Brainy's VFS-marker helper. + const metadata = (entity.metadata ?? {}) as Record + if (metadata.vfsType !== undefined) return false + if (metadata.isVFSEntity === true || metadata.isVFS === true) return false + } + + if (params.where !== undefined) { + if (!whereMatches(entity, params.where as Record)) return false + } + + return true +} diff --git a/src/index.ts b/src/index.ts index 7546786a..0416f283 100644 --- a/src/index.ts +++ b/src/index.ts @@ -128,6 +128,32 @@ export type { Migration, MigrationState, MigrationPreview, MigrationResult, Migr // Export optimistic-concurrency types (7.31.0) export { RevisionConflictError } from './transaction/RevisionConflictError.js' +// ============= 8.0 Db API — generational MVCC ============= +// Immutable database values: brain.now() / brain.transact() / brain.asOf() / +// db.with() / db.persist() / Brainy.load(). See src/db/ for the record layer. +export { Db } from './db/db.js' +export { + GenerationConflictError, + NotYetSupportedAtHistoricalGenerationError, + GenerationCompactedError +} from './db/errors.js' +export type { + TxOperation, + TxAddOperation, + TxUpdateOperation, + TxRemoveOperation, + TxRelateOperation, + TxUnrelateOperation, + TransactOptions, + TransactReceipt, + CompactHistoryOptions, + CompactHistoryResult, + ChangedIds +} from './db/types.js' +// Optional provider capability for generation-aware native indexes +export { isVersionedIndexProvider } from './plugin.js' +export type { VersionedIndexProvider } from './plugin.js' + // Export embedding functionality import { UniversalSentenceEncoder, diff --git a/src/plugin.ts b/src/plugin.ts index ff5f481a..f2ddf7e3 100644 --- a/src/plugin.ts +++ b/src/plugin.ts @@ -259,6 +259,108 @@ export interface GraphIndexProvider { getAllRelationshipCounts(): Map } +/** + * Optional capability interface for index providers that maintain versioned + * (generation-aware) internal state — the provider-side half of Brainy 8.0's + * generational MVCC contract. Implemented by native providers whose storage + * engines keep immutable versions (e.g. LSM snapshots); Brainy's own JS + * indexes do not implement it (they are rebuilt from the storage records, + * which carry the versioning). + * + * **Detection.** Brainy feature-detects this interface on every registered + * index provider (`'vector'`, `'metadataIndex'`, `'graphIndex'`): when a + * provider exposes `pin`/`release` functions, Brainy calls them in lockstep + * with `Db` lifecycle — `pin(g)` when a `Db` value pins generation `g` + * (`brain.now()`, `brain.transact()`, `brain.asOf()`), `release(g)` when that + * `Db` is released (explicitly or via the GC backstop). Pins are refcounted + * on the Brainy side; a provider may receive multiple `pin(g)` calls for the + * same generation and will receive exactly one matching `release(g)` per pin. + * + * **Consistency model (locked cross-team design).** Index providers are + * *post-commit appliers*: the storage-record commit (atomic manifest rename) + * is the source of truth, and provider index state is derived, applied after + * the commit point. On open, a provider compares its own persisted + * `generation()` against the store's committed generation and replays the + * gap from the storage records (or requests a rebuild) — there are no + * provider rollback hooks, because an uncommitted transaction is repaired at + * the storage layer before any index is opened. The explicit pin/release + * lifetime OVERRIDES any time-based snapshot retention the provider has + * (e.g. an LSM snapshot TTL): a pinned generation must stay readable until + * released, regardless of age. + * + * **Speculative reads.** `db.with(txData)` overlays are Brainy-side only — + * providers are never asked to read uncommitted/speculative state; they + * always serve committed generations. + */ +export interface VersionedIndexProvider { + /** + * @description The newest generation this provider's persisted index state + * reflects. Brainy compares it against the storage layer's committed + * generation on open to detect a replay gap (index behind storage after a + * crash between commit and index apply). + * @returns The provider's current generation as a `bigint` (u64 at the + * native boundary; Brainy's generation counter is a safe integer today). + */ + generation(): bigint + + /** + * @description True ⇒ the provider can serve consistent reads at + * `generation` (segments retained). Combined with pin: pinning a VISIBLE + * generation guarantees it stays servable until release. Pinning an + * invisible generation is permitted (refcount-only) — Brainy serves that + * generation from canonical storage instead. + * + * Brainy consults this at pin time (the read-routing rule: a `Db` at + * generation `g` uses provider-accelerated reads when + * `isGenerationVisible(g)` was true at pin time; otherwise canonical + * generation records). + * @param generation - The generation a `Db` value is about to pin. + */ + isGenerationVisible(generation: bigint): boolean + + /** + * @description Pin a generation: the provider must keep index state for + * `generation` readable until the matching {@link VersionedIndexProvider.release} + * call, overriding any time-based snapshot retention. Called once per + * Brainy-side pin (refcounted upstream — expect balanced pin/release pairs). + * @param generation - The generation a live `Db` value just pinned. + */ + pin(generation: bigint): void + + /** + * @description Release one pin on `generation`. After the last release the + * provider may reclaim resources for that generation at its discretion. + * @param generation - The generation being released (matches a prior + * {@link VersionedIndexProvider.pin} call). + */ + release(generation: bigint): void +} + +/** + * @description Feature-detection guard for {@link VersionedIndexProvider}. + * Brainy applies it to every registered index provider (vector, metadata, + * graph) when a `Db` value pins or releases a generation: providers exposing + * all four capability methods receive lockstep `pin`/`release` calls; + * everything else (including Brainy's own JS indexes) is skipped. + * + * @param candidate - A registered index provider instance. + * @returns Whether the candidate implements the versioned capability. + */ +export function isVersionedIndexProvider( + candidate: unknown +): candidate is VersionedIndexProvider { + if (candidate === null || typeof candidate !== 'object') { + return false + } + const c = candidate as Record + return ( + typeof c.generation === 'function' && + typeof c.isGenerationVisible === 'function' && + typeof c.pin === 'function' && + typeof c.release === 'function' + ) +} + /** * The object returned by the `'vector'` provider factory — Brainy's vector * index contract. Implementations include Brainy's own JS HNSW index and any diff --git a/src/storage/adapters/fileSystemStorage.ts b/src/storage/adapters/fileSystemStorage.ts index a6e87e44..535c9e77 100644 --- a/src/storage/adapters/fileSystemStorage.ts +++ b/src/storage/adapters/fileSystemStorage.ts @@ -954,6 +954,254 @@ export class FileSystemStorage extends BaseStorage { } } + // =========================================================================== + // Generational record layer (8.0 MVCC) — durability + snapshot primitives + // =========================================================================== + + /** + * Storage-root-relative paths that are mutated **in place** (appended to) + * rather than replaced via atomic tmp+rename. `snapshotToDirectory()` must + * byte-copy these instead of hard-linking them: a hard link shares the + * inode, so a post-snapshot append to the live file would silently mutate + * the snapshot. Every other persisted file in this adapter is written via + * tmp+rename (objects, blobs, counts, locks are excluded entirely), which + * makes hard links safe — a rewrite swaps in a new inode and the snapshot + * keeps the old one. + */ + private static readonly SNAPSHOT_BYTE_COPY_PATHS = new Set([ + `${SYSTEM_DIR}/tx-log.jsonl` + ]) + + /** + * Top-level directories excluded from snapshots: process-local lock state + * (writer lock, flush-request RPC files) must never travel with the data. + */ + private static readonly SNAPSHOT_EXCLUDED_TOP_DIRS = new Set(['locks']) + + /** + * Remove every object under a storage-root-relative prefix — one recursive + * directory removal instead of the base class's list+delete loop. + * + * @param prefix - Storage-root-relative directory prefix to remove. + */ + public async removeRawPrefix(prefix: string): Promise { + await this.ensureInitialized() + await fs.promises.rm(path.join(this.rootDir, prefix), { recursive: true, force: true }) + } + + /** + * Durability barrier: `fsync` each listed object file (resolving the + * compressed `.gz` variant when present) and then the set of parent + * directories, so both the file contents and the rename directory entries + * are durable before the commit protocol proceeds. Paths whose file no + * longer exists are skipped (a later step may have replaced them). + * + * @param paths - Storage-root-relative object paths previously written. + */ + public async syncRawObjects(paths: string[]): Promise { + await this.ensureInitialized() + const parentDirs = new Set() + + for (const objectPath of paths) { + const fullPath = path.join(this.rootDir, objectPath) + for (const candidate of [`${fullPath}.gz`, fullPath]) { + let handle: any + try { + handle = await fs.promises.open(candidate, 'r') + } catch (error: any) { + if (error.code === 'ENOENT') continue + throw error + } + try { + await handle.sync() + } finally { + await handle.close() + } + parentDirs.add(path.dirname(fullPath)) + break + } + } + + for (const dir of parentDirs) { + let handle: any + try { + handle = await fs.promises.open(dir, 'r') + } catch { + continue // directory vanished or platform disallows opening dirs + } + try { + await handle.sync() + } catch { + // Some platforms (and some filesystems) reject directory fsync — + // file-level fsync above already covers the data itself. + } finally { + await handle.close() + } + } + } + + /** + * Append one line to `_system/tx-log.jsonl`. Plain `appendFile` — the + * tx-log is the one append-in-place file in the store (and is byte-copied, + * never hard-linked, by {@link FileSystemStorage.snapshotToDirectory}). + * + * @param line - One complete JSON document, without trailing newline. + */ + public async appendTxLogLine(line: string): Promise { + await this.ensureInitialized() + const logPath = path.join(this.systemDir, 'tx-log.jsonl') + await fs.promises.appendFile(logPath, `${line}\n`, 'utf-8') + } + + /** + * Read all tx-log lines, oldest first (empty array when no log exists). + * Torn trailing lines from a crashed append are returned as-is — callers + * tolerate unparseable lines. + */ + public async readTxLogLines(): Promise { + await this.ensureInitialized() + const logPath = path.join(this.systemDir, 'tx-log.jsonl') + try { + const content: string = await fs.promises.readFile(logPath, 'utf-8') + return content.split('\n').filter((l: string) => l.length > 0) + } catch (error: any) { + if (error.code === 'ENOENT') return [] + throw error + } + } + + /** + * Snapshot the entire store into `targetPath` as a hard-link farm + * (Cassandra-style: instant, space-shared). Safe because every data file + * is immutable-by-rename — rewrites swap in new inodes, leaving the + * snapshot's links pointing at the old bytes. The two exceptions are + * handled explicitly: append-in-place files + * ({@link FileSystemStorage.SNAPSHOT_BYTE_COPY_PATHS}) are byte-copied, and + * process-local lock state + * ({@link FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS}) is excluded. + * Cross-device targets (where `link(2)` fails with `EXDEV`) fall back to + * byte copies per file. + * + * @param targetPath - Absolute directory for the snapshot. Created if + * missing; must be empty or absent (refuses to overwrite). + */ + public async snapshotToDirectory(targetPath: string): Promise { + await this.ensureInitialized() + + try { + const existing = await fs.promises.readdir(targetPath) + if (existing.length > 0) { + throw new Error( + `snapshotToDirectory: target ${targetPath} already exists and is not empty. ` + + `Choose a fresh directory per snapshot.` + ) + } + } catch (error: any) { + if (error.code !== 'ENOENT') throw error + } + await fs.promises.mkdir(targetPath, { recursive: true }) + + const files: string[] = [] + await this.collectSnapshotFiles(this.rootDir, '', files) + + for (const relPath of files) { + const sourceFile = path.join(this.rootDir, relPath) + const targetFile = path.join(targetPath, relPath) + await fs.promises.mkdir(path.dirname(targetFile), { recursive: true }) + + // Byte-copy list: compare against the normalized (extension-preserving) + // relative path with separators unified, so `_system/tx-log.jsonl` + // matches on every platform. + const normalized = relPath.split(path.sep).join('/') + if (FileSystemStorage.SNAPSHOT_BYTE_COPY_PATHS.has(normalized)) { + await fs.promises.copyFile(sourceFile, targetFile) + continue + } + + try { + await fs.promises.link(sourceFile, targetFile) + } catch (error: any) { + // EXDEV: cross-device; EPERM/ENOTSUP: filesystem forbids links. + // ENOENT: the live file was atomically replaced mid-walk — retry as + // a copy of whatever is current (single-writer discipline means this + // only happens for derived files being flushed concurrently). + if (['EXDEV', 'EPERM', 'ENOTSUP', 'ENOENT'].includes(error.code)) { + try { + await fs.promises.copyFile(sourceFile, targetFile) + } catch (copyError: any) { + if (copyError.code !== 'ENOENT') throw copyError + } + } else { + throw error + } + } + } + } + + /** + * Recursively collect snapshot-eligible files under `dirAbs`, excluding the + * lock directory and in-flight `*.tmp.*` write files. + */ + private async collectSnapshotFiles(dirAbs: string, relPrefix: string, out: string[]): Promise { + let entries: any[] + try { + entries = await fs.promises.readdir(dirAbs, { withFileTypes: true }) + } catch (error: any) { + if (error.code === 'ENOENT') return + throw error + } + + for (const entry of entries) { + const rel = relPrefix ? path.join(relPrefix, entry.name) : entry.name + if (entry.isDirectory()) { + if (relPrefix === '' && FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS.has(entry.name)) { + continue + } + await this.collectSnapshotFiles(path.join(dirAbs, entry.name), rel, out) + } else if (entry.isFile()) { + if (entry.name.includes('.tmp.')) continue // in-flight atomic write + out.push(rel) + } + } + } + + /** + * Replace the store's contents from a snapshot directory: every current + * top-level entry except `locks/` (the live writer lock must survive) is + * removed, the snapshot is byte-copied in (`fs.cp` — never hard-linked, so + * the snapshot stays independent of the restored store), and all + * adapter-internal derived state is reloaded. + * + * @param sourcePath - Absolute path of a directory produced by + * {@link FileSystemStorage.snapshotToDirectory}. + */ + public async restoreFromDirectory(sourcePath: string): Promise { + await this.ensureInitialized() + + const sourceStat = await fs.promises.stat(sourcePath).catch(() => null) + if (!sourceStat || !sourceStat.isDirectory()) { + throw new Error(`restoreFromDirectory: ${sourcePath} is not a directory`) + } + + // Clear current contents, preserving live lock state. + const currentEntries = await fs.promises.readdir(this.rootDir) + for (const entry of currentEntries) { + if (FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS.has(entry)) continue + await fs.promises.rm(path.join(this.rootDir, entry), { recursive: true, force: true }) + } + + // Byte-copy the snapshot in (defensively skipping lock dirs in old snapshots). + const sourceEntries = await fs.promises.readdir(sourcePath) + for (const entry of sourceEntries) { + if (FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS.has(entry)) continue + await fs.promises.cp(path.join(sourcePath, entry), path.join(this.rootDir, entry), { + recursive: true + }) + } + + await this.reloadDerivedState() + } + // =========================================================================== // Raw binary-blob primitive (mmap-friendly) // =========================================================================== diff --git a/src/storage/adapters/historicalStorageAdapter.ts b/src/storage/adapters/historicalStorageAdapter.ts index 4d177637..6ff29710 100644 --- a/src/storage/adapters/historicalStorageAdapter.ts +++ b/src/storage/adapters/historicalStorageAdapter.ts @@ -525,4 +525,47 @@ export class HistoricalStorageAdapter extends BaseStorage { protected async persistCounts(): Promise { // No-op: Historical storage is read-only } + + // =========================================================================== + // Generational record layer (8.0 MVCC) — not applicable to this adapter + // + // HistoricalStorageAdapter is the pre-8.0 commit-based time-travel view and + // is read-only by construction. The generation store never registers its + // write hooks on reader-mode instances, so these methods are unreachable in + // normal operation; they throw explicitly rather than fake behavior. + // =========================================================================== + + /** + * WRITE BLOCKED: the tx-log belongs to the live store, not a historical view. + */ + public async appendTxLogLine(_line: string): Promise { + throw new Error('Historical storage is read-only. Cannot append to the transaction log.') + } + + /** + * Not supported: historical commit views predate the generational tx-log. + */ + public async readTxLogLines(): Promise { + throw new Error( + 'Transaction-log reads are not supported on a historical commit view. ' + + 'Use the live store (or a Db from brain.now()/brain.asOf()).' + ) + } + + /** + * Not supported: snapshot a live store via db.persist(path) instead. + */ + public async snapshotToDirectory(_targetPath: string): Promise { + throw new Error( + 'snapshotToDirectory is not supported on a historical commit view. ' + + 'Call db.persist(path) on the live store instead.' + ) + } + + /** + * WRITE BLOCKED: cannot restore into a read-only historical view. + */ + public async restoreFromDirectory(_sourcePath: string): Promise { + throw new Error('Historical storage is read-only. Cannot restore from a snapshot.') + } } diff --git a/src/storage/adapters/memoryStorage.ts b/src/storage/adapters/memoryStorage.ts index cb2c5bb9..f7ea1125 100644 --- a/src/storage/adapters/memoryStorage.ts +++ b/src/storage/adapters/memoryStorage.ts @@ -3,6 +3,9 @@ * In-memory storage adapter for environments where persistent storage is not available or needed */ +import * as fs from 'node:fs' +import * as nodePath from 'node:path' +import * as zlib from 'node:zlib' import { GraphVerb, HNSWNoun, @@ -35,6 +38,11 @@ export class MemoryStorage extends BaseStorage { // storage has no local file, so getBinaryBlobPath() returns null. private blobStore: Map = new Map() + // Transaction log (8.0 MVCC) — the in-memory equivalent of the filesystem + // adapter's `_system/tx-log.jsonl`. One JSON document per committed + // transact(); serialized to a real JSONL file by snapshotToDirectory(). + private txLogLines: string[] = [] + // Backward compatibility aliases private get metadata(): Map { return this.objectStore @@ -189,6 +197,132 @@ export class MemoryStorage extends BaseStorage { return null } + // =========================================================================== + // Generational record layer (8.0 MVCC) — tx-log + snapshot primitives + // =========================================================================== + + /** + * Append one line to the in-memory transaction log (mirror of the + * filesystem adapter's `_system/tx-log.jsonl` append). + * + * @param line - One complete JSON document, without trailing newline. + */ + public async appendTxLogLine(line: string): Promise { + this.txLogLines.push(line) + } + + /** + * Read all transaction-log lines, oldest first (a copy — callers cannot + * mutate the log). + */ + public async readTxLogLines(): Promise { + return [...this.txLogLines] + } + + /** + * Serialize the entire in-memory store to a directory in the exact layout + * the filesystem adapter uses (uncompressed JSON objects, `_blobs/*.bin` + * payloads, `_system/tx-log.jsonl`). The resulting directory is a + * self-contained store openable via `Brainy.load(path)` — persisting an + * in-memory brain produces a real, durable snapshot. + * + * @param targetPath - Absolute directory for the snapshot. Created if + * missing; must be empty or absent (refuses to overwrite). + */ + public async snapshotToDirectory(targetPath: string): Promise { + try { + const existing = await fs.promises.readdir(targetPath) + if (existing.length > 0) { + throw new Error( + `snapshotToDirectory: target ${targetPath} already exists and is not empty. ` + + `Choose a fresh directory per snapshot.` + ) + } + } catch (error: any) { + if (error.code !== 'ENOENT') throw error + } + await fs.promises.mkdir(targetPath, { recursive: true }) + + for (const [key, value] of this.objectStore.entries()) { + const filePath = nodePath.join(targetPath, ...key.split('/')) + await fs.promises.mkdir(nodePath.dirname(filePath), { recursive: true }) + await fs.promises.writeFile(filePath, JSON.stringify(value, null, 2), 'utf-8') + } + + for (const [key, data] of this.blobStore.entries()) { + const blobPath = nodePath.join(targetPath, '_blobs', ...key.split('/')) + '.bin' + await fs.promises.mkdir(nodePath.dirname(blobPath), { recursive: true }) + await fs.promises.writeFile(blobPath, data) + } + + if (this.txLogLines.length > 0) { + const logPath = nodePath.join(targetPath, '_system', 'tx-log.jsonl') + await fs.promises.mkdir(nodePath.dirname(logPath), { recursive: true }) + await fs.promises.writeFile(logPath, this.txLogLines.map((l) => `${l}\n`).join(''), 'utf-8') + } + } + + /** + * Replace the in-memory store's contents from a snapshot directory — + * accepts snapshots produced by either adapter (handles the filesystem + * adapter's gzip-compressed `.json.gz` objects transparently), then reloads + * all derived state. + * + * @param sourcePath - Absolute path of a snapshot directory. + */ + public async restoreFromDirectory(sourcePath: string): Promise { + const sourceStat = await fs.promises.stat(sourcePath).catch(() => null) + if (!sourceStat || !sourceStat.isDirectory()) { + throw new Error(`restoreFromDirectory: ${sourcePath} is not a directory`) + } + + this.objectStore.clear() + this.blobStore.clear() + this.txLogLines = [] + this.statistics = null + + await this.loadDirectoryIntoStore(sourcePath, '') + await this.reloadDerivedState() + } + + /** + * Recursively load a snapshot directory into the object/blob/tx-log stores. + * Lock directories from filesystem-adapter snapshots are skipped (process- + * local state, meaningless in memory). + */ + private async loadDirectoryIntoStore(dirAbs: string, relPrefix: string): Promise { + const entries = await fs.promises.readdir(dirAbs, { withFileTypes: true }) + + for (const entry of entries) { + const rel = relPrefix ? `${relPrefix}/${entry.name}` : entry.name + const abs = nodePath.join(dirAbs, entry.name) + + if (entry.isDirectory()) { + if (relPrefix === '' && entry.name === 'locks') continue + await this.loadDirectoryIntoStore(abs, rel) + continue + } + if (!entry.isFile()) continue + + if (rel === '_system/tx-log.jsonl') { + const content = await fs.promises.readFile(abs, 'utf-8') + this.txLogLines = content.split('\n').filter((l) => l.length > 0) + } else if (relPrefix.startsWith('_blobs') && rel.endsWith('.bin')) { + const key = rel.slice('_blobs/'.length, -'.bin'.length) + this.blobStore.set(key, await fs.promises.readFile(abs)) + } else if (rel.endsWith('.json.gz') || rel.endsWith('.gz')) { + const raw = await fs.promises.readFile(abs) + const key = rel.slice(0, -'.gz'.length) + this.objectStore.set(key, JSON.parse(zlib.gunzipSync(raw).toString('utf-8'))) + } else if (entry.name.includes('.tmp.')) { + // In-flight atomic-write remnant from a crashed filesystem store — skip. + } else { + const content = await fs.promises.readFile(abs, 'utf-8') + this.objectStore.set(rel, JSON.parse(content)) + } + } + } + /** * Get multiple metadata objects in batches (CRITICAL: Prevents socket exhaustion) * Memory storage implementation is simple since all data is already in memory @@ -217,6 +351,7 @@ export class MemoryStorage extends BaseStorage { public async clear(): Promise { this.objectStore.clear() this.blobStore.clear() + this.txLogLines = [] this.statistics = null this.totalNounCount = 0 this.totalVerbCount = 0 diff --git a/src/storage/baseStorage.ts b/src/storage/baseStorage.ts index 0a72c15b..c7380caf 100644 --- a/src/storage/baseStorage.ts +++ b/src/storage/baseStorage.ts @@ -881,6 +881,271 @@ export abstract class BaseStorage extends BaseStorageAdapter { return Array.from(pathsSet) } + // ============================================================================ + // GENERATIONAL RECORD LAYER PRIMITIVES (8.0 MVCC) + // + // The narrow surface `GenerationStore` (src/db/generationStore.ts) needs from + // an adapter — see the `GenerationStorage` contract in src/db/types.ts. + // + // Raw-object methods operate on storage-root-relative paths and deliberately + // bypass branch scoping and the write cache: the record layer owns the + // `_system/` + `_generations/` areas outright and its files must never be + // shadowed by branch resolution. Entity-raw methods, by contrast, go through + // the branch-scoped, write-cache-coherent helpers so before-images capture + // exactly the bytes the live read paths see. + // ============================================================================ + + /** + * Hook invoked after every entity-visible single-operation write (noun/verb + * metadata save or delete). Registered by the generation store so + * `brain.generation()` advances on writes performed outside `transact()`. + * See {@link BaseStorage.setGenerationBumpHook}. + */ + protected generationBumpHook?: () => void + + /** + * Register (or detach, with `undefined`) the generation-bump hook. + * + * The hook fires once per entity-visible metadata mutation — noun/verb + * metadata saves and deletes, the one storage write every logical Brainy + * mutation (`add`, `update`, `delete`, `relate`, `updateRelation`, + * `unrelate`) performs exactly once per entity it touches. It does NOT fire + * for derived-index writes (HNSW node data, metadata-index chunks, LSM + * segments, statistics), so the generation counter tracks *data* mutations, + * not index maintenance. The counter is a monotonic watermark, not an + * operation count: a cascade delete bumps once per removed record. + * + * @param hook - Callback invoked synchronously after each qualifying write, + * or `undefined` to detach. + */ + public setGenerationBumpHook(hook: (() => void) | undefined): void { + this.generationBumpHook = hook + } + + /** + * Read a raw object at a storage-root-relative path. Bypasses branch + * scoping and the write cache (record-layer files are written through + * {@link BaseStorage.writeRawObject} only). + * + * @param path - Storage-root-relative object path (e.g. `_system/manifest.json`). + * @returns The parsed object, or `null` if absent. + */ + public async readRawObject(path: string): Promise { + await this.ensureInitialized() + return this.readObjectFromPath(path) + } + + /** + * Write a raw object at a storage-root-relative path. On disk this is an + * atomic tmp+rename (the filesystem adapter's primitive), which is what + * makes the manifest rename a valid commit point. + * + * @param path - Storage-root-relative object path. + * @param data - JSON-serializable object to persist. + */ + public async writeRawObject(path: string, data: any): Promise { + await this.ensureInitialized() + await this.writeObjectToPath(path, data) + } + + /** + * Delete a raw object at a storage-root-relative path (no-op if absent). + * + * @param path - Storage-root-relative object path. + */ + public async deleteRawObject(path: string): Promise { + await this.ensureInitialized() + await this.deleteObjectFromPath(path) + } + + /** + * List raw object paths under a storage-root-relative prefix (normalized, + * `.gz`-stripped — the adapter primitives already normalize). + * + * @param prefix - Storage-root-relative directory prefix. + * @returns Normalized object paths under the prefix (empty when none). + */ + public async listRawObjects(prefix: string): Promise { + await this.ensureInitialized() + return this.listObjectsUnderPath(prefix) + } + + /** + * Remove every object under a storage-root-relative prefix. The filesystem + * adapter overrides this with a recursive directory removal; this default + * lists and deletes individually (exactly what the in-memory adapter needs). + * + * @param prefix - Storage-root-relative directory prefix to remove. + */ + public async removeRawPrefix(prefix: string): Promise { + await this.ensureInitialized() + const paths = await this.listObjectsUnderPath(prefix) + for (const p of paths) { + await this.deleteObjectFromPath(p) + } + } + + /** + * Durability barrier for the commit protocol: ensure the listed raw-object + * paths are durable before the caller proceeds. The base implementation is + * a no-op (in-memory writes are durable-by-definition within the process); + * the filesystem adapter overrides it with real `fsync` of the files and + * their parent directories. + * + * @param paths - Storage-root-relative object paths previously written via + * {@link BaseStorage.writeRawObject}. + */ + public async syncRawObjects(paths: string[]): Promise { + void paths + } + + /** + * Read an entity's raw stored objects — the exact bytes at its canonical + * metadata + vector paths (branch-scoped, write-cache coherent). Used by + * the generation store to capture before-images. + * + * @param id - The entity id. + * @returns The raw stored metadata and vector objects (`null` per part when + * the corresponding file is absent). + */ + public async readNounRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> { + await this.ensureInitialized() + const [metadata, vector] = await Promise.all([ + this.readWithInheritance(getNounMetadataPath(id)), + this.readWithInheritance(getNounVectorPath(id)) + ]) + return { metadata: metadata ?? null, vector: vector ?? null } + } + + /** + * Restore an entity's raw stored objects byte-for-byte (a `null` part + * deletes that file). Used by crash recovery and transaction aborts to + * restore before-images. + * + * Bypasses the statistics/count bookkeeping of the normal save paths on + * purpose: restores must reproduce the exact prior bytes, and the count + * rollups are derived state with their own rebuild paths + * (`rebuildTypeCounts()` / `rebuildSubtypeCounts()`). + * + * @param id - The entity id. + * @param record - Raw stored objects as returned by {@link BaseStorage.readNounRaw}. + */ + public async writeNounRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise { + await this.ensureInitialized() + if (record.metadata === null) { + await this.deleteObjectFromBranch(getNounMetadataPath(id)) + } else { + await this.writeObjectToBranch(getNounMetadataPath(id), record.metadata) + } + if (record.vector === null) { + await this.deleteObjectFromBranch(getNounVectorPath(id)) + } else { + await this.writeObjectToBranch(getNounVectorPath(id), record.vector) + } + } + + /** + * Read a relationship's raw stored objects (verb-side mirror of + * {@link BaseStorage.readNounRaw}). + * + * @param id - The relationship id. + * @returns The raw stored metadata and vector objects (`null` per part when + * the corresponding file is absent). + */ + public async readVerbRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> { + await this.ensureInitialized() + const [metadata, vector] = await Promise.all([ + this.readWithInheritance(getVerbMetadataPath(id)), + this.readWithInheritance(getVerbVectorPath(id)) + ]) + return { metadata: metadata ?? null, vector: vector ?? null } + } + + /** + * Restore a relationship's raw stored objects byte-for-byte (verb-side + * mirror of {@link BaseStorage.writeNounRaw}; same bookkeeping caveats). + * + * @param id - The relationship id. + * @param record - Raw stored objects as returned by {@link BaseStorage.readVerbRaw}. + */ + public async writeVerbRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise { + await this.ensureInitialized() + if (record.metadata === null) { + await this.deleteObjectFromBranch(getVerbMetadataPath(id)) + } else { + await this.writeObjectToBranch(getVerbMetadataPath(id), record.metadata) + } + if (record.vector === null) { + await this.deleteObjectFromBranch(getVerbVectorPath(id)) + } else { + await this.writeObjectToBranch(getVerbVectorPath(id), record.vector) + } + } + + /** + * Append one line to the transaction log (`_system/tx-log.jsonl`). The + * filesystem adapter appends to a real JSONL file; the in-memory adapter + * keeps a line array (serialized by `snapshotToDirectory`). + * + * @param line - One complete JSON document, without trailing newline. + */ + public abstract appendTxLogLine(line: string): Promise + + /** + * Read all transaction-log lines, oldest first (empty array when no log + * exists). A torn trailing line from a crashed append is returned as-is — + * callers tolerate unparseable lines. + */ + public abstract readTxLogLines(): Promise + + /** + * Snapshot the entire store into `targetPath`. The filesystem adapter + * builds a hard-link farm (instant, space-shared — safe because data files + * are immutable-by-rename); the in-memory adapter serializes its object + * store to a filesystem-storage-compatible directory. The result is a + * self-contained store openable via `Brainy.load(path)`. + * + * @param targetPath - Absolute directory path for the snapshot (created if + * missing; must be empty or absent). + */ + public abstract snapshotToDirectory(targetPath: string): Promise + + /** + * Replace the entire store's contents from a snapshot directory previously + * produced by {@link BaseStorage.snapshotToDirectory}. Implementations + * clear current contents (preserving live lock files), copy the snapshot + * in (byte copy — never hard links, so the snapshot stays independent), + * and then call {@link BaseStorage.reloadDerivedState}. + * + * @param sourcePath - Absolute path of the snapshot directory. + */ + public abstract restoreFromDirectory(sourcePath: string): Promise + + /** + * Reset and reload every piece of adapter-internal derived state after the + * underlying objects changed wholesale (restore-from-snapshot): the + * write-through cache, the id→type/subtype caches, type/subtype statistics, + * total counts, and the graph-index singleton (invalidated so the next + * accessor rebuilds from the restored verbs). + */ + protected async reloadDerivedState(): Promise { + this.clearWriteCache() + this.nounTypeByIdCache.clear() + this.nounSubtypeByIdCache.clear() + this.verbSubtypeByIdCache.clear() + this.nounCountsByType.fill(0) + this.verbCountsByType.fill(0) + this.subtypeCountsByType.clear() + this.verbSubtypeCountsByType.clear() + this.statisticsCache = null + this.statisticsModified = false + this.invalidateGraphIndex() + await this.loadTypeStatistics() + await this.loadSubtypeStatistics() + await this.loadVerbSubtypeStatistics() + await this.initializeCounts() + } + /** * Save a noun to storage (vector only, metadata saved separately) * @param noun Pure HNSW vector data (no metadata) @@ -2315,6 +2580,10 @@ export abstract class BaseStorage extends BaseStorageAdapter { // Ignore persist errors - will retry on next operation }) } + + // 8.0 MVCC: entity-visible write — advance the generation watermark + // (suppressed inside transact batches by the generation store). + this.generationBumpHook?.() } /** @@ -2717,6 +2986,10 @@ export abstract class BaseStorage extends BaseStorageAdapter { this.decrementSubtypeCount(priorType, priorSubtype) } } + + // 8.0 MVCC: entity-visible write — advance the generation watermark + // (suppressed inside transact batches by the generation store). + this.generationBumpHook?.() } /** @@ -2750,6 +3023,8 @@ export abstract class BaseStorage extends BaseStorageAdapter { // Backward compatibility: fallback to old path if no verb type const keyInfo = this.analyzeKey(id, 'verb-metadata') await this.writeObjectToBranch(keyInfo.fullPath, metadata) + // 8.0 MVCC: still an entity-visible write — advance the watermark. + this.generationBumpHook?.() return } @@ -2804,6 +3079,10 @@ export abstract class BaseStorage extends BaseStorageAdapter { // Ignore persist errors - will retry on next operation }) } + + // 8.0 MVCC: entity-visible write — advance the generation watermark + // (suppressed inside transact batches by the generation store). + this.generationBumpHook?.() } /** @@ -2841,6 +3120,10 @@ export abstract class BaseStorage extends BaseStorageAdapter { this.verbSubtypeByIdCache.delete(id) this.decrementVerbSubtypeCount(priorEntry.verb, priorEntry.subtype) } + + // 8.0 MVCC: entity-visible write — advance the generation watermark + // (suppressed inside transact batches by the generation store). + this.generationBumpHook?.() } // ============================================================================ diff --git a/tests/integration/db-mvcc.test.ts b/tests/integration/db-mvcc.test.ts new file mode 100644 index 00000000..26b482a9 --- /dev/null +++ b/tests/integration/db-mvcc.test.ts @@ -0,0 +1,955 @@ +/** + * @module tests/integration/db-mvcc + * @description Credibility-bar proof suite for Brainy 8.0's generational MVCC + * storage core + Datomic-style Db API. Each test PROVES a stated guarantee + * (not merely exercises the API): + * + * 1. Isolation — a pinned Db reads its pinned state exactly, across 200 + * subsequent transact mutations (updates + removes). + * 2. Atomicity — a batch with a failing later operation applies NOTHING + * (generation unchanged, no partial records), for both plan-time and + * execution-phase failures (the latter via real fault injection into the + * storage adapter, exercising the production rollback path). + * 3. CAS — `ifAtGeneration` commits when fresh, throws + * `GenerationConflictError` with correct expected/actual when stale. + * 4. Snapshot integrity — persist → mutate source heavily → `Brainy.load()` + * is byte-identical to the pre-mutation state (hard-link safety), and an + * in-memory brain persists to a loadable directory. + * 5. Compaction safety — pinned reads stay correct across `compactHistory()`; + * after release + compact the superseded record-sets are reclaimed. + * 6. with() — speculative overlays are visible on the overlay Db only; the + * underlying brain and the original Db are untouched. + * 7. Generation monotonicity across close/reopen. + * 8. Crash consistency — a simulated crash between record staging and the + * manifest rename (the commit point) recovers to the exact pre-transaction + * state on reopen, through the REAL production recovery path. + * 9. Versioned provider wiring — a provider implementing the 4-method + * `VersionedIndexProvider` contract receives balanced pin/release (and a + * visibility consult per pin) as Db values are created and released. + * + * All entities carry explicit vectors so the suite never invokes the + * embedding engine (semantic search is exercised only for its documented + * historical-generation error). + */ + +import { describe, it, expect, afterEach, vi } from 'vitest' +import * as fs from 'node:fs' +import * as os from 'node:os' +import * as path from 'node:path' +import { Brainy } from '../../src/brainy.js' +import { Db } from '../../src/db/db.js' +import { + GenerationCompactedError, + GenerationConflictError, + NotYetSupportedAtHistoricalGenerationError +} from '../../src/db/errors.js' +import type { GenerationStore } from '../../src/db/generationStore.js' +import { GraphAdjacencyIndex } from '../../src/graph/graphAdjacencyIndex.js' +import { RevisionConflictError } from '../../src/transaction/RevisionConflictError.js' +import type { StorageAdapter } from '../../src/coreTypes.js' +import { NounType, VerbType } from '../../src/types/graphTypes.js' + +/** Deterministic 384-dim vector so no test ever invokes the embedder. */ +function vec(seed: number): number[] { + return Array.from({ length: 384 }, (_, i) => ((seed * 31 + i * 7) % 100) / 100) +} + +/** + * Map a readable label to a deterministic UUID-shaped id (entity ids must be + * UUIDs — the sharded storage layout derives the shard from the UUID hex). + * Two independent FNV-style mixes keep 7+ distinct labels collision-free. + */ +function uid(label: string): string { + let h1 = 0x811c9dc5 + for (let i = 0; i < label.length; i++) { + h1 = Math.imul(h1 ^ label.charCodeAt(i), 0x01000193) >>> 0 + } + let h2 = 0xdeadbeef + for (let i = label.length - 1; i >= 0; i--) { + h2 = Math.imul(h2 ^ label.charCodeAt(i), 0x85ebca6b) >>> 0 + } + const hex = h1.toString(16).padStart(8, '0') + h2.toString(16).padStart(8, '0') + return `00000000-0000-4000-8000-${hex.slice(0, 12)}` +} + +/** Typed access to the brain's private generation store (test injection point). */ +function generationStoreOf(brain: Brainy): GenerationStore { + return (brain as unknown as { generationStore: GenerationStore }).generationStore +} + +describe('8.0 Db API — generational MVCC', () => { + const brains: Brainy[] = [] + const tempDirs: string[] = [] + + /** Track a brain for afterEach teardown. */ + async function openMemoryBrain(): Promise { + const brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' } }) + await brain.init() + brains.push(brain) + return brain + } + + /** Open (and track) a filesystem brain rooted at a fresh temp directory. */ + async function openFsBrain(dir?: string): Promise<{ brain: Brainy; dir: string }> { + const rootDirectory = dir ?? makeTempDir() + const brain = new Brainy({ + requireSubtype: false, + storage: { type: 'filesystem', rootDirectory } + }) + await brain.init() + brains.push(brain) + return { brain, dir: rootDirectory } + } + + function makeTempDir(): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-db-mvcc-')) + tempDirs.push(dir) + return dir + } + + afterEach(async () => { + for (const brain of brains.splice(0)) { + try { + await brain.close() + } catch { + // already closed by the test + } + } + for (const dir of tempDirs.splice(0)) { + await fs.promises.rm(dir, { recursive: true, force: true }) + } + }) + + // ========================================================================== + // 1. Isolation + // ========================================================================== + it('proof 1 — a pinned Db reads its pinned state exactly across 200 transact mutations', async () => { + const brain = await openMemoryBrain() + + const ids = Array.from({ length: 20 }, (_, i) => uid(`iso-e${i}`)) + await brain.transact( + ids.map((id, i) => ({ + op: 'add' as const, + id, + type: NounType.Document, + data: `doc-${i}`, + vector: vec(i), + metadata: { v: 0, slot: i } + })) + ) + + const db = brain.now() + const pinned = new Map() + for (const id of ids) { + pinned.set(id, await db.get(id)) + expect(pinned.get(id)).not.toBeNull() + } + const pinnedFind = await db.find({ type: NounType.Document, limit: 100 }) + expect(pinnedFind.length).toBe(20) + + // 200 mutations: 198 updates over the first 18 ids + 2 removes. + for (let i = 0; i < 200; i++) { + if (i === 100) { + await brain.transact([{ op: 'remove', id: uid('iso-e18') }]) + } else if (i === 150) { + await brain.transact([{ op: 'remove', id: uid('iso-e19') }]) + } else { + await brain.transact([ + { op: 'update', id: ids[i % 18], metadata: { v: i + 1, touchedAt: i } } + ]) + } + } + + // Live state moved… + const liveUpdated = await brain.get(ids[0]) + expect((liveUpdated?.metadata as { v: number }).v).toBeGreaterThan(0) + expect(await brain.get(uid('iso-e18'))).toBeNull() + expect(await brain.get(uid('iso-e19'))).toBeNull() + + // …but EVERY read through the pinned Db is exactly the pinned state, + // including the two entities removed after the pin. + for (const id of ids) { + expect(await db.get(id)).toEqual(pinned.get(id)) + } + + // Set-level isolation: metadata find() at the pin still sees all 20. + const histFind = await db.find({ type: NounType.Document, limit: 100 }) + expect(histFind.length).toBe(20) + for (const row of histFind) { + expect((row.metadata as { v: number }).v).toBe(0) + } + + await db.release() + }) + + // ========================================================================== + // 2. Atomicity + // ========================================================================== + it('proof 2a — a plan-time failure in a later op applies nothing', async () => { + const brain = await openMemoryBrain() + const g0 = brain.generation() + + await expect( + brain.transact([ + { + op: 'add', + id: uid('atomic-a'), + type: NounType.Document, + data: 'a', + vector: vec(1), + metadata: { ok: true } + }, + { op: 'update', id: uid('never-created'), metadata: { boom: true } } + ]) + ).rejects.toThrow(`Entity ${uid('never-created')} not found`) + + expect(brain.generation()).toBe(g0) + expect(await brain.get(uid('atomic-a'))).toBeNull() + expect((await brain.find({ type: NounType.Document, limit: 10 })).length).toBe(0) + }) + + it('proof 2b — an execution-phase failure rolls back every applied op (generation unchanged, zero partial records)', async () => { + const brain = await openMemoryBrain() + + await brain.transact([ + { + op: 'add', + id: uid('atomic-base'), + type: NounType.Document, + data: 'base', + vector: vec(0), + metadata: { base: true } + } + ]) + const g0 = brain.generation() + const storage = brain.storageAdapter + + // Real fault injection: the verb-metadata write of the LAST operation + // fails once, after the earlier add operations have already executed — + // exercising the production TransactionManager rollback path. + const spy = vi + .spyOn(storage, 'saveVerbMetadata') + .mockImplementationOnce(async () => { + throw new Error('disk full (injected)') + }) + + await expect( + brain.transact([ + { + op: 'add', + id: uid('atomic-b'), + type: NounType.Person, + data: 'b', + vector: vec(2), + metadata: { name: 'b' } + }, + { + op: 'relate', + from: uid('atomic-b'), + to: uid('atomic-base'), + type: VerbType.RelatedTo + } + ]) + ).rejects.toThrow('disk full (injected)') + spy.mockRestore() + + // ZERO ops applied: generation unchanged, the add rolled back, no edge. + expect(brain.generation()).toBe(g0) + expect(await brain.get(uid('atomic-b'))).toBeNull() + expect((await brain.getRelations({ from: uid('atomic-b') })).length).toBe(0) + expect((await brain.find({ type: NounType.Person, limit: 10 })).length).toBe(0) + + // No partial generation records: the staging directory was removed. + const records = await storage.listRawObjects(`_generations/${g0 + 1}`) + expect(records).toEqual([]) + + // The store remains fully writable and consistent after the rollback. + const db = await brain.transact([ + { + op: 'add', + id: uid('atomic-c'), + type: NounType.Person, + data: 'c', + vector: vec(3), + metadata: { name: 'c' } + } + ]) + expect(db.generation).toBe(g0 + 1) + await db.release() + }) + + it('proof 2c — an ifRev conflict on a later update rejects the whole batch', async () => { + const brain = await openMemoryBrain() + await brain.transact([ + { + op: 'add', + id: uid('rev-e'), + type: NounType.Document, + data: 'rev', + vector: vec(4), + metadata: { v: 1 } + } + ]) + await brain.update({ id: uid('rev-e'), metadata: { v: 2 } }) // _rev is now 2 + const g0 = brain.generation() + + await expect( + brain.transact([ + { + op: 'add', + id: uid('rev-new'), + type: NounType.Document, + data: 'new', + vector: vec(5), + metadata: {} + }, + { op: 'update', id: uid('rev-e'), metadata: { v: 3 }, ifRev: 1 } // stale + ]) + ).rejects.toThrow(RevisionConflictError) + + expect(brain.generation()).toBe(g0) + expect(await brain.get(uid('rev-new'))).toBeNull() + expect(((await brain.get(uid('rev-e')))?.metadata as { v: number }).v).toBe(2) + }) + + // ========================================================================== + // 3. CAS (ifAtGeneration) + // ========================================================================== + it('proof 3 — ifAtGeneration commits when fresh and conflicts when stale', async () => { + const brain = await openMemoryBrain() + const db = brain.now() + + const committed = await brain.transact( + [ + { + op: 'add', + id: uid('cas-e'), + type: NounType.Document, + data: 'cas', + vector: vec(6), + metadata: { n: 1 } + } + ], + { ifAtGeneration: db.generation } + ) + expect(committed.generation).toBe(db.generation + 1) + + try { + await brain.transact( + [{ op: 'update', id: uid('cas-e'), metadata: { n: 2 } }], + { ifAtGeneration: db.generation } // stale — committed moved past it + ) + expect.unreachable('should have thrown GenerationConflictError') + } catch (err) { + expect(err).toBeInstanceOf(GenerationConflictError) + expect((err as GenerationConflictError).expected).toBe(db.generation) + expect((err as GenerationConflictError).actual).toBe(committed.generation) + } + + // Nothing committed by the conflicting attempt. + expect(((await brain.get(uid('cas-e')))?.metadata as { n: number }).n).toBe(1) + expect(brain.generation()).toBe(committed.generation) + + await db.release() + await committed.release() + }) + + // ========================================================================== + // 4. Snapshot integrity (persist → mutate → load) + // ========================================================================== + it('proof 4a — a persisted snapshot is immune to heavy source mutation (hard-link safety)', async () => { + const { brain } = await openFsBrain() + + const ids = Array.from({ length: 10 }, (_, i) => uid(`snap-e${i}`)) + await brain.transact( + ids.map((id, i) => ({ + op: 'add' as const, + id, + type: NounType.Document, + data: `snapshot-doc-${i}`, + vector: vec(i + 10), + metadata: { generation: 'pre', slot: i } + })) + ) + const relDb = await brain.transact([ + { op: 'relate', from: ids[0], to: ids[1], type: VerbType.References }, + { op: 'relate', from: ids[1], to: ids[2], type: VerbType.References } + ]) + await relDb.release() + + // Capture pre-mutation state, then snapshot. + const preState = new Map() + for (const id of ids) { + preState.set(id, await brain.get(id)) + } + const preRelations = await brain.getRelations({ from: ids[0] }) + expect(preRelations.length).toBe(1) + + const snapDir = path.join(makeTempDir(), 'snapshot') + const db = brain.now() + await db.persist(snapDir) + await db.release() + + // Mutate the source heavily: rewrite every entity, delete some, add new. + for (const [i, id] of ids.entries()) { + await brain.transact([ + { op: 'update', id, data: `mutated-${i}`, metadata: { generation: 'post' } } + ]) + } + await brain.transact([{ op: 'remove', id: ids[0] }]) + await brain.transact([ + { + op: 'add', + id: uid('snap-after'), + type: NounType.Document, + data: 'added after snapshot', + vector: vec(99), + metadata: { generation: 'post' } + } + ]) + await brain.flush() + + // The snapshot opens as a self-contained store with the PRE state. + const loaded = await Brainy.load(snapDir) + for (const id of ids) { + const entity = await loaded.get(id) + expect(entity).toEqual(preState.get(id)) + } + expect(await loaded.get(uid('snap-after'))).toBeNull() + const loadedFind = await loaded.find({ type: NounType.Document, limit: 100 }) + expect(loadedFind.length).toBe(10) + for (const row of loadedFind) { + expect((row.metadata as { generation: string }).generation).toBe('pre') + } + const loadedRelations = await loaded.related({ from: ids[0] }) + expect(loadedRelations.length).toBe(1) + expect(loadedRelations[0].to).toBe(ids[1]) + await loaded.release() + + // And asOf(path) is the instance-method route to the same snapshot. + const viaAsOf = await brain.asOf(snapDir) + expect(((await viaAsOf.get(ids[3])) as { metadata: { generation: string } }).metadata.generation).toBe('pre') + await viaAsOf.release() + }) + + it('proof 4b — an in-memory brain persists to a directory loadable as a real store', async () => { + const brain = await openMemoryBrain() + await brain.transact([ + { + op: 'add', + id: uid('mem-e1'), + type: NounType.Document, + data: 'memory-persisted', + vector: vec(21), + metadata: { from: 'memory' } + } + ]) + + const snapDir = path.join(makeTempDir(), 'memory-snapshot') + const db = brain.now() + await db.persist(snapDir) + await db.release() + + const loaded = await Brainy.load(snapDir) + const entity = await loaded.get(uid('mem-e1')) + expect(entity).not.toBeNull() + expect((entity?.metadata as { from: string }).from).toBe('memory') + await loaded.release() + }) + + it('proof 4c — persist() refuses a view that history has moved past (honest boundary)', async () => { + const brain = await openMemoryBrain() + const db = brain.now() + await brain.transact([ + { + op: 'add', + id: uid('stale-persist'), + type: NounType.Document, + data: 'x', + vector: vec(30), + metadata: {} + } + ]) + + const snapDir = path.join(makeTempDir(), 'stale-snapshot') + await expect(db.persist(snapDir)).rejects.toThrow( + NotYetSupportedAtHistoricalGenerationError + ) + await db.release() + }) + + // ========================================================================== + // 5. Compaction safety + // ========================================================================== + it('proof 5 — compaction never breaks pinned reads; after release the records are reclaimed', async () => { + const { brain } = await openFsBrain() + const storage = brain.storageAdapter + + // Every transact() returns a PINNED Db — release the ones this test + // does not keep, so compaction eligibility is determined solely by `db`. + await ( + await brain.transact([ + { + op: 'add', + id: uid('compact-e'), + type: NounType.Document, + data: 'v1', + vector: vec(40), + metadata: { v: 1 } + } + ]) + ).release() + await ( + await brain.transact([{ op: 'update', id: uid('compact-e'), metadata: { v: 2 } }]) + ).release() + const db = brain.now() // pinned at v2 + const pinnedEntity = await db.get(uid('compact-e')) + await ( + await brain.transact([{ op: 'update', id: uid('compact-e'), metadata: { v: 3 } }]) + ).release() + await ( + await brain.transact([{ op: 'update', id: uid('compact-e'), metadata: { v: 4 } }]) + ).release() + + const recordsBefore = (await storage.listRawObjects('_generations')).length + expect(recordsBefore).toBeGreaterThan(0) + + // Compact while pinned: record-sets above the pin survive, pinned reads stay correct. + const first = await brain.compactHistory() + expect(await db.get(uid('compact-e'))).toEqual(pinnedEntity) + expect(((await db.get(uid('compact-e')))?.metadata as { v: number }).v).toBe(2) + + // After release, everything is reclaimable. + await db.release() + const second = await brain.compactHistory() + expect(first.removedGenerations + second.removedGenerations).toBeGreaterThan(0) + + const recordsAfter = (await storage.listRawObjects('_generations')).length + expect(recordsAfter).toBeLessThan(recordsBefore) + expect(recordsAfter).toBe(0) + + // Compacted generations are honestly unreachable. + await expect(brain.asOf(1)).rejects.toThrow(GenerationCompactedError) + }) + + // ========================================================================== + // 6. with() — speculative overlays + // ========================================================================== + it('proof 6 — with() overlays are visible only on the overlay Db; brain and base Db untouched', async () => { + const brain = await openMemoryBrain() + await brain.transact([ + { + op: 'add', + id: uid('with-a'), + type: NounType.Person, + data: 'Ada', + vector: vec(50), + metadata: { v: 1 } + }, + { + op: 'add', + id: uid('with-b'), + type: NounType.Person, + data: 'Grace', + vector: vec(51), + metadata: { v: 1 } + } + ]) + const g0 = brain.generation() + const db = brain.now() + + const spec = await db.with([ + { op: 'update', id: uid('with-a'), metadata: { v: 2 } }, + { + op: 'add', + id: uid('with-new'), + type: NounType.Project, + data: 'Apollo', + metadata: { speculative: true } + }, + { op: 'remove', id: uid('with-b') }, + { op: 'relate', from: uid('with-a'), to: uid('with-new'), type: VerbType.WorksOn } + ]) + + // The overlay view sees everything… + expect(spec.speculative).toBe(true) + expect(((await spec.get(uid('with-a')))?.metadata as { v: number }).v).toBe(2) + expect(await spec.get(uid('with-new'))).not.toBeNull() + expect(await spec.get(uid('with-b'))).toBeNull() + const specEdges = await spec.related({ from: uid('with-a') }) + expect(specEdges.length).toBe(1) + expect(specEdges[0].to).toBe(uid('with-new')) + const specFind = await spec.find({ type: NounType.Project, limit: 10 }) + expect(specFind.length).toBe(1) + + // …the base Db sees none of it… + expect(db.speculative).toBe(false) + expect(((await db.get(uid('with-a')))?.metadata as { v: number }).v).toBe(1) + expect(await db.get(uid('with-new'))).toBeNull() + expect(await db.get(uid('with-b'))).not.toBeNull() + + // …and the brain (disk, indexes, generation counter) is untouched. + expect(brain.generation()).toBe(g0) + expect(((await brain.get(uid('with-a')))?.metadata as { v: number }).v).toBe(1) + expect(await brain.get(uid('with-new'))).toBeNull() + expect((await brain.getRelations({ from: uid('with-a') })).length).toBe(0) + + await spec.release() + await db.release() + }) + + // ========================================================================== + // 7. Generation monotonicity across close/reopen + // ========================================================================== + it('proof 7 — the generation counter never moves backwards across close/reopen', async () => { + const dir = makeTempDir() + const { brain: first } = await openFsBrain(dir) + + await first.transact([ + { + op: 'add', + id: uid('mono-e'), + type: NounType.Document, + data: 'v1', + vector: vec(60), + metadata: { v: 1 } + } + ]) + await first.transact([{ op: 'update', id: uid('mono-e'), metadata: { v: 2 } }]) + // Single-operation write — bumps the counter outside transact. + await first.update({ id: uid('mono-e'), metadata: { v: 3 } }) + const beforeClose = first.generation() + expect(beforeClose).toBeGreaterThanOrEqual(3) + await first.close() + + const { brain: second } = await openFsBrain(dir) + expect(second.generation()).toBeGreaterThanOrEqual(beforeClose) + + const db = await second.transact([{ op: 'update', id: uid('mono-e'), metadata: { v: 4 } }]) + expect(db.generation).toBeGreaterThan(beforeClose) + await db.release() + }) + + // ========================================================================== + // 8. Crash consistency + // ========================================================================== + it('proof 8 — a crash before the manifest rename recovers to the exact pre-transaction state', async () => { + const dir = makeTempDir() + const { brain: first } = await openFsBrain(dir) + + await first.transact([ + { + op: 'add', + id: uid('crash-e'), + type: NounType.Document, + data: 'stable', + vector: vec(70), + metadata: { v: 1 } + } + ]) + const committedGen = first.generation() + + // Simulated crash at the worst point of the REAL commit protocol: every + // record is staged and the batch has executed, but the atomic manifest + // rename — the commit point — never happens. The throwing injector skips + // the abort cleanup exactly as a dead process would. + generationStoreOf(first).setCommitFaultInjector((phase) => { + if (phase === 'before-manifest-rename') { + throw new Error('simulated process crash') + } + }) + await expect( + first.transact([ + { op: 'update', id: uid('crash-e'), metadata: { v: 2 } }, + { + op: 'add', + id: uid('crash-new'), + type: NounType.Document, + data: 'uncommitted', + vector: vec(71), + metadata: {} + } + ]) + ).rejects.toThrow('simulated process crash') + generationStoreOf(first).setCommitFaultInjector(undefined) + + // The in-process state is dirty (batch executed) — close flushes it all, + // the realistic worst case for the recovery path. + await first.close() + + // Reopen: recovery rolls the uncommitted generation back and rebuilds + // the indexes from the repaired records. + const { brain: second } = await openFsBrain(dir) + const recovered = await second.get(uid('crash-e')) + expect((recovered?.metadata as { v: number }).v).toBe(1) + expect(await second.get(uid('crash-new'))).toBeNull() + const found = await second.find({ type: NounType.Document, limit: 10 }) + expect(found.length).toBe(1) + expect((found[0].metadata as { v: number }).v).toBe(1) + + // The crashed generation number is never reissued. + expect(second.generation()).toBeGreaterThanOrEqual(committedGen + 1) + const db = await second.transact([{ op: 'update', id: uid('crash-e'), metadata: { v: 5 } }]) + expect(db.generation).toBeGreaterThan(committedGen + 1) + await db.release() + }) + + // ========================================================================== + // 9. Versioned provider wiring + // ========================================================================== + it('proof 9 — a VersionedIndexProvider receives balanced pin/release as Db values live and die', async () => { + const calls = { + pins: [] as bigint[], + releases: [] as bigint[], + visibility: [] as bigint[] + } + + const brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' } }) + brain.use({ + name: 'test-versioned-graph-index', + activate: async (ctx) => { + ctx.registerProvider('graphIndex', (storage: StorageAdapter) => + Object.assign(new GraphAdjacencyIndex(storage), { + generation: () => 0n, + isGenerationVisible: (g: bigint) => { + calls.visibility.push(g) + return true + }, + pin: (g: bigint) => { + calls.pins.push(g) + }, + release: (g: bigint) => { + calls.releases.push(g) + } + }) + ) + return true + } + }) + await brain.init() + brains.push(brain) + + const base = brain.generation() + const db1 = brain.now() + expect(calls.pins).toEqual([BigInt(base)]) + expect(calls.visibility).toEqual([BigInt(base)]) // consulted at pin time + + const db2 = await brain.transact([ + { + op: 'add', + id: uid('vp-e'), + type: NounType.Document, + data: 'pinned', + vector: vec(80), + metadata: {} + } + ]) + expect(calls.pins).toEqual([BigInt(base), BigInt(base + 1)]) + + const db3 = await brain.asOf(base) + expect(calls.pins).toEqual([BigInt(base), BigInt(base + 1), BigInt(base)]) + + // A with() overlay takes its own pin on the same generation. + const spec = await db2.with([{ op: 'update', id: uid('vp-e'), metadata: { spec: true } }]) + expect(calls.pins.length).toBe(4) + + await db1.release() + await db2.release() + await db3.release() + await spec.release() + await spec.release() // idempotent — must NOT double-release the pin + + expect(calls.releases.length).toBe(calls.pins.length) + expect([...calls.releases].sort()).toEqual([...calls.pins].sort()) + }) + + // ========================================================================== + // Supporting guarantees: asOf, since, receipts, tx-log, honest errors + // ========================================================================== + it('asOf(generation) and asOf(Date) resolve historical state; future/negative are rejected', async () => { + const brain = await openMemoryBrain() + + const tx1 = await brain.transact( + [ + { + op: 'add', + id: uid('asof-e'), + type: NounType.Document, + data: 'v1', + vector: vec(90), + metadata: { v: 1 } + } + ], + { meta: { reason: 'first commit' } } + ) + await new Promise((resolve) => setTimeout(resolve, 5)) + const betweenCommits = new Date() + await new Promise((resolve) => setTimeout(resolve, 5)) + await brain.transact([{ op: 'update', id: uid('asof-e'), metadata: { v: 2 } }]) + + // By generation. + const atFirst = await brain.asOf(tx1.generation) + expect(((await atFirst.get(uid('asof-e')))?.metadata as { v: number }).v).toBe(1) + expect(atFirst.timestamp).toBe(tx1.timestamp) + await atFirst.release() + + // By timestamp: resolves to the newest commit at-or-before the instant. + const atInstant = await brain.asOf(betweenCommits) + expect(atInstant.generation).toBe(tx1.generation) + expect(((await atInstant.get(uid('asof-e')))?.metadata as { v: number }).v).toBe(1) + await atInstant.release() + + // Honest range errors. + await expect(brain.asOf(brain.generation() + 100)).rejects.toThrow(RangeError) + await expect(brain.asOf(-1)).rejects.toThrow(RangeError) + await expect(brain.asOf('/no/such/snapshot/dir')).rejects.toThrow('not a snapshot directory') + + // The transact meta was reified into the tx-log. + const lines = await brain.storageAdapter.readTxLogLines() + const entries = lines.map((line) => JSON.parse(line)) + const first = entries.find((entry) => entry.generation === tx1.generation) + expect(first?.meta).toEqual({ reason: 'first commit' }) + + await tx1.release() + }) + + it('receipts resolve ids in input order, including intra-batch references and relate dedupe', async () => { + const brain = await openMemoryBrain() + + const db = await brain.transact([ + { op: 'add', id: uid('rcpt-aa'), type: NounType.Person, data: 'a', vector: vec(91), metadata: {} }, + { op: 'add', id: uid('rcpt-bb'), type: NounType.Person, data: 'b', vector: vec(92), metadata: {} }, + { op: 'relate', from: uid('rcpt-aa'), to: uid('rcpt-bb'), type: VerbType.Knows }, + { op: 'update', id: uid('rcpt-bb'), metadata: { seen: true } } + ]) + expect(db.receipt).toBeDefined() + expect(db.receipt!.generation).toBe(db.generation) + expect(db.receipt!.ids.length).toBe(4) + expect(db.receipt!.ids[0]).toBe(uid('rcpt-aa')) + expect(db.receipt!.ids[1]).toBe(uid('rcpt-bb')) + expect(db.receipt!.ids[3]).toBe(uid('rcpt-bb')) + const relationId = db.receipt!.ids[2] + expect((await brain.getRelations({ from: uid('rcpt-aa') }))[0].id).toBe(relationId) + + // Duplicate relate (same from/to/type) dedupes to the SAME id — both + // within a batch and against committed state, mirroring relate(). + const dup = await brain.transact([ + { op: 'relate', from: uid('rcpt-aa'), to: uid('rcpt-bb'), type: VerbType.Knows } + ]) + expect(dup.receipt!.ids[0]).toBe(relationId) + expect((await brain.getRelations({ from: uid('rcpt-aa') })).length).toBe(1) + + await db.release() + await dup.release() + }) + + it('since() reports the ids changed between two pinned views', async () => { + const brain = await openMemoryBrain() + await brain.transact([ + { op: 'add', id: uid('since-a'), type: NounType.Document, data: 'a', vector: vec(93), metadata: {} } + ]) + const before = brain.now() + + await brain.transact([ + { op: 'add', id: uid('since-b'), type: NounType.Document, data: 'b', vector: vec(94), metadata: {} }, + { op: 'relate', from: uid('since-a'), to: uid('since-b'), type: VerbType.References } + ]) + await brain.transact([{ op: 'update', id: uid('since-a'), metadata: { touched: true } }]) + const after = brain.now() + + const changed = await after.since(before) + expect([...changed.nouns].sort()).toEqual([uid('since-a'), uid('since-b')].sort()) + expect(changed.verbs.length).toBe(1) + expect(changed.fromGeneration).toBe(before.generation) + expect(changed.toGeneration).toBe(after.generation) + + // Direction is enforced. + await expect(before.since(after)).rejects.toThrow(RangeError) + + await before.release() + await after.release() + }) + + it('index-accelerated queries at a historical generation throw the documented error — never silently-wrong results', async () => { + const brain = await openMemoryBrain() + await brain.transact([ + { op: 'add', id: uid('hist-e'), type: NounType.Document, data: 'x', vector: vec(95), metadata: { v: 1 } } + ]) + const db = brain.now() + await brain.transact([{ op: 'update', id: uid('hist-e'), metadata: { v: 2 } }]) + + // Storage-record reads remain fully supported… + expect(((await db.get(uid('hist-e')))?.metadata as { v: number }).v).toBe(1) + expect((await db.find({ type: NounType.Document, where: { v: 1 } })).length).toBe(1) + + // …index-accelerated dimensions throw the documented, named error. + await expect(db.search('anything')).rejects.toThrow( + NotYetSupportedAtHistoricalGenerationError + ) + await expect(db.find({ vector: vec(95) })).rejects.toThrow( + NotYetSupportedAtHistoricalGenerationError + ) + await expect(db.find({ connected: { from: uid('hist-e') } })).rejects.toThrow( + NotYetSupportedAtHistoricalGenerationError + ) + + // Released views refuse all reads. + await db.release() + await expect(db.get(uid('hist-e'))).rejects.toThrow('released Db') + }) + + it('restore() requires confirmation and replaces state from a snapshot (generation floor preserved)', async () => { + const { brain } = await openFsBrain() + await brain.transact([ + { op: 'add', id: uid('restore-e'), type: NounType.Document, data: 'keep', vector: vec(96), metadata: { v: 1 } } + ]) + + const snapDir = path.join(makeTempDir(), 'restore-snapshot') + const db = brain.now() + await db.persist(snapDir) + await db.release() + + await brain.transact([{ op: 'update', id: uid('restore-e'), metadata: { v: 2 } }]) + await brain.transact([ + { op: 'add', id: uid('restore-extra'), type: NounType.Document, data: 'extra', vector: vec(97), metadata: {} } + ]) + const beforeRestore = brain.generation() + + await expect( + brain.restore(snapDir, { confirm: false }) + ).rejects.toThrow('confirm') + + await brain.restore(snapDir, { confirm: true }) + expect(((await brain.get(uid('restore-e')))?.metadata as { v: number }).v).toBe(1) + expect(await brain.get(uid('restore-extra'))).toBeNull() + expect((await brain.find({ type: NounType.Document, limit: 10 })).length).toBe(1) + + // The counter never moves backwards — pre-restore generations are not reissued. + expect(brain.generation()).toBeGreaterThanOrEqual(beforeRestore) + const after = await brain.transact([{ op: 'update', id: uid('restore-e'), metadata: { v: 9 } }]) + expect(after.generation).toBeGreaterThan(beforeRestore) + await after.release() + }) + + it('Db values are first-class: Brainy.open() + chained with() overlays compose', async () => { + const brain = await Brainy.open({ requireSubtype: false, storage: { type: 'memory' } }) + brains.push(brain) + expect(brain.isInitialized).toBe(true) + + await brain.transact([ + { op: 'add', id: uid('chain-e'), type: NounType.Document, data: 'x', vector: vec(98), metadata: { v: 1 } } + ]) + const db = brain.now() + const spec1 = await db.with([{ op: 'update', id: uid('chain-e'), metadata: { v: 2 } }]) + const spec2 = await spec1.with([{ op: 'update', id: uid('chain-e'), metadata: { v: 3 } }]) + + expect(((await db.get(uid('chain-e')))?.metadata as { v: number }).v).toBe(1) + expect(((await spec1.get(uid('chain-e')))?.metadata as { v: number }).v).toBe(2) + expect(((await spec2.get(uid('chain-e')))?.metadata as { v: number }).v).toBe(3) + expect(spec2).toBeInstanceOf(Db) + + await spec2.release() + await spec1.release() + await db.release() + }) +}) diff --git a/tests/unit/db/generationStore.test.ts b/tests/unit/db/generationStore.test.ts new file mode 100644 index 00000000..f847b92a --- /dev/null +++ b/tests/unit/db/generationStore.test.ts @@ -0,0 +1,333 @@ +/** + * @module tests/unit/db/generationStore + * @description Unit tests for the generational MVCC record layer + * (`src/db/generationStore.ts`) against a real `MemoryStorage` adapter — + * counter monotonicity, the commit protocol (staging → execute → manifest + * rename → tx-log), point-in-time resolution from before-images, pin + * refcounting, compaction retention rules, CAS conflicts, timestamp + * resolution, and crash rollback of uncommitted generations. + */ + +import { describe, it, expect, beforeEach } from 'vitest' +import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js' +import { + GenerationStore, + GENERATIONS_PREFIX, + MANIFEST_PATH +} from '../../../src/db/generationStore.js' +import { + GenerationCompactedError, + GenerationConflictError +} from '../../../src/db/errors.js' +import { NounType } from '../../../src/types/graphTypes.js' + +/** Stored-metadata fixture in the canonical shape the live write paths use. */ +function metadataFixture(version: number): Record { + return { + noun: NounType.Document, + subtype: 'note', + data: `payload-v${version}`, + version, + createdAt: 1000, + updatedAt: 1000 + version, + _rev: version + } +} + +// Entity ids must be UUID-shaped (the sharded storage layout derives the +// shard from the UUID hex). Fixed values keep assertions deterministic; +// suffix ordering (…aa < …bb < …cc) matches `changedBetween`'s sorted output. +const ID_A = '00000000-0000-4000-8000-0000000000aa' +const ID_B = '00000000-0000-4000-8000-0000000000bb' +const ID_C = '00000000-0000-4000-8000-0000000000cc' +const ID_SOLO = '00000000-0000-4000-8000-0000000000d0' +const ID_LATER = '00000000-0000-4000-8000-0000000000e0' + +describe('db/GenerationStore', () => { + let storage: MemoryStorage + let store: GenerationStore + + beforeEach(async () => { + storage = new MemoryStorage() + await storage.init() + store = new GenerationStore(storage) + await store.open() + }) + + /** Commit one transaction that writes `metadata` for `id`. */ + async function commitWrite( + id: string, + version: number, + options?: { meta?: Record; ifAtGeneration?: number } + ): Promise<{ generation: number; timestamp: number }> { + return store.commitTransaction({ + touched: { nouns: [id], verbs: [] }, + meta: options?.meta, + ifAtGeneration: options?.ifAtGeneration, + execute: async () => { + await storage.saveNounMetadata(id, metadataFixture(version)) + } + }) + } + + describe('generation counter', () => { + it('starts at 0 on a fresh store', () => { + expect(store.generation()).toBe(0) + expect(store.committedGeneration()).toBe(0) + }) + + it('advances by exactly one per committed transaction', async () => { + await commitWrite(ID_A, 1) + expect(store.generation()).toBe(1) + await commitWrite(ID_A, 2) + expect(store.generation()).toBe(2) + expect(store.committedGeneration()).toBe(2) + }) + + it('bumps on single-operation writes via the storage hook (suppressed inside transact)', async () => { + // Single-op write outside any transaction → hook bumps. + await storage.saveNounMetadata(ID_SOLO, metadataFixture(1)) + expect(store.generation()).toBe(1) + expect(store.committedGeneration()).toBe(0) // no transact committed + + // A transact batch writing twice bumps ONCE (the batch is one generation). + await store.commitTransaction({ + touched: { nouns: [ID_A, ID_B], verbs: [] }, + execute: async () => { + await storage.saveNounMetadata(ID_A, metadataFixture(1)) + await storage.saveNounMetadata(ID_B, metadataFixture(1)) + } + }) + expect(store.generation()).toBe(2) + }) + + it('survives close + reopen monotonically', async () => { + await commitWrite(ID_A, 1) + await storage.saveNounMetadata(ID_SOLO, metadataFixture(1)) // single-op bump + const before = store.generation() + await store.close() + + const reopened = new GenerationStore(storage) + await reopened.open() + expect(reopened.generation()).toBeGreaterThanOrEqual(before) + expect(reopened.committedGeneration()).toBe(1) + }) + }) + + describe('CAS (ifAtGeneration)', () => { + it('commits when the expectation matches and rejects with expected/actual when stale', async () => { + await commitWrite(ID_A, 1) + const pinned = store.generation() + + await commitWrite(ID_A, 2, { ifAtGeneration: pinned }) // fresh → commits + expect(store.generation()).toBe(pinned + 1) + + try { + await commitWrite(ID_A, 3, { ifAtGeneration: pinned }) // stale → rejects + expect.unreachable('should have thrown GenerationConflictError') + } catch (err) { + expect(err).toBeInstanceOf(GenerationConflictError) + expect((err as GenerationConflictError).expected).toBe(pinned) + expect((err as GenerationConflictError).actual).toBe(pinned + 1) + } + expect(store.generation()).toBe(pinned + 1) // nothing moved + }) + }) + + describe('failed transactions', () => { + it('removes the staging directory and returns the generation reservation', async () => { + await commitWrite(ID_A, 1) + const before = store.generation() + + await expect( + store.commitTransaction({ + touched: { nouns: [ID_A], verbs: [] }, + execute: async () => { + throw new Error('boom mid-batch') + } + }) + ).rejects.toThrow('boom mid-batch') + + expect(store.generation()).toBe(before) + const leftovers = await storage.listRawObjects(`${GENERATIONS_PREFIX}/${before + 1}`) + expect(leftovers).toEqual([]) + }) + }) + + describe('point-in-time resolution', () => { + it('resolves untouched ids to the live fast path', async () => { + await commitWrite(ID_A, 1) + const pinned = store.generation() + await commitWrite(ID_B, 1) // later commit touches a DIFFERENT id + + expect(await store.resolveAt('noun', ID_A, pinned)).toEqual({ source: 'current' }) + }) + + it('resolves changed ids from the first before-image after the pin', async () => { + await commitWrite(ID_A, 1) + const pinned = store.generation() + await commitWrite(ID_A, 2) + await commitWrite(ID_A, 3) + + const resolved = await store.resolveAt('noun', ID_A, pinned) + expect(resolved.source).toBe('record') + if (resolved.source === 'record') { + expect(resolved.metadata.version).toBe(1) // state as of the pin + } + }) + + it('resolves ids created after the pin as absent', async () => { + await commitWrite(ID_A, 1) + const pinned = store.generation() + await commitWrite(ID_LATER, 1) + + expect(await store.resolveAt('noun', ID_LATER, pinned)).toEqual({ source: 'absent' }) + }) + + it('changedBetween reports the ids touched in the interval', async () => { + await commitWrite(ID_A, 1) + const from = store.generation() + await commitWrite(ID_B, 1) + await commitWrite(ID_C, 1) + const to = store.generation() + + const changed = await store.changedBetween(from, to) + expect(changed.nouns).toEqual([ID_B, ID_C]) + expect(changed.verbs).toEqual([]) + expect(changed.fromGeneration).toBe(from) + expect(changed.toGeneration).toBe(to) + }) + }) + + describe('timestamp resolution + tx-log', () => { + it('records meta in the tx-log and resolves timestamps to generations', async () => { + const t0 = Date.now() - 10 + const first = await commitWrite(ID_A, 1, { meta: { author: 'unit-test' } }) + // Distinct commit timestamps: resolution picks the NEWEST generation + // committed at-or-before the instant, so same-millisecond commits tie. + await new Promise((resolve) => setTimeout(resolve, 5)) + const second = await commitWrite(ID_A, 2) + + const lines = await storage.readTxLogLines() + expect(lines.length).toBe(2) + const firstEntry = JSON.parse(lines[0]) + expect(firstEntry.generation).toBe(first.generation) + expect(firstEntry.meta).toEqual({ author: 'unit-test' }) + + expect((await store.resolveTimestamp(t0)).generation).toBe(0) + expect((await store.resolveTimestamp(first.timestamp)).generation).toBe(first.generation) + expect((await store.resolveTimestamp(Date.now() + 1000)).generation).toBe( + second.generation + ) + }) + }) + + describe('pins + compaction', () => { + it('refcounts pins and never reclaims records a live pin needs', async () => { + await commitWrite(ID_A, 1) + const pinned = store.generation() + store.pin(pinned) + store.pin(pinned) // refcount 2 + await commitWrite(ID_A, 2) + await commitWrite(ID_A, 3) + + // Pin at G protects every record-set ABOVE G (resolution reads + // before-images from generations strictly greater than the pin). + await store.compact() + const resolved = await store.resolveAt('noun', ID_A, pinned) + expect(resolved.source).toBe('record') + if (resolved.source === 'record') { + expect(resolved.metadata.version).toBe(1) + } + + store.release(pinned) + await store.compact() + // Still one live pin — records above it still resolve correctly. + const stillResolved = await store.resolveAt('noun', ID_A, pinned) + expect(stillResolved.source).toBe('record') + + store.release(pinned) + const result = await store.compact() + expect(result.removedGenerations).toBeGreaterThan(0) + const remaining = await storage.listRawObjects(GENERATIONS_PREFIX) + expect(remaining).toEqual([]) + }) + + it('retainGenerations keeps the most recent record-sets', async () => { + await commitWrite(ID_A, 1) + await commitWrite(ID_A, 2) + await commitWrite(ID_A, 3) + + const result = await store.compact({ retainGenerations: 2 }) + expect(result.removedGenerations).toBe(1) + expect(result.horizon).toBe(1) + + // Removing record-set N breaks pins BELOW N (their reads need N's + // before-images) — the horizon itself stays reachable: state at the + // horizon resolves from the retained record-sets above it. + expect(() => store.assertReachable(0)).toThrow(GenerationCompactedError) + expect(store.assertReachable(1)).toBe(1) + expect(store.assertReachable(2)).toBe(2) + }) + + it('rejects future and negative generations', async () => { + await commitWrite(ID_A, 1) + expect(() => store.assertReachable(99)).toThrow(RangeError) + expect(() => store.assertReachable(-1)).toThrow(RangeError) + expect(() => store.assertReachable(1.5)).toThrow(RangeError) + }) + }) + + describe('crash recovery', () => { + it('rolls back an uncommitted generation on open (manifest rename is the commit point)', async () => { + await commitWrite(ID_A, 1) + + // Simulate a crash between record staging and the manifest rename: a + // throwing fault injector skips the abort cleanup, exactly as a dead + // process would, leaving canonical files mutated and the staging + // directory present — but the manifest still at the prior generation. + store.setCommitFaultInjector((phase) => { + if (phase === 'before-manifest-rename') { + throw new Error('simulated crash') + } + }) + await expect(commitWrite(ID_A, 2)).rejects.toThrow('simulated crash') + store.setCommitFaultInjector(undefined) + + // The canonical file currently holds the EXECUTED (uncommitted) state. + const dirty = await storage.readNounRaw(ID_A) + expect((dirty.metadata as { version: number }).version).toBe(2) + + // Recovery on the next open restores the before-image byte-identically. + const reopened = new GenerationStore(storage) + const openResult = await reopened.open() + expect(openResult.rolledBackGenerations).toBe(1) + const repaired = await storage.readNounRaw(ID_A) + expect((repaired.metadata as { version: number }).version).toBe(1) + expect(reopened.committedGeneration()).toBe(1) + // The crashed generation number is never reused. + expect(reopened.generation()).toBeGreaterThanOrEqual(2) + }) + + it('keeps a transaction whose manifest rename landed (tx-log append is advisory)', async () => { + await commitWrite(ID_A, 1) + store.setCommitFaultInjector((phase) => { + if (phase === 'after-manifest-rename') { + throw new Error('simulated crash after commit point') + } + }) + await expect(commitWrite(ID_A, 2)).rejects.toThrow('simulated crash after commit point') + store.setCommitFaultInjector(undefined) + + const reopened = new GenerationStore(storage) + const openResult = await reopened.open() + expect(openResult.rolledBackGenerations).toBe(0) + expect(reopened.committedGeneration()).toBe(2) + const kept = await storage.readNounRaw(ID_A) + expect((kept.metadata as { version: number }).version).toBe(2) + + const manifest = await storage.readRawObject(MANIFEST_PATH) + expect(manifest.generation).toBe(2) + }) + }) +}) diff --git a/tests/unit/db/whereMatcher.test.ts b/tests/unit/db/whereMatcher.test.ts new file mode 100644 index 00000000..42f88a7a --- /dev/null +++ b/tests/unit/db/whereMatcher.test.ts @@ -0,0 +1,223 @@ +/** + * @module tests/unit/db/whereMatcher + * @description Unit tests for the in-memory `find()` filter evaluator behind + * historical and speculative `Db` reads (`src/db/whereMatcher.ts`). The + * evaluator must mirror the metadata index's operator semantics exactly — an + * entity matches in-memory if and only if it would have matched through the + * index — and must throw `UnsupportedWhereOperatorError` for anything it + * does not recognize (never guess on a historical read). + */ + +import { describe, it, expect } from 'vitest' +import { + entityMatchesFind, + resolveEntityField, + whereMatches, + UnsupportedWhereOperatorError +} from '../../../src/db/whereMatcher.js' +import type { Entity } from '../../../src/types/brainy.types.js' +import { NounType } from '../../../src/types/graphTypes.js' + +/** Build a minimal entity for matcher tests. */ +function entity(overrides: Partial = {}): Entity { + return { + id: 'e-1', + vector: [], + type: NounType.Document, + createdAt: 1000, + updatedAt: 2000, + metadata: {}, + ...overrides + } +} + +describe('db/whereMatcher — resolveEntityField', () => { + it('resolves standard top-level fields', () => { + const e = entity({ + subtype: 'invoice', + service: 'billing', + confidence: 0.9, + weight: 0.5, + _rev: 3, + data: 'payload' + }) + expect(resolveEntityField(e, 'id')).toBe('e-1') + expect(resolveEntityField(e, 'type')).toBe(NounType.Document) + expect(resolveEntityField(e, 'noun')).toBe(NounType.Document) // alias + expect(resolveEntityField(e, 'subtype')).toBe('invoice') + expect(resolveEntityField(e, 'service')).toBe('billing') + expect(resolveEntityField(e, 'confidence')).toBe(0.9) + expect(resolveEntityField(e, 'weight')).toBe(0.5) + expect(resolveEntityField(e, '_rev')).toBe(3) + expect(resolveEntityField(e, 'createdAt')).toBe(1000) + expect(resolveEntityField(e, 'updatedAt')).toBe(2000) + expect(resolveEntityField(e, 'data')).toBe('payload') + }) + + it('resolves custom fields from the metadata bag', () => { + const e = entity({ metadata: { status: 'open', priority: 2 } }) + expect(resolveEntityField(e, 'status')).toBe('open') + expect(resolveEntityField(e, 'priority')).toBe(2) + expect(resolveEntityField(e, 'absent')).toBeUndefined() + }) + + it('resolves dotted paths against the entity and the metadata bag', () => { + const e = entity({ metadata: { address: { city: 'Lyon' }, priority: 7 } }) + expect(resolveEntityField(e, 'metadata.priority')).toBe(7) + expect(resolveEntityField(e, 'address.city')).toBe('Lyon') + expect(resolveEntityField(e, 'address.zip')).toBeUndefined() + }) +}) + +describe('db/whereMatcher — operators', () => { + const e = entity({ + subtype: 'invoice', + metadata: { amount: 250, status: 'open', tags: ['urgent', 'q3'], city: 'Lyon' } + }) + + it('shorthand equality', () => { + expect(whereMatches(e, { status: 'open' })).toBe(true) + expect(whereMatches(e, { status: 'closed' })).toBe(false) + }) + + it('eq / equals / is aliases', () => { + expect(whereMatches(e, { amount: { eq: 250 } })).toBe(true) + expect(whereMatches(e, { amount: { equals: 250 } })).toBe(true) + expect(whereMatches(e, { amount: { is: 250 } })).toBe(true) + expect(whereMatches(e, { amount: { eq: 99 } })).toBe(false) + }) + + it('array-membership equality (index posting semantics)', () => { + expect(whereMatches(e, { tags: 'urgent' })).toBe(true) + expect(whereMatches(e, { tags: { contains: 'q3' } })).toBe(true) + expect(whereMatches(e, { tags: 'missing-tag' })).toBe(false) + }) + + it('ne / notEquals / isNot aliases', () => { + expect(whereMatches(e, { status: { ne: 'closed' } })).toBe(true) + expect(whereMatches(e, { status: { notEquals: 'open' } })).toBe(false) + expect(whereMatches(e, { status: { isNot: 'open' } })).toBe(false) + }) + + it('in / oneOf set membership', () => { + expect(whereMatches(e, { status: { in: ['open', 'closed'] } })).toBe(true) + expect(whereMatches(e, { status: { oneOf: ['archived'] } })).toBe(false) + }) + + it('numeric range operators with aliases', () => { + expect(whereMatches(e, { amount: { gt: 200 } })).toBe(true) + expect(whereMatches(e, { amount: { greaterThan: 250 } })).toBe(false) + expect(whereMatches(e, { amount: { gte: 250 } })).toBe(true) + expect(whereMatches(e, { amount: { greaterThanOrEqual: 251 } })).toBe(false) + expect(whereMatches(e, { amount: { greaterEqual: 250 } })).toBe(true) + expect(whereMatches(e, { amount: { lt: 251 } })).toBe(true) + expect(whereMatches(e, { amount: { lessThan: 250 } })).toBe(false) + expect(whereMatches(e, { amount: { lte: 250 } })).toBe(true) + expect(whereMatches(e, { amount: { lessThanOrEqual: 249 } })).toBe(false) + expect(whereMatches(e, { amount: { lessEqual: 250 } })).toBe(true) + }) + + it('string range comparison is lexicographic', () => { + expect(whereMatches(e, { city: { gt: 'Aix' } })).toBe(true) + expect(whereMatches(e, { city: { lt: 'Aix' } })).toBe(false) + }) + + it('mixed-type range comparison never matches', () => { + expect(whereMatches(e, { city: { gt: 5 } })).toBe(false) + expect(whereMatches(e, { amount: { lt: 'zzz' } })).toBe(false) + }) + + it('between', () => { + expect(whereMatches(e, { amount: { between: [200, 300] } })).toBe(true) + expect(whereMatches(e, { amount: { between: [251, 300] } })).toBe(false) + expect(whereMatches(e, { amount: { between: [200] } })).toBe(false) // malformed operand + }) + + it('exists / missing', () => { + expect(whereMatches(e, { status: { exists: true } })).toBe(true) + expect(whereMatches(e, { nope: { exists: true } })).toBe(false) + expect(whereMatches(e, { nope: { exists: false } })).toBe(true) + expect(whereMatches(e, { nope: { missing: true } })).toBe(true) + expect(whereMatches(e, { status: { missing: true } })).toBe(false) + expect(whereMatches(e, { status: { missing: false } })).toBe(true) + }) + + it('multiple operators on one field AND together', () => { + expect(whereMatches(e, { amount: { gte: 200, lte: 300 } })).toBe(true) + expect(whereMatches(e, { amount: { gte: 200, lte: 249 } })).toBe(false) + }) + + it('allOf / anyOf / not logical composition', () => { + expect(whereMatches(e, { allOf: [{ status: 'open' }, { amount: { gt: 100 } }] })).toBe(true) + expect(whereMatches(e, { allOf: [{ status: 'open' }, { amount: { gt: 999 } }] })).toBe(false) + expect(whereMatches(e, { anyOf: [{ status: 'closed' }, { amount: 250 }] })).toBe(true) + expect(whereMatches(e, { anyOf: [{ status: 'closed' }, { amount: 9 }] })).toBe(false) + expect(whereMatches(e, { not: { status: 'closed' } })).toBe(true) + expect(whereMatches(e, { not: { status: 'open' } })).toBe(false) + }) + + it('throws UnsupportedWhereOperatorError for unknown operators — never guesses', () => { + expect(() => whereMatches(e, { amount: { approximately: 250 } })).toThrow( + UnsupportedWhereOperatorError + ) + try { + whereMatches(e, { amount: { approximately: 250 } }) + expect.unreachable('should have thrown') + } catch (err) { + expect(err).toBeInstanceOf(UnsupportedWhereOperatorError) + expect((err as UnsupportedWhereOperatorError).operator).toBe('approximately') + } + }) +}) + +describe('db/whereMatcher — entityMatchesFind', () => { + const e = entity({ + subtype: 'invoice', + service: 'billing', + metadata: { amount: 250 } + }) + + it('filters by type (single + array)', () => { + expect(entityMatchesFind(e, { type: NounType.Document })).toBe(true) + expect(entityMatchesFind(e, { type: [NounType.Person, NounType.Document] })).toBe(true) + expect(entityMatchesFind(e, { type: NounType.Person })).toBe(false) + }) + + it('filters by subtype (single + array; entities without subtype excluded)', () => { + expect(entityMatchesFind(e, { subtype: 'invoice' })).toBe(true) + expect(entityMatchesFind(e, { subtype: ['invoice', 'receipt'] })).toBe(true) + expect(entityMatchesFind(e, { subtype: 'receipt' })).toBe(false) + expect(entityMatchesFind(entity(), { subtype: 'invoice' })).toBe(false) + }) + + it('filters by service', () => { + expect(entityMatchesFind(e, { service: 'billing' })).toBe(true) + expect(entityMatchesFind(e, { service: 'crm' })).toBe(false) + }) + + it('excludeVFS drops VFS-marked entities', () => { + const vfsEntity = entity({ metadata: { vfsType: 'file' } }) + const markedEntity = entity({ metadata: { isVFSEntity: true } }) + expect(entityMatchesFind(vfsEntity, { excludeVFS: true })).toBe(false) + expect(entityMatchesFind(markedEntity, { excludeVFS: true })).toBe(false) + expect(entityMatchesFind(e, { excludeVFS: true })).toBe(true) + expect(entityMatchesFind(vfsEntity, {})).toBe(true) + }) + + it('composes type + subtype + where', () => { + expect( + entityMatchesFind(e, { + type: NounType.Document, + subtype: 'invoice', + where: { amount: { gte: 200 } } + }) + ).toBe(true) + expect( + entityMatchesFind(e, { + type: NounType.Document, + subtype: 'invoice', + where: { amount: { gte: 999 } } + }) + ).toBe(false) + }) +})