diff --git a/docs/ADR-001-generational-mvcc.md b/docs/ADR-001-generational-mvcc.md new file mode 100644 index 00000000..8f7d6f36 --- /dev/null +++ b/docs/ADR-001-generational-mvcc.md @@ -0,0 +1,262 @@ +# ADR-001: Generational MVCC storage and the immutable Db API + +**Status:** Accepted (ships in 8.0) +**Date:** 2026-06-10 + +## Context + +Before 8.0, Brainy carried two overlapping version-control subsystems: a +copy-on-write branching layer (`fork`/`checkout`/`commit`/`branches`) and a +separate versioning subsystem (`versions.save/list/compare/restore/prune`), +plus a read-only historical adapter for commit-based time travel. Together +they were ~5,100 LOC of mechanism for one product need: *read a consistent +past state while the store keeps moving, and snapshot/restore cheaply.* + +Neither subsystem gave a precise isolation guarantee. Reads raced in-place +JSON overwrites, so a "snapshot" was only as immutable as the bytes it +happened to share with the live store. + +8.0 replaces both with **one mechanism**: generational MVCC over immutable, +generation-stamped records, exposed through a Datomic-style immutable +database value (`Db`). The same model is implemented natively by versioned +index providers (LSM snapshots), so semantics are identical on the pure-JS +path and the native path. + +## Decision + +### The model + +- A **monotonic u64 generation counter** is the store's logical clock. It + advances once per committed `transact()` batch and once per + single-operation write (`add`/`update`/`delete`/`relate`/…), so + `brain.generation()` is always a meaningful watermark. It is persisted in + `_system/generation.json` and never reissued for anything durable. +- `brain.now()` **pins** the current generation in O(1) and returns a `Db` — + an immutable view. Pins are refcounted; `db.release()` (with a + `FinalizationRegistry` backstop for leaked values) ends the pin. +- `brain.transact(ops, { meta, ifAtGeneration })` commits a declarative + batch atomically as **exactly one generation**, with whole-store + compare-and-swap (`ifAtGeneration` → `GenerationConflictError`) and + reified transaction metadata appended to `_system/tx-log.jsonl`. +- `brain.asOf(generation | Date | snapshotPath)` opens past state; + `db.with(ops)` layers a speculative in-memory overlay (never touching + disk, the counter, or index providers); `db.persist(path)` cuts an + instant snapshot; `brain.restore(path, { confirm: true })` replaces state + from one; `Brainy.load(path)` opens a snapshot read-only with the full + query surface. + +### Persisted layout + +All paths are storage-root-relative: + +``` +_system/generation.json { generation, updatedAt } atomic tmp+rename +_system/manifest.json { version, generation, atomic tmp+rename + committedAt, horizon } (the commit point) +_system/tx-log.jsonl one line per committed append-only + transact: { generation, + timestamp, meta? } +_generations//tx.json the generation-N delta: immutable + touched noun/verb ids + meta +_generations//prev/.json before-image of as of immutable + commit N (raw stored bytes; + null parts = file was absent) +``` + +**Why per-generation deltas instead of a global `id → latest generation` +map in the manifest:** a global map makes every commit O(all ids) — the +whole map must be rewritten to swap it atomically. The delta layout makes a +commit O(ids touched) and keeps the manifest a fixed-size watermark, while +point-in-time resolution stays correct (see "Read resolution" below). The +trade is that resolution at a pinned generation scans the deltas of later +commits — bounded by the number of commits since the pin, which is exactly +the window compaction keeps short. + +Before-images are deliberately the *only* per-id records. They serve both +roles the layer needs — the crash-recovery undo log and the point-in-time +read source. After-images would duplicate state that is already readable +(the canonical entity files hold the latest bytes; earlier states resolve +from later before-images) and would double record I/O per commit. + +### Commit protocol (durability) + +`transact()` commits under a store-wide mutex: + +1. **CAS check.** A stale `ifAtGeneration` throws `GenerationConflictError` + before anything is staged. +2. **Reserve** generation `N` (counter increment). +3. **Stage the undo log:** write the before-image of every touched id plus + `tx.json` under `_generations/N/`, then **fsync** the files and their + directories. From this point, any crash is recoverable to the exact + pre-transaction bytes. +4. **Execute** the planned batch through the TransactionManager (which has + its own operation-level rollback for in-flight failures). +5. **Commit point:** persist the counter, then write `_system/manifest.json` + via atomic tmp+rename and fsync it. The rename *is* the commit: a + generation directory is committed if and only if `N ≤ + manifest.generation`. +6. Append the tx-log line (advisory metadata — a crash between 5 and 6 + keeps the transaction). + +**Crash recovery (on open):** any `_generations/` directory with +`N > manifest.generation` is an uncommitted transaction. Its before-images +are restored to the canonical paths (idempotently — recovery itself can +crash and rerun) and the directory is removed. Because recovery runs before +any index is built, and a recovery that rolled something back forces a full +index rebuild, derived indexes never observe rolled-back state. Reader-mode +instances skip recovery (readers never write; the next writer repairs). + +A failed (non-crash) transaction takes the same staging directory down the +abort path: the TransactionManager rolls back applied operations, the +staging directory is removed, and the generation reservation is returned — +a failed batch leaves the generation counter unchanged. + +### Read resolution at a pinned generation + +The state of id X at pinned generation G is: + +- the before-image stored by the **first committed generation after G that + touched X**, or +- the live canonical bytes, when nothing after G touched X. + +While nothing has committed past G, *every* read on the `Db` delegates to +the live fast paths untouched — `now()` adds no read overhead until history +actually moves. + +**Honesty boundary.** `get()`, metadata-level `find()`, and `related()` are +fully correct at any reachable pinned generation. Index-accelerated +dimensions (semantic/vector search, graph traversal, cursors, aggregation) +are served by live indexes that represent only the current generation, so +at a historical generation they throw +`NotYetSupportedAtHistoricalGenerationError` — never silently-wrong +results. The escape hatch is `persist()` + `Brainy.load()`, which rebuilds +indexes from the snapshot and supports the full query surface. Rebuilding +indexes from a pinned record set in-place (the standard LSM answer) is the +documented follow-up; versioned native providers already expose +`isGenerationVisible()` for provider-accelerated historical reads. + +**History granularity.** Generation *records* are written per `transact()` +batch only. Single-operation writes advance the counter (so watermarks and +CAS stay sound) but do not stage before-images: they remain visible through +earlier pins and are not reported by `db.since()`. Code that needs pinned +isolation across its writes uses `transact()` — that is the documented 8.0 +contract, stated rather than papered over. + +### Pinning, retention, compaction + +- Each live `Db` holds one refcounted pin on its generation (plus a + `pin(generation)` on every registered `VersionedIndexProvider`, whose + explicit pin lifetime overrides any time-based snapshot retention the + provider has). +- `compactHistory({ retainGenerations?, retainMs? })` reclaims record-sets. + A record-set `N` is reclaimed only when `N` is at or below **every** live + pin — deleting `N` can only break readers pinned *below* `N`, because + resolution reads before-images from generations strictly greater than the + pin. With both retention options supplied, both must allow the reclaim. +- The manifest records the **horizon** (highest reclaimed generation). + Generations below the horizon are unreachable; `asOf()` on them throws + `GenerationCompactedError`. The horizon itself stays reachable, resolved + from the record-sets above it. To keep a generation readable forever, + `persist()` it first — snapshots are self-contained. + +### Snapshots and restore + +`db.persist(path)` flushes indexes, then cuts the snapshot under the +store's commit mutex (no commit, compaction, or counter write can +interleave). On filesystem storage it is a **hard-link farm**: every data +file is immutable-by-rename, so linking is safe — later rewrites swap +inodes and the snapshot keeps the old bytes. The two exceptions are handled +explicitly: the append-in-place tx-log is byte-copied, and process-local +lock state is excluded. Cross-device targets (and filesystems that refuse +links) fall back to per-file byte copies. In-memory stores serialize to the +same directory layout, so persisting a memory brain produces a real, +durable, loadable store. + +`persist()` requires the view to still be the store's latest generation +(a snapshot captures current bytes); a view that history has moved past +throws rather than persisting the wrong state. + +`restore(path, { confirm: true })` replaces the store's contents from a +snapshot via byte copy (never links — the snapshot stays independent), +reloads all adapter-internal derived state, rebuilds all indexes, and +floors the generation counter at its pre-restore value so observed +generation numbers are never reissued. Live pins do not survive a restore; +a warning is logged when any exist. + +### Versioned index providers + +Native index providers may implement the optional 4-method +`VersionedIndexProvider` capability (`generation()`, +`isGenerationVisible()`, `pin()`, `release()` — BigInt generations at the +boundary). The locked consistency model: providers are **post-commit +appliers**. The storage-record commit is the source of truth; provider +index state is derived. On open, a provider behind the committed watermark +replays the gap from storage (or requests a rebuild) — there are no +provider rollback hooks, because uncommitted transactions are repaired at +the storage layer before any index opens. Speculative `with()` overlays +never reach providers. + +## Guarantees (and their proofs) + +Each stated guarantee has a test that proves it, not merely exercises it +(`tests/integration/db-mvcc.test.ts`, plus +`tests/unit/db/generationStore.test.ts` for the record layer in isolation): + +| Guarantee | Proof | +|---|---| +| Snapshot isolation: a pinned `Db` reads exactly its pinned state, forever | proof 1 (200 mutations, including deletes, against a pinned view) | +| Atomicity: a failing batch applies nothing; generation unchanged | proofs 2a/2b/2c (plan-time failure, injected execution-phase storage failure, `ifRev` conflict) | +| Whole-store CAS | proof 3 (`ifAtGeneration` success + conflict with exact expected/actual) | +| Snapshot integrity under source mutation (hard-link safety) | proofs 4a/4b/4c | +| Compaction never breaks a pinned read; release enables reclaim | proof 5 | +| `with()` overlays touch nothing durable | proof 6 | +| Generation monotonicity across close/reopen | proof 7 | +| Crash before the manifest rename recovers to exact pre-transaction state through the real recovery path | proof 8 (fault injection that skips abort cleanup, exactly as a dead process would) | +| Balanced provider pin/release lockstep | proof 9 | + +One deliberate softness: single-operation generation bumps persist the +counter coalesced (per write burst), not per write. Durable artifacts — +records, manifests, snapshots — always persist the counter synchronously at +their own commit points, so a crash inside the coalescing window can lose +only counter values that nothing durable ever referenced. + +## Failure modes + +| Failure | Outcome | +|---|---| +| Crash before staging completes | Partial staging directory > manifest watermark → removed on next open; canonical state untouched. | +| Crash after staging, before/during batch execution | Before-images restored on next open; indexes rebuilt; byte-identical pre-transaction state. | +| Crash after execution, before manifest rename | Same as above — the rename is the only commit point. | +| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. | +| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. | +| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. | +| Index-accelerated query at a historical pin | `NotYetSupportedAtHistoricalGenerationError` — explicit, never silently-wrong results. | +| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. | + +## Lineage + +The design is an assembly of well-understood prior art, chosen for being +boring where it counts: + +- **Datomic** — the database-as-a-value: an immutable `Db` you query, with + `with()` for speculation and reified transaction metadata instead of + commit messages. +- **LMDB** — reader pins: readers never block writers; a reader's view + stays valid because nothing overwrites the pages (here: records) it + references; reclamation waits for the last reader. +- **LSM trees / Cassandra** — immutable segments make snapshots hard links + and make compaction a retention policy instead of a locking problem. + +## Consequences + +- One mechanism replaces the COW and versioning subsystems (their removal + is the companion change to this ADR). +- In-place branch switching (`checkout`) is gone by design; the replacement + is opening a persisted snapshot as a separate instance — a name→path + mapping where a product needs named branches. +- Every commit pays O(ids touched) extra writes (before-images + delta + + manifest). Single-operation writes pay only an in-memory counter bump + with coalesced persistence. +- Historical index-accelerated queries are an explicit error until the + asOf-rebuild ships; correctness-critical historical reads (`get`, + metadata `find`, `related`) work today and forever at any reachable pin. diff --git a/src/brainy.ts b/src/brainy.ts index f10eedde..1e229e1a 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -32,11 +32,11 @@ import { CommitBuilder } from './storage/cow/CommitObject.js' import { BlobStorage } from './storage/cow/BlobStorage.js' import { NULL_HASH } from './storage/cow/constants.js' import { createPipeline } from './streaming/pipeline.js' -import { configureLogger, LogLevel } from './utils/logger.js' +import { configureLogger, LogLevel, prodLog } from './utils/logger.js' import { setGlobalCache } from './utils/unifiedCache.js' import type { UnifiedCache } from './utils/unifiedCache.js' import { rankIndicesByScore, reorderByIndices } from './utils/resultRanking.js' -import { PluginRegistry } from './plugin.js' +import { PluginRegistry, isVersionedIndexProvider } from './plugin.js' import type { BrainyPlugin, BrainyPluginContext, @@ -109,6 +109,22 @@ import { AggregationIndex } from './aggregation/AggregationIndex.js' import { AggregateMaterializer } from './aggregation/materializer.js' import type { AggregateDefinition, AggregateQueryParams, AggregateResult } from './types/brainy.types.js' import { resolveJsHnswConfig } from './utils/recallPreset.js' +import * as fs from 'node:fs' +import { Db, type DbHost } from './db/db.js' +import { GenerationStore } from './db/generationStore.js' +import { + GenerationConflictError, + NotYetSupportedAtHistoricalGenerationError +} from './db/errors.js' +import type { + CompactHistoryOptions, + CompactHistoryResult, + TransactOptions, + TransactReceipt, + TxOperation +} from './db/types.js' +import type { VersionedIndexProvider } from './plugin.js' +import type { Operation } from './transaction/types.js' /** * Stopwords for semantic highlighting @@ -143,6 +159,47 @@ export interface DiagnosticsResult { } } +/** + * In-flight planning state for one `brain.transact()` batch (8.0 MVCC). + * + * Operations inside a batch may reference ids written by EARLIER operations + * of the same batch (`add` an entity, then `relate` to it) — but nothing has + * touched storage yet during planning. This state carries the + * would-be-written stored objects so later planners read + * "current state + batch so far", exactly what the executed batch produces. + */ +interface TxPlanState { + /** Stored metadata + vector of entities written earlier in this batch. */ + nouns: Map; vector: Vector }> + /** Entity ids removed earlier in this batch. */ + removedNouns: Set + /** Verbs created earlier in this batch (keyed by relationship id). */ + verbs: Map + /** Relationship ids removed earlier in this batch. */ + removedVerbs: Set +} + +/** + * The fully planned form of one `brain.transact()` batch: the ordered + * `TransactionManager` operations, the resolved id per input operation (the + * receipt), the touched-id sets the generation store stages before-images + * for, and the post-commit hooks (aggregation-index maintenance — derived + * data, applied outside the atomic batch exactly as the single-op methods + * do). + */ +interface PlannedTransact { + /** Ordered operations for one atomic TransactionManager execution. */ + operations: Operation[] + /** Resolved id per input operation, in input order. */ + ids: string[] + /** Entity ids the batch writes or deletes. */ + touchedNouns: string[] + /** Relationship ids the batch writes or deletes. */ + touchedVerbs: string[] + /** Aggregation-index hooks to run after the commit point. */ + postCommit: Array<() => void> +} + /** * The main Brainy class - Clean, Beautiful, Powerful * REAL IMPLEMENTATION - No stubs, no mocks @@ -160,6 +217,27 @@ export class Brainy implements BrainyInterface { private metadataIndex!: MetadataIndexManager private graphIndex!: GraphAdjacencyIndex private transactionManager: TransactionManager + + /** + * 8.0 generational MVCC record layer (created in `init()`, before any index + * is built — crash recovery may rewrite canonical entity files). Owns the + * generation counter, the transact commit protocol, pin refcounts, and + * point-in-time record resolution. Backs `now()`/`transact()`/`asOf()`/ + * `compactHistory()` and the `Db` value type. + */ + private generationStore!: GenerationStore + /** Lazily built host surface shared by every `Db` value of this brain. */ + private _dbHost?: DbHost + /** + * GC backstop for leaked `Db` pins: when an unreleased `Db` is collected, + * its generation pin (store + versioned providers) is released and an + * owned snapshot brain (from `Brainy.load()`) is closed. Explicit + * `db.release()` unregisters first, so pins are never double-released. + */ + private _dbFinalizationRegistry?: FinalizationRegistry<{ + generation: number + closeOnRelease?: () => Promise + }> private embedder: EmbeddingFunction private distance: DistanceFunction private config: Required @@ -513,6 +591,18 @@ export class Brainy implements BrainyInterface { (this.storage as any).enableCOWLightweight((this.config.storage as any)?.branch || 'main') } + // 8.0 generational MVCC: open the record layer BEFORE any index is + // created or loaded. Crash recovery may rewrite canonical entity files + // (restoring before-images of an uncommitted transaction), and every + // index below loads from those files — opening the store first + // guarantees indexes never observe rolled-back state. Reader-mode + // instances skip recovery (readers never write; the next writer + // repairs). + this.generationStore = new GenerationStore(this.storage) + const generationOpenResult = await this.generationStore.open({ + readOnly: this.config.mode === 'reader' + }) + // Provider: embeddings (reassign embedder if plugin provides one) const embeddingProvider = this.pluginRegistry.getProvider('embeddings') if (embeddingProvider) { @@ -625,6 +715,40 @@ export class Brainy implements BrainyInterface { // mmap-vector backend this layer engages even on cloud adapters. this.wireConnectionsCodec() + // 8.0 generational MVCC: if crash recovery rolled back an uncommitted + // transaction, every derived index is suspect — persisted index state + // (flushed before the crash) may reference the rolled-back writes. + // Rebuild all three from the repaired canonical records (the JS-index + // equivalent of the locked design's "manifest-vs-index generation + // comparison + replay on open"). + if (generationOpenResult.rolledBackGenerations > 0) { + prodLog.warn( + `[Brainy] Rebuilding indexes after crash recovery rolled back ` + + `${generationOpenResult.rolledBackGenerations} uncommitted transaction(s)` + ) + await Promise.all([ + this.metadataIndex.rebuild(), + this.index.rebuild(), + this.graphIndex.rebuild() + ]) + } + + // 8.0 versioned-provider replay-gap check: a provider whose persisted + // index generation is behind the storage layer's committed generation + // replays the gap itself (post-commit applier contract) — surface the + // gap for observability. + for (const provider of this.versionedIndexProviders()) { + const providerGen = provider.generation() + const committed = BigInt(this.generationStore.committedGeneration()) + if (providerGen < committed) { + prodLog.info( + `[Brainy] Versioned index provider is at generation ${providerGen} ` + + `(storage committed: ${committed}) — provider replays the gap per ` + + `the post-commit applier contract` + ) + } + } + // Rebuild indexes if needed for existing data await this.rebuildIndexesIfNeeded() @@ -4721,6 +4845,8 @@ export class Brainy implements BrainyInterface { /** * Create a read-only snapshot of the workspace at a specific commit + * (pre-8.0 commit-based time travel — the generational `asOf()` is the + * 8.0 replacement; this method is retired with the COW surface). * * Time-travel API for historical queries. Returns a new Brainy instance that: * - Contains all entities and relationships from that commit @@ -4742,7 +4868,7 @@ export class Brainy implements BrainyInterface { * @example * ```typescript * // Create snapshot at specific commit - * const snapshot = await brain.asOf(commitId) + * const snapshot = await brain.asOfCommit(commitId) * * // Query historical state (full triple intelligence works!) * const files = await snapshot.find({ @@ -4757,7 +4883,7 @@ export class Brainy implements BrainyInterface { * await snapshot.close() * ``` */ - async asOf(commitId: string, options?: { + async asOfCommit(commitId: string, options?: { cacheSize?: number // LRU cache size (default: 10000) }): Promise { await this.ensureInitialized() @@ -5168,6 +5294,1096 @@ export class Brainy implements BrainyInterface { } } + // ============= 8.0 DB API — GENERATIONAL MVCC ============= + // + // Datomic-style immutable database values over the generational record + // layer (src/db/). `now()` pins the current generation in O(1); + // `transact()` commits a declarative batch atomically as exactly one + // generation; `asOf()` opens past generations, timestamps, or persisted + // snapshots; `compactHistory()` reclaims unpinned history. The `Db` value + // type lives in src/db/db.ts; this region is the brain-side host: + // pin/release plumbing (storage + versioned index providers), record + // materialization, the transact planner, and snapshot/restore. + + /** + * @description The store's current generation — a monotonic u64 watermark + * bumped once per committed `transact()` batch and once per + * single-operation write batch (`add`/`update`/`delete`/`relate`/…). + * Persisted in `_system/generation.json`; never reused, including across + * restarts and `restore()`. + * + * @returns The current generation number. + * @throws Error when called before `init()` completes. + * @example + * const before = brain.generation() + * await brain.transact([{ op: 'add', type: NounType.Document, data: 'x', subtype: 'note' }]) + * brain.generation() // before + 1 + */ + generation(): number { + this.assertGenerationStoreReady('generation') + return this.generationStore.generation() + } + + /** + * @description Pin the CURRENT generation and return an immutable `Db` + * view of it — O(1), no I/O. The returned view keeps reading exactly this + * state no matter what commits afterwards: `get()` and metadata-level + * `find()`/`related()` stay correct at the pinned generation forever + * (resolved from immutable generation records), while index-accelerated + * queries throw the documented historical-query error once history moves + * past the pin. + * + * Call `db.release()` when done — pins gate `compactHistory()`. A + * `FinalizationRegistry` backstop releases leaked pins at GC time, but + * explicit release is what makes compaction deterministic. + * + * @returns A `Db` pinned at the current generation. + * @throws Error when called before `init()` completes (pinning is + * synchronous, so it cannot await initialization). + * @example + * const db = brain.now() + * await brain.transact([{ op: 'update', id, metadata: { v: 2 } }]) + * await db.get(id) // still sees v: 1 + * await brain.get(id) // sees v: 2 + * await db.release() + */ + now(): Db { + this.assertGenerationStoreReady('now') + return this.createPinnedDb({ + generation: this.generationStore.generation(), + timestamp: Date.now() + }) + } + + /** + * @description Execute a declarative operation batch atomically: either + * every operation applies and the store advances exactly ONE generation, + * or none apply and the store is byte-identical to its pre-transaction + * state. The semantics of each operation mirror the corresponding + * single-operation method (`add`/`update`/`delete`/`relate`/`unrelate`), + * including validation, subtype enforcement, per-entity `ifRev` CAS, + * relationship deduplication, and delete cascades. Operations may + * reference ids created by earlier operations of the same batch. + * + * Durability protocol (see `src/db/generationStore.ts`): before-images of + * every touched id are staged and fsynced, the batch executes through the + * TransactionManager, and the atomic rename of `_system/manifest.json` is + * the commit point — a crash anywhere before the rename is rolled back to + * the exact pre-transaction bytes on the next open. + * + * Isolation: concurrent `transact()` calls commit serially + * (snapshot-isolated batches). For strict lost-update protection across a + * read-modify-write cycle, pass `ifAtGeneration` (whole-store CAS) or use + * per-entity `ifRev` on update operations. + * + * @param ops - Declarative operations: `{ op: 'add', ... }`, + * `{ op: 'update', ... }`, `{ op: 'remove', id }`, + * `{ op: 'relate', ... }`, `{ op: 'unrelate', id }`. + * @param options - `meta` (reified transaction metadata, recorded in + * `_system/tx-log.jsonl`) and `ifAtGeneration` (whole-store CAS). + * @returns A `Db` pinned at the freshly committed generation, carrying a + * `receipt` with the resolved id per input operation. + * @throws GenerationConflictError when `ifAtGeneration` does not match the + * store's current generation. + * @throws RevisionConflictError when an update operation's `ifRev` does + * not match the entity's persisted revision (whole batch rejected). + * @example + * const db = await brain.transact([ + * { op: 'add', id: aId, type: NounType.Person, subtype: 'employee', data: 'Ada' }, + * { op: 'add', id: bId, type: NounType.Project, subtype: 'milestone', data: 'Apollo' }, + * { op: 'relate', from: aId, to: bId, type: VerbType.WorksOn, subtype: 'assignment' } + * ], { meta: { author: 'import-job-42' } }) + * db.receipt.ids // [aId, bId, relationshipId] + */ + async transact(ops: TxOperation[], options?: TransactOptions): Promise> { + this.assertWritable('transact') + await this.ensureInitialized() + + if (!Array.isArray(ops) || ops.length === 0) { + throw new Error('transact(): provide a non-empty array of transaction operations') + } + + // Early CAS check — avoids expensive planning (embedding) on a stale + // expectation. The generation store re-checks authoritatively under the + // commit mutex; this one is purely a fast-fail. + if ( + options?.ifAtGeneration !== undefined && + options.ifAtGeneration !== this.generationStore.generation() + ) { + throw new GenerationConflictError( + options.ifAtGeneration, + this.generationStore.generation() + ) + } + + const plan = await this.planTransact(ops) + + const { generation, timestamp } = await this.generationStore.commitTransaction({ + touched: { nouns: plan.touchedNouns, verbs: plan.touchedVerbs }, + meta: options?.meta, + ifAtGeneration: options?.ifAtGeneration, + execute: async () => { + await this.transactionManager.executeTransaction(async (tx) => { + for (const operation of plan.operations) { + tx.addOperation(operation) + } + }) + } + }) + + // Aggregation-index maintenance: derived data, applied after the commit + // point — exactly where the single-operation methods apply it. + for (const hook of plan.postCommit) { + hook() + } + + const receipt: TransactReceipt = { generation, timestamp, ids: plan.ids } + return this.createPinnedDb({ generation, timestamp, receipt }) + } + + /** + * @description Open an immutable `Db` view of PAST state: + * + * - **`number`** — a generation: pins it on this store; reads resolve + * through the generational record layer. History granularity is + * `transact()` commits — single-operation writes between commits do not + * produce records and remain visible through earlier pins. + * - **`Date`** — a wall-clock instant: resolved via the transaction log to + * the newest generation committed at or before it, then pinned as above. + * - **`string`** — a snapshot directory previously produced by + * `db.persist(path)`: opened as a self-contained read-only store + * (equivalent to {@link Brainy.load}) with the FULL query surface, + * including vector search. + * + * Release the returned `Db` when done. (For pre-8.0 commit-hash time + * travel, see `asOfCommit()` — retired with the COW surface.) + * + * @param target - Generation number, `Date`, or snapshot directory path. + * @returns A `Db` pinned at the resolved state. + * @throws GenerationCompactedError when the generation's records were + * reclaimed by `compactHistory()`. + * @throws RangeError for negative or future generations. + * @example + * const yesterday = await brain.asOf(new Date(Date.now() - 86_400_000)) + * const atGen42 = await brain.asOf(42) + * const fromSnapshot = await brain.asOf('/backups/2026-06-01') + */ + async asOf(target: number | Date | string): Promise> { + await this.ensureInitialized() + + if (typeof target === 'string') { + const stat = await fs.promises.stat(target).catch(() => null) + if (!stat || !stat.isDirectory()) { + throw new Error( + `asOf(): '${target}' is not a snapshot directory. Pass a generation number, ` + + `a Date, or a directory created by db.persist(path).` + ) + } + return Brainy.load(target) + } + + let generation: number + let timestamp: number + if (target instanceof Date) { + const resolved = await this.generationStore.resolveTimestamp(target.getTime()) + this.generationStore.assertReachable(resolved.generation) + generation = resolved.generation + timestamp = resolved.entry?.timestamp ?? target.getTime() + } else { + this.generationStore.assertReachable(target) + generation = target + timestamp = + (await this.generationStore.commitTimestampAtOrBefore(target)) ?? Date.now() + } + + return this.createPinnedDb({ generation, timestamp }) + } + + /** + * @description Reclaim the immutable generation record-sets that no + * retention rule and no live pin protects. Records are what serve + * historical reads (`asOf()`, pinned `Db` values); compaction trades that + * history for disk space. Generations at or below the resulting horizon + * become unreachable (`asOf()` throws `GenerationCompactedError`). + * + * Safety invariant: a record-set is never removed while any live `Db` pin + * could need it — pinned reads stay correct across compaction, always. + * Note that `transact()` returns a PINNED `Db`: release views you do not + * keep (the GC backstop eventually releases leaked ones, but until then + * compaction retains the history they protect). + * + * @param options - `retainGenerations` (keep the N most recent committed + * generations) and/or `retainMs` (keep everything committed within the + * window). Both supplied = both must allow reclaim. Neither = reclaim + * everything unpinned. + * @returns Count of reclaimed record-sets and the new horizon. + * @example + * await brain.compactHistory({ retainGenerations: 100, retainMs: 7 * 86_400_000 }) + */ + async compactHistory(options?: CompactHistoryOptions): Promise { + this.assertWritable('compactHistory') + await this.ensureInitialized() + return this.generationStore.compact(options) + } + + /** + * @description Replace this store's ENTIRE state from a snapshot directory + * previously produced by `db.persist(path)`. Destructive: current + * entities, relationships, indexes, and history records are all replaced + * by the snapshot's. The generation counter is floored at its pre-restore + * value, so generation numbers observed before the restore are never + * reissued. + * + * Live `Db` values pinned before a restore are NOT remapped — their + * snapshot-isolation guarantee does not survive a wholesale state + * replacement. Release them first; a warning is logged when live pins + * exist. + * + * @param path - Snapshot directory to restore from. + * @param options - Must be `{ confirm: true }` — an explicit acknowledgment + * that current state is destroyed. + * @throws Error when `confirm` is not `true`, or `path` is not a snapshot + * directory. + * @example + * await brain.restore('/backups/2026-06-01', { confirm: true }) + */ + async restore(path: string, options: { confirm: boolean }): Promise { + this.assertWritable('restore') + await this.ensureInitialized() + + if (options?.confirm !== true) { + throw new Error( + `restore() replaces the store's entire current state with the snapshot at ` + + `'${path}'. Pass { confirm: true } to proceed.` + ) + } + + const pinCount = this.generationStore.activePinCount() + if (pinCount > 0) { + prodLog.warn( + `[Brainy] restore() with ${pinCount} live Db pin(s): pinned views do not ` + + `survive a restore — release Db values before restoring.` + ) + } + + const floorGeneration = this.generationStore.generation() + await this.storage.restoreFromDirectory(path) + await this.generationStore.reopenAfterRestore(floorGeneration) + + // Every in-memory index is now stale — rebuild all three from the + // restored canonical records (each rebuild clears its own state first). + await Promise.all([ + this.metadataIndex.rebuild(), + this.index.rebuild(), + this.graphIndex.rebuild() + ]) + } + + /** + * @description Construct AND initialize a `Brainy` instance in one call — + * `new Brainy(config)` + `await init()`. + * + * @param config - Standard `BrainyConfig`. + * @returns The initialized instance. + * @example + * const brain = await Brainy.open({ storage: { type: 'filesystem', rootDirectory: './data' } }) + */ + static async open(config?: BrainyConfig): Promise> { + const brain = new Brainy(config) + await brain.init() + return brain + } + + /** + * @description Open a persisted snapshot directory (from `db.persist()`) + * as a self-contained READ-ONLY store and return it as a `Db`. The + * snapshot's indexes load from its own files, so the full query surface — + * including vector search — works at the snapshot's generation. Releasing + * the returned `Db` closes the underlying read-only instance. + * + * @param path - Snapshot directory produced by `db.persist(path)`. + * @returns A `Db` over the snapshot (release it to free resources). + * @example + * const db = await Brainy.load('/backups/2026-06-01') + * const hits = await db.search('quarterly invoices') + * await db.release() + */ + static async load(path: string): Promise> { + const stat = await fs.promises.stat(path).catch(() => null) + if (!stat || !stat.isDirectory()) { + throw new Error( + `Brainy.load(): '${path}' is not a snapshot directory ` + + `(expected a directory created by db.persist(path))` + ) + } + + const brain = new Brainy({ + storage: { type: 'filesystem', rootDirectory: path }, + mode: 'reader' + }) + await brain.init() + return brain.createPinnedDb({ + generation: brain.generationStore.generation(), + timestamp: Date.now(), + closeOnRelease: () => brain.close() + }) + } + + // --- Db host plumbing ------------------------------------------------------ + + /** + * Guard for the synchronous Db entry points (`now()`, `generation()`): + * they cannot await initialization, so they demand it up front. + */ + private assertGenerationStoreReady(method: string): void { + if (!this.initialized || !this.generationStore) { + throw new Error( + `Cannot call ${method}() before init() completes. ` + + `Await brain.init() (or brain.ready, or use Brainy.open()).` + ) + } + } + + /** + * Pin `generation` and construct the `Db` value over the shared host. The + * `Db` constructor registers itself with the GC-backstop finalization + * registry; explicit `db.release()` unregisters first. + */ + private createPinnedDb(init: { + generation: number + timestamp: number + receipt?: TransactReceipt + closeOnRelease?: () => Promise + }): Db { + this.pinGeneration(init.generation) + return new Db({ host: this.dbHost, ...init }) + } + + /** + * The host surface every `Db` of this brain shares (lazily built once): + * live read fast paths, generation-record materialization, snapshot + * support, and pin lifecycle. See {@link DbHost} in src/db/db.ts. + */ + private get dbHost(): DbHost { + if (!this._dbHost) { + this._dbHost = { + store: this.generationStore, + get: (id, options) => this.get(id, options), + find: (query) => this.find(query), + getRelations: (paramsOrId) => this.getRelations(paramsOrId), + entityFromRecord: (id, record, includeVectors) => + this.entityFromGenerationRecord(id, record, includeVectors), + relationFromRecord: (id, record) => this.relationFromGenerationRecord(id, record), + persistPinned: (targetPath, generation) => + this.persistPinnedGeneration(targetPath, generation), + pinGeneration: (generation) => this.pinGeneration(generation), + releaseGeneration: (generation) => this.releaseGeneration(generation), + registerDbForFinalization: (db, generation, closeOnRelease) => { + this.dbFinalizationRegistry.register(db, { generation, closeOnRelease }, db) + }, + unregisterDbFromFinalization: (db) => { + this._dbFinalizationRegistry?.unregister(db) + } + } + } + return this._dbHost + } + + /** Lazily build the GC backstop for leaked (never-released) `Db` pins. */ + private get dbFinalizationRegistry(): FinalizationRegistry<{ + generation: number + closeOnRelease?: () => Promise + }> { + if (!this._dbFinalizationRegistry) { + this._dbFinalizationRegistry = new FinalizationRegistry((held) => { + this.releaseGeneration(held.generation) + if (held.closeOnRelease) { + void held.closeOnRelease().catch((err: Error) => { + prodLog.warn(`[Brainy] Db finalization close failed: ${err.message}`) + }) + } + }) + } + return this._dbFinalizationRegistry + } + + /** + * Registered index providers that implement the optional + * {@link VersionedIndexProvider} capability (feature-detected on the three + * index surfaces: vector, metadata, graph). Brainy's own JS indexes do not + * implement it. + */ + private versionedIndexProviders(): VersionedIndexProvider[] { + const candidates: unknown[] = [this.index, this.metadataIndex, this.graphIndex] + return candidates.filter(isVersionedIndexProvider) + } + + /** + * Take one refcounted pin on `generation`: storage-record pin (gates + * compaction) plus a `pin()` on every versioned index provider — the + * explicit pin lifetime overrides any time-based snapshot retention the + * provider has. Provider visibility (`isGenerationVisible`) is evaluated + * at pin time per the locked read-routing rule; provider-accelerated + * at-generation reads engage with the asOf-rebuild follow-up — until then + * historical reads always resolve from canonical generation records + * (strictly correct, provider-independent). + */ + private pinGeneration(generation: number): void { + this.generationStore.pin(generation) + const big = BigInt(generation) + for (const provider of this.versionedIndexProviders()) { + const visible = provider.isGenerationVisible(big) + provider.pin(big) + if (!visible) { + prodLog.debug( + `[Brainy] generation ${generation} pinned but not provider-visible — ` + + `historical reads at this pin resolve from canonical generation records` + ) + } + } + } + + /** Release one refcounted pin on `generation` (mirror of {@link pinGeneration}). */ + private releaseGeneration(generation: number): void { + this.generationStore.release(generation) + const big = BigInt(generation) + for (const provider of this.versionedIndexProviders()) { + provider.release(big) + } + } + + /** + * Materialize an `Entity` from a generation record's raw stored objects — + * the historical-read counterpart of the live `get()` paths. + * `record.metadata` is the exact stored metadata object; + * `record.vector` is the stored HNSW noun object (whose `.vector` carries + * the embedding) or `null` when the vector file was absent. + */ + private async entityFromGenerationRecord( + id: string, + record: { metadata: any; vector: any | null }, + includeVectors: boolean + ): Promise> { + const entity = await this.convertMetadataToEntity(id, record.metadata) + if (includeVectors && record.vector && Array.isArray(record.vector.vector)) { + entity.vector = record.vector.vector + } + return entity + } + + /** + * Materialize a `Relation` from a generation record's raw stored objects. + * Field split mirrors `storage.getVerb()` + `verbsToRelations()`: the + * stored vector object carries the structural core (`sourceId`/`targetId`/ + * `verb`), the stored metadata object carries typed fields plus the custom + * metadata bag. Returns `null` when the metadata part is absent (the + * relationship did not exist as a live edge). + * @throws Error when metadata exists but the structural core is missing — + * that state is unreachable through the API (relate() writes both parts + * in one atomic batch) and indicates external tampering; throwing beats + * silently dropping an edge from historical results. + */ + private relationFromGenerationRecord( + id: string, + record: { metadata: any; vector: any | null } + ): Relation | null { + if (record.metadata === null || record.metadata === undefined) { + return null + } + const core = record.vector as { sourceId?: string; targetId?: string; verb?: string } | null + if (!core || typeof core.sourceId !== 'string' || typeof core.targetId !== 'string') { + throw new Error( + `Generation record for relationship ${id} has metadata but no structural core ` + + `(sourceId/targetId) — store corrupted or records modified outside Brainy` + ) + } + + const { + verb, + subtype, + createdAt, + updatedAt, + confidence, + weight, + service, + data, + ...customMetadata + } = record.metadata as Record + + return { + id, + from: core.sourceId, + to: core.targetId, + type: (verb ?? core.verb) as VerbType, + ...(subtype !== undefined && { subtype: subtype as string }), + weight: (weight as number) ?? 1.0, + data, + metadata: customMetadata as T, + service: service as string, + createdAt: typeof createdAt === 'number' ? createdAt : Date.now(), + ...(typeof updatedAt === 'number' && { updatedAt }), + ...(typeof confidence === 'number' && { confidence }) + } + } + + /** + * Snapshot the store at `generation` into `targetPath` — the brain-side + * half of `db.persist()`. Indexes are flushed first (so the snapshot's + * persisted index files match its records), then the snapshot is cut + * under the generation store's commit lock: no transact commit, no + * compaction, and no counter write can interleave with the hard-link + * walk, and the counter is durably persisted into the snapshot. + * + * @throws NotYetSupportedAtHistoricalGenerationError when `generation` is + * no longer the store's latest — a snapshot captures current bytes, so + * persist the view before further writes (pin with `brain.now()`, + * persist, then mutate). + */ + private async persistPinnedGeneration(targetPath: string, generation: number): Promise { + await this.ensureInitialized() + await this.flush() + await this.generationStore.snapshotWith(async () => { + if (generation !== this.generationStore.generation()) { + throw new NotYetSupportedAtHistoricalGenerationError( + 'persist of a historical generation', + generation, + this.generationStore.generation() + ) + } + await this.storage.snapshotToDirectory(targetPath) + }) + } + + // --- Transact planner ------------------------------------------------------ + + /** + * Plan a full `transact()` batch: validate and resolve every operation + * against "current state + batch so far", producing the ordered + * TransactionManager operations, the receipt ids, the touched-id sets + * (which the generation store stages before-images for), and the + * post-commit aggregation hooks. Planning performs reads and embedding + * only — nothing touches storage until the generation store runs the + * batch. + */ + private async planTransact(ops: TxOperation[]): Promise { + const state: TxPlanState = { + nouns: new Map(), + removedNouns: new Set(), + verbs: new Map(), + removedVerbs: new Set() + } + const plan: PlannedTransact = { + operations: [], + ids: [], + touchedNouns: [], + touchedVerbs: [], + postCommit: [] + } + + for (const op of ops) { + switch (op.op) { + case 'add': + plan.ids.push(await this.planTxAdd(op, state, plan)) + break + case 'update': + plan.ids.push(await this.planTxUpdate(op, state, plan)) + break + case 'remove': + plan.ids.push(await this.planTxRemove(op, state, plan)) + break + case 'relate': + plan.ids.push(await this.planTxRelate(op, state, plan)) + break + case 'unrelate': + plan.ids.push(await this.planTxUnrelate(op, state, plan)) + break + default: { + const exhaustive: never = op + throw new Error(`transact(): unknown operation ${JSON.stringify(exhaustive)}`) + } + } + } + + return plan + } + + /** + * Resolve an entity during planning: batch-pending writes win, then the + * live store. `includeVectors: false` returns the stub vector exactly as + * the live metadata-only `get()` does, so planned relationship vectors + * match `relate()` byte-for-byte. + */ + private async planGetEntity( + state: TxPlanState, + id: string, + options?: GetOptions + ): Promise | null> { + if (state.removedNouns.has(id)) { + return null + } + const pending = state.nouns.get(id) + if (pending) { + const entity = await this.convertMetadataToEntity(id, pending.metadata) + if (options?.includeVectors) { + entity.vector = pending.vector + } + return entity + } + return this.get(id, options) + } + + /** Plan one `{ op: 'add' }` — mirror of `add()`. Returns the entity id. */ + private async planTxAdd( + op: Extract, { op: 'add' }>, + state: TxPlanState, + plan: PlannedTransact + ): Promise { + const { op: _discriminator, ...params } = op + validateAddParams(params as AddParams) + this.enforceTrackedFieldValues(params.metadata as Record | undefined, 'metadata') + this.enforceTrackedFieldValues({ subtype: params.subtype } as Record, 'top-level') + this.enforceSubtypeOnAdd('add', params.type, params.subtype, params.metadata) + + // ifAbsent — idempotent by-id insert, resolved against batch + store. + if (params.id && params.ifAbsent) { + const existing = await this.planGetEntity(state, params.id) + if (existing) { + return params.id + } + } + + const id = params.id || uuidv4() + const vector = params.vector || (await this.embed(params.data)) + if (!this.dimensions) { + this.dimensions = vector.length + } else if (vector.length !== this.dimensions) { + throw new Error( + `Vector dimension mismatch: expected ${this.dimensions}, got ${vector.length}` + ) + } + + // isNew controls the operation's rollback strategy: a custom id may + // collide with an existing entity (add() overwrite semantics), and a + // failed batch must restore the overwritten bytes. + const isNew = params.id + ? !state.nouns.has(id) && (await this.storage.getNounMetadata(id)) === null + : true + + const now = Date.now() + const storageMetadata = { + ...params.metadata, + data: params.data, + noun: params.type, + ...(params.subtype !== undefined && { subtype: params.subtype }), + service: params.service, + createdAt: now, + updatedAt: now, + _rev: 1, + ...(params.confidence !== undefined && { confidence: params.confidence }), + ...(params.weight !== undefined && { weight: params.weight }), + ...(params.createdBy && { createdBy: params.createdBy }) + } + const entityForIndexing = { + id, + vector, + connections: new Map(), + level: 0, + type: params.type, + ...(params.subtype !== undefined && { subtype: params.subtype }), + ...(params.confidence !== undefined && { confidence: params.confidence }), + ...(params.weight !== undefined && { weight: params.weight }), + createdAt: now, + updatedAt: now, + service: params.service, + data: params.data, + ...(params.createdBy && { createdBy: params.createdBy }), + metadata: params.metadata || {} + } + + plan.operations.push( + new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew), + new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew), + new AddToHNSWOperation(this.index as any, id, vector), + new AddToMetadataIndexOperation(this.metadataIndex, id, entityForIndexing) + ) + plan.touchedNouns.push(id) + plan.postCommit.push(() => { + if (this._aggregationIndex) { + this._aggregationIndex.onEntityAdded(id, entityForIndexing) + } + }) + + state.nouns.set(id, { metadata: storageMetadata, vector }) + state.removedNouns.delete(id) + return id + } + + /** Plan one `{ op: 'update' }` — mirror of `update()`. Returns the entity id. */ + private async planTxUpdate( + op: Extract, { op: 'update' }>, + state: TxPlanState, + plan: PlannedTransact + ): Promise { + const { op: _discriminator, ...params } = op + validateUpdateParams(params as UpdateParams) + this.enforceTrackedFieldValues(params.metadata as Record | undefined, 'metadata') + if (params.subtype !== undefined) { + this.enforceTrackedFieldValues({ subtype: params.subtype } as Record, 'top-level') + } + + const existing = await this.planGetEntity(state, params.id, { includeVectors: true }) + if (!existing) { + throw new Error(`Entity ${params.id} not found`) + } + + if (params.subtype !== undefined || params.type !== undefined) { + const effectiveType = params.type ?? existing.type + const effectiveSubtype = params.subtype !== undefined ? params.subtype : existing.subtype + const effectiveMetadata = params.metadata ?? existing.metadata + this.enforceSubtypeOnAdd('update', effectiveType, effectiveSubtype, effectiveMetadata) + } + + // ifRev CAS — identical resolution to update(); a conflict rejects the + // WHOLE batch before anything is staged or applied. + const currentRev = + typeof (existing.metadata as any)?._rev === 'number' + ? (existing.metadata as any)._rev + : typeof existing._rev === 'number' + ? existing._rev + : 1 + if (typeof params.ifRev === 'number' && params.ifRev !== currentRev) { + throw new RevisionConflictError(params.id, params.ifRev, currentRev) + } + + let vector = existing.vector + const needsReindexing = params.data || params.type + if (params.data) { + vector = params.vector || (await this.embed(params.data)) + } + + const newMetadata = + params.merge !== false + ? { ...existing.metadata, ...params.metadata } + : params.metadata || existing.metadata + const now = Date.now() + const updatedMetadata = { + ...newMetadata, + data: params.data !== undefined ? params.data : existing.data, + noun: params.type || existing.type, + service: existing.service, + createdAt: existing.createdAt, + updatedAt: now, + _rev: currentRev + 1, + ...(params.confidence !== undefined && { confidence: params.confidence }), + ...(params.weight !== undefined && { weight: params.weight }), + ...(params.confidence === undefined && + existing.confidence !== undefined && { confidence: existing.confidence }), + ...(params.weight === undefined && + existing.weight !== undefined && { weight: existing.weight }), + ...(params.subtype !== undefined && { subtype: params.subtype }), + ...(params.subtype === undefined && + existing.subtype !== undefined && { subtype: existing.subtype }) + } + const entityForIndexing = { + id: params.id, + vector, + connections: new Map(), + level: 0, + type: params.type || existing.type, + subtype: params.subtype !== undefined ? params.subtype : existing.subtype, + confidence: params.confidence !== undefined ? params.confidence : existing.confidence, + weight: params.weight !== undefined ? params.weight : existing.weight, + createdAt: existing.createdAt, + updatedAt: now, + service: existing.service, + data: params.data !== undefined ? params.data : existing.data, + createdBy: existing.createdBy, + metadata: newMetadata + } + const removalMetadata = { + type: existing.type, + confidence: existing.confidence, + weight: existing.weight, + createdAt: existing.createdAt, + updatedAt: existing.updatedAt, + service: existing.service, + data: existing.data, + createdBy: existing.createdBy, + metadata: existing.metadata + } + + plan.operations.push( + new UpdateNounMetadataOperation(this.storage, params.id, updatedMetadata), + new SaveNounOperation(this.storage, { + id: params.id, + vector, + connections: new Map(), + level: 0 + }) + ) + if (needsReindexing) { + plan.operations.push( + new RemoveFromHNSWOperation(this.index as any, params.id, existing.vector), + new AddToHNSWOperation(this.index as any, params.id, vector) + ) + } + plan.operations.push( + new RemoveFromMetadataIndexOperation(this.metadataIndex, params.id, removalMetadata), + new AddToMetadataIndexOperation(this.metadataIndex, params.id, entityForIndexing) + ) + plan.touchedNouns.push(params.id) + + const oldEntityForAgg = { + type: existing.type, + service: existing.service, + data: existing.data, + metadata: existing.metadata + } + plan.postCommit.push(() => { + if (this._aggregationIndex) { + this._aggregationIndex.onEntityUpdated(params.id, entityForIndexing, oldEntityForAgg) + } + }) + + state.nouns.set(params.id, { metadata: updatedMetadata, vector }) + return params.id + } + + /** + * Plan one `{ op: 'remove' }` — mirror of `delete()`, including the + * relationship cascade (every edge where the entity is source or target, + * plus edges created earlier in this batch). Returns the entity id. + */ + private async planTxRemove( + op: Extract, { op: 'remove' }>, + state: TxPlanState, + plan: PlannedTransact + ): Promise { + if (!op.id || typeof op.id !== 'string') { + throw new Error(`transact(): remove operation requires an entity id (got ${JSON.stringify(op.id)})`) + } + const id = op.id + + const pending = state.nouns.get(id) + const metadata = pending ? pending.metadata : await this.storage.getNounMetadata(id) + const noun = pending + ? { id, vector: pending.vector, connections: new Map(), level: 0 } + : await this.storage.getNoun(id) + + // Cascade set: stored edges touching the entity, plus batch-pending + // edges, minus edges already removed in this batch. + const storedVerbs = [ + ...(await this.storage.getVerbsBySource(id)), + ...(await this.storage.getVerbsByTarget(id)) + ] + const cascade = new Map() + for (const verb of storedVerbs) { + if (!state.removedVerbs.has(verb.id)) { + cascade.set(verb.id, verb) + } + } + for (const [verbId, verb] of state.verbs) { + if ( + !state.removedVerbs.has(verbId) && + (verb.sourceId === id || verb.targetId === id) + ) { + cascade.set(verbId, verb) + } + } + + if (noun) { + plan.operations.push(new RemoveFromHNSWOperation(this.index as any, id, noun.vector)) + } + if (metadata) { + plan.operations.push(new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata)) + } + plan.operations.push(new DeleteNounMetadataOperation(this.storage, id)) + for (const verb of cascade.values()) { + const { sourceInt, targetInt } = this.resolveVerbEndpointInts(verb) + plan.operations.push( + new RemoveFromGraphIndexOperation(this.graphIndex, verb, sourceInt, targetInt), + new DeleteVerbMetadataOperation(this.storage, verb.id) + ) + plan.touchedVerbs.push(verb.id) + state.verbs.delete(verb.id) + state.removedVerbs.add(verb.id) + } + plan.touchedNouns.push(id) + + if (metadata) { + const { noun: nounType, createdAt, updatedAt, confidence, weight, service, data, createdBy, ...customMetadata } = metadata + const entityForAgg = { type: nounType, service, data, metadata: customMetadata } + plan.postCommit.push(() => { + if (this._aggregationIndex) { + this._aggregationIndex.onEntityDeleted(id, entityForAgg) + } + }) + } + + state.nouns.delete(id) + state.removedNouns.add(id) + return id + } + + /** + * Plan one `{ op: 'relate' }` — mirror of `relate()`, including duplicate + * deduplication (against the store AND earlier batch operations) and + * `bidirectional`. Returns the (primary) relationship id — the existing id + * when deduplicated. + */ + private async planTxRelate( + op: Extract, { op: 'relate' }>, + state: TxPlanState, + plan: PlannedTransact + ): Promise { + const { op: _discriminator, ...params } = op + validateRelateParams(params as RelateParams) + this.enforceSubtypeOnRelate('relate', params.type, params.subtype, params.metadata) + + const fromEntity = await this.planGetEntity(state, params.from) + const toEntity = await this.planGetEntity(state, params.to) + if (!fromEntity) { + throw new Error(`Source entity ${params.from} not found`) + } + if (!toEntity) { + throw new Error(`Target entity ${params.to} not found`) + } + + // Dedupe against earlier operations of this batch first… + for (const [verbId, verb] of state.verbs) { + if ( + !state.removedVerbs.has(verbId) && + verb.sourceId === params.from && + verb.targetId === params.to && + verb.verb === params.type + ) { + return verbId + } + } + // …then against the committed graph (same lookup relate() performs). + const dupSourceInt = this.graphEntityInt(params.from) + const verbIds = + dupSourceInt === undefined + ? [] + : await this.resolveVerbIntsToIds(await this.graphIndex.getVerbIdsBySource(dupSourceInt)) + if (verbIds.length > 0) { + const verbsMap = await this.graphIndex.getVerbsBatchCached(verbIds) + for (const verb of verbsMap.values()) { + if ( + !state.removedVerbs.has(verb.id) && + verb.targetId === params.to && + verb.verb === params.type + ) { + return verb.id + } + } + } + + const id = uuidv4() + const relationVector = fromEntity.vector.map((v, i) => (v + toEntity.vector[i]) / 2) + const now = Date.now() + const verbMetadata = { + ...(params.metadata || {}), + verb: params.type, + ...(params.subtype !== undefined && { subtype: params.subtype }), + weight: params.weight ?? 1.0, + createdAt: now, + ...(params.data !== undefined && { data: params.data }) + } + const verb: GraphVerb = { + id, + vector: relationVector, + sourceId: params.from, + targetId: params.to, + verb: params.type, + type: params.type, + ...(params.subtype !== undefined && { subtype: params.subtype }), + weight: params.weight ?? 1.0, + metadata: params.metadata, + data: params.data, + createdAt: now + } + const { sourceInt, targetInt } = this.resolveVerbEndpointInts(verb) + + plan.operations.push( + new SaveVerbOperation(this.storage, { + id, + vector: relationVector, + connections: new Map(), + verb: params.type, + sourceId: params.from, + targetId: params.to + }), + new SaveVerbMetadataOperation(this.storage, id, verbMetadata), + new AddToGraphIndexOperation(this.graphIndex, verb, sourceInt, targetInt, (verbInt) => + this.cacheVerbInt(verbInt, id) + ) + ) + plan.touchedVerbs.push(id) + state.verbs.set(id, verb) + state.removedVerbs.delete(id) + + if (params.bidirectional) { + const reverseId = uuidv4() + const reverseVerb: GraphVerb = { + ...verb, + id: reverseId, + sourceId: params.to, + targetId: params.from, + sourceInt: targetInt, + targetInt: sourceInt + } + plan.operations.push( + new SaveVerbOperation(this.storage, { + id: reverseId, + vector: relationVector, + connections: new Map(), + verb: params.type, + sourceId: params.to, + targetId: params.from + }), + new SaveVerbMetadataOperation(this.storage, reverseId, verbMetadata), + new AddToGraphIndexOperation(this.graphIndex, reverseVerb, targetInt, sourceInt, (verbInt) => + this.cacheVerbInt(verbInt, reverseId) + ) + ) + plan.touchedVerbs.push(reverseId) + state.verbs.set(reverseId, reverseVerb) + state.removedVerbs.delete(reverseId) + } + + return id + } + + /** Plan one `{ op: 'unrelate' }` — mirror of `unrelate()`. Returns the relationship id. */ + private async planTxUnrelate( + op: Extract, { op: 'unrelate' }>, + state: TxPlanState, + plan: PlannedTransact + ): Promise { + if (!op.id || typeof op.id !== 'string') { + throw new Error(`transact(): unrelate operation requires a relationship id (got ${JSON.stringify(op.id)})`) + } + const id = op.id + + const verb = state.removedVerbs.has(id) + ? null + : (state.verbs.get(id) ?? (await this.storage.getVerb(id))) + + if (verb) { + const { sourceInt, targetInt } = this.resolveVerbEndpointInts(verb) + plan.operations.push( + new RemoveFromGraphIndexOperation(this.graphIndex, verb, sourceInt, targetInt) + ) + } + plan.operations.push(new DeleteVerbMetadataOperation(this.storage, id)) + plan.touchedVerbs.push(id) + + state.verbs.delete(id) + state.removedVerbs.add(id) + return id + } + /** * Get total count of nouns - O(1) operation * @returns Promise that resolves to the total number of nouns @@ -5905,7 +7121,11 @@ export class Brainy implements BrainyInterface { if (this.index && typeof (this.index as any).flush === 'function') { await (this.index as any).flush() } - })() + })(), + + // 5. Persist the generation counter (8.0 MVCC — coalesced single-op + // bumps become durable on every explicit flush) + this.generationStore.persistCounterNow() ]) const elapsed = Date.now() - startTime @@ -9796,6 +11016,12 @@ export class Brainy implements BrainyInterface { if (this._aggregationIndex) { await this._aggregationIndex.flush() } + })(), + // 8.0 MVCC: detach the generation-bump hook and persist the counter + (async () => { + if (this.generationStore) { + await this.generationStore.close() + } })() ]) diff --git a/src/db/db.ts b/src/db/db.ts new file mode 100644 index 00000000..48d1aa53 --- /dev/null +++ b/src/db/db.ts @@ -0,0 +1,806 @@ +/** + * @module db/db + * @description `Db` — Brainy 8.0's immutable, Datomic-style database value. + * + * A `Db` is a **readonly view pinned at one generation** of a Brainy store. + * It is produced by `brain.now()` (O(1) pin of the current generation), + * `brain.transact()` (pinned at the freshly committed generation), + * `brain.asOf()` (a past generation, a timestamp, or a persisted snapshot + * path), `Brainy.load(path)` (a snapshot opened read-only), and `db.with()` + * (a speculative in-memory overlay). + * + * **Read semantics.** While no transaction has committed past the pinned + * generation, every read delegates straight to the live brain — the pinned + * view IS the current state, and the existing fast paths serve it untouched. + * Once later transactions commit, the `Db` resolves reads through the + * generational record layer: `get()` and metadata-level `find()`/`related()` + * remain fully correct at the pinned generation (changed ids are resolved + * from immutable before-images; unchanged ids still ride the live fast + * path), while index-accelerated queries (semantic/vector search) throw the + * documented {@link NotYetSupportedAtHistoricalGenerationError} — an honest + * boundary, never silently-wrong results. + * + * **History granularity.** Generation records are written per `transact()` + * batch. Single-operation writes (`add`/`update`/`relate`/… outside + * `transact()`) advance the generation counter but do not produce historical + * records, so they remain visible through earlier pins — the documented 8.0 + * granularity (see `docs/ADR-001-generational-mvcc.md`, "History + * granularity"). + * + * **Lifecycle.** Each `Db` holds one refcounted pin on its generation (and + * on every registered {@link ../plugin.js VersionedIndexProvider}). Call + * {@link Db.release} when done — a `FinalizationRegistry` backstop releases + * leaked pins on garbage collection, but explicit release is what makes + * `compactHistory()` deterministic. + */ + +import type { + Entity, + FindParams, + GetOptions, + GetRelationsParams, + Relation, + Result +} from '../types/brainy.types.js' +import { v4 as uuidv4 } from '../universal/uuid.js' +import { NotYetSupportedAtHistoricalGenerationError } from './errors.js' +import type { GenerationStore } from './generationStore.js' +import type { ChangedIds, TransactReceipt, TxOperation } from './types.js' +import { entityMatchesFind, resolveEntityField, UnsupportedWhereOperatorError } from './whereMatcher.js' + +/** + * @description The speculative overlay carried by a `db.with()` value: + * per-id replacement entities/relations (`null` = tombstone). Reads on the + * overlay `Db` consult these maps first, then fall through to the pinned + * base view. Overlays are pure in-memory values — they never touch disk or + * index providers. + */ +export interface SpeculativeOverlay { + /** Entity overlays: replacement entity, or `null` for a speculative delete. */ + nouns: Map | null> + /** Relation overlays: replacement relation, or `null` for a speculative delete. */ + verbs: Map | null> +} + +/** + * @description The host surface a `Db` needs from its `Brainy` instance — + * constructed once per brain (see `brainy.ts`). Internal: consumers never + * build one; `Db` values are only created by `Brainy` and `db.with()`. + */ +export interface DbHost { + /** The generational record layer of the host brain. */ + readonly store: GenerationStore + /** Live `brain.get()` (current-generation fast path). */ + get(id: string, options?: GetOptions): Promise | null> + /** Live `brain.find()` (current-generation fast path). */ + find(query: string | FindParams): Promise[]> + /** Live `brain.getRelations()` (current-generation fast path). */ + getRelations(paramsOrId?: string | GetRelationsParams): Promise[]> + /** Materialize an entity from a generation record's raw stored objects. */ + entityFromRecord( + id: string, + record: { metadata: any; vector: any | null }, + includeVectors: boolean + ): Promise> + /** Materialize a relation from a generation record's raw stored objects. */ + relationFromRecord(id: string, record: { metadata: any; vector: any | null }): Relation | null + /** Snapshot the store at `generation` into `targetPath` (throws if the pin is stale). */ + persistPinned(targetPath: string, generation: number): Promise + /** Add one refcounted pin (store + versioned providers) on `generation`. */ + pinGeneration(generation: number): void + /** Release one refcounted pin (store + versioned providers) on `generation`. */ + releaseGeneration(generation: number): void + /** Register a `Db` with the GC-backstop finalization registry. */ + registerDbForFinalization(db: Db, generation: number, closeOnRelease?: () => Promise): void + /** Unregister a `Db` after an explicit release (no double-release on GC). */ + unregisterDbFromFinalization(db: Db): void +} + +/** + * @description Internal constructor arguments for {@link Db}. The pin on + * `generation` is taken by the creator (`Brainy` or `db.with()`) **before** + * construction; the constructor only registers the GC backstop. + */ +export interface DbInit { + /** The host surface (one per brain). */ + host: DbHost + /** The pinned generation. */ + generation: number + /** The view's timestamp (ms since epoch) — see {@link Db.timestamp}. */ + timestamp: number + /** Per-operation receipt, present only on `transact()`-produced values. */ + receipt?: TransactReceipt + /** Speculative overlay, present only on `with()`-produced values. */ + overlay?: SpeculativeOverlay + /** Extra teardown on release (e.g. closing an owned snapshot brain). */ + closeOnRelease?: () => Promise +} + +/** + * @description An immutable, generation-pinned view of a Brainy store. See + * the module documentation for the full read-semantics contract. + * + * @example + * const db = brain.now() // O(1) pin + * await brain.transact([{ op: 'update', id, metadata: { v: 2 } }]) + * await db.get(id) // still sees v: 1 — pinned + * await brain.get(id) // sees v: 2 — live + * await db.release() // unpin (enables compaction) + */ +export class Db { + private readonly host: DbHost + private readonly gen: number + private readonly ts: number + private readonly overlay?: SpeculativeOverlay + private readonly closeOnRelease?: () => Promise + private isReleased = false + + /** + * Per-operation receipt (committed generation, timestamp, resolved ids in + * input order). Present only on values returned by `brain.transact()`. + */ + public readonly receipt?: TransactReceipt + + /** + * @param init - Internal construction arguments. `Db` values are created + * by `Brainy` (`now`/`transact`/`asOf`/`load`) and `db.with()` — never + * directly by consumers. + */ + constructor(init: DbInit) { + this.host = init.host + this.gen = init.generation + this.ts = init.timestamp + this.receipt = init.receipt + this.overlay = init.overlay + this.closeOnRelease = init.closeOnRelease + this.host.registerDbForFinalization(this, this.gen, this.closeOnRelease) + } + + /** The generation this view is pinned at. */ + get generation(): number { + return this.gen + } + + /** + * The view's wall-clock timestamp (ms since epoch): pin time for `now()`, + * commit time for `transact()`, and the resolved commit time for `asOf()`. + */ + get timestamp(): number { + return this.ts + } + + /** Whether {@link Db.release} has been called (released views throw on use). */ + get released(): boolean { + return this.isReleased + } + + /** Whether this view carries a speculative `with()` overlay. */ + get speculative(): boolean { + return this.overlay !== undefined + } + + // ========================================================================== + // Reads + // ========================================================================== + + /** + * @description Get an entity by id **as of this view's generation**. + * Overlay entries win (speculative views); ids untouched since the pin ride + * the live fast path; ids changed by later transactions resolve from + * immutable generation records. Always correct at the pinned generation. + * + * @param id - The entity id. + * @param options - Same options as `brain.get()` (`includeVectors`). + * @returns The entity as of this generation, or `null`. + */ + async get(id: string, options?: GetOptions): Promise | null> { + this.assertUsable('get') + + if (this.overlay && this.overlay.nouns.has(id)) { + return this.overlay.nouns.get(id) ?? null + } + + if (!this.isHistorical()) { + return this.host.get(id, options) + } + + const resolved = await this.host.store.resolveAt('noun', id, this.gen) + if (resolved.source === 'current') { + return this.host.get(id, options) + } + if (resolved.source === 'absent' || resolved.metadata === null) { + // Live semantics: an entity without metadata is not a live entity. + return null + } + return this.host.entityFromRecord( + id, + { metadata: resolved.metadata, vector: resolved.vector }, + options?.includeVectors ?? false + ) + } + + /** + * @description Metadata-level `find()` **as of this view's generation**. + * + * At the current generation (nothing committed past the pin, no overlay) + * this delegates to `brain.find()` untouched — the full query surface, + * including semantic search. At a historical generation, or on a + * speculative view, only the metadata dimensions are supported (`type`, + * `subtype`, `where`, `service`, `excludeVFS`, `orderBy`/`order`, + * `limit`/`offset`): results are computed by combining the live index for + * unchanged ids with per-entity evaluation of the same filters against + * generation records / overlay entries — set-correct at the pinned + * generation. Index-only dimensions (`query`, `vector`, `near`, + * `connected`, `cursor`, `aggregate`, …) throw + * {@link NotYetSupportedAtHistoricalGenerationError}. + * + * Result ordering is deterministic when `orderBy` is supplied; without it, + * overlay/record-resolved matches follow the live-index matches. + * + * @param query - A `FindParams` object, or a string (semantic search — + * current-generation views only). + * @returns Matching results as of this generation (score `1.0` for + * record-resolved metadata matches, mirroring live metadata-only finds). + */ + async find(query: string | FindParams): Promise[]> { + this.assertUsable('find') + + const params: FindParams = typeof query === 'string' ? { query } : query + const historical = this.isHistorical() + + if (!historical && !this.overlay) { + return this.host.find(query) + } + + this.assertMetadataOnlyFind(params) + + const limit = params.limit ?? 10 + const offset = params.offset ?? 0 + + // Ids whose live-index answer is NOT valid at this generation. + const changedNouns = historical + ? (await this.host.store.changedBetween(this.gen, this.host.store.committedGeneration())).nouns + : [] + const overlayNouns = this.overlay?.nouns ?? new Map | null>() + const excluded = new Set([...changedNouns, ...overlayNouns.keys()]) + + try { + // Over-fetch from the live index so dropping superseded ids cannot + // starve the requested page, then merge and re-window. + const base = await this.host.find({ + ...params, + limit: limit + offset + excluded.size, + offset: 0 + }) + const merged = base.filter((result) => !excluded.has(result.id)) + + for (const id of changedNouns) { + if (overlayNouns.has(id)) continue // The overlay supersedes history. + const resolved = await this.host.store.resolveAt('noun', id, this.gen) + let entity: Entity | null = null + if (resolved.source === 'record' && resolved.metadata !== null) { + entity = await this.host.entityFromRecord( + id, + { metadata: resolved.metadata, vector: resolved.vector }, + false + ) + } else if (resolved.source === 'current') { + // Defensive: changed ids always resolve to a record or absent, but + // a 'current' answer is still served correctly from the live path. + entity = await this.host.get(id) + } + if (entity && entityMatchesFind(entity as Entity, params as FindParams)) { + merged.push(resultFromEntity(entity)) + } + } + + for (const [, entity] of overlayNouns) { + if (entity === null) continue // Speculative delete. + if (entityMatchesFind(entity as Entity, params as FindParams)) { + merged.push(resultFromEntity(entity)) + } + } + + if (params.orderBy) { + sortResultsBy(merged, params.orderBy, params.order ?? 'asc') + } + return merged.slice(offset, offset + limit) + } catch (err) { + if (err instanceof UnsupportedWhereOperatorError) { + throw new NotYetSupportedAtHistoricalGenerationError( + `metadata find with where-operator '${err.operator}'`, + this.gen, + this.host.store.generation() + ) + } + throw err + } + } + + /** + * @description Semantic (vector) search **at the current generation only**. + * Convenience for `find({ query, ...options })`. Vector indexes serve the + * live state; at a historical generation, or on a speculative overlay + * (whose entities carry no embeddings), this throws the documented + * {@link NotYetSupportedAtHistoricalGenerationError} — persist the + * generation you need and `Brainy.load()` it for the full query surface. + * + * @param query - Natural-language search query. + * @param options - Additional `FindParams` (limit, filters, …). + * @returns Semantic search results (current-generation views only). + * @throws NotYetSupportedAtHistoricalGenerationError at historical + * generations and on speculative overlays. + */ + async search(query: string, options?: Omit, 'query'>): Promise[]> { + this.assertUsable('search') + + if (!this.isHistorical() && !this.overlay) { + return this.host.find({ ...(options ?? {}), query }) + } + throw new NotYetSupportedAtHistoricalGenerationError( + this.overlay ? 'vector search (speculative overlay)' : 'vector search', + this.gen, + this.host.store.generation() + ) + } + + /** + * @description Relationships **as of this view's generation** — the `Db` + * counterpart of `brain.getRelations()` (same string-id shorthand and + * filter surface). Relations whose edges changed after the pin are + * resolved from generation records; overlay relations (speculative + * `relate`/`unrelate`) and cascade tombstones (relations touching a + * speculatively deleted entity) are applied on top. Cursor pagination is + * index-only and throws the documented historical error on + * non-current/speculative views. + * + * @param paramsOrId - An entity id (shorthand for `{ from: id }`) or a + * `GetRelationsParams` filter object. + * @returns Relations as of this generation. + */ + async related(paramsOrId?: string | GetRelationsParams): Promise[]> { + this.assertUsable('related') + + const params: GetRelationsParams = + typeof paramsOrId === 'string' ? { from: paramsOrId } : (paramsOrId ?? {}) + const historical = this.isHistorical() + + if (!historical && !this.overlay) { + return this.host.getRelations(paramsOrId) + } + + if (params.cursor !== undefined) { + throw new NotYetSupportedAtHistoricalGenerationError( + 'cursor pagination on related()', + this.gen, + this.host.store.generation() + ) + } + + const limit = params.limit ?? 100 + const offset = params.offset ?? 0 + + const changedVerbs = historical + ? (await this.host.store.changedBetween(this.gen, this.host.store.committedGeneration())).verbs + : [] + const overlayVerbs = this.overlay?.verbs ?? new Map | null>() + const excluded = new Set([...changedVerbs, ...overlayVerbs.keys()]) + + const base = await this.host.getRelations({ + ...params, + limit: limit + offset + excluded.size, + offset: 0 + }) + let merged = base.filter((relation) => !excluded.has(relation.id)) + + for (const id of changedVerbs) { + if (overlayVerbs.has(id)) continue + const resolved = await this.host.store.resolveAt('verb', id, this.gen) + if (resolved.source !== 'record') continue + const relation = this.host.relationFromRecord(id, { + metadata: resolved.metadata, + vector: resolved.vector + }) + if (relation && relationMatchesParams(relation, params)) { + merged.push(relation) + } + } + + for (const [, relation] of overlayVerbs) { + if (relation === null) continue // Speculative unrelate. + if (relationMatchesParams(relation, params)) { + merged.push(relation) + } + } + + // Cascade semantics for speculative deletes: a relation whose endpoint + // is tombstoned in the overlay is gone in this view, exactly as + // brain.delete() cascades on the durable path. + if (this.overlay) { + const tombstoned = new Set() + for (const [id, entity] of this.overlay.nouns) { + if (entity === null) tombstoned.add(id) + } + if (tombstoned.size > 0) { + merged = merged.filter( + (relation) => !tombstoned.has(relation.from) && !tombstoned.has(relation.to) + ) + } + } + + return merged.slice(offset, offset + limit) + } + + // ========================================================================== + // Derived views + // ========================================================================== + + /** + * @description Speculative write: returns a NEW `Db` whose reads see the + * given transaction data applied **in memory, on top of this view** — + * Datomic's `with`. Nothing touches disk, the generation counter, or index + * providers; the underlying brain and this view are untouched. Use it to + * answer "what would the store look like if…" questions, then call + * `brain.transact()` with the same operations to make it real. + * + * Speculative semantics (documented deltas from the durable paths): + * - `add` without an explicit `vector` produces an entity with an empty + * stub vector — speculative entities are not semantically searchable + * (`with()` never invokes the embedder). + * - `remove` cascades on the read side: relations touching the removed + * entity disappear from `related()` without being enumerated. + * - `relate` deduplicates against edges visible in this view when verb + * history allows the check, mirroring `brain.relate()`. + * - Brain-level vocabulary/subtype enforcement does not run — overlays + * never persist, so there is nothing to protect. + * + * The returned view takes its own pin on this generation; release both + * views independently. + * + * @param ops - The same declarative operations `brain.transact()` accepts. + * @returns A new speculative `Db` over this view. + * @throws Error when an operation references a missing entity/relation, + * mirroring the durable path's planning errors. + */ + async with(ops: TxOperation[]): Promise> { + this.assertUsable('with') + if (!Array.isArray(ops) || ops.length === 0) { + throw new Error('with(): provide a non-empty array of transaction operations') + } + + // Child overlay starts as a copy of this view's overlay (chained with()). + const overlay: SpeculativeOverlay = { + nouns: new Map(this.overlay?.nouns ?? []), + verbs: new Map(this.overlay?.verbs ?? []) + } + + // Reads during planning must see "this view + overlay so far". + const speculativeGet = async (id: string): Promise | null> => { + if (overlay.nouns.has(id)) return overlay.nouns.get(id) ?? null + return this.get(id) + } + + for (const op of ops) { + switch (op.op) { + case 'add': { + const id = op.id ?? uuidv4() + const now = Date.now() + overlay.nouns.set(id, { + id, + vector: op.vector ?? [], + type: op.type, + ...(op.subtype !== undefined && { subtype: op.subtype }), + data: op.data, + metadata: (op.metadata ?? {}) as T, + ...(op.service !== undefined && { service: op.service }), + createdAt: now, + updatedAt: now, + ...(op.confidence !== undefined && { confidence: op.confidence }), + ...(op.weight !== undefined && { weight: op.weight }), + _rev: 1 + }) + break + } + case 'update': { + const base = await speculativeGet(op.id) + if (!base) { + throw new Error(`with(): entity ${op.id} not found at generation ${this.gen}`) + } + const mergedMetadata = + op.merge !== false + ? ({ ...(base.metadata as object), ...(op.metadata as object) } as T) + : ((op.metadata ?? base.metadata) as T) + overlay.nouns.set(op.id, { + ...base, + ...(op.type !== undefined && { type: op.type }), + ...(op.subtype !== undefined && { subtype: op.subtype }), + ...(op.data !== undefined && { data: op.data }), + ...(op.vector !== undefined && { vector: op.vector }), + ...(op.confidence !== undefined && { confidence: op.confidence }), + ...(op.weight !== undefined && { weight: op.weight }), + metadata: mergedMetadata, + updatedAt: Date.now(), + _rev: (base._rev ?? 1) + 1 + }) + break + } + case 'remove': { + overlay.nouns.set(op.id, null) + // Tombstone overlay relations touching the removed entity; base + // relations are cascade-filtered at read time in related(). + for (const [verbId, relation] of overlay.verbs) { + if (relation && (relation.from === op.id || relation.to === op.id)) { + overlay.verbs.set(verbId, null) + } + } + break + } + case 'relate': { + const from = await speculativeGet(op.from) + const to = await speculativeGet(op.to) + if (!from) throw new Error(`with(): source entity ${op.from} not found`) + if (!to) throw new Error(`with(): target entity ${op.to} not found`) + + // Dedupe against the view (overlay first, then committed edges + // when verb history allows the read) — mirror of relate(). + let duplicate: Relation | undefined + for (const relation of overlay.verbs.values()) { + if (relation && relation.from === op.from && relation.to === op.to && relation.type === op.type) { + duplicate = relation + break + } + } + if (!duplicate) { + try { + const existing = await this.related({ from: op.from, type: op.type }) + duplicate = existing.find((relation) => relation.to === op.to) + } catch (err) { + if (!(err instanceof NotYetSupportedAtHistoricalGenerationError)) throw err + // Verb history can't serve the check at this generation — + // proceed with overlay-only dedupe (documented). + } + } + if (duplicate) break + + const id = uuidv4() + overlay.verbs.set(id, { + id, + from: op.from, + to: op.to, + type: op.type, + ...(op.subtype !== undefined && { subtype: op.subtype }), + weight: op.weight ?? 1.0, + ...(op.data !== undefined && { data: op.data }), + metadata: op.metadata, + ...(op.service !== undefined && { service: op.service }), + createdAt: Date.now() + }) + if (op.bidirectional) { + const reverseId = uuidv4() + overlay.verbs.set(reverseId, { + id: reverseId, + from: op.to, + to: op.from, + type: op.type, + ...(op.subtype !== undefined && { subtype: op.subtype }), + weight: op.weight ?? 1.0, + ...(op.data !== undefined && { data: op.data }), + metadata: op.metadata, + ...(op.service !== undefined && { service: op.service }), + createdAt: Date.now() + }) + } + break + } + case 'unrelate': { + overlay.verbs.set(op.id, null) + break + } + default: { + const exhaustive: never = op + throw new Error(`with(): unknown operation ${JSON.stringify(exhaustive)}`) + } + } + } + + this.host.pinGeneration(this.gen) + return new Db({ + host: this.host, + generation: this.gen, + timestamp: Date.now(), + overlay + }) + } + + /** + * @description The ids changed by committed transactions in the interval + * `(priorDb.generation, this.generation]` — the diff between two pinned + * views of the same store. Single-operation writes between commits are not + * reflected (documented 8.0 history granularity). + * + * @param priorDb - An earlier view of the SAME store. + * @returns Changed entity and relation ids, sorted. + * @throws RangeError when `priorDb` is newer than this view, or Error when + * the views belong to different stores. + */ + async since(priorDb: Db): Promise { + this.assertUsable('since') + if (priorDb.host.store !== this.host.store) { + throw new Error('since(): both Db values must come from the same Brainy instance') + } + if (priorDb.gen > this.gen) { + throw new RangeError( + `since(): priorDb is at generation ${priorDb.gen}, which is newer than this view ` + + `(generation ${this.gen}). Call newerDb.since(olderDb).` + ) + } + return this.host.store.changedBetween(priorDb.gen, this.gen) + } + + // ========================================================================== + // Durability + // ========================================================================== + + /** + * @description Snapshot this view to `path` as a self-contained store: + * flush, then hard-link every immutable file into the target + * (Cassandra-style — instant and space-shared; cross-device targets fall + * back to byte copies, and the small append-in-place file list is always + * byte-copied). The result opens with `Brainy.load(path)` with the full + * query surface, and later mutations of the source can never alter it — + * rewrites swap inodes, the snapshot keeps the old bytes. + * + * `persist()` requires this view to still be the store's **latest** + * generation (snapshotting captures current bytes): pin with `brain.now()` + * and persist before further writes. A view that history has moved past + * throws {@link NotYetSupportedAtHistoricalGenerationError}; speculative + * overlays cannot be persisted (commit them with `brain.transact()` first). + * + * @param path - Absolute directory for the snapshot (created; must be + * empty or absent). + * @throws NotYetSupportedAtHistoricalGenerationError when this view is no + * longer the latest generation, or is speculative. + */ + async persist(path: string): Promise { + this.assertUsable('persist') + if (this.overlay) { + throw new NotYetSupportedAtHistoricalGenerationError( + 'persist of a speculative overlay', + this.gen, + this.host.store.generation() + ) + } + await this.host.persistPinned(path, this.gen) + } + + // ========================================================================== + // Lifecycle + // ========================================================================== + + /** + * @description Release this view's pin (store + versioned index + * providers). Idempotent. After release, every read throws. Views from + * `Brainy.load()` / `asOf(path)` also close their underlying read-only + * brain here. A `FinalizationRegistry` backstop releases leaked pins when + * a `Db` is garbage-collected, but explicit release is what makes + * `compactHistory()` deterministic — prefer it. + */ + async release(): Promise { + if (this.isReleased) return + this.isReleased = true + this.host.unregisterDbFromFinalization(this) + this.host.releaseGeneration(this.gen) + if (this.closeOnRelease) { + await this.closeOnRelease() + } + } + + // ========================================================================== + // Internals + // ========================================================================== + + /** Whether any transaction committed past the pinned generation. */ + private isHistorical(): boolean { + return this.host.store.hasCommittedAfter(this.gen) + } + + /** Throw on use-after-release. */ + private assertUsable(method: string): void { + if (this.isReleased) { + throw new Error( + `Cannot call ${method}() on a released Db (generation ${this.gen}). ` + + `Pin a fresh view with brain.now() or brain.asOf().` + ) + } + } + + /** + * Reject find() dimensions that require index machinery the record layer + * cannot answer at a historical generation / for a speculative overlay. + */ + private assertMetadataOnlyFind(params: FindParams): void { + const unsupported: Array<[string, boolean]> = [ + ['semantic query', params.query !== undefined], + ['vector search', params.vector !== undefined], + ['proximity search (near)', params.near !== undefined], + ['graph traversal (connected)', params.connected !== undefined], + ['cursor pagination', params.cursor !== undefined], + ['aggregation', params.aggregate !== undefined], + ['relation expansion (includeRelations)', params.includeRelations === true], + [ + `search mode '${params.mode ?? params.searchMode}'`, + (params.mode !== undefined && params.mode !== 'metadata' && params.mode !== 'auto') || + (params.searchMode !== undefined && params.searchMode !== 'auto') + ] + ] + for (const [capability, present] of unsupported) { + if (present) { + throw new NotYetSupportedAtHistoricalGenerationError( + capability, + this.gen, + this.host.store.generation() + ) + } + } + } +} + +/** + * @description Build a `Result` row from a record/overlay-resolved entity, + * mirroring the live metadata-only find path (flattened fields, score `1.0`). + */ +function resultFromEntity(entity: Entity): Result { + return { + id: entity.id, + score: 1.0, + type: entity.type, + subtype: entity.subtype, + metadata: entity.metadata, + data: entity.data, + confidence: entity.confidence, + weight: entity.weight, + _rev: entity._rev, + entity + } +} + +/** + * @description In-place stable sort of merged results by a find() `orderBy` + * field, mirroring live ordering semantics (`asc` default, undefined values + * last in both directions). + */ +function sortResultsBy(results: Result[], orderBy: string, order: 'asc' | 'desc'): void { + const direction = order === 'desc' ? -1 : 1 + results.sort((a, b) => { + const va = resolveEntityField(a.entity as Entity, orderBy) + const vb = resolveEntityField(b.entity as Entity, orderBy) + if (va === undefined && vb === undefined) return 0 + if (va === undefined) return 1 + if (vb === undefined) return -1 + if (typeof va === 'number' && typeof vb === 'number') return (va - vb) * direction + const sa = String(va) + const sb = String(vb) + return (sa < sb ? -1 : sa > sb ? 1 : 0) * direction + }) +} + +/** + * @description Filter one relation against `GetRelationsParams`, mirroring + * the storage-level filter `brain.getRelations()` builds (`from` → sourceId, + * `to` → targetId, type/subtype set membership, service equality). + */ +function relationMatchesParams(relation: Relation, params: GetRelationsParams): boolean { + if (params.from !== undefined && relation.from !== params.from) return false + if (params.to !== undefined && relation.to !== params.to) return false + if (params.type !== undefined) { + const types = Array.isArray(params.type) ? params.type : [params.type] + if (!types.includes(relation.type)) return false + } + if (params.subtype !== undefined) { + const subtypes = Array.isArray(params.subtype) ? params.subtype : [params.subtype] + if (relation.subtype === undefined || !subtypes.includes(relation.subtype)) return false + } + if (params.service !== undefined && relation.service !== params.service) return false + return true +} + diff --git a/src/db/errors.ts b/src/db/errors.ts new file mode 100644 index 00000000..f369c1e3 --- /dev/null +++ b/src/db/errors.ts @@ -0,0 +1,146 @@ +/** + * @module db/errors + * @description Error types for the 8.0 generational-MVCC Db API. + * + * Three failure modes have dedicated classes so callers can branch on them + * with `instanceof` instead of string matching: + * + * - {@link GenerationConflictError} — a `transact()` compare-and-swap + * (`ifAtGeneration`) observed a different current generation than the + * caller expected. The standard retry pattern is: re-read via + * `brain.now()`, re-derive the transaction, and re-submit. + * - {@link NotYetSupportedAtHistoricalGenerationError} — an index-accelerated + * query (vector / hybrid / graph-traversal search) was issued against a + * `Db` pinned at a generation the live indexes no longer represent. + * `get()` and metadata-filter `find()` remain fully supported at pinned + * generations; index-accelerated queries at historical generations require + * an index rebuild from the pinned record set, which ships as a follow-up + * (`asOf` rebuild). The error message names the unsupported capability and + * points at the supported alternatives. + * - {@link GenerationCompactedError} — `asOf()` asked for a generation whose + * immutable records were reclaimed by `compactHistory()`. + * + * All three are exported from the package root (`@soulcraft/brainy`). + */ + +/** + * @description Thrown by `brain.transact(ops, { ifAtGeneration })` when the + * store's current generation does not equal the caller-supplied expectation. + * This is the whole-store compare-and-swap counterpart to the per-entity + * `ifRev` / `RevisionConflictError` pair: it guarantees that *nothing* was + * committed between the caller's read (`brain.now()`) and this transaction. + * + * The transaction is rejected before any record is staged — the store is + * untouched and the generation counter is unchanged. + * + * @example + * const db = brain.now() + * try { + * await brain.transact(ops, { ifAtGeneration: db.generation }) + * } catch (err) { + * if (err instanceof GenerationConflictError) { + * // Someone committed since we pinned — re-read and retry. + * console.log(`expected ${err.expected}, store is at ${err.actual}`) + * } + * } + */ +export class GenerationConflictError extends Error { + /** The generation the caller expected the store to be at. */ + public readonly expected: number + /** The generation the store was actually at when the transaction arrived. */ + public readonly actual: number + + /** + * @param expected - The generation supplied via `ifAtGeneration`. + * @param actual - The store's current generation at submission time. + */ + constructor(expected: number, actual: number) { + super( + `Generation conflict: transact() was submitted with ifAtGeneration: ${expected}, ` + + `but the store is at generation ${actual}. Another write committed in between. ` + + `Re-read with brain.now(), rebuild the transaction, and retry.` + ) + this.name = 'GenerationConflictError' + this.expected = expected + this.actual = actual + } +} + +/** + * @description Thrown when an operation on a pinned `Db` requires + * index-accelerated machinery (vector search, hybrid search, graph + * traversal, cursor pagination, aggregation) that the live indexes cannot + * answer for a *historical* generation. Storage-record reads (`get()`) and + * metadata-filter `find()` are always supported at pinned generations. + * + * Rebuilding indexes from a pinned record set (the standard LSM answer for + * historical index queries) is the documented `asOf`-rebuild follow-up; this + * error is the honest boundary until it ships. + * + * @example + * const db = brain.now() + * await brain.transact([{ op: 'update', id, metadata: { v: 2 } }]) + * await db.get(id) // ✅ pinned read — supported + * await db.find({ where: { v: 1 } }) // ✅ metadata find — supported + * await db.search('semantic query') // ❌ throws this error + */ +export class NotYetSupportedAtHistoricalGenerationError extends Error { + /** The pinned generation the operation was attempted against. */ + public readonly generation: number + /** The capability that is not supported at historical generations. */ + public readonly capability: string + + /** + * @param capability - Human-readable name of the unsupported capability + * (e.g. `'vector search'`, `'graph traversal (connected)'`). + * @param generation - The Db's pinned generation. + * @param currentGeneration - The store's current generation, for context. + */ + constructor(capability: string, generation: number, currentGeneration: number) { + super( + `${capability} is not yet supported at a historical generation ` + + `(Db pinned at generation ${generation}; store is at ${currentGeneration}). ` + + `Supported at pinned generations: get() and metadata-filter find(). ` + + `For index-accelerated queries over historical state, persist() the generation ` + + `you need and open it with Brainy.load(path) — the loaded snapshot rebuilds ` + + `its indexes and supports the full query surface.` + ) + this.name = 'NotYetSupportedAtHistoricalGenerationError' + this.capability = capability + this.generation = generation + } +} + +/** + * @description Thrown by `brain.asOf(generationOrTimestamp)` when the + * requested generation's history was reclaimed by `brain.compactHistory()`. + * The store records the compaction horizon (the highest reclaimed + * generation) in its manifest; any generation BELOW the horizon is + * unreachable — its reads would need the reclaimed before-images. The + * horizon itself stays reachable, resolved from the record-sets above it. + * + * To keep a generation readable forever, `persist()` it to a snapshot + * directory before compacting — snapshots are self-contained and unaffected + * by compaction of the source store. + */ +export class GenerationCompactedError extends Error { + /** The generation that was requested. */ + public readonly requested: number + /** The compaction horizon — generations < this value are unreachable. */ + public readonly horizon: number + + /** + * @param requested - The generation the caller asked for. + * @param horizon - The store's current compaction horizon. + */ + constructor(requested: number, horizon: number) { + super( + `Generation ${requested} has been compacted away (compaction horizon: ${horizon}). ` + + `Only generations at or above the horizon are reachable via asOf(). ` + + `Use db.persist(path) before compactHistory() to keep a generation readable.` + ) + this.name = 'GenerationCompactedError' + this.requested = requested + this.horizon = horizon + } +} diff --git a/src/db/generationStore.ts b/src/db/generationStore.ts new file mode 100644 index 00000000..e56d43c7 --- /dev/null +++ b/src/db/generationStore.ts @@ -0,0 +1,838 @@ +/** + * @module db/generationStore + * @description The generational record layer behind Brainy 8.0's MVCC Db API. + * + * One `GenerationStore` is attached to every writable `Brainy` instance. It + * owns: + * + * - the **monotonic generation counter** (`_system/generation.json`), + * reserved per `transact()` commit and bumped per single-operation write + * batch via the storage-layer hook; + * - the **commit protocol** for `brain.transact()`: capture before-images → + * write the generation delta → fsync → execute the operation batch → + * atomically rename `_system/manifest.json` (the rename IS the commit + * point) → append the tx-log line; + * - **crash recovery**: on open, any `_generations/` directory with + * `N > manifest.generation` is an uncommitted transaction — its + * before-images are restored to the canonical entity paths and the + * directory is removed, so a crash anywhere before the manifest rename + * leaves the store byte-identical to its pre-transaction state; + * - **pin/release refcounting** for live `Db` values, which is what makes + * `compactHistory()` safe: a generation record-set is reclaimed only when + * no live pin could ever need it; + * - **point-in-time resolution**: the state of id X at pinned generation G + * is the before-image stored by the *first* committed generation after G + * that touched X — or the live storage state when nothing after G touched + * it (so current-generation reads stay on the existing fast paths, + * untouched). + * + * Durability and atomicity guarantees, failure modes, and the intellectual + * lineage (Datomic database-as-value, LMDB reader pins, LSM immutable + * segments) are specified in `docs/ADR-001-generational-mvcc.md`. + */ + +import { prodLog } from '../utils/logger.js' +import { GenerationCompactedError, GenerationConflictError } from './errors.js' +import type { + ChangedIds, + CompactHistoryOptions, + CompactHistoryResult, + GenerationDelta, + GenerationManifest, + GenerationRecord, + GenerationStorage, + TxLogEntry +} from './types.js' + +/** Storage-root-relative path of the persisted generation counter. */ +export const GENERATION_COUNTER_PATH = '_system/generation.json' +/** Storage-root-relative path of the commit manifest. */ +export const MANIFEST_PATH = '_system/manifest.json' +/** Storage-root-relative prefix of the per-generation record directories. */ +export const GENERATIONS_PREFIX = '_generations' + +/** + * @description Phases of the {@link GenerationStore.commitTransaction} commit + * protocol at which a test-only fault injector can simulate a process crash. + * The phases map 1:1 onto the protocol steps documented on + * `commitTransaction`: + * + * - `'after-staging'` — before-images + the `tx.json` delta are written and + * fsynced; the operation batch has NOT executed yet. + * - `'after-execute'` — the operation batch has been applied to canonical + * storage and indexes; the manifest still points at the prior generation. + * - `'before-manifest-rename'` — the batch has executed and the counter is + * persisted; the atomic manifest rename — the commit point — has NOT + * happened. A crash here is the canonical "fully staged, never committed" + * case that recovery must roll back byte-identically. + * - `'after-manifest-rename'` — the manifest rename landed (the transaction + * IS committed); the tx-log append has NOT happened yet. A crash here must + * keep the transaction (the tx-log is advisory metadata, not the source of + * commit truth). + */ +export type CommitFaultPhase = + | 'after-staging' + | 'after-execute' + | 'before-manifest-rename' + | 'after-manifest-rename' + +/** + * @description Identifies which ids a transaction touches, split by kind. + */ +export interface TouchedIds { + /** Entity ids the transaction writes or deletes. */ + nouns: string[] + /** Relationship ids the transaction writes or deletes. */ + verbs: string[] +} + +/** + * @description Result of {@link GenerationStore.open} — how many uncommitted + * generations crash recovery rolled back (a non-zero value tells `Brainy` to + * force an index reconciliation, since derived index state may reference the + * rolled-back writes). + */ +export interface GenerationStoreOpenResult { + /** Count of uncommitted generation directories rolled back and removed. */ + rolledBackGenerations: number +} + +/** + * @description The generational MVCC record layer. See the module + * documentation for the full protocol; every public method documents its + * own contract. + */ +export class GenerationStore { + private readonly storage: GenerationStorage + + /** Latest reserved/observed generation (≥ {@link committed}). */ + private counter = 0 + /** Committed-transaction watermark (manifest generation). */ + private committed = 0 + /** Compaction horizon — record-sets ≤ this are reclaimed. */ + private horizonGen = 0 + + /** Sorted list of committed generations whose record dirs exist. */ + private committedGens: number[] = [] + /** Delta cache, keyed by generation (lazily loaded from `tx.json`). */ + private readonly deltaCache = new Map< + number, + { nouns: Set; verbs: Set; timestamp: number } + >() + + /** Live pin refcounts, keyed by pinned generation. */ + private readonly pins = new Map() + + /** Serializes transact commits, compaction, and counter persistence. */ + private mutexTail: Promise = Promise.resolve() + + /** True while a transact batch is executing (suppresses the bump hook). */ + private inTransact = false + + /** Coalescing state for lazy persistence of single-op counter bumps. */ + private counterPersistScheduled = false + + /** + * Test-only fault injector for the commit protocol (see + * {@link GenerationStore.setCommitFaultInjector}). `undefined` in production. + */ + private commitFaultInjector?: (phase: CommitFaultPhase) => void + + private opened = false + + /** + * @param storage - The storage adapter, narrowed to the + * {@link GenerationStorage} contract (`BaseStorage` implements it). + */ + constructor(storage: GenerationStorage) { + this.storage = storage + } + + // ========================================================================== + // Lifecycle + // ========================================================================== + + /** + * @description Load the persisted counter + manifest and run crash + * recovery. Must be called once, before indexes are built, because + * recovery may rewrite canonical entity files (restoring before-images of + * an uncommitted transaction). + * + * @param options.readOnly - When true (reader-mode brains), recovery is + * skipped: readers never write. An un-repaired crashed directory is + * repaired by the next *writer* open. + * @returns How many uncommitted generations were rolled back. + */ + async open(options?: { readOnly?: boolean }): Promise { + const counterFile = (await this.storage.readRawObject(GENERATION_COUNTER_PATH)) as + | { generation?: number } + | null + const manifest = (await this.storage.readRawObject(MANIFEST_PATH)) as GenerationManifest | null + + this.committed = manifest?.generation ?? 0 + this.horizonGen = manifest?.horizon ?? 0 + this.counter = Math.max(counterFile?.generation ?? 0, this.committed) + + // Discover existing generation record directories. + const recordPaths = await this.storage.listRawObjects(GENERATIONS_PREFIX) + const seenGens = new Set() + for (const p of recordPaths) { + const gen = parseGenerationFromPath(p) + if (gen !== null) seenGens.add(gen) + } + + let rolledBack = 0 + const committedGens: number[] = [] + for (const gen of [...seenGens].sort((a, b) => a - b)) { + if (gen <= this.committed) { + committedGens.push(gen) + } else if (!options?.readOnly) { + // Uncommitted (crashed) transaction: restore before-images, drop the dir. + await this.rollBackUncommittedGeneration(gen) + rolledBack++ + } + // Reader mode: leave orphan dirs alone; resolution ignores them because + // committedGens only includes generations ≤ the manifest watermark. + this.counter = Math.max(this.counter, gen) + } + this.committedGens = committedGens + + if (rolledBack > 0) { + prodLog.warn( + `[GenerationStore] Crash recovery rolled back ${rolledBack} uncommitted ` + + `transaction(s); store restored to generation ${this.committed}.` + ) + } + + this.opened = true + + // Hook single-op write batches so generation() is always meaningful. + // Suppressed while a transact batch executes (the batch is ONE generation). + if (!options?.readOnly) { + this.storage.setGenerationBumpHook(() => this.noteSingleOpWrite()) + } + + return { rolledBackGenerations: rolledBack } + } + + /** + * @description Detach the storage hook and persist the counter. Called + * from `brain.close()`. + */ + async close(): Promise { + this.storage.setGenerationBumpHook(undefined) + await this.persistCounterNow() + } + + /** + * @description TEST-ONLY: install (or clear, with `undefined`) a fault + * injector that is invoked at each {@link CommitFaultPhase} of the commit + * protocol. A **throwing** injector simulates a process crash at that + * phase: the abort cleanup (staging-directory removal, generation-counter + * reservation return) is deliberately skipped — exactly as if the process + * had died — so crash-consistency tests exercise the REAL recovery path + * ({@link GenerationStore.open} rolling back the uncommitted generation on + * the next open). Never installed in production code; the injector exists + * solely so the durability protocol can be proven, not trusted. + * + * @param injector - Callback invoked synchronously at each phase, or + * `undefined` to detach. + */ + setCommitFaultInjector(injector: ((phase: CommitFaultPhase) => void) | undefined): void { + this.commitFaultInjector = injector + } + + // ========================================================================== + // Generation counter + // ========================================================================== + + /** @returns The store's current generation (latest write watermark). */ + generation(): number { + return this.counter + } + + /** @returns The committed-transaction watermark (manifest generation). */ + committedGeneration(): number { + return this.committed + } + + /** @returns The compaction horizon (generations ≤ horizon are reclaimed). */ + horizon(): number { + return this.horizonGen + } + + /** + * @description Single-operation write hook (registered with the storage + * layer in {@link open}). Bumps the in-memory counter and schedules a + * coalesced persist of `_system/generation.json` — durable artifacts + * (records, manifests, snapshots) always persist the counter + * synchronously at their own commit points, so a crash inside the + * coalescing window can never move the counter backwards relative to + * anything durable. + */ + private noteSingleOpWrite(): void { + if (this.inTransact) return + this.counter++ + this.schedulePersistCounter() + } + + /** Schedule one coalesced counter persist for the current write burst. */ + private schedulePersistCounter(): void { + if (this.counterPersistScheduled) return + this.counterPersistScheduled = true + setImmediate(() => { + this.counterPersistScheduled = false + void this.withMutex(() => this.persistCounterUnlocked()).catch((err) => { + prodLog.warn(`[GenerationStore] generation counter persist failed: ${(err as Error).message}`) + }) + }) + } + + /** + * @description Persist the generation counter now (atomic tmp+rename). + * Called from `flush()`, `close()`, snapshotting, and the commit protocol. + */ + async persistCounterNow(): Promise { + if (!this.opened) return + await this.withMutex(() => this.persistCounterUnlocked()) + } + + private async persistCounterUnlocked(): Promise { + await this.storage.writeRawObject(GENERATION_COUNTER_PATH, { + generation: this.counter, + updatedAt: new Date().toISOString() + }) + } + + // ========================================================================== + // Pins + // ========================================================================== + + /** + * @description Pin a generation (refcounted). Compaction never reclaims a + * record-set a live pin could need (i.e. any generation directory `N` + * with `N > G` for some pinned `G`). + * @param gen - The generation to pin. + */ + pin(gen: number): void { + this.pins.set(gen, (this.pins.get(gen) ?? 0) + 1) + } + + /** + * @description Release one pin on a generation. Safe to call for a + * generation with no live pins (no-op with a warning). + * @param gen - The generation to release. + */ + release(gen: number): void { + const count = this.pins.get(gen) + if (count === undefined) { + prodLog.warn(`[GenerationStore] release(${gen}) called with no live pin at that generation`) + return + } + if (count <= 1) this.pins.delete(gen) + else this.pins.set(gen, count - 1) + } + + /** @returns Total number of live pins across all generations. */ + activePinCount(): number { + let total = 0 + for (const count of this.pins.values()) total += count + return total + } + + /** @returns The smallest pinned generation, or `Infinity` with no pins. */ + private minPinnedGeneration(): number { + let min = Infinity + for (const gen of this.pins.keys()) min = Math.min(min, gen) + return min + } + + // ========================================================================== + // Commit protocol + // ========================================================================== + + /** + * @description Commit one transaction. The caller (Brainy.transact) plans + * the batch up front — resolving every touched id, including generated + * entity/relationship ids and delete cascades — and hands this method the + * touched-id sets plus an `execute` closure that runs the already-built + * operation batch through the TransactionManager. + * + * Protocol (under the commit mutex): + * 1. `ifAtGeneration` CAS check (rejects before anything is staged). + * 2. Reserve generation `N` (counter increment). + * 3. Capture before-images of every touched id and write them, plus the + * `tx.json` delta, under `_generations/N/`; fsync. From this point a + * crash anywhere is recoverable to the exact pre-transaction bytes. + * 4. Run `execute()`. On failure the TransactionManager has already + * rolled back its applied operations; the staging directory is removed + * and the reservation is returned (when no concurrent bump consumed a + * later number), so a failed transaction leaves the generation + * unchanged. + * 5. Persist the counter, then atomically rename the manifest to + * generation `N` — **the rename is the commit point** — and fsync it. + * 6. Append the tx-log line and update in-memory bookkeeping. + * + * @param args.touched - Every id the batch writes or deletes. + * @param args.meta - Transaction metadata for the tx-log. + * @param args.ifAtGeneration - Whole-store CAS expectation. + * @param args.execute - Runs the planned operation batch atomically. + * @returns The committed generation and its commit timestamp. + * @throws GenerationConflictError when the CAS expectation fails. + */ + async commitTransaction(args: { + touched: TouchedIds + meta?: Record + ifAtGeneration?: number + execute: () => Promise + }): Promise<{ generation: number; timestamp: number }> { + return this.withMutex(async () => { + if (args.ifAtGeneration !== undefined && args.ifAtGeneration !== this.counter) { + throw new GenerationConflictError(args.ifAtGeneration, this.counter) + } + + const nouns = [...new Set(args.touched.nouns)] + const verbs = [...new Set(args.touched.verbs)] + const gen = ++this.counter + const dir = `${GENERATIONS_PREFIX}/${gen}` + const timestamp = Date.now() + + // Test-only crash simulation (see setCommitFaultInjector): a throwing + // injector must bypass the abort cleanup below, exactly as a real + // process death would, so recovery-on-open is what restores the store. + let crashSimulated = false + const faultPoint = (phase: CommitFaultPhase): void => { + if (!this.commitFaultInjector) return + try { + this.commitFaultInjector(phase) + } catch (err) { + crashSimulated = true + throw err + } + } + + try { + // -- 3. Before-images + delta (the durable undo log) ------------------ + const stagedPaths: string[] = [] + for (const id of nouns) { + const prev = await this.storage.readNounRaw(id) + const recordPath = `${dir}/prev/${id}.json` + const record: GenerationRecord = { kind: 'noun', metadata: prev.metadata, vector: prev.vector } + await this.storage.writeRawObject(recordPath, record) + stagedPaths.push(recordPath) + } + for (const id of verbs) { + const prev = await this.storage.readVerbRaw(id) + const recordPath = `${dir}/prev/${id}.json` + const record: GenerationRecord = { kind: 'verb', metadata: prev.metadata, vector: prev.vector } + await this.storage.writeRawObject(recordPath, record) + stagedPaths.push(recordPath) + } + const delta: GenerationDelta = { + generation: gen, + timestamp, + ...(args.meta && { meta: args.meta }), + nouns, + verbs + } + const deltaPath = `${dir}/tx.json` + await this.storage.writeRawObject(deltaPath, delta) + stagedPaths.push(deltaPath) + await this.storage.syncRawObjects(stagedPaths) + faultPoint('after-staging') + + // -- 4. Execute the planned batch ------------------------------------- + this.inTransact = true + try { + await args.execute() + } finally { + this.inTransact = false + } + faultPoint('after-execute') + + // -- 5. Counter + manifest rename (COMMIT POINT) ---------------------- + await this.persistCounterUnlocked() + faultPoint('before-manifest-rename') + const manifest: GenerationManifest = { + version: 1, + generation: gen, + committedAt: new Date(timestamp).toISOString(), + horizon: this.horizonGen + } + await this.storage.writeRawObject(MANIFEST_PATH, manifest) + await this.storage.syncRawObjects([MANIFEST_PATH]) + faultPoint('after-manifest-rename') + + // -- 6. Post-commit bookkeeping --------------------------------------- + this.committed = gen + this.committedGens.push(gen) + this.deltaCache.set(gen, { nouns: new Set(nouns), verbs: new Set(verbs), timestamp }) + const logEntry: TxLogEntry = { generation: gen, timestamp, ...(args.meta && { meta: args.meta }) } + await this.storage.appendTxLogLine(JSON.stringify(logEntry)) + + return { generation: gen, timestamp } + } catch (err) { + // Simulated crash (test-only fault injector): skip the abort cleanup + // entirely — a dead process cannot clean up. Recovery on the next + // open() is what must (and does) restore the store. + if (crashSimulated) { + throw err + } + // Failed before the manifest rename: nothing is committed. Remove the + // staging directory; the TransactionManager already restored any + // applied operation byte-identically. + try { + await this.storage.removeRawPrefix(dir) + } catch (cleanupErr) { + prodLog.warn( + `[GenerationStore] failed to remove staging dir ${dir} after aborted transaction: ` + + `${(cleanupErr as Error).message} (recovery will remove it on next open)` + ) + } + // Return the reservation when no concurrent bump consumed a later + // number, so a failed transaction leaves generation() unchanged. + if (this.counter === gen) this.counter = gen - 1 + throw err + } + }) + } + + // ========================================================================== + // Point-in-time resolution + // ========================================================================== + + /** + * @description Whether any transaction committed after generation `gen`. + * O(1) — this is the fast-path check that lets a `Db` pinned at the + * current generation delegate every read to the live brain untouched. + * @param gen - The pinned generation. + */ + hasCommittedAfter(gen: number): boolean { + const last = this.committedGens[this.committedGens.length - 1] + return last !== undefined && last > gen + } + + /** + * @description Resolve the state of an entity or relationship at pinned + * generation `gen`: the before-image stored by the first committed + * generation after `gen` that touched the id — or `{ source: 'current' }` + * when nothing after `gen` touched it (the live storage state *is* the + * state at `gen`). + * + * @param kind - `'noun'` for entities, `'verb'` for relationships. + * @param id - The id to resolve. + * @param gen - The pinned generation. + * @returns `{ source: 'current' }`, `{ source: 'absent' }` (did not exist + * at `gen`), or `{ source: 'record', metadata, vector }` with the raw + * stored objects as of `gen`. + */ + async resolveAt( + kind: 'noun' | 'verb', + id: string, + gen: number + ): Promise< + | { source: 'current' } + | { source: 'absent' } + | { source: 'record'; metadata: any; vector: any | null } + > { + for (const candidate of this.committedGens) { + if (candidate <= gen) continue + const delta = await this.getDelta(candidate) + const touched = kind === 'noun' ? delta.nouns : delta.verbs + if (!touched.has(id)) continue + const record = (await this.storage.readRawObject( + `${GENERATIONS_PREFIX}/${candidate}/prev/${id}.json` + )) as GenerationRecord | null + if (record === null) { + // The record-set exists (candidate is committed) but the id's + // before-image is missing — only possible through external tampering. + throw new Error( + `Generation record missing: ${GENERATIONS_PREFIX}/${candidate}/prev/${id}.json ` + + `(store corrupted or records removed outside compactHistory())` + ) + } + if (record.metadata === null && record.vector === null) { + return { source: 'absent' } + } + return { source: 'record', metadata: record.metadata, vector: record.vector } + } + return { source: 'current' } + } + + /** + * @description The ids touched by committed transactions in the interval + * `(fromGen, toGen]`. Used by `db.since()` and by historical `find()` to + * build its overlay of changed entities. + * @param fromGen - Exclusive lower bound. + * @param toGen - Inclusive upper bound. + */ + async changedBetween(fromGen: number, toGen: number): Promise { + const nouns = new Set() + const verbs = new Set() + for (const gen of this.committedGens) { + if (gen <= fromGen || gen > toGen) continue + const delta = await this.getDelta(gen) + for (const id of delta.nouns) nouns.add(id) + for (const id of delta.verbs) verbs.add(id) + } + return { + nouns: [...nouns].sort(), + verbs: [...verbs].sort(), + fromGeneration: fromGen, + toGeneration: toGen + } + } + + /** + * @description Assert that generation `gen` is reachable (not compacted + * away) and within the committed range, then return it. Used by `asOf()`. + * Reachability boundary: reclaiming record-set `N` removes the + * before-images that reads at generations BELOW `N` depend on, so + * generations `< horizon` are unreachable while the horizon itself remains + * servable from the record-sets above it. + * @param gen - The requested generation. + * @throws GenerationCompactedError when `gen` is below the horizon. + * @throws RangeError when `gen` is negative or beyond the current counter. + */ + assertReachable(gen: number): number { + if (!Number.isInteger(gen) || gen < 0) { + throw new RangeError(`asOf(): generation must be a non-negative integer (got ${gen})`) + } + if (gen > this.counter) { + throw new RangeError( + `asOf(): generation ${gen} is in the future (store is at generation ${this.counter})` + ) + } + if (gen < this.horizonGen) { + throw new GenerationCompactedError(gen, this.horizonGen) + } + return gen + } + + /** + * @description Resolve a wall-clock timestamp to the generation that was + * committed at-or-before it, via the tx-log. Resolution granularity is + * transact commits — single-operation writes between commits are not + * individually addressable (documented 8.0 history granularity). + * @param timestampMs - Milliseconds since epoch. + * @returns The resolved generation (`0` when `timestampMs` predates the + * first commit) and the matched tx-log entry when one exists. + */ + async resolveTimestamp( + timestampMs: number + ): Promise<{ generation: number; entry: TxLogEntry | null }> { + const lines = await this.storage.readTxLogLines() + let best: TxLogEntry | null = null + for (const line of lines) { + let entry: TxLogEntry + try { + entry = JSON.parse(line) as TxLogEntry + } catch { + continue // tolerate a torn trailing line from a crashed append + } + if (entry.timestamp <= timestampMs && entry.generation <= this.committed) { + if (best === null || entry.generation > best.generation) best = entry + } + } + return { generation: best?.generation ?? 0, entry: best } + } + + /** + * @description The commit timestamp of the newest committed generation at + * or below `gen`, when its record-set is still resident. Used to populate + * `Db.timestamp` for `asOf()` values. + * @param gen - The pinned generation. + */ + async commitTimestampAtOrBefore(gen: number): Promise { + for (let i = this.committedGens.length - 1; i >= 0; i--) { + const candidate = this.committedGens[i] + if (candidate > gen) continue + const delta = await this.getDelta(candidate) + return delta.timestamp + } + return null + } + + private async getDelta( + gen: number + ): Promise<{ nouns: Set; verbs: Set; timestamp: number }> { + const cached = this.deltaCache.get(gen) + if (cached) return cached + const delta = (await this.storage.readRawObject( + `${GENERATIONS_PREFIX}/${gen}/tx.json` + )) as GenerationDelta | null + if (delta === null) { + throw new Error( + `Generation delta missing: ${GENERATIONS_PREFIX}/${gen}/tx.json ` + + `(store corrupted or records removed outside compactHistory())` + ) + } + const entry = { + nouns: new Set(delta.nouns), + verbs: new Set(delta.verbs), + timestamp: delta.timestamp + } + this.deltaCache.set(gen, entry) + return entry + } + + // ========================================================================== + // Compaction + // ========================================================================== + + /** + * @description Reclaim generation record-sets past the retention policy. + * A generation directory `N` may be removed only when `N` is at or below + * **every** live pin (deleting `N` can only break readers pinned *below* + * `N`, because resolution reads before-images from generations strictly + * greater than the pin). With both retention options supplied, a + * generation must satisfy both to be eligible. + * + * @param options - Retention policy (see {@link CompactHistoryOptions}). + * @returns Count of removed record-sets and the new horizon. + */ + async compact(options?: CompactHistoryOptions): Promise { + return this.withMutex(async () => { + const minPinned = this.minPinnedGeneration() + const retainGenerations = options?.retainGenerations ?? 0 + const countWatermark = this.committed - retainGenerations + const timeCutoff = + options?.retainMs !== undefined ? Date.now() - options.retainMs : Infinity + + const removed: number[] = [] + for (const gen of [...this.committedGens]) { + if (gen > countWatermark) continue + if (gen > minPinned) continue + if (timeCutoff !== Infinity) { + const delta = await this.getDelta(gen) + if (delta.timestamp >= timeCutoff) continue + } + await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${gen}`) + this.deltaCache.delete(gen) + removed.push(gen) + } + + if (removed.length > 0) { + const removedSet = new Set(removed) + this.committedGens = this.committedGens.filter((gen) => !removedSet.has(gen)) + this.horizonGen = Math.max(this.horizonGen, ...removed) + const manifest: GenerationManifest = { + version: 1, + generation: this.committed, + committedAt: new Date().toISOString(), + horizon: this.horizonGen + } + await this.storage.writeRawObject(MANIFEST_PATH, manifest) + await this.storage.syncRawObjects([MANIFEST_PATH]) + } + + return { removedGenerations: removed.length, horizon: this.horizonGen } + }) + } + + // ========================================================================== + // Restore support + // ========================================================================== + + /** + * @description Re-read all persisted state after `brain.restore()` + * replaced the store's contents from a snapshot, enforcing counter + * monotonicity: the counter never moves below `floorGeneration` (the + * pre-restore counter), so generation numbers observed before a restore + * are never reissued. + * @param floorGeneration - The counter value before the restore. + */ + async reopenAfterRestore(floorGeneration: number): Promise { + await this.withMutex(async () => { + this.deltaCache.clear() + this.opened = false + // open() re-reads counter/manifest and re-registers the bump hook. + await this.open() + if (this.counter < floorGeneration) { + this.counter = floorGeneration + await this.persistCounterUnlocked() + } + }) + } + + // ========================================================================== + // Internals + // ========================================================================== + + /** + * Restore the before-images of an uncommitted generation to the canonical + * entity paths, then remove its record directory. Restores are idempotent + * (writing identical bytes / deleting already-absent files), so recovery + * is safe at every crash point of the commit protocol. + */ + private async rollBackUncommittedGeneration(gen: number): Promise { + const dir = `${GENERATIONS_PREFIX}/${gen}` + const prevPaths = await this.storage.listRawObjects(`${dir}/prev`) + for (const recordPath of prevPaths) { + const id = recordIdFromPath(recordPath) + if (id === null) continue + const record = (await this.storage.readRawObject(recordPath)) as GenerationRecord | null + if (record === null) continue + const image = { metadata: record.metadata, vector: record.vector } + if (record.kind === 'verb') { + await this.storage.writeVerbRaw(id, image) + } else { + await this.storage.writeNounRaw(id, image) + } + } + await this.storage.removeRawPrefix(dir) + prodLog.warn( + `[GenerationStore] rolled back uncommitted generation ${gen} ` + + `(${prevPaths.length} record(s) restored)` + ) + } + + /** + * @description Run a snapshot section serialized against transact commits, + * compaction, and counter persistence (the store's single commit mutex), + * with the generation counter durably persisted first. Used by + * `db.persist()`: the lock guarantees a snapshot can never interleave with + * a commit's record-write/manifest-rename sequence, and the up-front + * counter persist guarantees the snapshot's `_system/generation.json` + * reflects every single-operation bump (whose persistence is otherwise + * coalesced and may still be scheduled). + * @param fn - The exclusive snapshot section. + * @returns `fn`'s result. + */ + snapshotWith(fn: () => Promise): Promise { + return this.withMutex(async () => { + await this.persistCounterUnlocked() + return fn() + }) + } + + /** Run `fn` exclusively, chained behind every prior exclusive section. */ + private withMutex(fn: () => Promise): Promise { + const run = this.mutexTail.then(fn, fn) + // Keep the chain alive regardless of fn's outcome. + this.mutexTail = run.catch(() => {}) + return run + } +} + +/** + * @description Extract the generation number from a record path of the form + * `_generations//...`. Returns `null` for paths that don't match. + * @param path - A storage-root-relative object path. + */ +function parseGenerationFromPath(path: string): number | null { + const match = /^_generations[/\\](\d+)[/\\]/.exec(path) + if (!match) return null + const gen = Number(match[1]) + return Number.isSafeInteger(gen) ? gen : null +} + +/** + * @description Extract the record id (file basename without `.json`) from a + * `prev/` before-image record path. Returns `null` for non-record paths. + * @param path - A storage-root-relative object path. + */ +function recordIdFromPath(path: string): string | null { + const match = /[/\\]prev[/\\]([^/\\]+)\.json$/.exec(path) + return match ? match[1] : null +} diff --git a/src/db/types.ts b/src/db/types.ts new file mode 100644 index 00000000..26d19a27 --- /dev/null +++ b/src/db/types.ts @@ -0,0 +1,321 @@ +/** + * @module db/types + * @description Shared types for the 8.0 generational-MVCC Db API: the + * declarative transaction-operation union consumed by `brain.transact()`, + * the options bags for `transact()` / `compactHistory()`, the persisted + * record shapes of the generational record layer, and the narrow storage + * contract (`GenerationStorage`) the record layer needs from an adapter. + * + * Persisted layout (all paths relative to the storage root): + * + * ``` + * _system/generation.json — { generation, updatedAt } (atomic tmp+rename) + * _system/manifest.json — { version, generation, committedAt, horizon } + * (atomic tmp+rename — the rename IS the commit point) + * _system/tx-log.jsonl — one JSON line per committed transact: + * { generation, timestamp, meta? } (append-only) + * _generations//tx.json — the generation-N delta: touched noun/verb ids + meta + * _generations//prev/.json — immutable before-image of as of commit N + * ``` + * + * Design note: the manifest holds the committed watermark + compaction + * horizon rather than a full `id → latest record generation` map. The id + * mapping is distributed across the per-generation `tx.json` deltas, which + * makes each commit O(ids touched) instead of O(all ids) — see + * `docs/ADR-001-generational-mvcc.md` for the full justification. + */ + +import type { AddParams, UpdateParams, RelateParams } from '../types/brainy.types.js' + +// ============================================================================ +// Transaction operations (brain.transact input) +// ============================================================================ + +/** + * @description Create an entity. Carries the same parameters as + * `brain.add()`; supply `id` to choose the entity id (otherwise a UUID v4 is + * generated and returned in the operation's planning, exactly as `add()` + * does). + */ +export interface TxAddOperation extends AddParams { + /** Discriminator. */ + op: 'add' +} + +/** + * @description Update an entity. Carries the same parameters as + * `brain.update()` including per-entity CAS via `ifRev` — an `ifRev` + * conflict rejects the whole transaction before anything is applied. + */ +export interface TxUpdateOperation extends UpdateParams { + /** Discriminator. */ + op: 'update' +} + +/** + * @description Delete an entity (and, exactly like `brain.delete()`, every + * relationship where it is source or target — the cascade is part of the + * same atomic batch). + */ +export interface TxRemoveOperation { + /** Discriminator. */ + op: 'remove' + /** Id of the entity to delete. */ + id: string +} + +/** + * @description Create a relationship. Carries the same parameters as + * `brain.relate()` (including `bidirectional`). Duplicate relationships + * (same from/to/type) are deduplicated exactly as `relate()` does — the + * operation becomes a no-op that resolves to the existing relationship id. + */ +export interface TxRelateOperation extends RelateParams { + /** Discriminator. */ + op: 'relate' +} + +/** + * @description Delete a relationship by id (mirror of `brain.unrelate()`). + */ +export interface TxUnrelateOperation { + /** Discriminator. */ + op: 'unrelate' + /** Id of the relationship to delete. */ + id: string +} + +/** + * @description One declarative operation inside a `brain.transact()` batch. + * The batch executes atomically: either every operation applies and the + * store advances exactly one generation, or none apply and the store is + * byte-identical to its pre-transaction state. + */ +export type TxOperation = + | TxAddOperation + | TxUpdateOperation + | TxRemoveOperation + | TxRelateOperation + | TxUnrelateOperation + +/** + * @description Options for `brain.transact()`. + */ +export interface TransactOptions { + /** + * Transaction metadata, reified Datomic-style: recorded in + * `_system/tx-log.jsonl` alongside the generation number and commit + * timestamp. Use it for audit fields (author, reason, request id) — + * it replaces commit messages from the pre-8.0 versioning API. + */ + meta?: Record + /** + * Whole-store compare-and-swap. When provided, the transaction commits + * only if the store's current generation equals this value; otherwise it + * throws `GenerationConflictError` (with `expected`/`actual`) before any + * record is staged. + */ + ifAtGeneration?: number +} + +/** + * @description Per-operation results of a committed transaction, in input + * order. `add` resolves to the created entity id, `relate` to the created + * (or deduplicated existing) relationship id; `update`/`remove`/`unrelate` + * resolve to the id they acted on. + */ +export interface TransactReceipt { + /** The generation this transaction committed as. */ + generation: number + /** Commit timestamp (ms since epoch) recorded in the tx-log. */ + timestamp: number + /** Resolved id per input operation, in input order. */ + ids: string[] +} + +// ============================================================================ +// Compaction +// ============================================================================ + +/** + * @description Options for `brain.compactHistory()`. When both options are + * supplied a generation must satisfy *both* retention rules to be reclaimed + * (the stricter retention wins). With no options, every generation not + * protected by a live pin is eligible. + */ +export interface CompactHistoryOptions { + /** + * Keep at least this many of the most recent committed generations + * (`0` = keep none beyond what live pins protect). + */ + retainGenerations?: number + /** + * Keep every generation committed within the last `retainMs` milliseconds. + */ + retainMs?: number +} + +/** + * @description Result of `brain.compactHistory()`. + */ +export interface CompactHistoryResult { + /** Number of generation record-sets reclaimed by this call. */ + removedGenerations: number + /** + * The new compaction horizon — the highest generation whose record-set has + * been reclaimed. Generations BELOW the horizon are no longer reachable via + * `asOf()` (their reads would need the reclaimed before-images); the + * horizon itself stays reachable, resolved from the record-sets above it. + */ + horizon: number +} + +// ============================================================================ +// Db surfaces +// ============================================================================ + +/** + * @description Result of `db.since(priorDb)` — the ids touched by committed + * transactions in the generation interval `(priorDb.generation, + * db.generation]`. Granularity note: single-operation writes performed + * outside `transact()` bump the generation counter but do not produce + * generation records, so they are not reflected here (documented 8.0 + * history granularity — see `docs/ADR-001-generational-mvcc.md`). + */ +export interface ChangedIds { + /** Entity (noun) ids touched in the interval, sorted. */ + nouns: string[] + /** Relationship (verb) ids touched in the interval, sorted. */ + verbs: string[] + /** Exclusive lower bound of the interval. */ + fromGeneration: number + /** Inclusive upper bound of the interval. */ + toGeneration: number +} + +// ============================================================================ +// Persisted record shapes +// ============================================================================ + +/** + * @description An immutable before-image of one entity or relationship, + * persisted under `_generations//prev/.json` — the state of the id + * immediately before commit `N` touched it. `metadata`/`vector` hold the + * *raw stored objects* (exact bytes that live at the entity's canonical + * storage paths); `null` means the corresponding file did not exist (for + * before-images of created ids, both are `null`). Before-images are both + * the crash-recovery undo log and the point-in-time read source: the state + * of id X at pinned generation G is the before-image of the first committed + * generation after G that touched X. + */ +export interface GenerationRecord { + /** Whether this record images an entity (noun) or a relationship (verb). */ + kind: 'noun' | 'verb' + /** Raw stored metadata object, or `null` if the metadata file was absent. */ + metadata: unknown | null + /** Raw stored vector object, or `null` if the vector file was absent. */ + vector: unknown | null +} + +/** + * @description The per-generation delta persisted at + * `_generations//tx.json` — written and fsynced *before* any operation of + * the transaction executes, so crash recovery always knows exactly which ids + * to restore from before-images. + */ +export interface GenerationDelta { + /** The generation this delta belongs to. */ + generation: number + /** Commit timestamp (ms since epoch). */ + timestamp: number + /** Transaction metadata (mirrored into the tx-log on commit). */ + meta?: Record + /** Entity ids touched by this generation. */ + nouns: string[] + /** Relationship ids touched by this generation. */ + verbs: string[] +} + +/** + * @description Shape of `_system/manifest.json`. Replaced via atomic + * tmp+rename on every commit — the rename is the commit point: a generation + * directory is committed if and only if its number is ≤ `generation`. + */ +export interface GenerationManifest { + /** Schema version of the manifest file. */ + version: 1 + /** Committed-transaction watermark. */ + generation: number + /** ISO timestamp of the most recent commit (or compaction). */ + committedAt: string + /** Compaction horizon — record-sets for generations ≤ this are reclaimed. */ + horizon: number +} + +/** + * @description One line of `_system/tx-log.jsonl`. + */ +export interface TxLogEntry { + /** The committed generation. */ + generation: number + /** Commit timestamp (ms since epoch). */ + timestamp: number + /** Transaction metadata, when supplied to `transact()`. */ + meta?: Record +} + +// ============================================================================ +// Storage contract +// ============================================================================ + +/** + * @description The narrow storage contract the generational record layer + * needs. `BaseStorage` implements it structurally (both shipped adapters — + * filesystem and memory — inherit the implementation), so any + * `StorageAdapter` produced by the 8.0 storage factory satisfies it. + * + * Raw-object methods bypass branch scoping and operate on storage-root- + * relative paths (the record layer owns `_system/` + `_generations/`); + * entity-raw methods read/write the canonical entity files (branch-scoped, + * write-cache coherent) so before/after images capture exactly the bytes + * the live read paths see. + */ +export interface GenerationStorage { + /** Read a raw object at a storage-root-relative path (`null` if absent). */ + readRawObject(path: string): Promise + /** Write a raw object at a storage-root-relative path (atomic tmp+rename on disk). */ + writeRawObject(path: string, data: any): Promise + /** Delete a raw object (no-op if absent). */ + deleteRawObject(path: string): Promise + /** List raw object paths under a prefix (normalized, `.gz`-stripped). */ + listRawObjects(prefix: string): Promise + /** Remove every object under a prefix (and the directory itself on disk). */ + removeRawPrefix(prefix: string): Promise + /** Durability barrier: fsync the given object paths (no-op in memory). */ + syncRawObjects(paths: string[]): Promise + + /** Read an entity's raw stored metadata+vector objects. */ + readNounRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> + /** Restore an entity's raw stored objects (`null` part ⇒ delete that file). */ + writeNounRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise + /** Read a relationship's raw stored metadata+vector objects. */ + readVerbRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> + /** Restore a relationship's raw stored objects (`null` part ⇒ delete that file). */ + writeVerbRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise + + /** Append one line to `_system/tx-log.jsonl`. */ + appendTxLogLine(line: string): Promise + /** Read all lines of `_system/tx-log.jsonl` (empty array if absent). */ + readTxLogLines(): Promise + + /** + * Register the generation-bump hook invoked on every entity-visible + * single-operation write (see `BaseStorage.setGenerationBumpHook`). + */ + setGenerationBumpHook(hook: (() => void) | undefined): void + + /** Snapshot the entire store to a directory (hard-link farm on disk). */ + snapshotToDirectory(targetPath: string): Promise + /** Replace the entire store's contents from a snapshot directory. */ + restoreFromDirectory(sourcePath: string): Promise +} diff --git a/src/db/whereMatcher.ts b/src/db/whereMatcher.ts new file mode 100644 index 00000000..c4b6ed4d --- /dev/null +++ b/src/db/whereMatcher.ts @@ -0,0 +1,292 @@ +/** + * @module db/whereMatcher + * @description In-memory evaluation of `find()` metadata filters against a + * single resolved entity — the overlay half of historical and speculative + * reads on a `Db` value. + * + * A `Db` pinned at a past generation answers metadata-level `find()` by + * combining the live index's results (for entities untouched since the pin) + * with per-entity evaluation of the same filter against generation records + * (for entities that DID change). The same evaluator backs `db.with()` + * speculative overlays. The semantics here deliberately mirror the index + * evaluator in `src/utils/metadataIndex.ts` (`getIdsForFilter`) — operator + * aliases, array-membership equality, `exists`/`missing`, and the + * `allOf`/`anyOf`/`not` logical forms — so an entity matches in-memory if and + * only if it would have matched through the index. + * + * **Honesty contract:** an operator this module does not recognize throws + * {@link UnsupportedWhereOperatorError} instead of guessing — a historical + * read must never silently return wrong results. + */ + +import type { Entity, FindParams } from '../types/brainy.types.js' + +/** + * @description Thrown when a `where` clause uses an operator this in-memory + * evaluator does not implement. Callers (historical `find()` on a `Db`) + * convert this into the documented historical-query error rather than + * returning silently-wrong results. + */ +export class UnsupportedWhereOperatorError extends Error { + /** The unrecognized operator name. */ + public readonly operator: string + + /** + * @param operator - The operator name that could not be evaluated. + */ + constructor(operator: string) { + super( + `The where-operator '${operator}' is not supported for historical/speculative ` + + `in-memory evaluation. Supported: eq/equals/is, ne/notEquals/isNot, in/oneOf, ` + + `gt/greaterThan, gte/greaterThanOrEqual/greaterEqual, lt/lessThan, ` + + `lte/lessThanOrEqual/lessEqual, between, contains, exists, missing, ` + + `plus allOf/anyOf/not.` + ) + this.name = 'UnsupportedWhereOperatorError' + this.operator = operator + } +} + +/** + * @description Resolve a filter field name to its value on an entity, + * mirroring how `extractIndexableFields` lays entities out for the metadata + * index: standard fields live at the top level (with the `noun` ↔ `type` + * alias), everything else is a custom field inside the `metadata` bag. + * Dotted paths (`metadata.priority`, `address.city`) traverse nested objects. + * + * @param entity - The resolved entity to read from. + * @param field - The filter field name. + * @returns The field's value, or `undefined` when absent. + */ +export function resolveEntityField(entity: Entity, field: string): unknown { + switch (field) { + case 'noun': + case 'type': + return entity.type + case 'subtype': + return entity.subtype + case 'id': + return entity.id + case 'createdAt': + return entity.createdAt + case 'updatedAt': + return entity.updatedAt + case 'service': + return entity.service + case 'createdBy': + return entity.createdBy + case 'confidence': + return entity.confidence + case 'weight': + return entity.weight + case '_rev': + return entity._rev + case 'data': + return entity.data + } + + if (field.includes('.')) { + // Dotted path: resolve against the whole entity first (`metadata.x`), + // then against the metadata bag (`address.city` on nested metadata). + const fromEntity = resolvePath(entity as unknown as Record, field) + if (fromEntity !== undefined) return fromEntity + return resolvePath((entity.metadata ?? {}) as Record, field) + } + + return ((entity.metadata ?? {}) as Record)[field] +} + +/** Walk a dotted path through nested plain objects. */ +function resolvePath(obj: Record, path: string): unknown { + let current: unknown = obj + for (const segment of path.split('.')) { + if (current === null || typeof current !== 'object') return undefined + current = (current as Record)[segment] + } + return current +} + +/** + * @description Index-equality semantics: scalar strict equality, with + * array-valued fields matching by membership (the index stores one posting + * per array element, so `eq` on an array field means "contains"). + */ +function eqMatches(value: unknown, operand: unknown): boolean { + if (Array.isArray(value)) { + return value.some((element) => element === operand) + } + return value === operand +} + +/** Ordered comparison for range operators (numbers, or both-strings lexicographic). */ +function compare(value: unknown, operand: unknown): number | null { + if (typeof value === 'number' && typeof operand === 'number') { + return value - operand + } + if (typeof value === 'string' && typeof operand === 'string') { + return value < operand ? -1 : value > operand ? 1 : 0 + } + return null +} + +/** + * @description Evaluate one field condition (shorthand value or operator + * object) against a resolved field value, mirroring the operator table in + * `metadataIndex.getIdsForFilter`. + * + * @param value - The entity's field value (possibly `undefined`). + * @param condition - The filter condition for this field. + * @returns Whether the value satisfies the condition. + * @throws UnsupportedWhereOperatorError for unrecognized operators. + */ +function fieldConditionMatches(value: unknown, condition: unknown): boolean { + if (condition === null || typeof condition !== 'object' || Array.isArray(condition)) { + // Shorthand for 'eq'. + return eqMatches(value, condition) + } + + for (const [op, operand] of Object.entries(condition as Record)) { + let matches: boolean + switch (op) { + case 'is': + case 'equals': + case 'eq': + matches = eqMatches(value, operand) + break + case 'isNot': + case 'notEquals': + case 'ne': + matches = !eqMatches(value, operand) + break + case 'oneOf': + case 'in': + matches = Array.isArray(operand) && operand.some((candidate) => eqMatches(value, candidate)) + break + case 'greaterThan': + case 'gt': { + const cmp = compare(value, operand) + matches = cmp !== null && cmp > 0 + break + } + case 'greaterEqual': + case 'greaterThanOrEqual': + case 'gte': { + const cmp = compare(value, operand) + matches = cmp !== null && cmp >= 0 + break + } + case 'lessThan': + case 'lt': { + const cmp = compare(value, operand) + matches = cmp !== null && cmp < 0 + break + } + case 'lessEqual': + case 'lessThanOrEqual': + case 'lte': { + const cmp = compare(value, operand) + matches = cmp !== null && cmp <= 0 + break + } + case 'between': { + if (!Array.isArray(operand) || operand.length !== 2) { + matches = false + break + } + const lower = compare(value, operand[0]) + const upper = compare(value, operand[1]) + matches = lower !== null && upper !== null && lower >= 0 && upper <= 0 + break + } + case 'contains': + // Index semantics: array fields post one entry per element, so + // 'contains' is the same lookup as 'eq' (membership on arrays). + matches = eqMatches(value, operand) + break + case 'exists': + matches = operand ? value !== undefined : value === undefined + break + case 'missing': + matches = operand ? value === undefined : value !== undefined + break + default: + throw new UnsupportedWhereOperatorError(op) + } + if (!matches) return false // Multiple operators on one field AND together. + } + return true +} + +/** + * @description Evaluate a full `where` filter (field conditions plus + * `allOf`/`anyOf`/`not` logical composition) against one entity. + * + * @param entity - The resolved entity (historical record or speculative overlay). + * @param where - The `where` clause from `FindParams`. + * @returns Whether the entity satisfies every clause. + * @throws UnsupportedWhereOperatorError for unrecognized operators. + */ +export function whereMatches(entity: Entity, where: Record): boolean { + for (const [field, condition] of Object.entries(where)) { + if (field === 'allOf') { + if (!Array.isArray(condition)) return false + if (!condition.every((sub) => whereMatches(entity, sub as Record))) return false + continue + } + if (field === 'anyOf') { + if (!Array.isArray(condition)) return false + if (!condition.some((sub) => whereMatches(entity, sub as Record))) return false + continue + } + if (field === 'not') { + if (whereMatches(entity, condition as Record)) return false + continue + } + if (!fieldConditionMatches(resolveEntityField(entity, field), condition)) return false + } + return true +} + +/** + * @description Evaluate the metadata-level portions of a `find()` query + * (`type`, `subtype`, `where`, `service`, `excludeVFS`) against one resolved + * entity. Used by historical and speculative `find()` to decide whether a + * changed/overlaid entity belongs in the result set. The caller guarantees + * the query carries no index-only dimensions (semantic `query`/`vector`, + * `connected` traversal, …) — those are rejected with the documented + * historical-query error before evaluation starts. + * + * @param entity - The resolved entity. + * @param params - The metadata-level find parameters. + * @returns Whether the entity matches every requested filter. + * @throws UnsupportedWhereOperatorError for unrecognized `where` operators. + */ +export function entityMatchesFind(entity: Entity, params: FindParams): boolean { + if (params.type !== undefined) { + const types = Array.isArray(params.type) ? params.type : [params.type] + if (!types.includes(entity.type)) return false + } + + if (params.subtype !== undefined) { + const subtypes = Array.isArray(params.subtype) ? params.subtype : [params.subtype] + if (entity.subtype === undefined || !subtypes.includes(entity.subtype)) return false + } + + if (params.service !== undefined && entity.service !== params.service) { + return false + } + + if (params.excludeVFS === true) { + // Mirror of find()'s exclusion filter (`vfsType: { exists: false }`, + // `isVFSEntity: { ne: true }`) and Brainy's VFS-marker helper. + const metadata = (entity.metadata ?? {}) as Record + if (metadata.vfsType !== undefined) return false + if (metadata.isVFSEntity === true || metadata.isVFS === true) return false + } + + if (params.where !== undefined) { + if (!whereMatches(entity, params.where as Record)) return false + } + + return true +} diff --git a/src/index.ts b/src/index.ts index 7546786a..0416f283 100644 --- a/src/index.ts +++ b/src/index.ts @@ -128,6 +128,32 @@ export type { Migration, MigrationState, MigrationPreview, MigrationResult, Migr // Export optimistic-concurrency types (7.31.0) export { RevisionConflictError } from './transaction/RevisionConflictError.js' +// ============= 8.0 Db API — generational MVCC ============= +// Immutable database values: brain.now() / brain.transact() / brain.asOf() / +// db.with() / db.persist() / Brainy.load(). See src/db/ for the record layer. +export { Db } from './db/db.js' +export { + GenerationConflictError, + NotYetSupportedAtHistoricalGenerationError, + GenerationCompactedError +} from './db/errors.js' +export type { + TxOperation, + TxAddOperation, + TxUpdateOperation, + TxRemoveOperation, + TxRelateOperation, + TxUnrelateOperation, + TransactOptions, + TransactReceipt, + CompactHistoryOptions, + CompactHistoryResult, + ChangedIds +} from './db/types.js' +// Optional provider capability for generation-aware native indexes +export { isVersionedIndexProvider } from './plugin.js' +export type { VersionedIndexProvider } from './plugin.js' + // Export embedding functionality import { UniversalSentenceEncoder, diff --git a/src/plugin.ts b/src/plugin.ts index ff5f481a..f2ddf7e3 100644 --- a/src/plugin.ts +++ b/src/plugin.ts @@ -259,6 +259,108 @@ export interface GraphIndexProvider { getAllRelationshipCounts(): Map } +/** + * Optional capability interface for index providers that maintain versioned + * (generation-aware) internal state — the provider-side half of Brainy 8.0's + * generational MVCC contract. Implemented by native providers whose storage + * engines keep immutable versions (e.g. LSM snapshots); Brainy's own JS + * indexes do not implement it (they are rebuilt from the storage records, + * which carry the versioning). + * + * **Detection.** Brainy feature-detects this interface on every registered + * index provider (`'vector'`, `'metadataIndex'`, `'graphIndex'`): when a + * provider exposes `pin`/`release` functions, Brainy calls them in lockstep + * with `Db` lifecycle — `pin(g)` when a `Db` value pins generation `g` + * (`brain.now()`, `brain.transact()`, `brain.asOf()`), `release(g)` when that + * `Db` is released (explicitly or via the GC backstop). Pins are refcounted + * on the Brainy side; a provider may receive multiple `pin(g)` calls for the + * same generation and will receive exactly one matching `release(g)` per pin. + * + * **Consistency model (locked cross-team design).** Index providers are + * *post-commit appliers*: the storage-record commit (atomic manifest rename) + * is the source of truth, and provider index state is derived, applied after + * the commit point. On open, a provider compares its own persisted + * `generation()` against the store's committed generation and replays the + * gap from the storage records (or requests a rebuild) — there are no + * provider rollback hooks, because an uncommitted transaction is repaired at + * the storage layer before any index is opened. The explicit pin/release + * lifetime OVERRIDES any time-based snapshot retention the provider has + * (e.g. an LSM snapshot TTL): a pinned generation must stay readable until + * released, regardless of age. + * + * **Speculative reads.** `db.with(txData)` overlays are Brainy-side only — + * providers are never asked to read uncommitted/speculative state; they + * always serve committed generations. + */ +export interface VersionedIndexProvider { + /** + * @description The newest generation this provider's persisted index state + * reflects. Brainy compares it against the storage layer's committed + * generation on open to detect a replay gap (index behind storage after a + * crash between commit and index apply). + * @returns The provider's current generation as a `bigint` (u64 at the + * native boundary; Brainy's generation counter is a safe integer today). + */ + generation(): bigint + + /** + * @description True ⇒ the provider can serve consistent reads at + * `generation` (segments retained). Combined with pin: pinning a VISIBLE + * generation guarantees it stays servable until release. Pinning an + * invisible generation is permitted (refcount-only) — Brainy serves that + * generation from canonical storage instead. + * + * Brainy consults this at pin time (the read-routing rule: a `Db` at + * generation `g` uses provider-accelerated reads when + * `isGenerationVisible(g)` was true at pin time; otherwise canonical + * generation records). + * @param generation - The generation a `Db` value is about to pin. + */ + isGenerationVisible(generation: bigint): boolean + + /** + * @description Pin a generation: the provider must keep index state for + * `generation` readable until the matching {@link VersionedIndexProvider.release} + * call, overriding any time-based snapshot retention. Called once per + * Brainy-side pin (refcounted upstream — expect balanced pin/release pairs). + * @param generation - The generation a live `Db` value just pinned. + */ + pin(generation: bigint): void + + /** + * @description Release one pin on `generation`. After the last release the + * provider may reclaim resources for that generation at its discretion. + * @param generation - The generation being released (matches a prior + * {@link VersionedIndexProvider.pin} call). + */ + release(generation: bigint): void +} + +/** + * @description Feature-detection guard for {@link VersionedIndexProvider}. + * Brainy applies it to every registered index provider (vector, metadata, + * graph) when a `Db` value pins or releases a generation: providers exposing + * all four capability methods receive lockstep `pin`/`release` calls; + * everything else (including Brainy's own JS indexes) is skipped. + * + * @param candidate - A registered index provider instance. + * @returns Whether the candidate implements the versioned capability. + */ +export function isVersionedIndexProvider( + candidate: unknown +): candidate is VersionedIndexProvider { + if (candidate === null || typeof candidate !== 'object') { + return false + } + const c = candidate as Record + return ( + typeof c.generation === 'function' && + typeof c.isGenerationVisible === 'function' && + typeof c.pin === 'function' && + typeof c.release === 'function' + ) +} + /** * The object returned by the `'vector'` provider factory — Brainy's vector * index contract. Implementations include Brainy's own JS HNSW index and any diff --git a/src/storage/adapters/fileSystemStorage.ts b/src/storage/adapters/fileSystemStorage.ts index a6e87e44..535c9e77 100644 --- a/src/storage/adapters/fileSystemStorage.ts +++ b/src/storage/adapters/fileSystemStorage.ts @@ -954,6 +954,254 @@ export class FileSystemStorage extends BaseStorage { } } + // =========================================================================== + // Generational record layer (8.0 MVCC) — durability + snapshot primitives + // =========================================================================== + + /** + * Storage-root-relative paths that are mutated **in place** (appended to) + * rather than replaced via atomic tmp+rename. `snapshotToDirectory()` must + * byte-copy these instead of hard-linking them: a hard link shares the + * inode, so a post-snapshot append to the live file would silently mutate + * the snapshot. Every other persisted file in this adapter is written via + * tmp+rename (objects, blobs, counts, locks are excluded entirely), which + * makes hard links safe — a rewrite swaps in a new inode and the snapshot + * keeps the old one. + */ + private static readonly SNAPSHOT_BYTE_COPY_PATHS = new Set([ + `${SYSTEM_DIR}/tx-log.jsonl` + ]) + + /** + * Top-level directories excluded from snapshots: process-local lock state + * (writer lock, flush-request RPC files) must never travel with the data. + */ + private static readonly SNAPSHOT_EXCLUDED_TOP_DIRS = new Set(['locks']) + + /** + * Remove every object under a storage-root-relative prefix — one recursive + * directory removal instead of the base class's list+delete loop. + * + * @param prefix - Storage-root-relative directory prefix to remove. + */ + public async removeRawPrefix(prefix: string): Promise { + await this.ensureInitialized() + await fs.promises.rm(path.join(this.rootDir, prefix), { recursive: true, force: true }) + } + + /** + * Durability barrier: `fsync` each listed object file (resolving the + * compressed `.gz` variant when present) and then the set of parent + * directories, so both the file contents and the rename directory entries + * are durable before the commit protocol proceeds. Paths whose file no + * longer exists are skipped (a later step may have replaced them). + * + * @param paths - Storage-root-relative object paths previously written. + */ + public async syncRawObjects(paths: string[]): Promise { + await this.ensureInitialized() + const parentDirs = new Set() + + for (const objectPath of paths) { + const fullPath = path.join(this.rootDir, objectPath) + for (const candidate of [`${fullPath}.gz`, fullPath]) { + let handle: any + try { + handle = await fs.promises.open(candidate, 'r') + } catch (error: any) { + if (error.code === 'ENOENT') continue + throw error + } + try { + await handle.sync() + } finally { + await handle.close() + } + parentDirs.add(path.dirname(fullPath)) + break + } + } + + for (const dir of parentDirs) { + let handle: any + try { + handle = await fs.promises.open(dir, 'r') + } catch { + continue // directory vanished or platform disallows opening dirs + } + try { + await handle.sync() + } catch { + // Some platforms (and some filesystems) reject directory fsync — + // file-level fsync above already covers the data itself. + } finally { + await handle.close() + } + } + } + + /** + * Append one line to `_system/tx-log.jsonl`. Plain `appendFile` — the + * tx-log is the one append-in-place file in the store (and is byte-copied, + * never hard-linked, by {@link FileSystemStorage.snapshotToDirectory}). + * + * @param line - One complete JSON document, without trailing newline. + */ + public async appendTxLogLine(line: string): Promise { + await this.ensureInitialized() + const logPath = path.join(this.systemDir, 'tx-log.jsonl') + await fs.promises.appendFile(logPath, `${line}\n`, 'utf-8') + } + + /** + * Read all tx-log lines, oldest first (empty array when no log exists). + * Torn trailing lines from a crashed append are returned as-is — callers + * tolerate unparseable lines. + */ + public async readTxLogLines(): Promise { + await this.ensureInitialized() + const logPath = path.join(this.systemDir, 'tx-log.jsonl') + try { + const content: string = await fs.promises.readFile(logPath, 'utf-8') + return content.split('\n').filter((l: string) => l.length > 0) + } catch (error: any) { + if (error.code === 'ENOENT') return [] + throw error + } + } + + /** + * Snapshot the entire store into `targetPath` as a hard-link farm + * (Cassandra-style: instant, space-shared). Safe because every data file + * is immutable-by-rename — rewrites swap in new inodes, leaving the + * snapshot's links pointing at the old bytes. The two exceptions are + * handled explicitly: append-in-place files + * ({@link FileSystemStorage.SNAPSHOT_BYTE_COPY_PATHS}) are byte-copied, and + * process-local lock state + * ({@link FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS}) is excluded. + * Cross-device targets (where `link(2)` fails with `EXDEV`) fall back to + * byte copies per file. + * + * @param targetPath - Absolute directory for the snapshot. Created if + * missing; must be empty or absent (refuses to overwrite). + */ + public async snapshotToDirectory(targetPath: string): Promise { + await this.ensureInitialized() + + try { + const existing = await fs.promises.readdir(targetPath) + if (existing.length > 0) { + throw new Error( + `snapshotToDirectory: target ${targetPath} already exists and is not empty. ` + + `Choose a fresh directory per snapshot.` + ) + } + } catch (error: any) { + if (error.code !== 'ENOENT') throw error + } + await fs.promises.mkdir(targetPath, { recursive: true }) + + const files: string[] = [] + await this.collectSnapshotFiles(this.rootDir, '', files) + + for (const relPath of files) { + const sourceFile = path.join(this.rootDir, relPath) + const targetFile = path.join(targetPath, relPath) + await fs.promises.mkdir(path.dirname(targetFile), { recursive: true }) + + // Byte-copy list: compare against the normalized (extension-preserving) + // relative path with separators unified, so `_system/tx-log.jsonl` + // matches on every platform. + const normalized = relPath.split(path.sep).join('/') + if (FileSystemStorage.SNAPSHOT_BYTE_COPY_PATHS.has(normalized)) { + await fs.promises.copyFile(sourceFile, targetFile) + continue + } + + try { + await fs.promises.link(sourceFile, targetFile) + } catch (error: any) { + // EXDEV: cross-device; EPERM/ENOTSUP: filesystem forbids links. + // ENOENT: the live file was atomically replaced mid-walk — retry as + // a copy of whatever is current (single-writer discipline means this + // only happens for derived files being flushed concurrently). + if (['EXDEV', 'EPERM', 'ENOTSUP', 'ENOENT'].includes(error.code)) { + try { + await fs.promises.copyFile(sourceFile, targetFile) + } catch (copyError: any) { + if (copyError.code !== 'ENOENT') throw copyError + } + } else { + throw error + } + } + } + } + + /** + * Recursively collect snapshot-eligible files under `dirAbs`, excluding the + * lock directory and in-flight `*.tmp.*` write files. + */ + private async collectSnapshotFiles(dirAbs: string, relPrefix: string, out: string[]): Promise { + let entries: any[] + try { + entries = await fs.promises.readdir(dirAbs, { withFileTypes: true }) + } catch (error: any) { + if (error.code === 'ENOENT') return + throw error + } + + for (const entry of entries) { + const rel = relPrefix ? path.join(relPrefix, entry.name) : entry.name + if (entry.isDirectory()) { + if (relPrefix === '' && FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS.has(entry.name)) { + continue + } + await this.collectSnapshotFiles(path.join(dirAbs, entry.name), rel, out) + } else if (entry.isFile()) { + if (entry.name.includes('.tmp.')) continue // in-flight atomic write + out.push(rel) + } + } + } + + /** + * Replace the store's contents from a snapshot directory: every current + * top-level entry except `locks/` (the live writer lock must survive) is + * removed, the snapshot is byte-copied in (`fs.cp` — never hard-linked, so + * the snapshot stays independent of the restored store), and all + * adapter-internal derived state is reloaded. + * + * @param sourcePath - Absolute path of a directory produced by + * {@link FileSystemStorage.snapshotToDirectory}. + */ + public async restoreFromDirectory(sourcePath: string): Promise { + await this.ensureInitialized() + + const sourceStat = await fs.promises.stat(sourcePath).catch(() => null) + if (!sourceStat || !sourceStat.isDirectory()) { + throw new Error(`restoreFromDirectory: ${sourcePath} is not a directory`) + } + + // Clear current contents, preserving live lock state. + const currentEntries = await fs.promises.readdir(this.rootDir) + for (const entry of currentEntries) { + if (FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS.has(entry)) continue + await fs.promises.rm(path.join(this.rootDir, entry), { recursive: true, force: true }) + } + + // Byte-copy the snapshot in (defensively skipping lock dirs in old snapshots). + const sourceEntries = await fs.promises.readdir(sourcePath) + for (const entry of sourceEntries) { + if (FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS.has(entry)) continue + await fs.promises.cp(path.join(sourcePath, entry), path.join(this.rootDir, entry), { + recursive: true + }) + } + + await this.reloadDerivedState() + } + // =========================================================================== // Raw binary-blob primitive (mmap-friendly) // =========================================================================== diff --git a/src/storage/adapters/historicalStorageAdapter.ts b/src/storage/adapters/historicalStorageAdapter.ts index 4d177637..6ff29710 100644 --- a/src/storage/adapters/historicalStorageAdapter.ts +++ b/src/storage/adapters/historicalStorageAdapter.ts @@ -525,4 +525,47 @@ export class HistoricalStorageAdapter extends BaseStorage { protected async persistCounts(): Promise { // No-op: Historical storage is read-only } + + // =========================================================================== + // Generational record layer (8.0 MVCC) — not applicable to this adapter + // + // HistoricalStorageAdapter is the pre-8.0 commit-based time-travel view and + // is read-only by construction. The generation store never registers its + // write hooks on reader-mode instances, so these methods are unreachable in + // normal operation; they throw explicitly rather than fake behavior. + // =========================================================================== + + /** + * WRITE BLOCKED: the tx-log belongs to the live store, not a historical view. + */ + public async appendTxLogLine(_line: string): Promise { + throw new Error('Historical storage is read-only. Cannot append to the transaction log.') + } + + /** + * Not supported: historical commit views predate the generational tx-log. + */ + public async readTxLogLines(): Promise { + throw new Error( + 'Transaction-log reads are not supported on a historical commit view. ' + + 'Use the live store (or a Db from brain.now()/brain.asOf()).' + ) + } + + /** + * Not supported: snapshot a live store via db.persist(path) instead. + */ + public async snapshotToDirectory(_targetPath: string): Promise { + throw new Error( + 'snapshotToDirectory is not supported on a historical commit view. ' + + 'Call db.persist(path) on the live store instead.' + ) + } + + /** + * WRITE BLOCKED: cannot restore into a read-only historical view. + */ + public async restoreFromDirectory(_sourcePath: string): Promise { + throw new Error('Historical storage is read-only. Cannot restore from a snapshot.') + } } diff --git a/src/storage/adapters/memoryStorage.ts b/src/storage/adapters/memoryStorage.ts index cb2c5bb9..f7ea1125 100644 --- a/src/storage/adapters/memoryStorage.ts +++ b/src/storage/adapters/memoryStorage.ts @@ -3,6 +3,9 @@ * In-memory storage adapter for environments where persistent storage is not available or needed */ +import * as fs from 'node:fs' +import * as nodePath from 'node:path' +import * as zlib from 'node:zlib' import { GraphVerb, HNSWNoun, @@ -35,6 +38,11 @@ export class MemoryStorage extends BaseStorage { // storage has no local file, so getBinaryBlobPath() returns null. private blobStore: Map = new Map() + // Transaction log (8.0 MVCC) — the in-memory equivalent of the filesystem + // adapter's `_system/tx-log.jsonl`. One JSON document per committed + // transact(); serialized to a real JSONL file by snapshotToDirectory(). + private txLogLines: string[] = [] + // Backward compatibility aliases private get metadata(): Map { return this.objectStore @@ -189,6 +197,132 @@ export class MemoryStorage extends BaseStorage { return null } + // =========================================================================== + // Generational record layer (8.0 MVCC) — tx-log + snapshot primitives + // =========================================================================== + + /** + * Append one line to the in-memory transaction log (mirror of the + * filesystem adapter's `_system/tx-log.jsonl` append). + * + * @param line - One complete JSON document, without trailing newline. + */ + public async appendTxLogLine(line: string): Promise { + this.txLogLines.push(line) + } + + /** + * Read all transaction-log lines, oldest first (a copy — callers cannot + * mutate the log). + */ + public async readTxLogLines(): Promise { + return [...this.txLogLines] + } + + /** + * Serialize the entire in-memory store to a directory in the exact layout + * the filesystem adapter uses (uncompressed JSON objects, `_blobs/*.bin` + * payloads, `_system/tx-log.jsonl`). The resulting directory is a + * self-contained store openable via `Brainy.load(path)` — persisting an + * in-memory brain produces a real, durable snapshot. + * + * @param targetPath - Absolute directory for the snapshot. Created if + * missing; must be empty or absent (refuses to overwrite). + */ + public async snapshotToDirectory(targetPath: string): Promise { + try { + const existing = await fs.promises.readdir(targetPath) + if (existing.length > 0) { + throw new Error( + `snapshotToDirectory: target ${targetPath} already exists and is not empty. ` + + `Choose a fresh directory per snapshot.` + ) + } + } catch (error: any) { + if (error.code !== 'ENOENT') throw error + } + await fs.promises.mkdir(targetPath, { recursive: true }) + + for (const [key, value] of this.objectStore.entries()) { + const filePath = nodePath.join(targetPath, ...key.split('/')) + await fs.promises.mkdir(nodePath.dirname(filePath), { recursive: true }) + await fs.promises.writeFile(filePath, JSON.stringify(value, null, 2), 'utf-8') + } + + for (const [key, data] of this.blobStore.entries()) { + const blobPath = nodePath.join(targetPath, '_blobs', ...key.split('/')) + '.bin' + await fs.promises.mkdir(nodePath.dirname(blobPath), { recursive: true }) + await fs.promises.writeFile(blobPath, data) + } + + if (this.txLogLines.length > 0) { + const logPath = nodePath.join(targetPath, '_system', 'tx-log.jsonl') + await fs.promises.mkdir(nodePath.dirname(logPath), { recursive: true }) + await fs.promises.writeFile(logPath, this.txLogLines.map((l) => `${l}\n`).join(''), 'utf-8') + } + } + + /** + * Replace the in-memory store's contents from a snapshot directory — + * accepts snapshots produced by either adapter (handles the filesystem + * adapter's gzip-compressed `.json.gz` objects transparently), then reloads + * all derived state. + * + * @param sourcePath - Absolute path of a snapshot directory. + */ + public async restoreFromDirectory(sourcePath: string): Promise { + const sourceStat = await fs.promises.stat(sourcePath).catch(() => null) + if (!sourceStat || !sourceStat.isDirectory()) { + throw new Error(`restoreFromDirectory: ${sourcePath} is not a directory`) + } + + this.objectStore.clear() + this.blobStore.clear() + this.txLogLines = [] + this.statistics = null + + await this.loadDirectoryIntoStore(sourcePath, '') + await this.reloadDerivedState() + } + + /** + * Recursively load a snapshot directory into the object/blob/tx-log stores. + * Lock directories from filesystem-adapter snapshots are skipped (process- + * local state, meaningless in memory). + */ + private async loadDirectoryIntoStore(dirAbs: string, relPrefix: string): Promise { + const entries = await fs.promises.readdir(dirAbs, { withFileTypes: true }) + + for (const entry of entries) { + const rel = relPrefix ? `${relPrefix}/${entry.name}` : entry.name + const abs = nodePath.join(dirAbs, entry.name) + + if (entry.isDirectory()) { + if (relPrefix === '' && entry.name === 'locks') continue + await this.loadDirectoryIntoStore(abs, rel) + continue + } + if (!entry.isFile()) continue + + if (rel === '_system/tx-log.jsonl') { + const content = await fs.promises.readFile(abs, 'utf-8') + this.txLogLines = content.split('\n').filter((l) => l.length > 0) + } else if (relPrefix.startsWith('_blobs') && rel.endsWith('.bin')) { + const key = rel.slice('_blobs/'.length, -'.bin'.length) + this.blobStore.set(key, await fs.promises.readFile(abs)) + } else if (rel.endsWith('.json.gz') || rel.endsWith('.gz')) { + const raw = await fs.promises.readFile(abs) + const key = rel.slice(0, -'.gz'.length) + this.objectStore.set(key, JSON.parse(zlib.gunzipSync(raw).toString('utf-8'))) + } else if (entry.name.includes('.tmp.')) { + // In-flight atomic-write remnant from a crashed filesystem store — skip. + } else { + const content = await fs.promises.readFile(abs, 'utf-8') + this.objectStore.set(rel, JSON.parse(content)) + } + } + } + /** * Get multiple metadata objects in batches (CRITICAL: Prevents socket exhaustion) * Memory storage implementation is simple since all data is already in memory @@ -217,6 +351,7 @@ export class MemoryStorage extends BaseStorage { public async clear(): Promise { this.objectStore.clear() this.blobStore.clear() + this.txLogLines = [] this.statistics = null this.totalNounCount = 0 this.totalVerbCount = 0 diff --git a/src/storage/baseStorage.ts b/src/storage/baseStorage.ts index 0a72c15b..c7380caf 100644 --- a/src/storage/baseStorage.ts +++ b/src/storage/baseStorage.ts @@ -881,6 +881,271 @@ export abstract class BaseStorage extends BaseStorageAdapter { return Array.from(pathsSet) } + // ============================================================================ + // GENERATIONAL RECORD LAYER PRIMITIVES (8.0 MVCC) + // + // The narrow surface `GenerationStore` (src/db/generationStore.ts) needs from + // an adapter — see the `GenerationStorage` contract in src/db/types.ts. + // + // Raw-object methods operate on storage-root-relative paths and deliberately + // bypass branch scoping and the write cache: the record layer owns the + // `_system/` + `_generations/` areas outright and its files must never be + // shadowed by branch resolution. Entity-raw methods, by contrast, go through + // the branch-scoped, write-cache-coherent helpers so before-images capture + // exactly the bytes the live read paths see. + // ============================================================================ + + /** + * Hook invoked after every entity-visible single-operation write (noun/verb + * metadata save or delete). Registered by the generation store so + * `brain.generation()` advances on writes performed outside `transact()`. + * See {@link BaseStorage.setGenerationBumpHook}. + */ + protected generationBumpHook?: () => void + + /** + * Register (or detach, with `undefined`) the generation-bump hook. + * + * The hook fires once per entity-visible metadata mutation — noun/verb + * metadata saves and deletes, the one storage write every logical Brainy + * mutation (`add`, `update`, `delete`, `relate`, `updateRelation`, + * `unrelate`) performs exactly once per entity it touches. It does NOT fire + * for derived-index writes (HNSW node data, metadata-index chunks, LSM + * segments, statistics), so the generation counter tracks *data* mutations, + * not index maintenance. The counter is a monotonic watermark, not an + * operation count: a cascade delete bumps once per removed record. + * + * @param hook - Callback invoked synchronously after each qualifying write, + * or `undefined` to detach. + */ + public setGenerationBumpHook(hook: (() => void) | undefined): void { + this.generationBumpHook = hook + } + + /** + * Read a raw object at a storage-root-relative path. Bypasses branch + * scoping and the write cache (record-layer files are written through + * {@link BaseStorage.writeRawObject} only). + * + * @param path - Storage-root-relative object path (e.g. `_system/manifest.json`). + * @returns The parsed object, or `null` if absent. + */ + public async readRawObject(path: string): Promise { + await this.ensureInitialized() + return this.readObjectFromPath(path) + } + + /** + * Write a raw object at a storage-root-relative path. On disk this is an + * atomic tmp+rename (the filesystem adapter's primitive), which is what + * makes the manifest rename a valid commit point. + * + * @param path - Storage-root-relative object path. + * @param data - JSON-serializable object to persist. + */ + public async writeRawObject(path: string, data: any): Promise { + await this.ensureInitialized() + await this.writeObjectToPath(path, data) + } + + /** + * Delete a raw object at a storage-root-relative path (no-op if absent). + * + * @param path - Storage-root-relative object path. + */ + public async deleteRawObject(path: string): Promise { + await this.ensureInitialized() + await this.deleteObjectFromPath(path) + } + + /** + * List raw object paths under a storage-root-relative prefix (normalized, + * `.gz`-stripped — the adapter primitives already normalize). + * + * @param prefix - Storage-root-relative directory prefix. + * @returns Normalized object paths under the prefix (empty when none). + */ + public async listRawObjects(prefix: string): Promise { + await this.ensureInitialized() + return this.listObjectsUnderPath(prefix) + } + + /** + * Remove every object under a storage-root-relative prefix. The filesystem + * adapter overrides this with a recursive directory removal; this default + * lists and deletes individually (exactly what the in-memory adapter needs). + * + * @param prefix - Storage-root-relative directory prefix to remove. + */ + public async removeRawPrefix(prefix: string): Promise { + await this.ensureInitialized() + const paths = await this.listObjectsUnderPath(prefix) + for (const p of paths) { + await this.deleteObjectFromPath(p) + } + } + + /** + * Durability barrier for the commit protocol: ensure the listed raw-object + * paths are durable before the caller proceeds. The base implementation is + * a no-op (in-memory writes are durable-by-definition within the process); + * the filesystem adapter overrides it with real `fsync` of the files and + * their parent directories. + * + * @param paths - Storage-root-relative object paths previously written via + * {@link BaseStorage.writeRawObject}. + */ + public async syncRawObjects(paths: string[]): Promise { + void paths + } + + /** + * Read an entity's raw stored objects — the exact bytes at its canonical + * metadata + vector paths (branch-scoped, write-cache coherent). Used by + * the generation store to capture before-images. + * + * @param id - The entity id. + * @returns The raw stored metadata and vector objects (`null` per part when + * the corresponding file is absent). + */ + public async readNounRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> { + await this.ensureInitialized() + const [metadata, vector] = await Promise.all([ + this.readWithInheritance(getNounMetadataPath(id)), + this.readWithInheritance(getNounVectorPath(id)) + ]) + return { metadata: metadata ?? null, vector: vector ?? null } + } + + /** + * Restore an entity's raw stored objects byte-for-byte (a `null` part + * deletes that file). Used by crash recovery and transaction aborts to + * restore before-images. + * + * Bypasses the statistics/count bookkeeping of the normal save paths on + * purpose: restores must reproduce the exact prior bytes, and the count + * rollups are derived state with their own rebuild paths + * (`rebuildTypeCounts()` / `rebuildSubtypeCounts()`). + * + * @param id - The entity id. + * @param record - Raw stored objects as returned by {@link BaseStorage.readNounRaw}. + */ + public async writeNounRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise { + await this.ensureInitialized() + if (record.metadata === null) { + await this.deleteObjectFromBranch(getNounMetadataPath(id)) + } else { + await this.writeObjectToBranch(getNounMetadataPath(id), record.metadata) + } + if (record.vector === null) { + await this.deleteObjectFromBranch(getNounVectorPath(id)) + } else { + await this.writeObjectToBranch(getNounVectorPath(id), record.vector) + } + } + + /** + * Read a relationship's raw stored objects (verb-side mirror of + * {@link BaseStorage.readNounRaw}). + * + * @param id - The relationship id. + * @returns The raw stored metadata and vector objects (`null` per part when + * the corresponding file is absent). + */ + public async readVerbRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> { + await this.ensureInitialized() + const [metadata, vector] = await Promise.all([ + this.readWithInheritance(getVerbMetadataPath(id)), + this.readWithInheritance(getVerbVectorPath(id)) + ]) + return { metadata: metadata ?? null, vector: vector ?? null } + } + + /** + * Restore a relationship's raw stored objects byte-for-byte (verb-side + * mirror of {@link BaseStorage.writeNounRaw}; same bookkeeping caveats). + * + * @param id - The relationship id. + * @param record - Raw stored objects as returned by {@link BaseStorage.readVerbRaw}. + */ + public async writeVerbRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise { + await this.ensureInitialized() + if (record.metadata === null) { + await this.deleteObjectFromBranch(getVerbMetadataPath(id)) + } else { + await this.writeObjectToBranch(getVerbMetadataPath(id), record.metadata) + } + if (record.vector === null) { + await this.deleteObjectFromBranch(getVerbVectorPath(id)) + } else { + await this.writeObjectToBranch(getVerbVectorPath(id), record.vector) + } + } + + /** + * Append one line to the transaction log (`_system/tx-log.jsonl`). The + * filesystem adapter appends to a real JSONL file; the in-memory adapter + * keeps a line array (serialized by `snapshotToDirectory`). + * + * @param line - One complete JSON document, without trailing newline. + */ + public abstract appendTxLogLine(line: string): Promise + + /** + * Read all transaction-log lines, oldest first (empty array when no log + * exists). A torn trailing line from a crashed append is returned as-is — + * callers tolerate unparseable lines. + */ + public abstract readTxLogLines(): Promise + + /** + * Snapshot the entire store into `targetPath`. The filesystem adapter + * builds a hard-link farm (instant, space-shared — safe because data files + * are immutable-by-rename); the in-memory adapter serializes its object + * store to a filesystem-storage-compatible directory. The result is a + * self-contained store openable via `Brainy.load(path)`. + * + * @param targetPath - Absolute directory path for the snapshot (created if + * missing; must be empty or absent). + */ + public abstract snapshotToDirectory(targetPath: string): Promise + + /** + * Replace the entire store's contents from a snapshot directory previously + * produced by {@link BaseStorage.snapshotToDirectory}. Implementations + * clear current contents (preserving live lock files), copy the snapshot + * in (byte copy — never hard links, so the snapshot stays independent), + * and then call {@link BaseStorage.reloadDerivedState}. + * + * @param sourcePath - Absolute path of the snapshot directory. + */ + public abstract restoreFromDirectory(sourcePath: string): Promise + + /** + * Reset and reload every piece of adapter-internal derived state after the + * underlying objects changed wholesale (restore-from-snapshot): the + * write-through cache, the id→type/subtype caches, type/subtype statistics, + * total counts, and the graph-index singleton (invalidated so the next + * accessor rebuilds from the restored verbs). + */ + protected async reloadDerivedState(): Promise { + this.clearWriteCache() + this.nounTypeByIdCache.clear() + this.nounSubtypeByIdCache.clear() + this.verbSubtypeByIdCache.clear() + this.nounCountsByType.fill(0) + this.verbCountsByType.fill(0) + this.subtypeCountsByType.clear() + this.verbSubtypeCountsByType.clear() + this.statisticsCache = null + this.statisticsModified = false + this.invalidateGraphIndex() + await this.loadTypeStatistics() + await this.loadSubtypeStatistics() + await this.loadVerbSubtypeStatistics() + await this.initializeCounts() + } + /** * Save a noun to storage (vector only, metadata saved separately) * @param noun Pure HNSW vector data (no metadata) @@ -2315,6 +2580,10 @@ export abstract class BaseStorage extends BaseStorageAdapter { // Ignore persist errors - will retry on next operation }) } + + // 8.0 MVCC: entity-visible write — advance the generation watermark + // (suppressed inside transact batches by the generation store). + this.generationBumpHook?.() } /** @@ -2717,6 +2986,10 @@ export abstract class BaseStorage extends BaseStorageAdapter { this.decrementSubtypeCount(priorType, priorSubtype) } } + + // 8.0 MVCC: entity-visible write — advance the generation watermark + // (suppressed inside transact batches by the generation store). + this.generationBumpHook?.() } /** @@ -2750,6 +3023,8 @@ export abstract class BaseStorage extends BaseStorageAdapter { // Backward compatibility: fallback to old path if no verb type const keyInfo = this.analyzeKey(id, 'verb-metadata') await this.writeObjectToBranch(keyInfo.fullPath, metadata) + // 8.0 MVCC: still an entity-visible write — advance the watermark. + this.generationBumpHook?.() return } @@ -2804,6 +3079,10 @@ export abstract class BaseStorage extends BaseStorageAdapter { // Ignore persist errors - will retry on next operation }) } + + // 8.0 MVCC: entity-visible write — advance the generation watermark + // (suppressed inside transact batches by the generation store). + this.generationBumpHook?.() } /** @@ -2841,6 +3120,10 @@ export abstract class BaseStorage extends BaseStorageAdapter { this.verbSubtypeByIdCache.delete(id) this.decrementVerbSubtypeCount(priorEntry.verb, priorEntry.subtype) } + + // 8.0 MVCC: entity-visible write — advance the generation watermark + // (suppressed inside transact batches by the generation store). + this.generationBumpHook?.() } // ============================================================================ diff --git a/tests/integration/db-mvcc.test.ts b/tests/integration/db-mvcc.test.ts new file mode 100644 index 00000000..26b482a9 --- /dev/null +++ b/tests/integration/db-mvcc.test.ts @@ -0,0 +1,955 @@ +/** + * @module tests/integration/db-mvcc + * @description Credibility-bar proof suite for Brainy 8.0's generational MVCC + * storage core + Datomic-style Db API. Each test PROVES a stated guarantee + * (not merely exercises the API): + * + * 1. Isolation — a pinned Db reads its pinned state exactly, across 200 + * subsequent transact mutations (updates + removes). + * 2. Atomicity — a batch with a failing later operation applies NOTHING + * (generation unchanged, no partial records), for both plan-time and + * execution-phase failures (the latter via real fault injection into the + * storage adapter, exercising the production rollback path). + * 3. CAS — `ifAtGeneration` commits when fresh, throws + * `GenerationConflictError` with correct expected/actual when stale. + * 4. Snapshot integrity — persist → mutate source heavily → `Brainy.load()` + * is byte-identical to the pre-mutation state (hard-link safety), and an + * in-memory brain persists to a loadable directory. + * 5. Compaction safety — pinned reads stay correct across `compactHistory()`; + * after release + compact the superseded record-sets are reclaimed. + * 6. with() — speculative overlays are visible on the overlay Db only; the + * underlying brain and the original Db are untouched. + * 7. Generation monotonicity across close/reopen. + * 8. Crash consistency — a simulated crash between record staging and the + * manifest rename (the commit point) recovers to the exact pre-transaction + * state on reopen, through the REAL production recovery path. + * 9. Versioned provider wiring — a provider implementing the 4-method + * `VersionedIndexProvider` contract receives balanced pin/release (and a + * visibility consult per pin) as Db values are created and released. + * + * All entities carry explicit vectors so the suite never invokes the + * embedding engine (semantic search is exercised only for its documented + * historical-generation error). + */ + +import { describe, it, expect, afterEach, vi } from 'vitest' +import * as fs from 'node:fs' +import * as os from 'node:os' +import * as path from 'node:path' +import { Brainy } from '../../src/brainy.js' +import { Db } from '../../src/db/db.js' +import { + GenerationCompactedError, + GenerationConflictError, + NotYetSupportedAtHistoricalGenerationError +} from '../../src/db/errors.js' +import type { GenerationStore } from '../../src/db/generationStore.js' +import { GraphAdjacencyIndex } from '../../src/graph/graphAdjacencyIndex.js' +import { RevisionConflictError } from '../../src/transaction/RevisionConflictError.js' +import type { StorageAdapter } from '../../src/coreTypes.js' +import { NounType, VerbType } from '../../src/types/graphTypes.js' + +/** Deterministic 384-dim vector so no test ever invokes the embedder. */ +function vec(seed: number): number[] { + return Array.from({ length: 384 }, (_, i) => ((seed * 31 + i * 7) % 100) / 100) +} + +/** + * Map a readable label to a deterministic UUID-shaped id (entity ids must be + * UUIDs — the sharded storage layout derives the shard from the UUID hex). + * Two independent FNV-style mixes keep 7+ distinct labels collision-free. + */ +function uid(label: string): string { + let h1 = 0x811c9dc5 + for (let i = 0; i < label.length; i++) { + h1 = Math.imul(h1 ^ label.charCodeAt(i), 0x01000193) >>> 0 + } + let h2 = 0xdeadbeef + for (let i = label.length - 1; i >= 0; i--) { + h2 = Math.imul(h2 ^ label.charCodeAt(i), 0x85ebca6b) >>> 0 + } + const hex = h1.toString(16).padStart(8, '0') + h2.toString(16).padStart(8, '0') + return `00000000-0000-4000-8000-${hex.slice(0, 12)}` +} + +/** Typed access to the brain's private generation store (test injection point). */ +function generationStoreOf(brain: Brainy): GenerationStore { + return (brain as unknown as { generationStore: GenerationStore }).generationStore +} + +describe('8.0 Db API — generational MVCC', () => { + const brains: Brainy[] = [] + const tempDirs: string[] = [] + + /** Track a brain for afterEach teardown. */ + async function openMemoryBrain(): Promise { + const brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' } }) + await brain.init() + brains.push(brain) + return brain + } + + /** Open (and track) a filesystem brain rooted at a fresh temp directory. */ + async function openFsBrain(dir?: string): Promise<{ brain: Brainy; dir: string }> { + const rootDirectory = dir ?? makeTempDir() + const brain = new Brainy({ + requireSubtype: false, + storage: { type: 'filesystem', rootDirectory } + }) + await brain.init() + brains.push(brain) + return { brain, dir: rootDirectory } + } + + function makeTempDir(): string { + const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-db-mvcc-')) + tempDirs.push(dir) + return dir + } + + afterEach(async () => { + for (const brain of brains.splice(0)) { + try { + await brain.close() + } catch { + // already closed by the test + } + } + for (const dir of tempDirs.splice(0)) { + await fs.promises.rm(dir, { recursive: true, force: true }) + } + }) + + // ========================================================================== + // 1. Isolation + // ========================================================================== + it('proof 1 — a pinned Db reads its pinned state exactly across 200 transact mutations', async () => { + const brain = await openMemoryBrain() + + const ids = Array.from({ length: 20 }, (_, i) => uid(`iso-e${i}`)) + await brain.transact( + ids.map((id, i) => ({ + op: 'add' as const, + id, + type: NounType.Document, + data: `doc-${i}`, + vector: vec(i), + metadata: { v: 0, slot: i } + })) + ) + + const db = brain.now() + const pinned = new Map() + for (const id of ids) { + pinned.set(id, await db.get(id)) + expect(pinned.get(id)).not.toBeNull() + } + const pinnedFind = await db.find({ type: NounType.Document, limit: 100 }) + expect(pinnedFind.length).toBe(20) + + // 200 mutations: 198 updates over the first 18 ids + 2 removes. + for (let i = 0; i < 200; i++) { + if (i === 100) { + await brain.transact([{ op: 'remove', id: uid('iso-e18') }]) + } else if (i === 150) { + await brain.transact([{ op: 'remove', id: uid('iso-e19') }]) + } else { + await brain.transact([ + { op: 'update', id: ids[i % 18], metadata: { v: i + 1, touchedAt: i } } + ]) + } + } + + // Live state moved… + const liveUpdated = await brain.get(ids[0]) + expect((liveUpdated?.metadata as { v: number }).v).toBeGreaterThan(0) + expect(await brain.get(uid('iso-e18'))).toBeNull() + expect(await brain.get(uid('iso-e19'))).toBeNull() + + // …but EVERY read through the pinned Db is exactly the pinned state, + // including the two entities removed after the pin. + for (const id of ids) { + expect(await db.get(id)).toEqual(pinned.get(id)) + } + + // Set-level isolation: metadata find() at the pin still sees all 20. + const histFind = await db.find({ type: NounType.Document, limit: 100 }) + expect(histFind.length).toBe(20) + for (const row of histFind) { + expect((row.metadata as { v: number }).v).toBe(0) + } + + await db.release() + }) + + // ========================================================================== + // 2. Atomicity + // ========================================================================== + it('proof 2a — a plan-time failure in a later op applies nothing', async () => { + const brain = await openMemoryBrain() + const g0 = brain.generation() + + await expect( + brain.transact([ + { + op: 'add', + id: uid('atomic-a'), + type: NounType.Document, + data: 'a', + vector: vec(1), + metadata: { ok: true } + }, + { op: 'update', id: uid('never-created'), metadata: { boom: true } } + ]) + ).rejects.toThrow(`Entity ${uid('never-created')} not found`) + + expect(brain.generation()).toBe(g0) + expect(await brain.get(uid('atomic-a'))).toBeNull() + expect((await brain.find({ type: NounType.Document, limit: 10 })).length).toBe(0) + }) + + it('proof 2b — an execution-phase failure rolls back every applied op (generation unchanged, zero partial records)', async () => { + const brain = await openMemoryBrain() + + await brain.transact([ + { + op: 'add', + id: uid('atomic-base'), + type: NounType.Document, + data: 'base', + vector: vec(0), + metadata: { base: true } + } + ]) + const g0 = brain.generation() + const storage = brain.storageAdapter + + // Real fault injection: the verb-metadata write of the LAST operation + // fails once, after the earlier add operations have already executed — + // exercising the production TransactionManager rollback path. + const spy = vi + .spyOn(storage, 'saveVerbMetadata') + .mockImplementationOnce(async () => { + throw new Error('disk full (injected)') + }) + + await expect( + brain.transact([ + { + op: 'add', + id: uid('atomic-b'), + type: NounType.Person, + data: 'b', + vector: vec(2), + metadata: { name: 'b' } + }, + { + op: 'relate', + from: uid('atomic-b'), + to: uid('atomic-base'), + type: VerbType.RelatedTo + } + ]) + ).rejects.toThrow('disk full (injected)') + spy.mockRestore() + + // ZERO ops applied: generation unchanged, the add rolled back, no edge. + expect(brain.generation()).toBe(g0) + expect(await brain.get(uid('atomic-b'))).toBeNull() + expect((await brain.getRelations({ from: uid('atomic-b') })).length).toBe(0) + expect((await brain.find({ type: NounType.Person, limit: 10 })).length).toBe(0) + + // No partial generation records: the staging directory was removed. + const records = await storage.listRawObjects(`_generations/${g0 + 1}`) + expect(records).toEqual([]) + + // The store remains fully writable and consistent after the rollback. + const db = await brain.transact([ + { + op: 'add', + id: uid('atomic-c'), + type: NounType.Person, + data: 'c', + vector: vec(3), + metadata: { name: 'c' } + } + ]) + expect(db.generation).toBe(g0 + 1) + await db.release() + }) + + it('proof 2c — an ifRev conflict on a later update rejects the whole batch', async () => { + const brain = await openMemoryBrain() + await brain.transact([ + { + op: 'add', + id: uid('rev-e'), + type: NounType.Document, + data: 'rev', + vector: vec(4), + metadata: { v: 1 } + } + ]) + await brain.update({ id: uid('rev-e'), metadata: { v: 2 } }) // _rev is now 2 + const g0 = brain.generation() + + await expect( + brain.transact([ + { + op: 'add', + id: uid('rev-new'), + type: NounType.Document, + data: 'new', + vector: vec(5), + metadata: {} + }, + { op: 'update', id: uid('rev-e'), metadata: { v: 3 }, ifRev: 1 } // stale + ]) + ).rejects.toThrow(RevisionConflictError) + + expect(brain.generation()).toBe(g0) + expect(await brain.get(uid('rev-new'))).toBeNull() + expect(((await brain.get(uid('rev-e')))?.metadata as { v: number }).v).toBe(2) + }) + + // ========================================================================== + // 3. CAS (ifAtGeneration) + // ========================================================================== + it('proof 3 — ifAtGeneration commits when fresh and conflicts when stale', async () => { + const brain = await openMemoryBrain() + const db = brain.now() + + const committed = await brain.transact( + [ + { + op: 'add', + id: uid('cas-e'), + type: NounType.Document, + data: 'cas', + vector: vec(6), + metadata: { n: 1 } + } + ], + { ifAtGeneration: db.generation } + ) + expect(committed.generation).toBe(db.generation + 1) + + try { + await brain.transact( + [{ op: 'update', id: uid('cas-e'), metadata: { n: 2 } }], + { ifAtGeneration: db.generation } // stale — committed moved past it + ) + expect.unreachable('should have thrown GenerationConflictError') + } catch (err) { + expect(err).toBeInstanceOf(GenerationConflictError) + expect((err as GenerationConflictError).expected).toBe(db.generation) + expect((err as GenerationConflictError).actual).toBe(committed.generation) + } + + // Nothing committed by the conflicting attempt. + expect(((await brain.get(uid('cas-e')))?.metadata as { n: number }).n).toBe(1) + expect(brain.generation()).toBe(committed.generation) + + await db.release() + await committed.release() + }) + + // ========================================================================== + // 4. Snapshot integrity (persist → mutate → load) + // ========================================================================== + it('proof 4a — a persisted snapshot is immune to heavy source mutation (hard-link safety)', async () => { + const { brain } = await openFsBrain() + + const ids = Array.from({ length: 10 }, (_, i) => uid(`snap-e${i}`)) + await brain.transact( + ids.map((id, i) => ({ + op: 'add' as const, + id, + type: NounType.Document, + data: `snapshot-doc-${i}`, + vector: vec(i + 10), + metadata: { generation: 'pre', slot: i } + })) + ) + const relDb = await brain.transact([ + { op: 'relate', from: ids[0], to: ids[1], type: VerbType.References }, + { op: 'relate', from: ids[1], to: ids[2], type: VerbType.References } + ]) + await relDb.release() + + // Capture pre-mutation state, then snapshot. + const preState = new Map() + for (const id of ids) { + preState.set(id, await brain.get(id)) + } + const preRelations = await brain.getRelations({ from: ids[0] }) + expect(preRelations.length).toBe(1) + + const snapDir = path.join(makeTempDir(), 'snapshot') + const db = brain.now() + await db.persist(snapDir) + await db.release() + + // Mutate the source heavily: rewrite every entity, delete some, add new. + for (const [i, id] of ids.entries()) { + await brain.transact([ + { op: 'update', id, data: `mutated-${i}`, metadata: { generation: 'post' } } + ]) + } + await brain.transact([{ op: 'remove', id: ids[0] }]) + await brain.transact([ + { + op: 'add', + id: uid('snap-after'), + type: NounType.Document, + data: 'added after snapshot', + vector: vec(99), + metadata: { generation: 'post' } + } + ]) + await brain.flush() + + // The snapshot opens as a self-contained store with the PRE state. + const loaded = await Brainy.load(snapDir) + for (const id of ids) { + const entity = await loaded.get(id) + expect(entity).toEqual(preState.get(id)) + } + expect(await loaded.get(uid('snap-after'))).toBeNull() + const loadedFind = await loaded.find({ type: NounType.Document, limit: 100 }) + expect(loadedFind.length).toBe(10) + for (const row of loadedFind) { + expect((row.metadata as { generation: string }).generation).toBe('pre') + } + const loadedRelations = await loaded.related({ from: ids[0] }) + expect(loadedRelations.length).toBe(1) + expect(loadedRelations[0].to).toBe(ids[1]) + await loaded.release() + + // And asOf(path) is the instance-method route to the same snapshot. + const viaAsOf = await brain.asOf(snapDir) + expect(((await viaAsOf.get(ids[3])) as { metadata: { generation: string } }).metadata.generation).toBe('pre') + await viaAsOf.release() + }) + + it('proof 4b — an in-memory brain persists to a directory loadable as a real store', async () => { + const brain = await openMemoryBrain() + await brain.transact([ + { + op: 'add', + id: uid('mem-e1'), + type: NounType.Document, + data: 'memory-persisted', + vector: vec(21), + metadata: { from: 'memory' } + } + ]) + + const snapDir = path.join(makeTempDir(), 'memory-snapshot') + const db = brain.now() + await db.persist(snapDir) + await db.release() + + const loaded = await Brainy.load(snapDir) + const entity = await loaded.get(uid('mem-e1')) + expect(entity).not.toBeNull() + expect((entity?.metadata as { from: string }).from).toBe('memory') + await loaded.release() + }) + + it('proof 4c — persist() refuses a view that history has moved past (honest boundary)', async () => { + const brain = await openMemoryBrain() + const db = brain.now() + await brain.transact([ + { + op: 'add', + id: uid('stale-persist'), + type: NounType.Document, + data: 'x', + vector: vec(30), + metadata: {} + } + ]) + + const snapDir = path.join(makeTempDir(), 'stale-snapshot') + await expect(db.persist(snapDir)).rejects.toThrow( + NotYetSupportedAtHistoricalGenerationError + ) + await db.release() + }) + + // ========================================================================== + // 5. Compaction safety + // ========================================================================== + it('proof 5 — compaction never breaks pinned reads; after release the records are reclaimed', async () => { + const { brain } = await openFsBrain() + const storage = brain.storageAdapter + + // Every transact() returns a PINNED Db — release the ones this test + // does not keep, so compaction eligibility is determined solely by `db`. + await ( + await brain.transact([ + { + op: 'add', + id: uid('compact-e'), + type: NounType.Document, + data: 'v1', + vector: vec(40), + metadata: { v: 1 } + } + ]) + ).release() + await ( + await brain.transact([{ op: 'update', id: uid('compact-e'), metadata: { v: 2 } }]) + ).release() + const db = brain.now() // pinned at v2 + const pinnedEntity = await db.get(uid('compact-e')) + await ( + await brain.transact([{ op: 'update', id: uid('compact-e'), metadata: { v: 3 } }]) + ).release() + await ( + await brain.transact([{ op: 'update', id: uid('compact-e'), metadata: { v: 4 } }]) + ).release() + + const recordsBefore = (await storage.listRawObjects('_generations')).length + expect(recordsBefore).toBeGreaterThan(0) + + // Compact while pinned: record-sets above the pin survive, pinned reads stay correct. + const first = await brain.compactHistory() + expect(await db.get(uid('compact-e'))).toEqual(pinnedEntity) + expect(((await db.get(uid('compact-e')))?.metadata as { v: number }).v).toBe(2) + + // After release, everything is reclaimable. + await db.release() + const second = await brain.compactHistory() + expect(first.removedGenerations + second.removedGenerations).toBeGreaterThan(0) + + const recordsAfter = (await storage.listRawObjects('_generations')).length + expect(recordsAfter).toBeLessThan(recordsBefore) + expect(recordsAfter).toBe(0) + + // Compacted generations are honestly unreachable. + await expect(brain.asOf(1)).rejects.toThrow(GenerationCompactedError) + }) + + // ========================================================================== + // 6. with() — speculative overlays + // ========================================================================== + it('proof 6 — with() overlays are visible only on the overlay Db; brain and base Db untouched', async () => { + const brain = await openMemoryBrain() + await brain.transact([ + { + op: 'add', + id: uid('with-a'), + type: NounType.Person, + data: 'Ada', + vector: vec(50), + metadata: { v: 1 } + }, + { + op: 'add', + id: uid('with-b'), + type: NounType.Person, + data: 'Grace', + vector: vec(51), + metadata: { v: 1 } + } + ]) + const g0 = brain.generation() + const db = brain.now() + + const spec = await db.with([ + { op: 'update', id: uid('with-a'), metadata: { v: 2 } }, + { + op: 'add', + id: uid('with-new'), + type: NounType.Project, + data: 'Apollo', + metadata: { speculative: true } + }, + { op: 'remove', id: uid('with-b') }, + { op: 'relate', from: uid('with-a'), to: uid('with-new'), type: VerbType.WorksOn } + ]) + + // The overlay view sees everything… + expect(spec.speculative).toBe(true) + expect(((await spec.get(uid('with-a')))?.metadata as { v: number }).v).toBe(2) + expect(await spec.get(uid('with-new'))).not.toBeNull() + expect(await spec.get(uid('with-b'))).toBeNull() + const specEdges = await spec.related({ from: uid('with-a') }) + expect(specEdges.length).toBe(1) + expect(specEdges[0].to).toBe(uid('with-new')) + const specFind = await spec.find({ type: NounType.Project, limit: 10 }) + expect(specFind.length).toBe(1) + + // …the base Db sees none of it… + expect(db.speculative).toBe(false) + expect(((await db.get(uid('with-a')))?.metadata as { v: number }).v).toBe(1) + expect(await db.get(uid('with-new'))).toBeNull() + expect(await db.get(uid('with-b'))).not.toBeNull() + + // …and the brain (disk, indexes, generation counter) is untouched. + expect(brain.generation()).toBe(g0) + expect(((await brain.get(uid('with-a')))?.metadata as { v: number }).v).toBe(1) + expect(await brain.get(uid('with-new'))).toBeNull() + expect((await brain.getRelations({ from: uid('with-a') })).length).toBe(0) + + await spec.release() + await db.release() + }) + + // ========================================================================== + // 7. Generation monotonicity across close/reopen + // ========================================================================== + it('proof 7 — the generation counter never moves backwards across close/reopen', async () => { + const dir = makeTempDir() + const { brain: first } = await openFsBrain(dir) + + await first.transact([ + { + op: 'add', + id: uid('mono-e'), + type: NounType.Document, + data: 'v1', + vector: vec(60), + metadata: { v: 1 } + } + ]) + await first.transact([{ op: 'update', id: uid('mono-e'), metadata: { v: 2 } }]) + // Single-operation write — bumps the counter outside transact. + await first.update({ id: uid('mono-e'), metadata: { v: 3 } }) + const beforeClose = first.generation() + expect(beforeClose).toBeGreaterThanOrEqual(3) + await first.close() + + const { brain: second } = await openFsBrain(dir) + expect(second.generation()).toBeGreaterThanOrEqual(beforeClose) + + const db = await second.transact([{ op: 'update', id: uid('mono-e'), metadata: { v: 4 } }]) + expect(db.generation).toBeGreaterThan(beforeClose) + await db.release() + }) + + // ========================================================================== + // 8. Crash consistency + // ========================================================================== + it('proof 8 — a crash before the manifest rename recovers to the exact pre-transaction state', async () => { + const dir = makeTempDir() + const { brain: first } = await openFsBrain(dir) + + await first.transact([ + { + op: 'add', + id: uid('crash-e'), + type: NounType.Document, + data: 'stable', + vector: vec(70), + metadata: { v: 1 } + } + ]) + const committedGen = first.generation() + + // Simulated crash at the worst point of the REAL commit protocol: every + // record is staged and the batch has executed, but the atomic manifest + // rename — the commit point — never happens. The throwing injector skips + // the abort cleanup exactly as a dead process would. + generationStoreOf(first).setCommitFaultInjector((phase) => { + if (phase === 'before-manifest-rename') { + throw new Error('simulated process crash') + } + }) + await expect( + first.transact([ + { op: 'update', id: uid('crash-e'), metadata: { v: 2 } }, + { + op: 'add', + id: uid('crash-new'), + type: NounType.Document, + data: 'uncommitted', + vector: vec(71), + metadata: {} + } + ]) + ).rejects.toThrow('simulated process crash') + generationStoreOf(first).setCommitFaultInjector(undefined) + + // The in-process state is dirty (batch executed) — close flushes it all, + // the realistic worst case for the recovery path. + await first.close() + + // Reopen: recovery rolls the uncommitted generation back and rebuilds + // the indexes from the repaired records. + const { brain: second } = await openFsBrain(dir) + const recovered = await second.get(uid('crash-e')) + expect((recovered?.metadata as { v: number }).v).toBe(1) + expect(await second.get(uid('crash-new'))).toBeNull() + const found = await second.find({ type: NounType.Document, limit: 10 }) + expect(found.length).toBe(1) + expect((found[0].metadata as { v: number }).v).toBe(1) + + // The crashed generation number is never reissued. + expect(second.generation()).toBeGreaterThanOrEqual(committedGen + 1) + const db = await second.transact([{ op: 'update', id: uid('crash-e'), metadata: { v: 5 } }]) + expect(db.generation).toBeGreaterThan(committedGen + 1) + await db.release() + }) + + // ========================================================================== + // 9. Versioned provider wiring + // ========================================================================== + it('proof 9 — a VersionedIndexProvider receives balanced pin/release as Db values live and die', async () => { + const calls = { + pins: [] as bigint[], + releases: [] as bigint[], + visibility: [] as bigint[] + } + + const brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' } }) + brain.use({ + name: 'test-versioned-graph-index', + activate: async (ctx) => { + ctx.registerProvider('graphIndex', (storage: StorageAdapter) => + Object.assign(new GraphAdjacencyIndex(storage), { + generation: () => 0n, + isGenerationVisible: (g: bigint) => { + calls.visibility.push(g) + return true + }, + pin: (g: bigint) => { + calls.pins.push(g) + }, + release: (g: bigint) => { + calls.releases.push(g) + } + }) + ) + return true + } + }) + await brain.init() + brains.push(brain) + + const base = brain.generation() + const db1 = brain.now() + expect(calls.pins).toEqual([BigInt(base)]) + expect(calls.visibility).toEqual([BigInt(base)]) // consulted at pin time + + const db2 = await brain.transact([ + { + op: 'add', + id: uid('vp-e'), + type: NounType.Document, + data: 'pinned', + vector: vec(80), + metadata: {} + } + ]) + expect(calls.pins).toEqual([BigInt(base), BigInt(base + 1)]) + + const db3 = await brain.asOf(base) + expect(calls.pins).toEqual([BigInt(base), BigInt(base + 1), BigInt(base)]) + + // A with() overlay takes its own pin on the same generation. + const spec = await db2.with([{ op: 'update', id: uid('vp-e'), metadata: { spec: true } }]) + expect(calls.pins.length).toBe(4) + + await db1.release() + await db2.release() + await db3.release() + await spec.release() + await spec.release() // idempotent — must NOT double-release the pin + + expect(calls.releases.length).toBe(calls.pins.length) + expect([...calls.releases].sort()).toEqual([...calls.pins].sort()) + }) + + // ========================================================================== + // Supporting guarantees: asOf, since, receipts, tx-log, honest errors + // ========================================================================== + it('asOf(generation) and asOf(Date) resolve historical state; future/negative are rejected', async () => { + const brain = await openMemoryBrain() + + const tx1 = await brain.transact( + [ + { + op: 'add', + id: uid('asof-e'), + type: NounType.Document, + data: 'v1', + vector: vec(90), + metadata: { v: 1 } + } + ], + { meta: { reason: 'first commit' } } + ) + await new Promise((resolve) => setTimeout(resolve, 5)) + const betweenCommits = new Date() + await new Promise((resolve) => setTimeout(resolve, 5)) + await brain.transact([{ op: 'update', id: uid('asof-e'), metadata: { v: 2 } }]) + + // By generation. + const atFirst = await brain.asOf(tx1.generation) + expect(((await atFirst.get(uid('asof-e')))?.metadata as { v: number }).v).toBe(1) + expect(atFirst.timestamp).toBe(tx1.timestamp) + await atFirst.release() + + // By timestamp: resolves to the newest commit at-or-before the instant. + const atInstant = await brain.asOf(betweenCommits) + expect(atInstant.generation).toBe(tx1.generation) + expect(((await atInstant.get(uid('asof-e')))?.metadata as { v: number }).v).toBe(1) + await atInstant.release() + + // Honest range errors. + await expect(brain.asOf(brain.generation() + 100)).rejects.toThrow(RangeError) + await expect(brain.asOf(-1)).rejects.toThrow(RangeError) + await expect(brain.asOf('/no/such/snapshot/dir')).rejects.toThrow('not a snapshot directory') + + // The transact meta was reified into the tx-log. + const lines = await brain.storageAdapter.readTxLogLines() + const entries = lines.map((line) => JSON.parse(line)) + const first = entries.find((entry) => entry.generation === tx1.generation) + expect(first?.meta).toEqual({ reason: 'first commit' }) + + await tx1.release() + }) + + it('receipts resolve ids in input order, including intra-batch references and relate dedupe', async () => { + const brain = await openMemoryBrain() + + const db = await brain.transact([ + { op: 'add', id: uid('rcpt-aa'), type: NounType.Person, data: 'a', vector: vec(91), metadata: {} }, + { op: 'add', id: uid('rcpt-bb'), type: NounType.Person, data: 'b', vector: vec(92), metadata: {} }, + { op: 'relate', from: uid('rcpt-aa'), to: uid('rcpt-bb'), type: VerbType.Knows }, + { op: 'update', id: uid('rcpt-bb'), metadata: { seen: true } } + ]) + expect(db.receipt).toBeDefined() + expect(db.receipt!.generation).toBe(db.generation) + expect(db.receipt!.ids.length).toBe(4) + expect(db.receipt!.ids[0]).toBe(uid('rcpt-aa')) + expect(db.receipt!.ids[1]).toBe(uid('rcpt-bb')) + expect(db.receipt!.ids[3]).toBe(uid('rcpt-bb')) + const relationId = db.receipt!.ids[2] + expect((await brain.getRelations({ from: uid('rcpt-aa') }))[0].id).toBe(relationId) + + // Duplicate relate (same from/to/type) dedupes to the SAME id — both + // within a batch and against committed state, mirroring relate(). + const dup = await brain.transact([ + { op: 'relate', from: uid('rcpt-aa'), to: uid('rcpt-bb'), type: VerbType.Knows } + ]) + expect(dup.receipt!.ids[0]).toBe(relationId) + expect((await brain.getRelations({ from: uid('rcpt-aa') })).length).toBe(1) + + await db.release() + await dup.release() + }) + + it('since() reports the ids changed between two pinned views', async () => { + const brain = await openMemoryBrain() + await brain.transact([ + { op: 'add', id: uid('since-a'), type: NounType.Document, data: 'a', vector: vec(93), metadata: {} } + ]) + const before = brain.now() + + await brain.transact([ + { op: 'add', id: uid('since-b'), type: NounType.Document, data: 'b', vector: vec(94), metadata: {} }, + { op: 'relate', from: uid('since-a'), to: uid('since-b'), type: VerbType.References } + ]) + await brain.transact([{ op: 'update', id: uid('since-a'), metadata: { touched: true } }]) + const after = brain.now() + + const changed = await after.since(before) + expect([...changed.nouns].sort()).toEqual([uid('since-a'), uid('since-b')].sort()) + expect(changed.verbs.length).toBe(1) + expect(changed.fromGeneration).toBe(before.generation) + expect(changed.toGeneration).toBe(after.generation) + + // Direction is enforced. + await expect(before.since(after)).rejects.toThrow(RangeError) + + await before.release() + await after.release() + }) + + it('index-accelerated queries at a historical generation throw the documented error — never silently-wrong results', async () => { + const brain = await openMemoryBrain() + await brain.transact([ + { op: 'add', id: uid('hist-e'), type: NounType.Document, data: 'x', vector: vec(95), metadata: { v: 1 } } + ]) + const db = brain.now() + await brain.transact([{ op: 'update', id: uid('hist-e'), metadata: { v: 2 } }]) + + // Storage-record reads remain fully supported… + expect(((await db.get(uid('hist-e')))?.metadata as { v: number }).v).toBe(1) + expect((await db.find({ type: NounType.Document, where: { v: 1 } })).length).toBe(1) + + // …index-accelerated dimensions throw the documented, named error. + await expect(db.search('anything')).rejects.toThrow( + NotYetSupportedAtHistoricalGenerationError + ) + await expect(db.find({ vector: vec(95) })).rejects.toThrow( + NotYetSupportedAtHistoricalGenerationError + ) + await expect(db.find({ connected: { from: uid('hist-e') } })).rejects.toThrow( + NotYetSupportedAtHistoricalGenerationError + ) + + // Released views refuse all reads. + await db.release() + await expect(db.get(uid('hist-e'))).rejects.toThrow('released Db') + }) + + it('restore() requires confirmation and replaces state from a snapshot (generation floor preserved)', async () => { + const { brain } = await openFsBrain() + await brain.transact([ + { op: 'add', id: uid('restore-e'), type: NounType.Document, data: 'keep', vector: vec(96), metadata: { v: 1 } } + ]) + + const snapDir = path.join(makeTempDir(), 'restore-snapshot') + const db = brain.now() + await db.persist(snapDir) + await db.release() + + await brain.transact([{ op: 'update', id: uid('restore-e'), metadata: { v: 2 } }]) + await brain.transact([ + { op: 'add', id: uid('restore-extra'), type: NounType.Document, data: 'extra', vector: vec(97), metadata: {} } + ]) + const beforeRestore = brain.generation() + + await expect( + brain.restore(snapDir, { confirm: false }) + ).rejects.toThrow('confirm') + + await brain.restore(snapDir, { confirm: true }) + expect(((await brain.get(uid('restore-e')))?.metadata as { v: number }).v).toBe(1) + expect(await brain.get(uid('restore-extra'))).toBeNull() + expect((await brain.find({ type: NounType.Document, limit: 10 })).length).toBe(1) + + // The counter never moves backwards — pre-restore generations are not reissued. + expect(brain.generation()).toBeGreaterThanOrEqual(beforeRestore) + const after = await brain.transact([{ op: 'update', id: uid('restore-e'), metadata: { v: 9 } }]) + expect(after.generation).toBeGreaterThan(beforeRestore) + await after.release() + }) + + it('Db values are first-class: Brainy.open() + chained with() overlays compose', async () => { + const brain = await Brainy.open({ requireSubtype: false, storage: { type: 'memory' } }) + brains.push(brain) + expect(brain.isInitialized).toBe(true) + + await brain.transact([ + { op: 'add', id: uid('chain-e'), type: NounType.Document, data: 'x', vector: vec(98), metadata: { v: 1 } } + ]) + const db = brain.now() + const spec1 = await db.with([{ op: 'update', id: uid('chain-e'), metadata: { v: 2 } }]) + const spec2 = await spec1.with([{ op: 'update', id: uid('chain-e'), metadata: { v: 3 } }]) + + expect(((await db.get(uid('chain-e')))?.metadata as { v: number }).v).toBe(1) + expect(((await spec1.get(uid('chain-e')))?.metadata as { v: number }).v).toBe(2) + expect(((await spec2.get(uid('chain-e')))?.metadata as { v: number }).v).toBe(3) + expect(spec2).toBeInstanceOf(Db) + + await spec2.release() + await spec1.release() + await db.release() + }) +}) diff --git a/tests/unit/db/generationStore.test.ts b/tests/unit/db/generationStore.test.ts new file mode 100644 index 00000000..f847b92a --- /dev/null +++ b/tests/unit/db/generationStore.test.ts @@ -0,0 +1,333 @@ +/** + * @module tests/unit/db/generationStore + * @description Unit tests for the generational MVCC record layer + * (`src/db/generationStore.ts`) against a real `MemoryStorage` adapter — + * counter monotonicity, the commit protocol (staging → execute → manifest + * rename → tx-log), point-in-time resolution from before-images, pin + * refcounting, compaction retention rules, CAS conflicts, timestamp + * resolution, and crash rollback of uncommitted generations. + */ + +import { describe, it, expect, beforeEach } from 'vitest' +import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js' +import { + GenerationStore, + GENERATIONS_PREFIX, + MANIFEST_PATH +} from '../../../src/db/generationStore.js' +import { + GenerationCompactedError, + GenerationConflictError +} from '../../../src/db/errors.js' +import { NounType } from '../../../src/types/graphTypes.js' + +/** Stored-metadata fixture in the canonical shape the live write paths use. */ +function metadataFixture(version: number): Record { + return { + noun: NounType.Document, + subtype: 'note', + data: `payload-v${version}`, + version, + createdAt: 1000, + updatedAt: 1000 + version, + _rev: version + } +} + +// Entity ids must be UUID-shaped (the sharded storage layout derives the +// shard from the UUID hex). Fixed values keep assertions deterministic; +// suffix ordering (…aa < …bb < …cc) matches `changedBetween`'s sorted output. +const ID_A = '00000000-0000-4000-8000-0000000000aa' +const ID_B = '00000000-0000-4000-8000-0000000000bb' +const ID_C = '00000000-0000-4000-8000-0000000000cc' +const ID_SOLO = '00000000-0000-4000-8000-0000000000d0' +const ID_LATER = '00000000-0000-4000-8000-0000000000e0' + +describe('db/GenerationStore', () => { + let storage: MemoryStorage + let store: GenerationStore + + beforeEach(async () => { + storage = new MemoryStorage() + await storage.init() + store = new GenerationStore(storage) + await store.open() + }) + + /** Commit one transaction that writes `metadata` for `id`. */ + async function commitWrite( + id: string, + version: number, + options?: { meta?: Record; ifAtGeneration?: number } + ): Promise<{ generation: number; timestamp: number }> { + return store.commitTransaction({ + touched: { nouns: [id], verbs: [] }, + meta: options?.meta, + ifAtGeneration: options?.ifAtGeneration, + execute: async () => { + await storage.saveNounMetadata(id, metadataFixture(version)) + } + }) + } + + describe('generation counter', () => { + it('starts at 0 on a fresh store', () => { + expect(store.generation()).toBe(0) + expect(store.committedGeneration()).toBe(0) + }) + + it('advances by exactly one per committed transaction', async () => { + await commitWrite(ID_A, 1) + expect(store.generation()).toBe(1) + await commitWrite(ID_A, 2) + expect(store.generation()).toBe(2) + expect(store.committedGeneration()).toBe(2) + }) + + it('bumps on single-operation writes via the storage hook (suppressed inside transact)', async () => { + // Single-op write outside any transaction → hook bumps. + await storage.saveNounMetadata(ID_SOLO, metadataFixture(1)) + expect(store.generation()).toBe(1) + expect(store.committedGeneration()).toBe(0) // no transact committed + + // A transact batch writing twice bumps ONCE (the batch is one generation). + await store.commitTransaction({ + touched: { nouns: [ID_A, ID_B], verbs: [] }, + execute: async () => { + await storage.saveNounMetadata(ID_A, metadataFixture(1)) + await storage.saveNounMetadata(ID_B, metadataFixture(1)) + } + }) + expect(store.generation()).toBe(2) + }) + + it('survives close + reopen monotonically', async () => { + await commitWrite(ID_A, 1) + await storage.saveNounMetadata(ID_SOLO, metadataFixture(1)) // single-op bump + const before = store.generation() + await store.close() + + const reopened = new GenerationStore(storage) + await reopened.open() + expect(reopened.generation()).toBeGreaterThanOrEqual(before) + expect(reopened.committedGeneration()).toBe(1) + }) + }) + + describe('CAS (ifAtGeneration)', () => { + it('commits when the expectation matches and rejects with expected/actual when stale', async () => { + await commitWrite(ID_A, 1) + const pinned = store.generation() + + await commitWrite(ID_A, 2, { ifAtGeneration: pinned }) // fresh → commits + expect(store.generation()).toBe(pinned + 1) + + try { + await commitWrite(ID_A, 3, { ifAtGeneration: pinned }) // stale → rejects + expect.unreachable('should have thrown GenerationConflictError') + } catch (err) { + expect(err).toBeInstanceOf(GenerationConflictError) + expect((err as GenerationConflictError).expected).toBe(pinned) + expect((err as GenerationConflictError).actual).toBe(pinned + 1) + } + expect(store.generation()).toBe(pinned + 1) // nothing moved + }) + }) + + describe('failed transactions', () => { + it('removes the staging directory and returns the generation reservation', async () => { + await commitWrite(ID_A, 1) + const before = store.generation() + + await expect( + store.commitTransaction({ + touched: { nouns: [ID_A], verbs: [] }, + execute: async () => { + throw new Error('boom mid-batch') + } + }) + ).rejects.toThrow('boom mid-batch') + + expect(store.generation()).toBe(before) + const leftovers = await storage.listRawObjects(`${GENERATIONS_PREFIX}/${before + 1}`) + expect(leftovers).toEqual([]) + }) + }) + + describe('point-in-time resolution', () => { + it('resolves untouched ids to the live fast path', async () => { + await commitWrite(ID_A, 1) + const pinned = store.generation() + await commitWrite(ID_B, 1) // later commit touches a DIFFERENT id + + expect(await store.resolveAt('noun', ID_A, pinned)).toEqual({ source: 'current' }) + }) + + it('resolves changed ids from the first before-image after the pin', async () => { + await commitWrite(ID_A, 1) + const pinned = store.generation() + await commitWrite(ID_A, 2) + await commitWrite(ID_A, 3) + + const resolved = await store.resolveAt('noun', ID_A, pinned) + expect(resolved.source).toBe('record') + if (resolved.source === 'record') { + expect(resolved.metadata.version).toBe(1) // state as of the pin + } + }) + + it('resolves ids created after the pin as absent', async () => { + await commitWrite(ID_A, 1) + const pinned = store.generation() + await commitWrite(ID_LATER, 1) + + expect(await store.resolveAt('noun', ID_LATER, pinned)).toEqual({ source: 'absent' }) + }) + + it('changedBetween reports the ids touched in the interval', async () => { + await commitWrite(ID_A, 1) + const from = store.generation() + await commitWrite(ID_B, 1) + await commitWrite(ID_C, 1) + const to = store.generation() + + const changed = await store.changedBetween(from, to) + expect(changed.nouns).toEqual([ID_B, ID_C]) + expect(changed.verbs).toEqual([]) + expect(changed.fromGeneration).toBe(from) + expect(changed.toGeneration).toBe(to) + }) + }) + + describe('timestamp resolution + tx-log', () => { + it('records meta in the tx-log and resolves timestamps to generations', async () => { + const t0 = Date.now() - 10 + const first = await commitWrite(ID_A, 1, { meta: { author: 'unit-test' } }) + // Distinct commit timestamps: resolution picks the NEWEST generation + // committed at-or-before the instant, so same-millisecond commits tie. + await new Promise((resolve) => setTimeout(resolve, 5)) + const second = await commitWrite(ID_A, 2) + + const lines = await storage.readTxLogLines() + expect(lines.length).toBe(2) + const firstEntry = JSON.parse(lines[0]) + expect(firstEntry.generation).toBe(first.generation) + expect(firstEntry.meta).toEqual({ author: 'unit-test' }) + + expect((await store.resolveTimestamp(t0)).generation).toBe(0) + expect((await store.resolveTimestamp(first.timestamp)).generation).toBe(first.generation) + expect((await store.resolveTimestamp(Date.now() + 1000)).generation).toBe( + second.generation + ) + }) + }) + + describe('pins + compaction', () => { + it('refcounts pins and never reclaims records a live pin needs', async () => { + await commitWrite(ID_A, 1) + const pinned = store.generation() + store.pin(pinned) + store.pin(pinned) // refcount 2 + await commitWrite(ID_A, 2) + await commitWrite(ID_A, 3) + + // Pin at G protects every record-set ABOVE G (resolution reads + // before-images from generations strictly greater than the pin). + await store.compact() + const resolved = await store.resolveAt('noun', ID_A, pinned) + expect(resolved.source).toBe('record') + if (resolved.source === 'record') { + expect(resolved.metadata.version).toBe(1) + } + + store.release(pinned) + await store.compact() + // Still one live pin — records above it still resolve correctly. + const stillResolved = await store.resolveAt('noun', ID_A, pinned) + expect(stillResolved.source).toBe('record') + + store.release(pinned) + const result = await store.compact() + expect(result.removedGenerations).toBeGreaterThan(0) + const remaining = await storage.listRawObjects(GENERATIONS_PREFIX) + expect(remaining).toEqual([]) + }) + + it('retainGenerations keeps the most recent record-sets', async () => { + await commitWrite(ID_A, 1) + await commitWrite(ID_A, 2) + await commitWrite(ID_A, 3) + + const result = await store.compact({ retainGenerations: 2 }) + expect(result.removedGenerations).toBe(1) + expect(result.horizon).toBe(1) + + // Removing record-set N breaks pins BELOW N (their reads need N's + // before-images) — the horizon itself stays reachable: state at the + // horizon resolves from the retained record-sets above it. + expect(() => store.assertReachable(0)).toThrow(GenerationCompactedError) + expect(store.assertReachable(1)).toBe(1) + expect(store.assertReachable(2)).toBe(2) + }) + + it('rejects future and negative generations', async () => { + await commitWrite(ID_A, 1) + expect(() => store.assertReachable(99)).toThrow(RangeError) + expect(() => store.assertReachable(-1)).toThrow(RangeError) + expect(() => store.assertReachable(1.5)).toThrow(RangeError) + }) + }) + + describe('crash recovery', () => { + it('rolls back an uncommitted generation on open (manifest rename is the commit point)', async () => { + await commitWrite(ID_A, 1) + + // Simulate a crash between record staging and the manifest rename: a + // throwing fault injector skips the abort cleanup, exactly as a dead + // process would, leaving canonical files mutated and the staging + // directory present — but the manifest still at the prior generation. + store.setCommitFaultInjector((phase) => { + if (phase === 'before-manifest-rename') { + throw new Error('simulated crash') + } + }) + await expect(commitWrite(ID_A, 2)).rejects.toThrow('simulated crash') + store.setCommitFaultInjector(undefined) + + // The canonical file currently holds the EXECUTED (uncommitted) state. + const dirty = await storage.readNounRaw(ID_A) + expect((dirty.metadata as { version: number }).version).toBe(2) + + // Recovery on the next open restores the before-image byte-identically. + const reopened = new GenerationStore(storage) + const openResult = await reopened.open() + expect(openResult.rolledBackGenerations).toBe(1) + const repaired = await storage.readNounRaw(ID_A) + expect((repaired.metadata as { version: number }).version).toBe(1) + expect(reopened.committedGeneration()).toBe(1) + // The crashed generation number is never reused. + expect(reopened.generation()).toBeGreaterThanOrEqual(2) + }) + + it('keeps a transaction whose manifest rename landed (tx-log append is advisory)', async () => { + await commitWrite(ID_A, 1) + store.setCommitFaultInjector((phase) => { + if (phase === 'after-manifest-rename') { + throw new Error('simulated crash after commit point') + } + }) + await expect(commitWrite(ID_A, 2)).rejects.toThrow('simulated crash after commit point') + store.setCommitFaultInjector(undefined) + + const reopened = new GenerationStore(storage) + const openResult = await reopened.open() + expect(openResult.rolledBackGenerations).toBe(0) + expect(reopened.committedGeneration()).toBe(2) + const kept = await storage.readNounRaw(ID_A) + expect((kept.metadata as { version: number }).version).toBe(2) + + const manifest = await storage.readRawObject(MANIFEST_PATH) + expect(manifest.generation).toBe(2) + }) + }) +}) diff --git a/tests/unit/db/whereMatcher.test.ts b/tests/unit/db/whereMatcher.test.ts new file mode 100644 index 00000000..42f88a7a --- /dev/null +++ b/tests/unit/db/whereMatcher.test.ts @@ -0,0 +1,223 @@ +/** + * @module tests/unit/db/whereMatcher + * @description Unit tests for the in-memory `find()` filter evaluator behind + * historical and speculative `Db` reads (`src/db/whereMatcher.ts`). The + * evaluator must mirror the metadata index's operator semantics exactly — an + * entity matches in-memory if and only if it would have matched through the + * index — and must throw `UnsupportedWhereOperatorError` for anything it + * does not recognize (never guess on a historical read). + */ + +import { describe, it, expect } from 'vitest' +import { + entityMatchesFind, + resolveEntityField, + whereMatches, + UnsupportedWhereOperatorError +} from '../../../src/db/whereMatcher.js' +import type { Entity } from '../../../src/types/brainy.types.js' +import { NounType } from '../../../src/types/graphTypes.js' + +/** Build a minimal entity for matcher tests. */ +function entity(overrides: Partial = {}): Entity { + return { + id: 'e-1', + vector: [], + type: NounType.Document, + createdAt: 1000, + updatedAt: 2000, + metadata: {}, + ...overrides + } +} + +describe('db/whereMatcher — resolveEntityField', () => { + it('resolves standard top-level fields', () => { + const e = entity({ + subtype: 'invoice', + service: 'billing', + confidence: 0.9, + weight: 0.5, + _rev: 3, + data: 'payload' + }) + expect(resolveEntityField(e, 'id')).toBe('e-1') + expect(resolveEntityField(e, 'type')).toBe(NounType.Document) + expect(resolveEntityField(e, 'noun')).toBe(NounType.Document) // alias + expect(resolveEntityField(e, 'subtype')).toBe('invoice') + expect(resolveEntityField(e, 'service')).toBe('billing') + expect(resolveEntityField(e, 'confidence')).toBe(0.9) + expect(resolveEntityField(e, 'weight')).toBe(0.5) + expect(resolveEntityField(e, '_rev')).toBe(3) + expect(resolveEntityField(e, 'createdAt')).toBe(1000) + expect(resolveEntityField(e, 'updatedAt')).toBe(2000) + expect(resolveEntityField(e, 'data')).toBe('payload') + }) + + it('resolves custom fields from the metadata bag', () => { + const e = entity({ metadata: { status: 'open', priority: 2 } }) + expect(resolveEntityField(e, 'status')).toBe('open') + expect(resolveEntityField(e, 'priority')).toBe(2) + expect(resolveEntityField(e, 'absent')).toBeUndefined() + }) + + it('resolves dotted paths against the entity and the metadata bag', () => { + const e = entity({ metadata: { address: { city: 'Lyon' }, priority: 7 } }) + expect(resolveEntityField(e, 'metadata.priority')).toBe(7) + expect(resolveEntityField(e, 'address.city')).toBe('Lyon') + expect(resolveEntityField(e, 'address.zip')).toBeUndefined() + }) +}) + +describe('db/whereMatcher — operators', () => { + const e = entity({ + subtype: 'invoice', + metadata: { amount: 250, status: 'open', tags: ['urgent', 'q3'], city: 'Lyon' } + }) + + it('shorthand equality', () => { + expect(whereMatches(e, { status: 'open' })).toBe(true) + expect(whereMatches(e, { status: 'closed' })).toBe(false) + }) + + it('eq / equals / is aliases', () => { + expect(whereMatches(e, { amount: { eq: 250 } })).toBe(true) + expect(whereMatches(e, { amount: { equals: 250 } })).toBe(true) + expect(whereMatches(e, { amount: { is: 250 } })).toBe(true) + expect(whereMatches(e, { amount: { eq: 99 } })).toBe(false) + }) + + it('array-membership equality (index posting semantics)', () => { + expect(whereMatches(e, { tags: 'urgent' })).toBe(true) + expect(whereMatches(e, { tags: { contains: 'q3' } })).toBe(true) + expect(whereMatches(e, { tags: 'missing-tag' })).toBe(false) + }) + + it('ne / notEquals / isNot aliases', () => { + expect(whereMatches(e, { status: { ne: 'closed' } })).toBe(true) + expect(whereMatches(e, { status: { notEquals: 'open' } })).toBe(false) + expect(whereMatches(e, { status: { isNot: 'open' } })).toBe(false) + }) + + it('in / oneOf set membership', () => { + expect(whereMatches(e, { status: { in: ['open', 'closed'] } })).toBe(true) + expect(whereMatches(e, { status: { oneOf: ['archived'] } })).toBe(false) + }) + + it('numeric range operators with aliases', () => { + expect(whereMatches(e, { amount: { gt: 200 } })).toBe(true) + expect(whereMatches(e, { amount: { greaterThan: 250 } })).toBe(false) + expect(whereMatches(e, { amount: { gte: 250 } })).toBe(true) + expect(whereMatches(e, { amount: { greaterThanOrEqual: 251 } })).toBe(false) + expect(whereMatches(e, { amount: { greaterEqual: 250 } })).toBe(true) + expect(whereMatches(e, { amount: { lt: 251 } })).toBe(true) + expect(whereMatches(e, { amount: { lessThan: 250 } })).toBe(false) + expect(whereMatches(e, { amount: { lte: 250 } })).toBe(true) + expect(whereMatches(e, { amount: { lessThanOrEqual: 249 } })).toBe(false) + expect(whereMatches(e, { amount: { lessEqual: 250 } })).toBe(true) + }) + + it('string range comparison is lexicographic', () => { + expect(whereMatches(e, { city: { gt: 'Aix' } })).toBe(true) + expect(whereMatches(e, { city: { lt: 'Aix' } })).toBe(false) + }) + + it('mixed-type range comparison never matches', () => { + expect(whereMatches(e, { city: { gt: 5 } })).toBe(false) + expect(whereMatches(e, { amount: { lt: 'zzz' } })).toBe(false) + }) + + it('between', () => { + expect(whereMatches(e, { amount: { between: [200, 300] } })).toBe(true) + expect(whereMatches(e, { amount: { between: [251, 300] } })).toBe(false) + expect(whereMatches(e, { amount: { between: [200] } })).toBe(false) // malformed operand + }) + + it('exists / missing', () => { + expect(whereMatches(e, { status: { exists: true } })).toBe(true) + expect(whereMatches(e, { nope: { exists: true } })).toBe(false) + expect(whereMatches(e, { nope: { exists: false } })).toBe(true) + expect(whereMatches(e, { nope: { missing: true } })).toBe(true) + expect(whereMatches(e, { status: { missing: true } })).toBe(false) + expect(whereMatches(e, { status: { missing: false } })).toBe(true) + }) + + it('multiple operators on one field AND together', () => { + expect(whereMatches(e, { amount: { gte: 200, lte: 300 } })).toBe(true) + expect(whereMatches(e, { amount: { gte: 200, lte: 249 } })).toBe(false) + }) + + it('allOf / anyOf / not logical composition', () => { + expect(whereMatches(e, { allOf: [{ status: 'open' }, { amount: { gt: 100 } }] })).toBe(true) + expect(whereMatches(e, { allOf: [{ status: 'open' }, { amount: { gt: 999 } }] })).toBe(false) + expect(whereMatches(e, { anyOf: [{ status: 'closed' }, { amount: 250 }] })).toBe(true) + expect(whereMatches(e, { anyOf: [{ status: 'closed' }, { amount: 9 }] })).toBe(false) + expect(whereMatches(e, { not: { status: 'closed' } })).toBe(true) + expect(whereMatches(e, { not: { status: 'open' } })).toBe(false) + }) + + it('throws UnsupportedWhereOperatorError for unknown operators — never guesses', () => { + expect(() => whereMatches(e, { amount: { approximately: 250 } })).toThrow( + UnsupportedWhereOperatorError + ) + try { + whereMatches(e, { amount: { approximately: 250 } }) + expect.unreachable('should have thrown') + } catch (err) { + expect(err).toBeInstanceOf(UnsupportedWhereOperatorError) + expect((err as UnsupportedWhereOperatorError).operator).toBe('approximately') + } + }) +}) + +describe('db/whereMatcher — entityMatchesFind', () => { + const e = entity({ + subtype: 'invoice', + service: 'billing', + metadata: { amount: 250 } + }) + + it('filters by type (single + array)', () => { + expect(entityMatchesFind(e, { type: NounType.Document })).toBe(true) + expect(entityMatchesFind(e, { type: [NounType.Person, NounType.Document] })).toBe(true) + expect(entityMatchesFind(e, { type: NounType.Person })).toBe(false) + }) + + it('filters by subtype (single + array; entities without subtype excluded)', () => { + expect(entityMatchesFind(e, { subtype: 'invoice' })).toBe(true) + expect(entityMatchesFind(e, { subtype: ['invoice', 'receipt'] })).toBe(true) + expect(entityMatchesFind(e, { subtype: 'receipt' })).toBe(false) + expect(entityMatchesFind(entity(), { subtype: 'invoice' })).toBe(false) + }) + + it('filters by service', () => { + expect(entityMatchesFind(e, { service: 'billing' })).toBe(true) + expect(entityMatchesFind(e, { service: 'crm' })).toBe(false) + }) + + it('excludeVFS drops VFS-marked entities', () => { + const vfsEntity = entity({ metadata: { vfsType: 'file' } }) + const markedEntity = entity({ metadata: { isVFSEntity: true } }) + expect(entityMatchesFind(vfsEntity, { excludeVFS: true })).toBe(false) + expect(entityMatchesFind(markedEntity, { excludeVFS: true })).toBe(false) + expect(entityMatchesFind(e, { excludeVFS: true })).toBe(true) + expect(entityMatchesFind(vfsEntity, {})).toBe(true) + }) + + it('composes type + subtype + where', () => { + expect( + entityMatchesFind(e, { + type: NounType.Document, + subtype: 'invoice', + where: { amount: { gte: 200 } } + }) + ).toBe(true) + expect( + entityMatchesFind(e, { + type: NounType.Document, + subtype: 'invoice', + where: { amount: { gte: 999 } } + }) + ).toBe(false) + }) +})