feat(8.0): generational MVCC storage + Datomic-style Db API (now/transact/asOf/with/persist)

One mechanism replaces the COW and versioning subsystems: immutable
generation-stamped records behind a Datomic-style database value.

Record layer (src/db/generationStore.ts):
- Monotonic generation counter in _system/generation.json; bumped once per
  transact() commit and once per single-operation write (storage hook), so
  brain.generation() is always a meaningful watermark.
- Commit protocol: stage before-images + tx.json delta -> fsync -> execute
  batch via TransactionManager -> atomic tmp+rename of _system/manifest.json
  (the rename IS the commit point) -> append _system/tx-log.jsonl.
- Crash recovery on open rolls uncommitted generations back byte-identically
  and forces an index rebuild; refcounted pins gate compactHistory(), which
  records a horizon (asOf below it throws GenerationCompactedError).

Db API:
- Db: get/find/search/related pinned at a generation, with() speculative
  overlays, since() diffs, persist() hard-link-farm snapshots, timestamp,
  generation, release() + FinalizationRegistry backstop.
- Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch
  (GenerationConflictError CAS), asOf(generation|Date|path), restore(path,
  {confirm}), compactHistory(), generation(), static open() + load().
- get()/metadata find()/related() are fully correct at any reachable pinned
  generation; index-accelerated queries at historical generations throw
  NotYetSupportedAtHistoricalGenerationError - never silently-wrong results.
- VersionedIndexProvider (generation/isGenerationVisible/pin/release) in
  plugin.ts: feature-detected, balanced pin/release in lockstep with Db
  lifecycle, post-commit applier + replay-gap model documented.

Storage primitives (BaseStorage + filesystem/memory adapters): raw-object
read/write/list/remove, fsync barrier, noun/verb raw before-image capture,
tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and
append-in-place exceptions), restoreFromDirectory + derived-state reload.

Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation
across 200 mutations, batch atomicity under injected execution failure,
ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety
under pins, with() overlay isolation, generation monotonicity across
reopen, crash consistency through the real recovery path, and versioned
provider pin/release balance. Design record in
docs/ADR-001-generational-mvcc.md.
This commit is contained in:
David Snelling 2026-06-10 14:14:07 -07:00
parent 49e49483d1
commit 431cd64406
16 changed files with 6244 additions and 5 deletions

View file

@ -0,0 +1,262 @@
# ADR-001: Generational MVCC storage and the immutable Db API
**Status:** Accepted (ships in 8.0)
**Date:** 2026-06-10
## Context
Before 8.0, Brainy carried two overlapping version-control subsystems: a
copy-on-write branching layer (`fork`/`checkout`/`commit`/`branches`) and a
separate versioning subsystem (`versions.save/list/compare/restore/prune`),
plus a read-only historical adapter for commit-based time travel. Together
they were ~5,100 LOC of mechanism for one product need: *read a consistent
past state while the store keeps moving, and snapshot/restore cheaply.*
Neither subsystem gave a precise isolation guarantee. Reads raced in-place
JSON overwrites, so a "snapshot" was only as immutable as the bytes it
happened to share with the live store.
8.0 replaces both with **one mechanism**: generational MVCC over immutable,
generation-stamped records, exposed through a Datomic-style immutable
database value (`Db`). The same model is implemented natively by versioned
index providers (LSM snapshots), so semantics are identical on the pure-JS
path and the native path.
## Decision
### The model
- A **monotonic u64 generation counter** is the store's logical clock. It
advances once per committed `transact()` batch and once per
single-operation write (`add`/`update`/`delete`/`relate`/…), so
`brain.generation()` is always a meaningful watermark. It is persisted in
`_system/generation.json` and never reissued for anything durable.
- `brain.now()` **pins** the current generation in O(1) and returns a `Db`
an immutable view. Pins are refcounted; `db.release()` (with a
`FinalizationRegistry` backstop for leaked values) ends the pin.
- `brain.transact(ops, { meta, ifAtGeneration })` commits a declarative
batch atomically as **exactly one generation**, with whole-store
compare-and-swap (`ifAtGeneration``GenerationConflictError`) and
reified transaction metadata appended to `_system/tx-log.jsonl`.
- `brain.asOf(generation | Date | snapshotPath)` opens past state;
`db.with(ops)` layers a speculative in-memory overlay (never touching
disk, the counter, or index providers); `db.persist(path)` cuts an
instant snapshot; `brain.restore(path, { confirm: true })` replaces state
from one; `Brainy.load(path)` opens a snapshot read-only with the full
query surface.
### Persisted layout
All paths are storage-root-relative:
```
_system/generation.json { generation, updatedAt } atomic tmp+rename
_system/manifest.json { version, generation, atomic tmp+rename
committedAt, horizon } (the commit point)
_system/tx-log.jsonl one line per committed append-only
transact: { generation,
timestamp, meta? }
_generations/<N>/tx.json the generation-N delta: immutable
touched noun/verb ids + meta
_generations/<N>/prev/<id>.json before-image of <id> as of immutable
commit N (raw stored bytes;
null parts = file was absent)
```
**Why per-generation deltas instead of a global `id → latest generation`
map in the manifest:** a global map makes every commit O(all ids) — the
whole map must be rewritten to swap it atomically. The delta layout makes a
commit O(ids touched) and keeps the manifest a fixed-size watermark, while
point-in-time resolution stays correct (see "Read resolution" below). The
trade is that resolution at a pinned generation scans the deltas of later
commits — bounded by the number of commits since the pin, which is exactly
the window compaction keeps short.
Before-images are deliberately the *only* per-id records. They serve both
roles the layer needs — the crash-recovery undo log and the point-in-time
read source. After-images would duplicate state that is already readable
(the canonical entity files hold the latest bytes; earlier states resolve
from later before-images) and would double record I/O per commit.
### Commit protocol (durability)
`transact()` commits under a store-wide mutex:
1. **CAS check.** A stale `ifAtGeneration` throws `GenerationConflictError`
before anything is staged.
2. **Reserve** generation `N` (counter increment).
3. **Stage the undo log:** write the before-image of every touched id plus
`tx.json` under `_generations/N/`, then **fsync** the files and their
directories. From this point, any crash is recoverable to the exact
pre-transaction bytes.
4. **Execute** the planned batch through the TransactionManager (which has
its own operation-level rollback for in-flight failures).
5. **Commit point:** persist the counter, then write `_system/manifest.json`
via atomic tmp+rename and fsync it. The rename *is* the commit: a
generation directory is committed if and only if `N ≤
manifest.generation`.
6. Append the tx-log line (advisory metadata — a crash between 5 and 6
keeps the transaction).
**Crash recovery (on open):** any `_generations/<N>` directory with
`N > manifest.generation` is an uncommitted transaction. Its before-images
are restored to the canonical paths (idempotently — recovery itself can
crash and rerun) and the directory is removed. Because recovery runs before
any index is built, and a recovery that rolled something back forces a full
index rebuild, derived indexes never observe rolled-back state. Reader-mode
instances skip recovery (readers never write; the next writer repairs).
A failed (non-crash) transaction takes the same staging directory down the
abort path: the TransactionManager rolls back applied operations, the
staging directory is removed, and the generation reservation is returned —
a failed batch leaves the generation counter unchanged.
### Read resolution at a pinned generation
The state of id X at pinned generation G is:
- the before-image stored by the **first committed generation after G that
touched X**, or
- the live canonical bytes, when nothing after G touched X.
While nothing has committed past G, *every* read on the `Db` delegates to
the live fast paths untouched — `now()` adds no read overhead until history
actually moves.
**Honesty boundary.** `get()`, metadata-level `find()`, and `related()` are
fully correct at any reachable pinned generation. Index-accelerated
dimensions (semantic/vector search, graph traversal, cursors, aggregation)
are served by live indexes that represent only the current generation, so
at a historical generation they throw
`NotYetSupportedAtHistoricalGenerationError` — never silently-wrong
results. The escape hatch is `persist()` + `Brainy.load()`, which rebuilds
indexes from the snapshot and supports the full query surface. Rebuilding
indexes from a pinned record set in-place (the standard LSM answer) is the
documented follow-up; versioned native providers already expose
`isGenerationVisible()` for provider-accelerated historical reads.
**History granularity.** Generation *records* are written per `transact()`
batch only. Single-operation writes advance the counter (so watermarks and
CAS stay sound) but do not stage before-images: they remain visible through
earlier pins and are not reported by `db.since()`. Code that needs pinned
isolation across its writes uses `transact()` — that is the documented 8.0
contract, stated rather than papered over.
### Pinning, retention, compaction
- Each live `Db` holds one refcounted pin on its generation (plus a
`pin(generation)` on every registered `VersionedIndexProvider`, whose
explicit pin lifetime overrides any time-based snapshot retention the
provider has).
- `compactHistory({ retainGenerations?, retainMs? })` reclaims record-sets.
A record-set `N` is reclaimed only when `N` is at or below **every** live
pin — deleting `N` can only break readers pinned *below* `N`, because
resolution reads before-images from generations strictly greater than the
pin. With both retention options supplied, both must allow the reclaim.
- The manifest records the **horizon** (highest reclaimed generation).
Generations below the horizon are unreachable; `asOf()` on them throws
`GenerationCompactedError`. The horizon itself stays reachable, resolved
from the record-sets above it. To keep a generation readable forever,
`persist()` it first — snapshots are self-contained.
### Snapshots and restore
`db.persist(path)` flushes indexes, then cuts the snapshot under the
store's commit mutex (no commit, compaction, or counter write can
interleave). On filesystem storage it is a **hard-link farm**: every data
file is immutable-by-rename, so linking is safe — later rewrites swap
inodes and the snapshot keeps the old bytes. The two exceptions are handled
explicitly: the append-in-place tx-log is byte-copied, and process-local
lock state is excluded. Cross-device targets (and filesystems that refuse
links) fall back to per-file byte copies. In-memory stores serialize to the
same directory layout, so persisting a memory brain produces a real,
durable, loadable store.
`persist()` requires the view to still be the store's latest generation
(a snapshot captures current bytes); a view that history has moved past
throws rather than persisting the wrong state.
`restore(path, { confirm: true })` replaces the store's contents from a
snapshot via byte copy (never links — the snapshot stays independent),
reloads all adapter-internal derived state, rebuilds all indexes, and
floors the generation counter at its pre-restore value so observed
generation numbers are never reissued. Live pins do not survive a restore;
a warning is logged when any exist.
### Versioned index providers
Native index providers may implement the optional 4-method
`VersionedIndexProvider` capability (`generation()`,
`isGenerationVisible()`, `pin()`, `release()` — BigInt generations at the
boundary). The locked consistency model: providers are **post-commit
appliers**. The storage-record commit is the source of truth; provider
index state is derived. On open, a provider behind the committed watermark
replays the gap from storage (or requests a rebuild) — there are no
provider rollback hooks, because uncommitted transactions are repaired at
the storage layer before any index opens. Speculative `with()` overlays
never reach providers.
## Guarantees (and their proofs)
Each stated guarantee has a test that proves it, not merely exercises it
(`tests/integration/db-mvcc.test.ts`, plus
`tests/unit/db/generationStore.test.ts` for the record layer in isolation):
| Guarantee | Proof |
|---|---|
| Snapshot isolation: a pinned `Db` reads exactly its pinned state, forever | proof 1 (200 mutations, including deletes, against a pinned view) |
| Atomicity: a failing batch applies nothing; generation unchanged | proofs 2a/2b/2c (plan-time failure, injected execution-phase storage failure, `ifRev` conflict) |
| Whole-store CAS | proof 3 (`ifAtGeneration` success + conflict with exact expected/actual) |
| Snapshot integrity under source mutation (hard-link safety) | proofs 4a/4b/4c |
| Compaction never breaks a pinned read; release enables reclaim | proof 5 |
| `with()` overlays touch nothing durable | proof 6 |
| Generation monotonicity across close/reopen | proof 7 |
| Crash before the manifest rename recovers to exact pre-transaction state through the real recovery path | proof 8 (fault injection that skips abort cleanup, exactly as a dead process would) |
| Balanced provider pin/release lockstep | proof 9 |
One deliberate softness: single-operation generation bumps persist the
counter coalesced (per write burst), not per write. Durable artifacts —
records, manifests, snapshots — always persist the counter synchronously at
their own commit points, so a crash inside the coalescing window can lose
only counter values that nothing durable ever referenced.
## Failure modes
| Failure | Outcome |
|---|---|
| Crash before staging completes | Partial staging directory > manifest watermark → removed on next open; canonical state untouched. |
| Crash after staging, before/during batch execution | Before-images restored on next open; indexes rebuilt; byte-identical pre-transaction state. |
| Crash after execution, before manifest rename | Same as above — the rename is the only commit point. |
| Crash after manifest rename, before tx-log append | Transaction kept (committed); tx-log misses one advisory line; `asOf(Date)` resolution for that commit falls back to neighboring entries. |
| Batch fails mid-execution (no crash) | TransactionManager operation rollback + staging-directory removal + reservation return; generation unchanged. |
| `asOf()` below the compaction horizon | `GenerationCompactedError` — explicit, never partial data. |
| Index-accelerated query at a historical pin | `NotYetSupportedAtHistoricalGenerationError` — explicit, never silently-wrong results. |
| Torn trailing tx-log line (crashed append) | Tolerated; unparseable lines are skipped by readers. |
## Lineage
The design is an assembly of well-understood prior art, chosen for being
boring where it counts:
- **Datomic** — the database-as-a-value: an immutable `Db` you query, with
`with()` for speculation and reified transaction metadata instead of
commit messages.
- **LMDB** — reader pins: readers never block writers; a reader's view
stays valid because nothing overwrites the pages (here: records) it
references; reclamation waits for the last reader.
- **LSM trees / Cassandra** — immutable segments make snapshots hard links
and make compaction a retention policy instead of a locking problem.
## Consequences
- One mechanism replaces the COW and versioning subsystems (their removal
is the companion change to this ADR).
- In-place branch switching (`checkout`) is gone by design; the replacement
is opening a persisted snapshot as a separate instance — a name→path
mapping where a product needs named branches.
- Every commit pays O(ids touched) extra writes (before-images + delta +
manifest). Single-operation writes pay only an in-memory counter bump
with coalesced persistence.
- Historical index-accelerated queries are an explicit error until the
asOf-rebuild ships; correctness-critical historical reads (`get`,
metadata `find`, `related`) work today and forever at any reachable pin.

File diff suppressed because it is too large Load diff

806
src/db/db.ts Normal file
View file

@ -0,0 +1,806 @@
/**
* @module db/db
* @description `Db` Brainy 8.0's immutable, Datomic-style database value.
*
* A `Db` is a **readonly view pinned at one generation** of a Brainy store.
* It is produced by `brain.now()` (O(1) pin of the current generation),
* `brain.transact()` (pinned at the freshly committed generation),
* `brain.asOf()` (a past generation, a timestamp, or a persisted snapshot
* path), `Brainy.load(path)` (a snapshot opened read-only), and `db.with()`
* (a speculative in-memory overlay).
*
* **Read semantics.** While no transaction has committed past the pinned
* generation, every read delegates straight to the live brain the pinned
* view IS the current state, and the existing fast paths serve it untouched.
* Once later transactions commit, the `Db` resolves reads through the
* generational record layer: `get()` and metadata-level `find()`/`related()`
* remain fully correct at the pinned generation (changed ids are resolved
* from immutable before-images; unchanged ids still ride the live fast
* path), while index-accelerated queries (semantic/vector search) throw the
* documented {@link NotYetSupportedAtHistoricalGenerationError} an honest
* boundary, never silently-wrong results.
*
* **History granularity.** Generation records are written per `transact()`
* batch. Single-operation writes (`add`/`update`/`relate`/ outside
* `transact()`) advance the generation counter but do not produce historical
* records, so they remain visible through earlier pins the documented 8.0
* granularity (see `docs/ADR-001-generational-mvcc.md`, "History
* granularity").
*
* **Lifecycle.** Each `Db` holds one refcounted pin on its generation (and
* on every registered {@link ../plugin.js VersionedIndexProvider}). Call
* {@link Db.release} when done a `FinalizationRegistry` backstop releases
* leaked pins on garbage collection, but explicit release is what makes
* `compactHistory()` deterministic.
*/
import type {
Entity,
FindParams,
GetOptions,
GetRelationsParams,
Relation,
Result
} from '../types/brainy.types.js'
import { v4 as uuidv4 } from '../universal/uuid.js'
import { NotYetSupportedAtHistoricalGenerationError } from './errors.js'
import type { GenerationStore } from './generationStore.js'
import type { ChangedIds, TransactReceipt, TxOperation } from './types.js'
import { entityMatchesFind, resolveEntityField, UnsupportedWhereOperatorError } from './whereMatcher.js'
/**
* @description The speculative overlay carried by a `db.with()` value:
* per-id replacement entities/relations (`null` = tombstone). Reads on the
* overlay `Db` consult these maps first, then fall through to the pinned
* base view. Overlays are pure in-memory values they never touch disk or
* index providers.
*/
export interface SpeculativeOverlay<T = any> {
/** Entity overlays: replacement entity, or `null` for a speculative delete. */
nouns: Map<string, Entity<T> | null>
/** Relation overlays: replacement relation, or `null` for a speculative delete. */
verbs: Map<string, Relation<T> | null>
}
/**
* @description The host surface a `Db` needs from its `Brainy` instance
* constructed once per brain (see `brainy.ts`). Internal: consumers never
* build one; `Db` values are only created by `Brainy` and `db.with()`.
*/
export interface DbHost<T = any> {
/** The generational record layer of the host brain. */
readonly store: GenerationStore
/** Live `brain.get()` (current-generation fast path). */
get(id: string, options?: GetOptions): Promise<Entity<T> | null>
/** Live `brain.find()` (current-generation fast path). */
find(query: string | FindParams<T>): Promise<Result<T>[]>
/** Live `brain.getRelations()` (current-generation fast path). */
getRelations(paramsOrId?: string | GetRelationsParams): Promise<Relation<T>[]>
/** Materialize an entity from a generation record's raw stored objects. */
entityFromRecord(
id: string,
record: { metadata: any; vector: any | null },
includeVectors: boolean
): Promise<Entity<T>>
/** Materialize a relation from a generation record's raw stored objects. */
relationFromRecord(id: string, record: { metadata: any; vector: any | null }): Relation<T> | null
/** Snapshot the store at `generation` into `targetPath` (throws if the pin is stale). */
persistPinned(targetPath: string, generation: number): Promise<void>
/** Add one refcounted pin (store + versioned providers) on `generation`. */
pinGeneration(generation: number): void
/** Release one refcounted pin (store + versioned providers) on `generation`. */
releaseGeneration(generation: number): void
/** Register a `Db` with the GC-backstop finalization registry. */
registerDbForFinalization(db: Db<T>, generation: number, closeOnRelease?: () => Promise<void>): void
/** Unregister a `Db` after an explicit release (no double-release on GC). */
unregisterDbFromFinalization(db: Db<T>): void
}
/**
* @description Internal constructor arguments for {@link Db}. The pin on
* `generation` is taken by the creator (`Brainy` or `db.with()`) **before**
* construction; the constructor only registers the GC backstop.
*/
export interface DbInit<T = any> {
/** The host surface (one per brain). */
host: DbHost<T>
/** The pinned generation. */
generation: number
/** The view's timestamp (ms since epoch) — see {@link Db.timestamp}. */
timestamp: number
/** Per-operation receipt, present only on `transact()`-produced values. */
receipt?: TransactReceipt
/** Speculative overlay, present only on `with()`-produced values. */
overlay?: SpeculativeOverlay<T>
/** Extra teardown on release (e.g. closing an owned snapshot brain). */
closeOnRelease?: () => Promise<void>
}
/**
* @description An immutable, generation-pinned view of a Brainy store. See
* the module documentation for the full read-semantics contract.
*
* @example
* const db = brain.now() // O(1) pin
* await brain.transact([{ op: 'update', id, metadata: { v: 2 } }])
* await db.get(id) // still sees v: 1 — pinned
* await brain.get(id) // sees v: 2 — live
* await db.release() // unpin (enables compaction)
*/
export class Db<T = any> {
private readonly host: DbHost<T>
private readonly gen: number
private readonly ts: number
private readonly overlay?: SpeculativeOverlay<T>
private readonly closeOnRelease?: () => Promise<void>
private isReleased = false
/**
* Per-operation receipt (committed generation, timestamp, resolved ids in
* input order). Present only on values returned by `brain.transact()`.
*/
public readonly receipt?: TransactReceipt
/**
* @param init - Internal construction arguments. `Db` values are created
* by `Brainy` (`now`/`transact`/`asOf`/`load`) and `db.with()` never
* directly by consumers.
*/
constructor(init: DbInit<T>) {
this.host = init.host
this.gen = init.generation
this.ts = init.timestamp
this.receipt = init.receipt
this.overlay = init.overlay
this.closeOnRelease = init.closeOnRelease
this.host.registerDbForFinalization(this, this.gen, this.closeOnRelease)
}
/** The generation this view is pinned at. */
get generation(): number {
return this.gen
}
/**
* The view's wall-clock timestamp (ms since epoch): pin time for `now()`,
* commit time for `transact()`, and the resolved commit time for `asOf()`.
*/
get timestamp(): number {
return this.ts
}
/** Whether {@link Db.release} has been called (released views throw on use). */
get released(): boolean {
return this.isReleased
}
/** Whether this view carries a speculative `with()` overlay. */
get speculative(): boolean {
return this.overlay !== undefined
}
// ==========================================================================
// Reads
// ==========================================================================
/**
* @description Get an entity by id **as of this view's generation**.
* Overlay entries win (speculative views); ids untouched since the pin ride
* the live fast path; ids changed by later transactions resolve from
* immutable generation records. Always correct at the pinned generation.
*
* @param id - The entity id.
* @param options - Same options as `brain.get()` (`includeVectors`).
* @returns The entity as of this generation, or `null`.
*/
async get(id: string, options?: GetOptions): Promise<Entity<T> | null> {
this.assertUsable('get')
if (this.overlay && this.overlay.nouns.has(id)) {
return this.overlay.nouns.get(id) ?? null
}
if (!this.isHistorical()) {
return this.host.get(id, options)
}
const resolved = await this.host.store.resolveAt('noun', id, this.gen)
if (resolved.source === 'current') {
return this.host.get(id, options)
}
if (resolved.source === 'absent' || resolved.metadata === null) {
// Live semantics: an entity without metadata is not a live entity.
return null
}
return this.host.entityFromRecord(
id,
{ metadata: resolved.metadata, vector: resolved.vector },
options?.includeVectors ?? false
)
}
/**
* @description Metadata-level `find()` **as of this view's generation**.
*
* At the current generation (nothing committed past the pin, no overlay)
* this delegates to `brain.find()` untouched the full query surface,
* including semantic search. At a historical generation, or on a
* speculative view, only the metadata dimensions are supported (`type`,
* `subtype`, `where`, `service`, `excludeVFS`, `orderBy`/`order`,
* `limit`/`offset`): results are computed by combining the live index for
* unchanged ids with per-entity evaluation of the same filters against
* generation records / overlay entries set-correct at the pinned
* generation. Index-only dimensions (`query`, `vector`, `near`,
* `connected`, `cursor`, `aggregate`, ) throw
* {@link NotYetSupportedAtHistoricalGenerationError}.
*
* Result ordering is deterministic when `orderBy` is supplied; without it,
* overlay/record-resolved matches follow the live-index matches.
*
* @param query - A `FindParams` object, or a string (semantic search
* current-generation views only).
* @returns Matching results as of this generation (score `1.0` for
* record-resolved metadata matches, mirroring live metadata-only finds).
*/
async find(query: string | FindParams<T>): Promise<Result<T>[]> {
this.assertUsable('find')
const params: FindParams<T> = typeof query === 'string' ? { query } : query
const historical = this.isHistorical()
if (!historical && !this.overlay) {
return this.host.find(query)
}
this.assertMetadataOnlyFind(params)
const limit = params.limit ?? 10
const offset = params.offset ?? 0
// Ids whose live-index answer is NOT valid at this generation.
const changedNouns = historical
? (await this.host.store.changedBetween(this.gen, this.host.store.committedGeneration())).nouns
: []
const overlayNouns = this.overlay?.nouns ?? new Map<string, Entity<T> | null>()
const excluded = new Set<string>([...changedNouns, ...overlayNouns.keys()])
try {
// Over-fetch from the live index so dropping superseded ids cannot
// starve the requested page, then merge and re-window.
const base = await this.host.find({
...params,
limit: limit + offset + excluded.size,
offset: 0
})
const merged = base.filter((result) => !excluded.has(result.id))
for (const id of changedNouns) {
if (overlayNouns.has(id)) continue // The overlay supersedes history.
const resolved = await this.host.store.resolveAt('noun', id, this.gen)
let entity: Entity<T> | null = null
if (resolved.source === 'record' && resolved.metadata !== null) {
entity = await this.host.entityFromRecord(
id,
{ metadata: resolved.metadata, vector: resolved.vector },
false
)
} else if (resolved.source === 'current') {
// Defensive: changed ids always resolve to a record or absent, but
// a 'current' answer is still served correctly from the live path.
entity = await this.host.get(id)
}
if (entity && entityMatchesFind(entity as Entity, params as FindParams)) {
merged.push(resultFromEntity(entity))
}
}
for (const [, entity] of overlayNouns) {
if (entity === null) continue // Speculative delete.
if (entityMatchesFind(entity as Entity, params as FindParams)) {
merged.push(resultFromEntity(entity))
}
}
if (params.orderBy) {
sortResultsBy(merged, params.orderBy, params.order ?? 'asc')
}
return merged.slice(offset, offset + limit)
} catch (err) {
if (err instanceof UnsupportedWhereOperatorError) {
throw new NotYetSupportedAtHistoricalGenerationError(
`metadata find with where-operator '${err.operator}'`,
this.gen,
this.host.store.generation()
)
}
throw err
}
}
/**
* @description Semantic (vector) search **at the current generation only**.
* Convenience for `find({ query, ...options })`. Vector indexes serve the
* live state; at a historical generation, or on a speculative overlay
* (whose entities carry no embeddings), this throws the documented
* {@link NotYetSupportedAtHistoricalGenerationError} persist the
* generation you need and `Brainy.load()` it for the full query surface.
*
* @param query - Natural-language search query.
* @param options - Additional `FindParams` (limit, filters, ).
* @returns Semantic search results (current-generation views only).
* @throws NotYetSupportedAtHistoricalGenerationError at historical
* generations and on speculative overlays.
*/
async search(query: string, options?: Omit<FindParams<T>, 'query'>): Promise<Result<T>[]> {
this.assertUsable('search')
if (!this.isHistorical() && !this.overlay) {
return this.host.find({ ...(options ?? {}), query })
}
throw new NotYetSupportedAtHistoricalGenerationError(
this.overlay ? 'vector search (speculative overlay)' : 'vector search',
this.gen,
this.host.store.generation()
)
}
/**
* @description Relationships **as of this view's generation** the `Db`
* counterpart of `brain.getRelations()` (same string-id shorthand and
* filter surface). Relations whose edges changed after the pin are
* resolved from generation records; overlay relations (speculative
* `relate`/`unrelate`) and cascade tombstones (relations touching a
* speculatively deleted entity) are applied on top. Cursor pagination is
* index-only and throws the documented historical error on
* non-current/speculative views.
*
* @param paramsOrId - An entity id (shorthand for `{ from: id }`) or a
* `GetRelationsParams` filter object.
* @returns Relations as of this generation.
*/
async related(paramsOrId?: string | GetRelationsParams): Promise<Relation<T>[]> {
this.assertUsable('related')
const params: GetRelationsParams =
typeof paramsOrId === 'string' ? { from: paramsOrId } : (paramsOrId ?? {})
const historical = this.isHistorical()
if (!historical && !this.overlay) {
return this.host.getRelations(paramsOrId)
}
if (params.cursor !== undefined) {
throw new NotYetSupportedAtHistoricalGenerationError(
'cursor pagination on related()',
this.gen,
this.host.store.generation()
)
}
const limit = params.limit ?? 100
const offset = params.offset ?? 0
const changedVerbs = historical
? (await this.host.store.changedBetween(this.gen, this.host.store.committedGeneration())).verbs
: []
const overlayVerbs = this.overlay?.verbs ?? new Map<string, Relation<T> | null>()
const excluded = new Set<string>([...changedVerbs, ...overlayVerbs.keys()])
const base = await this.host.getRelations({
...params,
limit: limit + offset + excluded.size,
offset: 0
})
let merged = base.filter((relation) => !excluded.has(relation.id))
for (const id of changedVerbs) {
if (overlayVerbs.has(id)) continue
const resolved = await this.host.store.resolveAt('verb', id, this.gen)
if (resolved.source !== 'record') continue
const relation = this.host.relationFromRecord(id, {
metadata: resolved.metadata,
vector: resolved.vector
})
if (relation && relationMatchesParams(relation, params)) {
merged.push(relation)
}
}
for (const [, relation] of overlayVerbs) {
if (relation === null) continue // Speculative unrelate.
if (relationMatchesParams(relation, params)) {
merged.push(relation)
}
}
// Cascade semantics for speculative deletes: a relation whose endpoint
// is tombstoned in the overlay is gone in this view, exactly as
// brain.delete() cascades on the durable path.
if (this.overlay) {
const tombstoned = new Set<string>()
for (const [id, entity] of this.overlay.nouns) {
if (entity === null) tombstoned.add(id)
}
if (tombstoned.size > 0) {
merged = merged.filter(
(relation) => !tombstoned.has(relation.from) && !tombstoned.has(relation.to)
)
}
}
return merged.slice(offset, offset + limit)
}
// ==========================================================================
// Derived views
// ==========================================================================
/**
* @description Speculative write: returns a NEW `Db` whose reads see the
* given transaction data applied **in memory, on top of this view**
* Datomic's `with`. Nothing touches disk, the generation counter, or index
* providers; the underlying brain and this view are untouched. Use it to
* answer "what would the store look like if…" questions, then call
* `brain.transact()` with the same operations to make it real.
*
* Speculative semantics (documented deltas from the durable paths):
* - `add` without an explicit `vector` produces an entity with an empty
* stub vector speculative entities are not semantically searchable
* (`with()` never invokes the embedder).
* - `remove` cascades on the read side: relations touching the removed
* entity disappear from `related()` without being enumerated.
* - `relate` deduplicates against edges visible in this view when verb
* history allows the check, mirroring `brain.relate()`.
* - Brain-level vocabulary/subtype enforcement does not run overlays
* never persist, so there is nothing to protect.
*
* The returned view takes its own pin on this generation; release both
* views independently.
*
* @param ops - The same declarative operations `brain.transact()` accepts.
* @returns A new speculative `Db` over this view.
* @throws Error when an operation references a missing entity/relation,
* mirroring the durable path's planning errors.
*/
async with(ops: TxOperation<T>[]): Promise<Db<T>> {
this.assertUsable('with')
if (!Array.isArray(ops) || ops.length === 0) {
throw new Error('with(): provide a non-empty array of transaction operations')
}
// Child overlay starts as a copy of this view's overlay (chained with()).
const overlay: SpeculativeOverlay<T> = {
nouns: new Map(this.overlay?.nouns ?? []),
verbs: new Map(this.overlay?.verbs ?? [])
}
// Reads during planning must see "this view + overlay so far".
const speculativeGet = async (id: string): Promise<Entity<T> | null> => {
if (overlay.nouns.has(id)) return overlay.nouns.get(id) ?? null
return this.get(id)
}
for (const op of ops) {
switch (op.op) {
case 'add': {
const id = op.id ?? uuidv4()
const now = Date.now()
overlay.nouns.set(id, {
id,
vector: op.vector ?? [],
type: op.type,
...(op.subtype !== undefined && { subtype: op.subtype }),
data: op.data,
metadata: (op.metadata ?? {}) as T,
...(op.service !== undefined && { service: op.service }),
createdAt: now,
updatedAt: now,
...(op.confidence !== undefined && { confidence: op.confidence }),
...(op.weight !== undefined && { weight: op.weight }),
_rev: 1
})
break
}
case 'update': {
const base = await speculativeGet(op.id)
if (!base) {
throw new Error(`with(): entity ${op.id} not found at generation ${this.gen}`)
}
const mergedMetadata =
op.merge !== false
? ({ ...(base.metadata as object), ...(op.metadata as object) } as T)
: ((op.metadata ?? base.metadata) as T)
overlay.nouns.set(op.id, {
...base,
...(op.type !== undefined && { type: op.type }),
...(op.subtype !== undefined && { subtype: op.subtype }),
...(op.data !== undefined && { data: op.data }),
...(op.vector !== undefined && { vector: op.vector }),
...(op.confidence !== undefined && { confidence: op.confidence }),
...(op.weight !== undefined && { weight: op.weight }),
metadata: mergedMetadata,
updatedAt: Date.now(),
_rev: (base._rev ?? 1) + 1
})
break
}
case 'remove': {
overlay.nouns.set(op.id, null)
// Tombstone overlay relations touching the removed entity; base
// relations are cascade-filtered at read time in related().
for (const [verbId, relation] of overlay.verbs) {
if (relation && (relation.from === op.id || relation.to === op.id)) {
overlay.verbs.set(verbId, null)
}
}
break
}
case 'relate': {
const from = await speculativeGet(op.from)
const to = await speculativeGet(op.to)
if (!from) throw new Error(`with(): source entity ${op.from} not found`)
if (!to) throw new Error(`with(): target entity ${op.to} not found`)
// Dedupe against the view (overlay first, then committed edges
// when verb history allows the read) — mirror of relate().
let duplicate: Relation<T> | undefined
for (const relation of overlay.verbs.values()) {
if (relation && relation.from === op.from && relation.to === op.to && relation.type === op.type) {
duplicate = relation
break
}
}
if (!duplicate) {
try {
const existing = await this.related({ from: op.from, type: op.type })
duplicate = existing.find((relation) => relation.to === op.to)
} catch (err) {
if (!(err instanceof NotYetSupportedAtHistoricalGenerationError)) throw err
// Verb history can't serve the check at this generation —
// proceed with overlay-only dedupe (documented).
}
}
if (duplicate) break
const id = uuidv4()
overlay.verbs.set(id, {
id,
from: op.from,
to: op.to,
type: op.type,
...(op.subtype !== undefined && { subtype: op.subtype }),
weight: op.weight ?? 1.0,
...(op.data !== undefined && { data: op.data }),
metadata: op.metadata,
...(op.service !== undefined && { service: op.service }),
createdAt: Date.now()
})
if (op.bidirectional) {
const reverseId = uuidv4()
overlay.verbs.set(reverseId, {
id: reverseId,
from: op.to,
to: op.from,
type: op.type,
...(op.subtype !== undefined && { subtype: op.subtype }),
weight: op.weight ?? 1.0,
...(op.data !== undefined && { data: op.data }),
metadata: op.metadata,
...(op.service !== undefined && { service: op.service }),
createdAt: Date.now()
})
}
break
}
case 'unrelate': {
overlay.verbs.set(op.id, null)
break
}
default: {
const exhaustive: never = op
throw new Error(`with(): unknown operation ${JSON.stringify(exhaustive)}`)
}
}
}
this.host.pinGeneration(this.gen)
return new Db<T>({
host: this.host,
generation: this.gen,
timestamp: Date.now(),
overlay
})
}
/**
* @description The ids changed by committed transactions in the interval
* `(priorDb.generation, this.generation]` the diff between two pinned
* views of the same store. Single-operation writes between commits are not
* reflected (documented 8.0 history granularity).
*
* @param priorDb - An earlier view of the SAME store.
* @returns Changed entity and relation ids, sorted.
* @throws RangeError when `priorDb` is newer than this view, or Error when
* the views belong to different stores.
*/
async since(priorDb: Db<T>): Promise<ChangedIds> {
this.assertUsable('since')
if (priorDb.host.store !== this.host.store) {
throw new Error('since(): both Db values must come from the same Brainy instance')
}
if (priorDb.gen > this.gen) {
throw new RangeError(
`since(): priorDb is at generation ${priorDb.gen}, which is newer than this view ` +
`(generation ${this.gen}). Call newerDb.since(olderDb).`
)
}
return this.host.store.changedBetween(priorDb.gen, this.gen)
}
// ==========================================================================
// Durability
// ==========================================================================
/**
* @description Snapshot this view to `path` as a self-contained store:
* flush, then hard-link every immutable file into the target
* (Cassandra-style instant and space-shared; cross-device targets fall
* back to byte copies, and the small append-in-place file list is always
* byte-copied). The result opens with `Brainy.load(path)` with the full
* query surface, and later mutations of the source can never alter it
* rewrites swap inodes, the snapshot keeps the old bytes.
*
* `persist()` requires this view to still be the store's **latest**
* generation (snapshotting captures current bytes): pin with `brain.now()`
* and persist before further writes. A view that history has moved past
* throws {@link NotYetSupportedAtHistoricalGenerationError}; speculative
* overlays cannot be persisted (commit them with `brain.transact()` first).
*
* @param path - Absolute directory for the snapshot (created; must be
* empty or absent).
* @throws NotYetSupportedAtHistoricalGenerationError when this view is no
* longer the latest generation, or is speculative.
*/
async persist(path: string): Promise<void> {
this.assertUsable('persist')
if (this.overlay) {
throw new NotYetSupportedAtHistoricalGenerationError(
'persist of a speculative overlay',
this.gen,
this.host.store.generation()
)
}
await this.host.persistPinned(path, this.gen)
}
// ==========================================================================
// Lifecycle
// ==========================================================================
/**
* @description Release this view's pin (store + versioned index
* providers). Idempotent. After release, every read throws. Views from
* `Brainy.load()` / `asOf(path)` also close their underlying read-only
* brain here. A `FinalizationRegistry` backstop releases leaked pins when
* a `Db` is garbage-collected, but explicit release is what makes
* `compactHistory()` deterministic prefer it.
*/
async release(): Promise<void> {
if (this.isReleased) return
this.isReleased = true
this.host.unregisterDbFromFinalization(this)
this.host.releaseGeneration(this.gen)
if (this.closeOnRelease) {
await this.closeOnRelease()
}
}
// ==========================================================================
// Internals
// ==========================================================================
/** Whether any transaction committed past the pinned generation. */
private isHistorical(): boolean {
return this.host.store.hasCommittedAfter(this.gen)
}
/** Throw on use-after-release. */
private assertUsable(method: string): void {
if (this.isReleased) {
throw new Error(
`Cannot call ${method}() on a released Db (generation ${this.gen}). ` +
`Pin a fresh view with brain.now() or brain.asOf().`
)
}
}
/**
* Reject find() dimensions that require index machinery the record layer
* cannot answer at a historical generation / for a speculative overlay.
*/
private assertMetadataOnlyFind(params: FindParams<T>): void {
const unsupported: Array<[string, boolean]> = [
['semantic query', params.query !== undefined],
['vector search', params.vector !== undefined],
['proximity search (near)', params.near !== undefined],
['graph traversal (connected)', params.connected !== undefined],
['cursor pagination', params.cursor !== undefined],
['aggregation', params.aggregate !== undefined],
['relation expansion (includeRelations)', params.includeRelations === true],
[
`search mode '${params.mode ?? params.searchMode}'`,
(params.mode !== undefined && params.mode !== 'metadata' && params.mode !== 'auto') ||
(params.searchMode !== undefined && params.searchMode !== 'auto')
]
]
for (const [capability, present] of unsupported) {
if (present) {
throw new NotYetSupportedAtHistoricalGenerationError(
capability,
this.gen,
this.host.store.generation()
)
}
}
}
}
/**
* @description Build a `Result` row from a record/overlay-resolved entity,
* mirroring the live metadata-only find path (flattened fields, score `1.0`).
*/
function resultFromEntity<T>(entity: Entity<T>): Result<T> {
return {
id: entity.id,
score: 1.0,
type: entity.type,
subtype: entity.subtype,
metadata: entity.metadata,
data: entity.data,
confidence: entity.confidence,
weight: entity.weight,
_rev: entity._rev,
entity
}
}
/**
* @description In-place stable sort of merged results by a find() `orderBy`
* field, mirroring live ordering semantics (`asc` default, undefined values
* last in both directions).
*/
function sortResultsBy<T>(results: Result<T>[], orderBy: string, order: 'asc' | 'desc'): void {
const direction = order === 'desc' ? -1 : 1
results.sort((a, b) => {
const va = resolveEntityField(a.entity as Entity, orderBy)
const vb = resolveEntityField(b.entity as Entity, orderBy)
if (va === undefined && vb === undefined) return 0
if (va === undefined) return 1
if (vb === undefined) return -1
if (typeof va === 'number' && typeof vb === 'number') return (va - vb) * direction
const sa = String(va)
const sb = String(vb)
return (sa < sb ? -1 : sa > sb ? 1 : 0) * direction
})
}
/**
* @description Filter one relation against `GetRelationsParams`, mirroring
* the storage-level filter `brain.getRelations()` builds (`from` sourceId,
* `to` targetId, type/subtype set membership, service equality).
*/
function relationMatchesParams<T>(relation: Relation<T>, params: GetRelationsParams): boolean {
if (params.from !== undefined && relation.from !== params.from) return false
if (params.to !== undefined && relation.to !== params.to) return false
if (params.type !== undefined) {
const types = Array.isArray(params.type) ? params.type : [params.type]
if (!types.includes(relation.type)) return false
}
if (params.subtype !== undefined) {
const subtypes = Array.isArray(params.subtype) ? params.subtype : [params.subtype]
if (relation.subtype === undefined || !subtypes.includes(relation.subtype)) return false
}
if (params.service !== undefined && relation.service !== params.service) return false
return true
}

146
src/db/errors.ts Normal file
View file

@ -0,0 +1,146 @@
/**
* @module db/errors
* @description Error types for the 8.0 generational-MVCC Db API.
*
* Three failure modes have dedicated classes so callers can branch on them
* with `instanceof` instead of string matching:
*
* - {@link GenerationConflictError} a `transact()` compare-and-swap
* (`ifAtGeneration`) observed a different current generation than the
* caller expected. The standard retry pattern is: re-read via
* `brain.now()`, re-derive the transaction, and re-submit.
* - {@link NotYetSupportedAtHistoricalGenerationError} an index-accelerated
* query (vector / hybrid / graph-traversal search) was issued against a
* `Db` pinned at a generation the live indexes no longer represent.
* `get()` and metadata-filter `find()` remain fully supported at pinned
* generations; index-accelerated queries at historical generations require
* an index rebuild from the pinned record set, which ships as a follow-up
* (`asOf` rebuild). The error message names the unsupported capability and
* points at the supported alternatives.
* - {@link GenerationCompactedError} `asOf()` asked for a generation whose
* immutable records were reclaimed by `compactHistory()`.
*
* All three are exported from the package root (`@soulcraft/brainy`).
*/
/**
* @description Thrown by `brain.transact(ops, { ifAtGeneration })` when the
* store's current generation does not equal the caller-supplied expectation.
* This is the whole-store compare-and-swap counterpart to the per-entity
* `ifRev` / `RevisionConflictError` pair: it guarantees that *nothing* was
* committed between the caller's read (`brain.now()`) and this transaction.
*
* The transaction is rejected before any record is staged the store is
* untouched and the generation counter is unchanged.
*
* @example
* const db = brain.now()
* try {
* await brain.transact(ops, { ifAtGeneration: db.generation })
* } catch (err) {
* if (err instanceof GenerationConflictError) {
* // Someone committed since we pinned — re-read and retry.
* console.log(`expected ${err.expected}, store is at ${err.actual}`)
* }
* }
*/
export class GenerationConflictError extends Error {
/** The generation the caller expected the store to be at. */
public readonly expected: number
/** The generation the store was actually at when the transaction arrived. */
public readonly actual: number
/**
* @param expected - The generation supplied via `ifAtGeneration`.
* @param actual - The store's current generation at submission time.
*/
constructor(expected: number, actual: number) {
super(
`Generation conflict: transact() was submitted with ifAtGeneration: ${expected}, ` +
`but the store is at generation ${actual}. Another write committed in between. ` +
`Re-read with brain.now(), rebuild the transaction, and retry.`
)
this.name = 'GenerationConflictError'
this.expected = expected
this.actual = actual
}
}
/**
* @description Thrown when an operation on a pinned `Db` requires
* index-accelerated machinery (vector search, hybrid search, graph
* traversal, cursor pagination, aggregation) that the live indexes cannot
* answer for a *historical* generation. Storage-record reads (`get()`) and
* metadata-filter `find()` are always supported at pinned generations.
*
* Rebuilding indexes from a pinned record set (the standard LSM answer for
* historical index queries) is the documented `asOf`-rebuild follow-up; this
* error is the honest boundary until it ships.
*
* @example
* const db = brain.now()
* await brain.transact([{ op: 'update', id, metadata: { v: 2 } }])
* await db.get(id) // ✅ pinned read — supported
* await db.find({ where: { v: 1 } }) // ✅ metadata find — supported
* await db.search('semantic query') // ❌ throws this error
*/
export class NotYetSupportedAtHistoricalGenerationError extends Error {
/** The pinned generation the operation was attempted against. */
public readonly generation: number
/** The capability that is not supported at historical generations. */
public readonly capability: string
/**
* @param capability - Human-readable name of the unsupported capability
* (e.g. `'vector search'`, `'graph traversal (connected)'`).
* @param generation - The Db's pinned generation.
* @param currentGeneration - The store's current generation, for context.
*/
constructor(capability: string, generation: number, currentGeneration: number) {
super(
`${capability} is not yet supported at a historical generation ` +
`(Db pinned at generation ${generation}; store is at ${currentGeneration}). ` +
`Supported at pinned generations: get() and metadata-filter find(). ` +
`For index-accelerated queries over historical state, persist() the generation ` +
`you need and open it with Brainy.load(path) — the loaded snapshot rebuilds ` +
`its indexes and supports the full query surface.`
)
this.name = 'NotYetSupportedAtHistoricalGenerationError'
this.capability = capability
this.generation = generation
}
}
/**
* @description Thrown by `brain.asOf(generationOrTimestamp)` when the
* requested generation's history was reclaimed by `brain.compactHistory()`.
* The store records the compaction horizon (the highest reclaimed
* generation) in its manifest; any generation BELOW the horizon is
* unreachable its reads would need the reclaimed before-images. The
* horizon itself stays reachable, resolved from the record-sets above it.
*
* To keep a generation readable forever, `persist()` it to a snapshot
* directory before compacting snapshots are self-contained and unaffected
* by compaction of the source store.
*/
export class GenerationCompactedError extends Error {
/** The generation that was requested. */
public readonly requested: number
/** The compaction horizon — generations < this value are unreachable. */
public readonly horizon: number
/**
* @param requested - The generation the caller asked for.
* @param horizon - The store's current compaction horizon.
*/
constructor(requested: number, horizon: number) {
super(
`Generation ${requested} has been compacted away (compaction horizon: ${horizon}). ` +
`Only generations at or above the horizon are reachable via asOf(). ` +
`Use db.persist(path) before compactHistory() to keep a generation readable.`
)
this.name = 'GenerationCompactedError'
this.requested = requested
this.horizon = horizon
}
}

838
src/db/generationStore.ts Normal file
View file

@ -0,0 +1,838 @@
/**
* @module db/generationStore
* @description The generational record layer behind Brainy 8.0's MVCC Db API.
*
* One `GenerationStore` is attached to every writable `Brainy` instance. It
* owns:
*
* - the **monotonic generation counter** (`_system/generation.json`),
* reserved per `transact()` commit and bumped per single-operation write
* batch via the storage-layer hook;
* - the **commit protocol** for `brain.transact()`: capture before-images
* write the generation delta fsync execute the operation batch
* atomically rename `_system/manifest.json` (the rename IS the commit
* point) append the tx-log line;
* - **crash recovery**: on open, any `_generations/<N>` directory with
* `N > manifest.generation` is an uncommitted transaction its
* before-images are restored to the canonical entity paths and the
* directory is removed, so a crash anywhere before the manifest rename
* leaves the store byte-identical to its pre-transaction state;
* - **pin/release refcounting** for live `Db` values, which is what makes
* `compactHistory()` safe: a generation record-set is reclaimed only when
* no live pin could ever need it;
* - **point-in-time resolution**: the state of id X at pinned generation G
* is the before-image stored by the *first* committed generation after G
* that touched X or the live storage state when nothing after G touched
* it (so current-generation reads stay on the existing fast paths,
* untouched).
*
* Durability and atomicity guarantees, failure modes, and the intellectual
* lineage (Datomic database-as-value, LMDB reader pins, LSM immutable
* segments) are specified in `docs/ADR-001-generational-mvcc.md`.
*/
import { prodLog } from '../utils/logger.js'
import { GenerationCompactedError, GenerationConflictError } from './errors.js'
import type {
ChangedIds,
CompactHistoryOptions,
CompactHistoryResult,
GenerationDelta,
GenerationManifest,
GenerationRecord,
GenerationStorage,
TxLogEntry
} from './types.js'
/** Storage-root-relative path of the persisted generation counter. */
export const GENERATION_COUNTER_PATH = '_system/generation.json'
/** Storage-root-relative path of the commit manifest. */
export const MANIFEST_PATH = '_system/manifest.json'
/** Storage-root-relative prefix of the per-generation record directories. */
export const GENERATIONS_PREFIX = '_generations'
/**
* @description Phases of the {@link GenerationStore.commitTransaction} commit
* protocol at which a test-only fault injector can simulate a process crash.
* The phases map 1:1 onto the protocol steps documented on
* `commitTransaction`:
*
* - `'after-staging'` before-images + the `tx.json` delta are written and
* fsynced; the operation batch has NOT executed yet.
* - `'after-execute'` the operation batch has been applied to canonical
* storage and indexes; the manifest still points at the prior generation.
* - `'before-manifest-rename'` the batch has executed and the counter is
* persisted; the atomic manifest rename the commit point has NOT
* happened. A crash here is the canonical "fully staged, never committed"
* case that recovery must roll back byte-identically.
* - `'after-manifest-rename'` the manifest rename landed (the transaction
* IS committed); the tx-log append has NOT happened yet. A crash here must
* keep the transaction (the tx-log is advisory metadata, not the source of
* commit truth).
*/
export type CommitFaultPhase =
| 'after-staging'
| 'after-execute'
| 'before-manifest-rename'
| 'after-manifest-rename'
/**
* @description Identifies which ids a transaction touches, split by kind.
*/
export interface TouchedIds {
/** Entity ids the transaction writes or deletes. */
nouns: string[]
/** Relationship ids the transaction writes or deletes. */
verbs: string[]
}
/**
* @description Result of {@link GenerationStore.open} how many uncommitted
* generations crash recovery rolled back (a non-zero value tells `Brainy` to
* force an index reconciliation, since derived index state may reference the
* rolled-back writes).
*/
export interface GenerationStoreOpenResult {
/** Count of uncommitted generation directories rolled back and removed. */
rolledBackGenerations: number
}
/**
* @description The generational MVCC record layer. See the module
* documentation for the full protocol; every public method documents its
* own contract.
*/
export class GenerationStore {
private readonly storage: GenerationStorage
/** Latest reserved/observed generation (≥ {@link committed}). */
private counter = 0
/** Committed-transaction watermark (manifest generation). */
private committed = 0
/** Compaction horizon — record-sets ≤ this are reclaimed. */
private horizonGen = 0
/** Sorted list of committed generations whose record dirs exist. */
private committedGens: number[] = []
/** Delta cache, keyed by generation (lazily loaded from `tx.json`). */
private readonly deltaCache = new Map<
number,
{ nouns: Set<string>; verbs: Set<string>; timestamp: number }
>()
/** Live pin refcounts, keyed by pinned generation. */
private readonly pins = new Map<number, number>()
/** Serializes transact commits, compaction, and counter persistence. */
private mutexTail: Promise<unknown> = Promise.resolve()
/** True while a transact batch is executing (suppresses the bump hook). */
private inTransact = false
/** Coalescing state for lazy persistence of single-op counter bumps. */
private counterPersistScheduled = false
/**
* Test-only fault injector for the commit protocol (see
* {@link GenerationStore.setCommitFaultInjector}). `undefined` in production.
*/
private commitFaultInjector?: (phase: CommitFaultPhase) => void
private opened = false
/**
* @param storage - The storage adapter, narrowed to the
* {@link GenerationStorage} contract (`BaseStorage` implements it).
*/
constructor(storage: GenerationStorage) {
this.storage = storage
}
// ==========================================================================
// Lifecycle
// ==========================================================================
/**
* @description Load the persisted counter + manifest and run crash
* recovery. Must be called once, before indexes are built, because
* recovery may rewrite canonical entity files (restoring before-images of
* an uncommitted transaction).
*
* @param options.readOnly - When true (reader-mode brains), recovery is
* skipped: readers never write. An un-repaired crashed directory is
* repaired by the next *writer* open.
* @returns How many uncommitted generations were rolled back.
*/
async open(options?: { readOnly?: boolean }): Promise<GenerationStoreOpenResult> {
const counterFile = (await this.storage.readRawObject(GENERATION_COUNTER_PATH)) as
| { generation?: number }
| null
const manifest = (await this.storage.readRawObject(MANIFEST_PATH)) as GenerationManifest | null
this.committed = manifest?.generation ?? 0
this.horizonGen = manifest?.horizon ?? 0
this.counter = Math.max(counterFile?.generation ?? 0, this.committed)
// Discover existing generation record directories.
const recordPaths = await this.storage.listRawObjects(GENERATIONS_PREFIX)
const seenGens = new Set<number>()
for (const p of recordPaths) {
const gen = parseGenerationFromPath(p)
if (gen !== null) seenGens.add(gen)
}
let rolledBack = 0
const committedGens: number[] = []
for (const gen of [...seenGens].sort((a, b) => a - b)) {
if (gen <= this.committed) {
committedGens.push(gen)
} else if (!options?.readOnly) {
// Uncommitted (crashed) transaction: restore before-images, drop the dir.
await this.rollBackUncommittedGeneration(gen)
rolledBack++
}
// Reader mode: leave orphan dirs alone; resolution ignores them because
// committedGens only includes generations ≤ the manifest watermark.
this.counter = Math.max(this.counter, gen)
}
this.committedGens = committedGens
if (rolledBack > 0) {
prodLog.warn(
`[GenerationStore] Crash recovery rolled back ${rolledBack} uncommitted ` +
`transaction(s); store restored to generation ${this.committed}.`
)
}
this.opened = true
// Hook single-op write batches so generation() is always meaningful.
// Suppressed while a transact batch executes (the batch is ONE generation).
if (!options?.readOnly) {
this.storage.setGenerationBumpHook(() => this.noteSingleOpWrite())
}
return { rolledBackGenerations: rolledBack }
}
/**
* @description Detach the storage hook and persist the counter. Called
* from `brain.close()`.
*/
async close(): Promise<void> {
this.storage.setGenerationBumpHook(undefined)
await this.persistCounterNow()
}
/**
* @description TEST-ONLY: install (or clear, with `undefined`) a fault
* injector that is invoked at each {@link CommitFaultPhase} of the commit
* protocol. A **throwing** injector simulates a process crash at that
* phase: the abort cleanup (staging-directory removal, generation-counter
* reservation return) is deliberately skipped exactly as if the process
* had died so crash-consistency tests exercise the REAL recovery path
* ({@link GenerationStore.open} rolling back the uncommitted generation on
* the next open). Never installed in production code; the injector exists
* solely so the durability protocol can be proven, not trusted.
*
* @param injector - Callback invoked synchronously at each phase, or
* `undefined` to detach.
*/
setCommitFaultInjector(injector: ((phase: CommitFaultPhase) => void) | undefined): void {
this.commitFaultInjector = injector
}
// ==========================================================================
// Generation counter
// ==========================================================================
/** @returns The store's current generation (latest write watermark). */
generation(): number {
return this.counter
}
/** @returns The committed-transaction watermark (manifest generation). */
committedGeneration(): number {
return this.committed
}
/** @returns The compaction horizon (generations ≤ horizon are reclaimed). */
horizon(): number {
return this.horizonGen
}
/**
* @description Single-operation write hook (registered with the storage
* layer in {@link open}). Bumps the in-memory counter and schedules a
* coalesced persist of `_system/generation.json` durable artifacts
* (records, manifests, snapshots) always persist the counter
* synchronously at their own commit points, so a crash inside the
* coalescing window can never move the counter backwards relative to
* anything durable.
*/
private noteSingleOpWrite(): void {
if (this.inTransact) return
this.counter++
this.schedulePersistCounter()
}
/** Schedule one coalesced counter persist for the current write burst. */
private schedulePersistCounter(): void {
if (this.counterPersistScheduled) return
this.counterPersistScheduled = true
setImmediate(() => {
this.counterPersistScheduled = false
void this.withMutex(() => this.persistCounterUnlocked()).catch((err) => {
prodLog.warn(`[GenerationStore] generation counter persist failed: ${(err as Error).message}`)
})
})
}
/**
* @description Persist the generation counter now (atomic tmp+rename).
* Called from `flush()`, `close()`, snapshotting, and the commit protocol.
*/
async persistCounterNow(): Promise<void> {
if (!this.opened) return
await this.withMutex(() => this.persistCounterUnlocked())
}
private async persistCounterUnlocked(): Promise<void> {
await this.storage.writeRawObject(GENERATION_COUNTER_PATH, {
generation: this.counter,
updatedAt: new Date().toISOString()
})
}
// ==========================================================================
// Pins
// ==========================================================================
/**
* @description Pin a generation (refcounted). Compaction never reclaims a
* record-set a live pin could need (i.e. any generation directory `N`
* with `N > G` for some pinned `G`).
* @param gen - The generation to pin.
*/
pin(gen: number): void {
this.pins.set(gen, (this.pins.get(gen) ?? 0) + 1)
}
/**
* @description Release one pin on a generation. Safe to call for a
* generation with no live pins (no-op with a warning).
* @param gen - The generation to release.
*/
release(gen: number): void {
const count = this.pins.get(gen)
if (count === undefined) {
prodLog.warn(`[GenerationStore] release(${gen}) called with no live pin at that generation`)
return
}
if (count <= 1) this.pins.delete(gen)
else this.pins.set(gen, count - 1)
}
/** @returns Total number of live pins across all generations. */
activePinCount(): number {
let total = 0
for (const count of this.pins.values()) total += count
return total
}
/** @returns The smallest pinned generation, or `Infinity` with no pins. */
private minPinnedGeneration(): number {
let min = Infinity
for (const gen of this.pins.keys()) min = Math.min(min, gen)
return min
}
// ==========================================================================
// Commit protocol
// ==========================================================================
/**
* @description Commit one transaction. The caller (Brainy.transact) plans
* the batch up front resolving every touched id, including generated
* entity/relationship ids and delete cascades and hands this method the
* touched-id sets plus an `execute` closure that runs the already-built
* operation batch through the TransactionManager.
*
* Protocol (under the commit mutex):
* 1. `ifAtGeneration` CAS check (rejects before anything is staged).
* 2. Reserve generation `N` (counter increment).
* 3. Capture before-images of every touched id and write them, plus the
* `tx.json` delta, under `_generations/N/`; fsync. From this point a
* crash anywhere is recoverable to the exact pre-transaction bytes.
* 4. Run `execute()`. On failure the TransactionManager has already
* rolled back its applied operations; the staging directory is removed
* and the reservation is returned (when no concurrent bump consumed a
* later number), so a failed transaction leaves the generation
* unchanged.
* 5. Persist the counter, then atomically rename the manifest to
* generation `N` **the rename is the commit point** and fsync it.
* 6. Append the tx-log line and update in-memory bookkeeping.
*
* @param args.touched - Every id the batch writes or deletes.
* @param args.meta - Transaction metadata for the tx-log.
* @param args.ifAtGeneration - Whole-store CAS expectation.
* @param args.execute - Runs the planned operation batch atomically.
* @returns The committed generation and its commit timestamp.
* @throws GenerationConflictError when the CAS expectation fails.
*/
async commitTransaction(args: {
touched: TouchedIds
meta?: Record<string, unknown>
ifAtGeneration?: number
execute: () => Promise<void>
}): Promise<{ generation: number; timestamp: number }> {
return this.withMutex(async () => {
if (args.ifAtGeneration !== undefined && args.ifAtGeneration !== this.counter) {
throw new GenerationConflictError(args.ifAtGeneration, this.counter)
}
const nouns = [...new Set(args.touched.nouns)]
const verbs = [...new Set(args.touched.verbs)]
const gen = ++this.counter
const dir = `${GENERATIONS_PREFIX}/${gen}`
const timestamp = Date.now()
// Test-only crash simulation (see setCommitFaultInjector): a throwing
// injector must bypass the abort cleanup below, exactly as a real
// process death would, so recovery-on-open is what restores the store.
let crashSimulated = false
const faultPoint = (phase: CommitFaultPhase): void => {
if (!this.commitFaultInjector) return
try {
this.commitFaultInjector(phase)
} catch (err) {
crashSimulated = true
throw err
}
}
try {
// -- 3. Before-images + delta (the durable undo log) ------------------
const stagedPaths: string[] = []
for (const id of nouns) {
const prev = await this.storage.readNounRaw(id)
const recordPath = `${dir}/prev/${id}.json`
const record: GenerationRecord = { kind: 'noun', metadata: prev.metadata, vector: prev.vector }
await this.storage.writeRawObject(recordPath, record)
stagedPaths.push(recordPath)
}
for (const id of verbs) {
const prev = await this.storage.readVerbRaw(id)
const recordPath = `${dir}/prev/${id}.json`
const record: GenerationRecord = { kind: 'verb', metadata: prev.metadata, vector: prev.vector }
await this.storage.writeRawObject(recordPath, record)
stagedPaths.push(recordPath)
}
const delta: GenerationDelta = {
generation: gen,
timestamp,
...(args.meta && { meta: args.meta }),
nouns,
verbs
}
const deltaPath = `${dir}/tx.json`
await this.storage.writeRawObject(deltaPath, delta)
stagedPaths.push(deltaPath)
await this.storage.syncRawObjects(stagedPaths)
faultPoint('after-staging')
// -- 4. Execute the planned batch -------------------------------------
this.inTransact = true
try {
await args.execute()
} finally {
this.inTransact = false
}
faultPoint('after-execute')
// -- 5. Counter + manifest rename (COMMIT POINT) ----------------------
await this.persistCounterUnlocked()
faultPoint('before-manifest-rename')
const manifest: GenerationManifest = {
version: 1,
generation: gen,
committedAt: new Date(timestamp).toISOString(),
horizon: this.horizonGen
}
await this.storage.writeRawObject(MANIFEST_PATH, manifest)
await this.storage.syncRawObjects([MANIFEST_PATH])
faultPoint('after-manifest-rename')
// -- 6. Post-commit bookkeeping ---------------------------------------
this.committed = gen
this.committedGens.push(gen)
this.deltaCache.set(gen, { nouns: new Set(nouns), verbs: new Set(verbs), timestamp })
const logEntry: TxLogEntry = { generation: gen, timestamp, ...(args.meta && { meta: args.meta }) }
await this.storage.appendTxLogLine(JSON.stringify(logEntry))
return { generation: gen, timestamp }
} catch (err) {
// Simulated crash (test-only fault injector): skip the abort cleanup
// entirely — a dead process cannot clean up. Recovery on the next
// open() is what must (and does) restore the store.
if (crashSimulated) {
throw err
}
// Failed before the manifest rename: nothing is committed. Remove the
// staging directory; the TransactionManager already restored any
// applied operation byte-identically.
try {
await this.storage.removeRawPrefix(dir)
} catch (cleanupErr) {
prodLog.warn(
`[GenerationStore] failed to remove staging dir ${dir} after aborted transaction: ` +
`${(cleanupErr as Error).message} (recovery will remove it on next open)`
)
}
// Return the reservation when no concurrent bump consumed a later
// number, so a failed transaction leaves generation() unchanged.
if (this.counter === gen) this.counter = gen - 1
throw err
}
})
}
// ==========================================================================
// Point-in-time resolution
// ==========================================================================
/**
* @description Whether any transaction committed after generation `gen`.
* O(1) this is the fast-path check that lets a `Db` pinned at the
* current generation delegate every read to the live brain untouched.
* @param gen - The pinned generation.
*/
hasCommittedAfter(gen: number): boolean {
const last = this.committedGens[this.committedGens.length - 1]
return last !== undefined && last > gen
}
/**
* @description Resolve the state of an entity or relationship at pinned
* generation `gen`: the before-image stored by the first committed
* generation after `gen` that touched the id or `{ source: 'current' }`
* when nothing after `gen` touched it (the live storage state *is* the
* state at `gen`).
*
* @param kind - `'noun'` for entities, `'verb'` for relationships.
* @param id - The id to resolve.
* @param gen - The pinned generation.
* @returns `{ source: 'current' }`, `{ source: 'absent' }` (did not exist
* at `gen`), or `{ source: 'record', metadata, vector }` with the raw
* stored objects as of `gen`.
*/
async resolveAt(
kind: 'noun' | 'verb',
id: string,
gen: number
): Promise<
| { source: 'current' }
| { source: 'absent' }
| { source: 'record'; metadata: any; vector: any | null }
> {
for (const candidate of this.committedGens) {
if (candidate <= gen) continue
const delta = await this.getDelta(candidate)
const touched = kind === 'noun' ? delta.nouns : delta.verbs
if (!touched.has(id)) continue
const record = (await this.storage.readRawObject(
`${GENERATIONS_PREFIX}/${candidate}/prev/${id}.json`
)) as GenerationRecord | null
if (record === null) {
// The record-set exists (candidate is committed) but the id's
// before-image is missing — only possible through external tampering.
throw new Error(
`Generation record missing: ${GENERATIONS_PREFIX}/${candidate}/prev/${id}.json ` +
`(store corrupted or records removed outside compactHistory())`
)
}
if (record.metadata === null && record.vector === null) {
return { source: 'absent' }
}
return { source: 'record', metadata: record.metadata, vector: record.vector }
}
return { source: 'current' }
}
/**
* @description The ids touched by committed transactions in the interval
* `(fromGen, toGen]`. Used by `db.since()` and by historical `find()` to
* build its overlay of changed entities.
* @param fromGen - Exclusive lower bound.
* @param toGen - Inclusive upper bound.
*/
async changedBetween(fromGen: number, toGen: number): Promise<ChangedIds> {
const nouns = new Set<string>()
const verbs = new Set<string>()
for (const gen of this.committedGens) {
if (gen <= fromGen || gen > toGen) continue
const delta = await this.getDelta(gen)
for (const id of delta.nouns) nouns.add(id)
for (const id of delta.verbs) verbs.add(id)
}
return {
nouns: [...nouns].sort(),
verbs: [...verbs].sort(),
fromGeneration: fromGen,
toGeneration: toGen
}
}
/**
* @description Assert that generation `gen` is reachable (not compacted
* away) and within the committed range, then return it. Used by `asOf()`.
* Reachability boundary: reclaiming record-set `N` removes the
* before-images that reads at generations BELOW `N` depend on, so
* generations `< horizon` are unreachable while the horizon itself remains
* servable from the record-sets above it.
* @param gen - The requested generation.
* @throws GenerationCompactedError when `gen` is below the horizon.
* @throws RangeError when `gen` is negative or beyond the current counter.
*/
assertReachable(gen: number): number {
if (!Number.isInteger(gen) || gen < 0) {
throw new RangeError(`asOf(): generation must be a non-negative integer (got ${gen})`)
}
if (gen > this.counter) {
throw new RangeError(
`asOf(): generation ${gen} is in the future (store is at generation ${this.counter})`
)
}
if (gen < this.horizonGen) {
throw new GenerationCompactedError(gen, this.horizonGen)
}
return gen
}
/**
* @description Resolve a wall-clock timestamp to the generation that was
* committed at-or-before it, via the tx-log. Resolution granularity is
* transact commits single-operation writes between commits are not
* individually addressable (documented 8.0 history granularity).
* @param timestampMs - Milliseconds since epoch.
* @returns The resolved generation (`0` when `timestampMs` predates the
* first commit) and the matched tx-log entry when one exists.
*/
async resolveTimestamp(
timestampMs: number
): Promise<{ generation: number; entry: TxLogEntry | null }> {
const lines = await this.storage.readTxLogLines()
let best: TxLogEntry | null = null
for (const line of lines) {
let entry: TxLogEntry
try {
entry = JSON.parse(line) as TxLogEntry
} catch {
continue // tolerate a torn trailing line from a crashed append
}
if (entry.timestamp <= timestampMs && entry.generation <= this.committed) {
if (best === null || entry.generation > best.generation) best = entry
}
}
return { generation: best?.generation ?? 0, entry: best }
}
/**
* @description The commit timestamp of the newest committed generation at
* or below `gen`, when its record-set is still resident. Used to populate
* `Db.timestamp` for `asOf()` values.
* @param gen - The pinned generation.
*/
async commitTimestampAtOrBefore(gen: number): Promise<number | null> {
for (let i = this.committedGens.length - 1; i >= 0; i--) {
const candidate = this.committedGens[i]
if (candidate > gen) continue
const delta = await this.getDelta(candidate)
return delta.timestamp
}
return null
}
private async getDelta(
gen: number
): Promise<{ nouns: Set<string>; verbs: Set<string>; timestamp: number }> {
const cached = this.deltaCache.get(gen)
if (cached) return cached
const delta = (await this.storage.readRawObject(
`${GENERATIONS_PREFIX}/${gen}/tx.json`
)) as GenerationDelta | null
if (delta === null) {
throw new Error(
`Generation delta missing: ${GENERATIONS_PREFIX}/${gen}/tx.json ` +
`(store corrupted or records removed outside compactHistory())`
)
}
const entry = {
nouns: new Set(delta.nouns),
verbs: new Set(delta.verbs),
timestamp: delta.timestamp
}
this.deltaCache.set(gen, entry)
return entry
}
// ==========================================================================
// Compaction
// ==========================================================================
/**
* @description Reclaim generation record-sets past the retention policy.
* A generation directory `N` may be removed only when `N` is at or below
* **every** live pin (deleting `N` can only break readers pinned *below*
* `N`, because resolution reads before-images from generations strictly
* greater than the pin). With both retention options supplied, a
* generation must satisfy both to be eligible.
*
* @param options - Retention policy (see {@link CompactHistoryOptions}).
* @returns Count of removed record-sets and the new horizon.
*/
async compact(options?: CompactHistoryOptions): Promise<CompactHistoryResult> {
return this.withMutex(async () => {
const minPinned = this.minPinnedGeneration()
const retainGenerations = options?.retainGenerations ?? 0
const countWatermark = this.committed - retainGenerations
const timeCutoff =
options?.retainMs !== undefined ? Date.now() - options.retainMs : Infinity
const removed: number[] = []
for (const gen of [...this.committedGens]) {
if (gen > countWatermark) continue
if (gen > minPinned) continue
if (timeCutoff !== Infinity) {
const delta = await this.getDelta(gen)
if (delta.timestamp >= timeCutoff) continue
}
await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${gen}`)
this.deltaCache.delete(gen)
removed.push(gen)
}
if (removed.length > 0) {
const removedSet = new Set(removed)
this.committedGens = this.committedGens.filter((gen) => !removedSet.has(gen))
this.horizonGen = Math.max(this.horizonGen, ...removed)
const manifest: GenerationManifest = {
version: 1,
generation: this.committed,
committedAt: new Date().toISOString(),
horizon: this.horizonGen
}
await this.storage.writeRawObject(MANIFEST_PATH, manifest)
await this.storage.syncRawObjects([MANIFEST_PATH])
}
return { removedGenerations: removed.length, horizon: this.horizonGen }
})
}
// ==========================================================================
// Restore support
// ==========================================================================
/**
* @description Re-read all persisted state after `brain.restore()`
* replaced the store's contents from a snapshot, enforcing counter
* monotonicity: the counter never moves below `floorGeneration` (the
* pre-restore counter), so generation numbers observed before a restore
* are never reissued.
* @param floorGeneration - The counter value before the restore.
*/
async reopenAfterRestore(floorGeneration: number): Promise<void> {
await this.withMutex(async () => {
this.deltaCache.clear()
this.opened = false
// open() re-reads counter/manifest and re-registers the bump hook.
await this.open()
if (this.counter < floorGeneration) {
this.counter = floorGeneration
await this.persistCounterUnlocked()
}
})
}
// ==========================================================================
// Internals
// ==========================================================================
/**
* Restore the before-images of an uncommitted generation to the canonical
* entity paths, then remove its record directory. Restores are idempotent
* (writing identical bytes / deleting already-absent files), so recovery
* is safe at every crash point of the commit protocol.
*/
private async rollBackUncommittedGeneration(gen: number): Promise<void> {
const dir = `${GENERATIONS_PREFIX}/${gen}`
const prevPaths = await this.storage.listRawObjects(`${dir}/prev`)
for (const recordPath of prevPaths) {
const id = recordIdFromPath(recordPath)
if (id === null) continue
const record = (await this.storage.readRawObject(recordPath)) as GenerationRecord | null
if (record === null) continue
const image = { metadata: record.metadata, vector: record.vector }
if (record.kind === 'verb') {
await this.storage.writeVerbRaw(id, image)
} else {
await this.storage.writeNounRaw(id, image)
}
}
await this.storage.removeRawPrefix(dir)
prodLog.warn(
`[GenerationStore] rolled back uncommitted generation ${gen} ` +
`(${prevPaths.length} record(s) restored)`
)
}
/**
* @description Run a snapshot section serialized against transact commits,
* compaction, and counter persistence (the store's single commit mutex),
* with the generation counter durably persisted first. Used by
* `db.persist()`: the lock guarantees a snapshot can never interleave with
* a commit's record-write/manifest-rename sequence, and the up-front
* counter persist guarantees the snapshot's `_system/generation.json`
* reflects every single-operation bump (whose persistence is otherwise
* coalesced and may still be scheduled).
* @param fn - The exclusive snapshot section.
* @returns `fn`'s result.
*/
snapshotWith<R>(fn: () => Promise<R>): Promise<R> {
return this.withMutex(async () => {
await this.persistCounterUnlocked()
return fn()
})
}
/** Run `fn` exclusively, chained behind every prior exclusive section. */
private withMutex<R>(fn: () => Promise<R>): Promise<R> {
const run = this.mutexTail.then(fn, fn)
// Keep the chain alive regardless of fn's outcome.
this.mutexTail = run.catch(() => {})
return run
}
}
/**
* @description Extract the generation number from a record path of the form
* `_generations/<N>/...`. Returns `null` for paths that don't match.
* @param path - A storage-root-relative object path.
*/
function parseGenerationFromPath(path: string): number | null {
const match = /^_generations[/\\](\d+)[/\\]/.exec(path)
if (!match) return null
const gen = Number(match[1])
return Number.isSafeInteger(gen) ? gen : null
}
/**
* @description Extract the record id (file basename without `.json`) from a
* `prev/` before-image record path. Returns `null` for non-record paths.
* @param path - A storage-root-relative object path.
*/
function recordIdFromPath(path: string): string | null {
const match = /[/\\]prev[/\\]([^/\\]+)\.json$/.exec(path)
return match ? match[1] : null
}

321
src/db/types.ts Normal file
View file

@ -0,0 +1,321 @@
/**
* @module db/types
* @description Shared types for the 8.0 generational-MVCC Db API: the
* declarative transaction-operation union consumed by `brain.transact()`,
* the options bags for `transact()` / `compactHistory()`, the persisted
* record shapes of the generational record layer, and the narrow storage
* contract (`GenerationStorage`) the record layer needs from an adapter.
*
* Persisted layout (all paths relative to the storage root):
*
* ```
* _system/generation.json { generation, updatedAt } (atomic tmp+rename)
* _system/manifest.json { version, generation, committedAt, horizon }
* (atomic tmp+rename the rename IS the commit point)
* _system/tx-log.jsonl one JSON line per committed transact:
* { generation, timestamp, meta? } (append-only)
* _generations/<N>/tx.json the generation-N delta: touched noun/verb ids + meta
* _generations/<N>/prev/<id>.json immutable before-image of <id> as of commit N
* ```
*
* Design note: the manifest holds the committed watermark + compaction
* horizon rather than a full `id → latest record generation` map. The id
* mapping is distributed across the per-generation `tx.json` deltas, which
* makes each commit O(ids touched) instead of O(all ids) see
* `docs/ADR-001-generational-mvcc.md` for the full justification.
*/
import type { AddParams, UpdateParams, RelateParams } from '../types/brainy.types.js'
// ============================================================================
// Transaction operations (brain.transact input)
// ============================================================================
/**
* @description Create an entity. Carries the same parameters as
* `brain.add()`; supply `id` to choose the entity id (otherwise a UUID v4 is
* generated and returned in the operation's planning, exactly as `add()`
* does).
*/
export interface TxAddOperation<T = any> extends AddParams<T> {
/** Discriminator. */
op: 'add'
}
/**
* @description Update an entity. Carries the same parameters as
* `brain.update()` including per-entity CAS via `ifRev` an `ifRev`
* conflict rejects the whole transaction before anything is applied.
*/
export interface TxUpdateOperation<T = any> extends UpdateParams<T> {
/** Discriminator. */
op: 'update'
}
/**
* @description Delete an entity (and, exactly like `brain.delete()`, every
* relationship where it is source or target the cascade is part of the
* same atomic batch).
*/
export interface TxRemoveOperation {
/** Discriminator. */
op: 'remove'
/** Id of the entity to delete. */
id: string
}
/**
* @description Create a relationship. Carries the same parameters as
* `brain.relate()` (including `bidirectional`). Duplicate relationships
* (same from/to/type) are deduplicated exactly as `relate()` does the
* operation becomes a no-op that resolves to the existing relationship id.
*/
export interface TxRelateOperation<T = any> extends RelateParams<T> {
/** Discriminator. */
op: 'relate'
}
/**
* @description Delete a relationship by id (mirror of `brain.unrelate()`).
*/
export interface TxUnrelateOperation {
/** Discriminator. */
op: 'unrelate'
/** Id of the relationship to delete. */
id: string
}
/**
* @description One declarative operation inside a `brain.transact()` batch.
* The batch executes atomically: either every operation applies and the
* store advances exactly one generation, or none apply and the store is
* byte-identical to its pre-transaction state.
*/
export type TxOperation<T = any> =
| TxAddOperation<T>
| TxUpdateOperation<T>
| TxRemoveOperation
| TxRelateOperation<T>
| TxUnrelateOperation
/**
* @description Options for `brain.transact()`.
*/
export interface TransactOptions {
/**
* Transaction metadata, reified Datomic-style: recorded in
* `_system/tx-log.jsonl` alongside the generation number and commit
* timestamp. Use it for audit fields (author, reason, request id)
* it replaces commit messages from the pre-8.0 versioning API.
*/
meta?: Record<string, unknown>
/**
* Whole-store compare-and-swap. When provided, the transaction commits
* only if the store's current generation equals this value; otherwise it
* throws `GenerationConflictError` (with `expected`/`actual`) before any
* record is staged.
*/
ifAtGeneration?: number
}
/**
* @description Per-operation results of a committed transaction, in input
* order. `add` resolves to the created entity id, `relate` to the created
* (or deduplicated existing) relationship id; `update`/`remove`/`unrelate`
* resolve to the id they acted on.
*/
export interface TransactReceipt {
/** The generation this transaction committed as. */
generation: number
/** Commit timestamp (ms since epoch) recorded in the tx-log. */
timestamp: number
/** Resolved id per input operation, in input order. */
ids: string[]
}
// ============================================================================
// Compaction
// ============================================================================
/**
* @description Options for `brain.compactHistory()`. When both options are
* supplied a generation must satisfy *both* retention rules to be reclaimed
* (the stricter retention wins). With no options, every generation not
* protected by a live pin is eligible.
*/
export interface CompactHistoryOptions {
/**
* Keep at least this many of the most recent committed generations
* (`0` = keep none beyond what live pins protect).
*/
retainGenerations?: number
/**
* Keep every generation committed within the last `retainMs` milliseconds.
*/
retainMs?: number
}
/**
* @description Result of `brain.compactHistory()`.
*/
export interface CompactHistoryResult {
/** Number of generation record-sets reclaimed by this call. */
removedGenerations: number
/**
* The new compaction horizon the highest generation whose record-set has
* been reclaimed. Generations BELOW the horizon are no longer reachable via
* `asOf()` (their reads would need the reclaimed before-images); the
* horizon itself stays reachable, resolved from the record-sets above it.
*/
horizon: number
}
// ============================================================================
// Db surfaces
// ============================================================================
/**
* @description Result of `db.since(priorDb)` the ids touched by committed
* transactions in the generation interval `(priorDb.generation,
* db.generation]`. Granularity note: single-operation writes performed
* outside `transact()` bump the generation counter but do not produce
* generation records, so they are not reflected here (documented 8.0
* history granularity see `docs/ADR-001-generational-mvcc.md`).
*/
export interface ChangedIds {
/** Entity (noun) ids touched in the interval, sorted. */
nouns: string[]
/** Relationship (verb) ids touched in the interval, sorted. */
verbs: string[]
/** Exclusive lower bound of the interval. */
fromGeneration: number
/** Inclusive upper bound of the interval. */
toGeneration: number
}
// ============================================================================
// Persisted record shapes
// ============================================================================
/**
* @description An immutable before-image of one entity or relationship,
* persisted under `_generations/<N>/prev/<id>.json` the state of the id
* immediately before commit `N` touched it. `metadata`/`vector` hold the
* *raw stored objects* (exact bytes that live at the entity's canonical
* storage paths); `null` means the corresponding file did not exist (for
* before-images of created ids, both are `null`). Before-images are both
* the crash-recovery undo log and the point-in-time read source: the state
* of id X at pinned generation G is the before-image of the first committed
* generation after G that touched X.
*/
export interface GenerationRecord {
/** Whether this record images an entity (noun) or a relationship (verb). */
kind: 'noun' | 'verb'
/** Raw stored metadata object, or `null` if the metadata file was absent. */
metadata: unknown | null
/** Raw stored vector object, or `null` if the vector file was absent. */
vector: unknown | null
}
/**
* @description The per-generation delta persisted at
* `_generations/<N>/tx.json` written and fsynced *before* any operation of
* the transaction executes, so crash recovery always knows exactly which ids
* to restore from before-images.
*/
export interface GenerationDelta {
/** The generation this delta belongs to. */
generation: number
/** Commit timestamp (ms since epoch). */
timestamp: number
/** Transaction metadata (mirrored into the tx-log on commit). */
meta?: Record<string, unknown>
/** Entity ids touched by this generation. */
nouns: string[]
/** Relationship ids touched by this generation. */
verbs: string[]
}
/**
* @description Shape of `_system/manifest.json`. Replaced via atomic
* tmp+rename on every commit the rename is the commit point: a generation
* directory is committed if and only if its number is `generation`.
*/
export interface GenerationManifest {
/** Schema version of the manifest file. */
version: 1
/** Committed-transaction watermark. */
generation: number
/** ISO timestamp of the most recent commit (or compaction). */
committedAt: string
/** Compaction horizon — record-sets for generations ≤ this are reclaimed. */
horizon: number
}
/**
* @description One line of `_system/tx-log.jsonl`.
*/
export interface TxLogEntry {
/** The committed generation. */
generation: number
/** Commit timestamp (ms since epoch). */
timestamp: number
/** Transaction metadata, when supplied to `transact()`. */
meta?: Record<string, unknown>
}
// ============================================================================
// Storage contract
// ============================================================================
/**
* @description The narrow storage contract the generational record layer
* needs. `BaseStorage` implements it structurally (both shipped adapters
* filesystem and memory inherit the implementation), so any
* `StorageAdapter` produced by the 8.0 storage factory satisfies it.
*
* Raw-object methods bypass branch scoping and operate on storage-root-
* relative paths (the record layer owns `_system/` + `_generations/`);
* entity-raw methods read/write the canonical entity files (branch-scoped,
* write-cache coherent) so before/after images capture exactly the bytes
* the live read paths see.
*/
export interface GenerationStorage {
/** Read a raw object at a storage-root-relative path (`null` if absent). */
readRawObject(path: string): Promise<any | null>
/** Write a raw object at a storage-root-relative path (atomic tmp+rename on disk). */
writeRawObject(path: string, data: any): Promise<void>
/** Delete a raw object (no-op if absent). */
deleteRawObject(path: string): Promise<void>
/** List raw object paths under a prefix (normalized, `.gz`-stripped). */
listRawObjects(prefix: string): Promise<string[]>
/** Remove every object under a prefix (and the directory itself on disk). */
removeRawPrefix(prefix: string): Promise<void>
/** Durability barrier: fsync the given object paths (no-op in memory). */
syncRawObjects(paths: string[]): Promise<void>
/** Read an entity's raw stored metadata+vector objects. */
readNounRaw(id: string): Promise<{ metadata: any | null; vector: any | null }>
/** Restore an entity's raw stored objects (`null` part ⇒ delete that file). */
writeNounRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise<void>
/** Read a relationship's raw stored metadata+vector objects. */
readVerbRaw(id: string): Promise<{ metadata: any | null; vector: any | null }>
/** Restore a relationship's raw stored objects (`null` part ⇒ delete that file). */
writeVerbRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise<void>
/** Append one line to `_system/tx-log.jsonl`. */
appendTxLogLine(line: string): Promise<void>
/** Read all lines of `_system/tx-log.jsonl` (empty array if absent). */
readTxLogLines(): Promise<string[]>
/**
* Register the generation-bump hook invoked on every entity-visible
* single-operation write (see `BaseStorage.setGenerationBumpHook`).
*/
setGenerationBumpHook(hook: (() => void) | undefined): void
/** Snapshot the entire store to a directory (hard-link farm on disk). */
snapshotToDirectory(targetPath: string): Promise<void>
/** Replace the entire store's contents from a snapshot directory. */
restoreFromDirectory(sourcePath: string): Promise<void>
}

292
src/db/whereMatcher.ts Normal file
View file

@ -0,0 +1,292 @@
/**
* @module db/whereMatcher
* @description In-memory evaluation of `find()` metadata filters against a
* single resolved entity the overlay half of historical and speculative
* reads on a `Db` value.
*
* A `Db` pinned at a past generation answers metadata-level `find()` by
* combining the live index's results (for entities untouched since the pin)
* with per-entity evaluation of the same filter against generation records
* (for entities that DID change). The same evaluator backs `db.with()`
* speculative overlays. The semantics here deliberately mirror the index
* evaluator in `src/utils/metadataIndex.ts` (`getIdsForFilter`) operator
* aliases, array-membership equality, `exists`/`missing`, and the
* `allOf`/`anyOf`/`not` logical forms so an entity matches in-memory if and
* only if it would have matched through the index.
*
* **Honesty contract:** an operator this module does not recognize throws
* {@link UnsupportedWhereOperatorError} instead of guessing a historical
* read must never silently return wrong results.
*/
import type { Entity, FindParams } from '../types/brainy.types.js'
/**
* @description Thrown when a `where` clause uses an operator this in-memory
* evaluator does not implement. Callers (historical `find()` on a `Db`)
* convert this into the documented historical-query error rather than
* returning silently-wrong results.
*/
export class UnsupportedWhereOperatorError extends Error {
/** The unrecognized operator name. */
public readonly operator: string
/**
* @param operator - The operator name that could not be evaluated.
*/
constructor(operator: string) {
super(
`The where-operator '${operator}' is not supported for historical/speculative ` +
`in-memory evaluation. Supported: eq/equals/is, ne/notEquals/isNot, in/oneOf, ` +
`gt/greaterThan, gte/greaterThanOrEqual/greaterEqual, lt/lessThan, ` +
`lte/lessThanOrEqual/lessEqual, between, contains, exists, missing, ` +
`plus allOf/anyOf/not.`
)
this.name = 'UnsupportedWhereOperatorError'
this.operator = operator
}
}
/**
* @description Resolve a filter field name to its value on an entity,
* mirroring how `extractIndexableFields` lays entities out for the metadata
* index: standard fields live at the top level (with the `noun` `type`
* alias), everything else is a custom field inside the `metadata` bag.
* Dotted paths (`metadata.priority`, `address.city`) traverse nested objects.
*
* @param entity - The resolved entity to read from.
* @param field - The filter field name.
* @returns The field's value, or `undefined` when absent.
*/
export function resolveEntityField(entity: Entity, field: string): unknown {
switch (field) {
case 'noun':
case 'type':
return entity.type
case 'subtype':
return entity.subtype
case 'id':
return entity.id
case 'createdAt':
return entity.createdAt
case 'updatedAt':
return entity.updatedAt
case 'service':
return entity.service
case 'createdBy':
return entity.createdBy
case 'confidence':
return entity.confidence
case 'weight':
return entity.weight
case '_rev':
return entity._rev
case 'data':
return entity.data
}
if (field.includes('.')) {
// Dotted path: resolve against the whole entity first (`metadata.x`),
// then against the metadata bag (`address.city` on nested metadata).
const fromEntity = resolvePath(entity as unknown as Record<string, unknown>, field)
if (fromEntity !== undefined) return fromEntity
return resolvePath((entity.metadata ?? {}) as Record<string, unknown>, field)
}
return ((entity.metadata ?? {}) as Record<string, unknown>)[field]
}
/** Walk a dotted path through nested plain objects. */
function resolvePath(obj: Record<string, unknown>, path: string): unknown {
let current: unknown = obj
for (const segment of path.split('.')) {
if (current === null || typeof current !== 'object') return undefined
current = (current as Record<string, unknown>)[segment]
}
return current
}
/**
* @description Index-equality semantics: scalar strict equality, with
* array-valued fields matching by membership (the index stores one posting
* per array element, so `eq` on an array field means "contains").
*/
function eqMatches(value: unknown, operand: unknown): boolean {
if (Array.isArray(value)) {
return value.some((element) => element === operand)
}
return value === operand
}
/** Ordered comparison for range operators (numbers, or both-strings lexicographic). */
function compare(value: unknown, operand: unknown): number | null {
if (typeof value === 'number' && typeof operand === 'number') {
return value - operand
}
if (typeof value === 'string' && typeof operand === 'string') {
return value < operand ? -1 : value > operand ? 1 : 0
}
return null
}
/**
* @description Evaluate one field condition (shorthand value or operator
* object) against a resolved field value, mirroring the operator table in
* `metadataIndex.getIdsForFilter`.
*
* @param value - The entity's field value (possibly `undefined`).
* @param condition - The filter condition for this field.
* @returns Whether the value satisfies the condition.
* @throws UnsupportedWhereOperatorError for unrecognized operators.
*/
function fieldConditionMatches(value: unknown, condition: unknown): boolean {
if (condition === null || typeof condition !== 'object' || Array.isArray(condition)) {
// Shorthand for 'eq'.
return eqMatches(value, condition)
}
for (const [op, operand] of Object.entries(condition as Record<string, unknown>)) {
let matches: boolean
switch (op) {
case 'is':
case 'equals':
case 'eq':
matches = eqMatches(value, operand)
break
case 'isNot':
case 'notEquals':
case 'ne':
matches = !eqMatches(value, operand)
break
case 'oneOf':
case 'in':
matches = Array.isArray(operand) && operand.some((candidate) => eqMatches(value, candidate))
break
case 'greaterThan':
case 'gt': {
const cmp = compare(value, operand)
matches = cmp !== null && cmp > 0
break
}
case 'greaterEqual':
case 'greaterThanOrEqual':
case 'gte': {
const cmp = compare(value, operand)
matches = cmp !== null && cmp >= 0
break
}
case 'lessThan':
case 'lt': {
const cmp = compare(value, operand)
matches = cmp !== null && cmp < 0
break
}
case 'lessEqual':
case 'lessThanOrEqual':
case 'lte': {
const cmp = compare(value, operand)
matches = cmp !== null && cmp <= 0
break
}
case 'between': {
if (!Array.isArray(operand) || operand.length !== 2) {
matches = false
break
}
const lower = compare(value, operand[0])
const upper = compare(value, operand[1])
matches = lower !== null && upper !== null && lower >= 0 && upper <= 0
break
}
case 'contains':
// Index semantics: array fields post one entry per element, so
// 'contains' is the same lookup as 'eq' (membership on arrays).
matches = eqMatches(value, operand)
break
case 'exists':
matches = operand ? value !== undefined : value === undefined
break
case 'missing':
matches = operand ? value === undefined : value !== undefined
break
default:
throw new UnsupportedWhereOperatorError(op)
}
if (!matches) return false // Multiple operators on one field AND together.
}
return true
}
/**
* @description Evaluate a full `where` filter (field conditions plus
* `allOf`/`anyOf`/`not` logical composition) against one entity.
*
* @param entity - The resolved entity (historical record or speculative overlay).
* @param where - The `where` clause from `FindParams`.
* @returns Whether the entity satisfies every clause.
* @throws UnsupportedWhereOperatorError for unrecognized operators.
*/
export function whereMatches(entity: Entity, where: Record<string, unknown>): boolean {
for (const [field, condition] of Object.entries(where)) {
if (field === 'allOf') {
if (!Array.isArray(condition)) return false
if (!condition.every((sub) => whereMatches(entity, sub as Record<string, unknown>))) return false
continue
}
if (field === 'anyOf') {
if (!Array.isArray(condition)) return false
if (!condition.some((sub) => whereMatches(entity, sub as Record<string, unknown>))) return false
continue
}
if (field === 'not') {
if (whereMatches(entity, condition as Record<string, unknown>)) return false
continue
}
if (!fieldConditionMatches(resolveEntityField(entity, field), condition)) return false
}
return true
}
/**
* @description Evaluate the metadata-level portions of a `find()` query
* (`type`, `subtype`, `where`, `service`, `excludeVFS`) against one resolved
* entity. Used by historical and speculative `find()` to decide whether a
* changed/overlaid entity belongs in the result set. The caller guarantees
* the query carries no index-only dimensions (semantic `query`/`vector`,
* `connected` traversal, ) those are rejected with the documented
* historical-query error before evaluation starts.
*
* @param entity - The resolved entity.
* @param params - The metadata-level find parameters.
* @returns Whether the entity matches every requested filter.
* @throws UnsupportedWhereOperatorError for unrecognized `where` operators.
*/
export function entityMatchesFind(entity: Entity, params: FindParams): boolean {
if (params.type !== undefined) {
const types = Array.isArray(params.type) ? params.type : [params.type]
if (!types.includes(entity.type)) return false
}
if (params.subtype !== undefined) {
const subtypes = Array.isArray(params.subtype) ? params.subtype : [params.subtype]
if (entity.subtype === undefined || !subtypes.includes(entity.subtype)) return false
}
if (params.service !== undefined && entity.service !== params.service) {
return false
}
if (params.excludeVFS === true) {
// Mirror of find()'s exclusion filter (`vfsType: { exists: false }`,
// `isVFSEntity: { ne: true }`) and Brainy's VFS-marker helper.
const metadata = (entity.metadata ?? {}) as Record<string, unknown>
if (metadata.vfsType !== undefined) return false
if (metadata.isVFSEntity === true || metadata.isVFS === true) return false
}
if (params.where !== undefined) {
if (!whereMatches(entity, params.where as Record<string, unknown>)) return false
}
return true
}

View file

@ -128,6 +128,32 @@ export type { Migration, MigrationState, MigrationPreview, MigrationResult, Migr
// Export optimistic-concurrency types (7.31.0) // Export optimistic-concurrency types (7.31.0)
export { RevisionConflictError } from './transaction/RevisionConflictError.js' export { RevisionConflictError } from './transaction/RevisionConflictError.js'
// ============= 8.0 Db API — generational MVCC =============
// Immutable database values: brain.now() / brain.transact() / brain.asOf() /
// db.with() / db.persist() / Brainy.load(). See src/db/ for the record layer.
export { Db } from './db/db.js'
export {
GenerationConflictError,
NotYetSupportedAtHistoricalGenerationError,
GenerationCompactedError
} from './db/errors.js'
export type {
TxOperation,
TxAddOperation,
TxUpdateOperation,
TxRemoveOperation,
TxRelateOperation,
TxUnrelateOperation,
TransactOptions,
TransactReceipt,
CompactHistoryOptions,
CompactHistoryResult,
ChangedIds
} from './db/types.js'
// Optional provider capability for generation-aware native indexes
export { isVersionedIndexProvider } from './plugin.js'
export type { VersionedIndexProvider } from './plugin.js'
// Export embedding functionality // Export embedding functionality
import { import {
UniversalSentenceEncoder, UniversalSentenceEncoder,

View file

@ -259,6 +259,108 @@ export interface GraphIndexProvider {
getAllRelationshipCounts(): Map<string, number> getAllRelationshipCounts(): Map<string, number>
} }
/**
* Optional capability interface for index providers that maintain versioned
* (generation-aware) internal state the provider-side half of Brainy 8.0's
* generational MVCC contract. Implemented by native providers whose storage
* engines keep immutable versions (e.g. LSM snapshots); Brainy's own JS
* indexes do not implement it (they are rebuilt from the storage records,
* which carry the versioning).
*
* **Detection.** Brainy feature-detects this interface on every registered
* index provider (`'vector'`, `'metadataIndex'`, `'graphIndex'`): when a
* provider exposes `pin`/`release` functions, Brainy calls them in lockstep
* with `Db` lifecycle `pin(g)` when a `Db` value pins generation `g`
* (`brain.now()`, `brain.transact()`, `brain.asOf()`), `release(g)` when that
* `Db` is released (explicitly or via the GC backstop). Pins are refcounted
* on the Brainy side; a provider may receive multiple `pin(g)` calls for the
* same generation and will receive exactly one matching `release(g)` per pin.
*
* **Consistency model (locked cross-team design).** Index providers are
* *post-commit appliers*: the storage-record commit (atomic manifest rename)
* is the source of truth, and provider index state is derived, applied after
* the commit point. On open, a provider compares its own persisted
* `generation()` against the store's committed generation and replays the
* gap from the storage records (or requests a rebuild) there are no
* provider rollback hooks, because an uncommitted transaction is repaired at
* the storage layer before any index is opened. The explicit pin/release
* lifetime OVERRIDES any time-based snapshot retention the provider has
* (e.g. an LSM snapshot TTL): a pinned generation must stay readable until
* released, regardless of age.
*
* **Speculative reads.** `db.with(txData)` overlays are Brainy-side only
* providers are never asked to read uncommitted/speculative state; they
* always serve committed generations.
*/
export interface VersionedIndexProvider {
/**
* @description The newest generation this provider's persisted index state
* reflects. Brainy compares it against the storage layer's committed
* generation on open to detect a replay gap (index behind storage after a
* crash between commit and index apply).
* @returns The provider's current generation as a `bigint` (u64 at the
* native boundary; Brainy's generation counter is a safe integer today).
*/
generation(): bigint
/**
* @description True the provider can serve consistent reads at
* `generation` (segments retained). Combined with pin: pinning a VISIBLE
* generation guarantees it stays servable until release. Pinning an
* invisible generation is permitted (refcount-only) Brainy serves that
* generation from canonical storage instead.
*
* Brainy consults this at pin time (the read-routing rule: a `Db` at
* generation `g` uses provider-accelerated reads when
* `isGenerationVisible(g)` was true at pin time; otherwise canonical
* generation records).
* @param generation - The generation a `Db` value is about to pin.
*/
isGenerationVisible(generation: bigint): boolean
/**
* @description Pin a generation: the provider must keep index state for
* `generation` readable until the matching {@link VersionedIndexProvider.release}
* call, overriding any time-based snapshot retention. Called once per
* Brainy-side pin (refcounted upstream expect balanced pin/release pairs).
* @param generation - The generation a live `Db` value just pinned.
*/
pin(generation: bigint): void
/**
* @description Release one pin on `generation`. After the last release the
* provider may reclaim resources for that generation at its discretion.
* @param generation - The generation being released (matches a prior
* {@link VersionedIndexProvider.pin} call).
*/
release(generation: bigint): void
}
/**
* @description Feature-detection guard for {@link VersionedIndexProvider}.
* Brainy applies it to every registered index provider (vector, metadata,
* graph) when a `Db` value pins or releases a generation: providers exposing
* all four capability methods receive lockstep `pin`/`release` calls;
* everything else (including Brainy's own JS indexes) is skipped.
*
* @param candidate - A registered index provider instance.
* @returns Whether the candidate implements the versioned capability.
*/
export function isVersionedIndexProvider(
candidate: unknown
): candidate is VersionedIndexProvider {
if (candidate === null || typeof candidate !== 'object') {
return false
}
const c = candidate as Record<string, unknown>
return (
typeof c.generation === 'function' &&
typeof c.isGenerationVisible === 'function' &&
typeof c.pin === 'function' &&
typeof c.release === 'function'
)
}
/** /**
* The object returned by the `'vector'` provider factory Brainy's vector * The object returned by the `'vector'` provider factory Brainy's vector
* index contract. Implementations include Brainy's own JS HNSW index and any * index contract. Implementations include Brainy's own JS HNSW index and any

View file

@ -954,6 +954,254 @@ export class FileSystemStorage extends BaseStorage {
} }
} }
// ===========================================================================
// Generational record layer (8.0 MVCC) — durability + snapshot primitives
// ===========================================================================
/**
* Storage-root-relative paths that are mutated **in place** (appended to)
* rather than replaced via atomic tmp+rename. `snapshotToDirectory()` must
* byte-copy these instead of hard-linking them: a hard link shares the
* inode, so a post-snapshot append to the live file would silently mutate
* the snapshot. Every other persisted file in this adapter is written via
* tmp+rename (objects, blobs, counts, locks are excluded entirely), which
* makes hard links safe a rewrite swaps in a new inode and the snapshot
* keeps the old one.
*/
private static readonly SNAPSHOT_BYTE_COPY_PATHS = new Set<string>([
`${SYSTEM_DIR}/tx-log.jsonl`
])
/**
* Top-level directories excluded from snapshots: process-local lock state
* (writer lock, flush-request RPC files) must never travel with the data.
*/
private static readonly SNAPSHOT_EXCLUDED_TOP_DIRS = new Set<string>(['locks'])
/**
* Remove every object under a storage-root-relative prefix one recursive
* directory removal instead of the base class's list+delete loop.
*
* @param prefix - Storage-root-relative directory prefix to remove.
*/
public async removeRawPrefix(prefix: string): Promise<void> {
await this.ensureInitialized()
await fs.promises.rm(path.join(this.rootDir, prefix), { recursive: true, force: true })
}
/**
* Durability barrier: `fsync` each listed object file (resolving the
* compressed `.gz` variant when present) and then the set of parent
* directories, so both the file contents and the rename directory entries
* are durable before the commit protocol proceeds. Paths whose file no
* longer exists are skipped (a later step may have replaced them).
*
* @param paths - Storage-root-relative object paths previously written.
*/
public async syncRawObjects(paths: string[]): Promise<void> {
await this.ensureInitialized()
const parentDirs = new Set<string>()
for (const objectPath of paths) {
const fullPath = path.join(this.rootDir, objectPath)
for (const candidate of [`${fullPath}.gz`, fullPath]) {
let handle: any
try {
handle = await fs.promises.open(candidate, 'r')
} catch (error: any) {
if (error.code === 'ENOENT') continue
throw error
}
try {
await handle.sync()
} finally {
await handle.close()
}
parentDirs.add(path.dirname(fullPath))
break
}
}
for (const dir of parentDirs) {
let handle: any
try {
handle = await fs.promises.open(dir, 'r')
} catch {
continue // directory vanished or platform disallows opening dirs
}
try {
await handle.sync()
} catch {
// Some platforms (and some filesystems) reject directory fsync —
// file-level fsync above already covers the data itself.
} finally {
await handle.close()
}
}
}
/**
* Append one line to `_system/tx-log.jsonl`. Plain `appendFile` the
* tx-log is the one append-in-place file in the store (and is byte-copied,
* never hard-linked, by {@link FileSystemStorage.snapshotToDirectory}).
*
* @param line - One complete JSON document, without trailing newline.
*/
public async appendTxLogLine(line: string): Promise<void> {
await this.ensureInitialized()
const logPath = path.join(this.systemDir, 'tx-log.jsonl')
await fs.promises.appendFile(logPath, `${line}\n`, 'utf-8')
}
/**
* Read all tx-log lines, oldest first (empty array when no log exists).
* Torn trailing lines from a crashed append are returned as-is callers
* tolerate unparseable lines.
*/
public async readTxLogLines(): Promise<string[]> {
await this.ensureInitialized()
const logPath = path.join(this.systemDir, 'tx-log.jsonl')
try {
const content: string = await fs.promises.readFile(logPath, 'utf-8')
return content.split('\n').filter((l: string) => l.length > 0)
} catch (error: any) {
if (error.code === 'ENOENT') return []
throw error
}
}
/**
* Snapshot the entire store into `targetPath` as a hard-link farm
* (Cassandra-style: instant, space-shared). Safe because every data file
* is immutable-by-rename rewrites swap in new inodes, leaving the
* snapshot's links pointing at the old bytes. The two exceptions are
* handled explicitly: append-in-place files
* ({@link FileSystemStorage.SNAPSHOT_BYTE_COPY_PATHS}) are byte-copied, and
* process-local lock state
* ({@link FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS}) is excluded.
* Cross-device targets (where `link(2)` fails with `EXDEV`) fall back to
* byte copies per file.
*
* @param targetPath - Absolute directory for the snapshot. Created if
* missing; must be empty or absent (refuses to overwrite).
*/
public async snapshotToDirectory(targetPath: string): Promise<void> {
await this.ensureInitialized()
try {
const existing = await fs.promises.readdir(targetPath)
if (existing.length > 0) {
throw new Error(
`snapshotToDirectory: target ${targetPath} already exists and is not empty. ` +
`Choose a fresh directory per snapshot.`
)
}
} catch (error: any) {
if (error.code !== 'ENOENT') throw error
}
await fs.promises.mkdir(targetPath, { recursive: true })
const files: string[] = []
await this.collectSnapshotFiles(this.rootDir, '', files)
for (const relPath of files) {
const sourceFile = path.join(this.rootDir, relPath)
const targetFile = path.join(targetPath, relPath)
await fs.promises.mkdir(path.dirname(targetFile), { recursive: true })
// Byte-copy list: compare against the normalized (extension-preserving)
// relative path with separators unified, so `_system/tx-log.jsonl`
// matches on every platform.
const normalized = relPath.split(path.sep).join('/')
if (FileSystemStorage.SNAPSHOT_BYTE_COPY_PATHS.has(normalized)) {
await fs.promises.copyFile(sourceFile, targetFile)
continue
}
try {
await fs.promises.link(sourceFile, targetFile)
} catch (error: any) {
// EXDEV: cross-device; EPERM/ENOTSUP: filesystem forbids links.
// ENOENT: the live file was atomically replaced mid-walk — retry as
// a copy of whatever is current (single-writer discipline means this
// only happens for derived files being flushed concurrently).
if (['EXDEV', 'EPERM', 'ENOTSUP', 'ENOENT'].includes(error.code)) {
try {
await fs.promises.copyFile(sourceFile, targetFile)
} catch (copyError: any) {
if (copyError.code !== 'ENOENT') throw copyError
}
} else {
throw error
}
}
}
}
/**
* Recursively collect snapshot-eligible files under `dirAbs`, excluding the
* lock directory and in-flight `*.tmp.*` write files.
*/
private async collectSnapshotFiles(dirAbs: string, relPrefix: string, out: string[]): Promise<void> {
let entries: any[]
try {
entries = await fs.promises.readdir(dirAbs, { withFileTypes: true })
} catch (error: any) {
if (error.code === 'ENOENT') return
throw error
}
for (const entry of entries) {
const rel = relPrefix ? path.join(relPrefix, entry.name) : entry.name
if (entry.isDirectory()) {
if (relPrefix === '' && FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS.has(entry.name)) {
continue
}
await this.collectSnapshotFiles(path.join(dirAbs, entry.name), rel, out)
} else if (entry.isFile()) {
if (entry.name.includes('.tmp.')) continue // in-flight atomic write
out.push(rel)
}
}
}
/**
* Replace the store's contents from a snapshot directory: every current
* top-level entry except `locks/` (the live writer lock must survive) is
* removed, the snapshot is byte-copied in (`fs.cp` never hard-linked, so
* the snapshot stays independent of the restored store), and all
* adapter-internal derived state is reloaded.
*
* @param sourcePath - Absolute path of a directory produced by
* {@link FileSystemStorage.snapshotToDirectory}.
*/
public async restoreFromDirectory(sourcePath: string): Promise<void> {
await this.ensureInitialized()
const sourceStat = await fs.promises.stat(sourcePath).catch(() => null)
if (!sourceStat || !sourceStat.isDirectory()) {
throw new Error(`restoreFromDirectory: ${sourcePath} is not a directory`)
}
// Clear current contents, preserving live lock state.
const currentEntries = await fs.promises.readdir(this.rootDir)
for (const entry of currentEntries) {
if (FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS.has(entry)) continue
await fs.promises.rm(path.join(this.rootDir, entry), { recursive: true, force: true })
}
// Byte-copy the snapshot in (defensively skipping lock dirs in old snapshots).
const sourceEntries = await fs.promises.readdir(sourcePath)
for (const entry of sourceEntries) {
if (FileSystemStorage.SNAPSHOT_EXCLUDED_TOP_DIRS.has(entry)) continue
await fs.promises.cp(path.join(sourcePath, entry), path.join(this.rootDir, entry), {
recursive: true
})
}
await this.reloadDerivedState()
}
// =========================================================================== // ===========================================================================
// Raw binary-blob primitive (mmap-friendly) // Raw binary-blob primitive (mmap-friendly)
// =========================================================================== // ===========================================================================

View file

@ -525,4 +525,47 @@ export class HistoricalStorageAdapter extends BaseStorage {
protected async persistCounts(): Promise<void> { protected async persistCounts(): Promise<void> {
// No-op: Historical storage is read-only // No-op: Historical storage is read-only
} }
// ===========================================================================
// Generational record layer (8.0 MVCC) — not applicable to this adapter
//
// HistoricalStorageAdapter is the pre-8.0 commit-based time-travel view and
// is read-only by construction. The generation store never registers its
// write hooks on reader-mode instances, so these methods are unreachable in
// normal operation; they throw explicitly rather than fake behavior.
// ===========================================================================
/**
* WRITE BLOCKED: the tx-log belongs to the live store, not a historical view.
*/
public async appendTxLogLine(_line: string): Promise<void> {
throw new Error('Historical storage is read-only. Cannot append to the transaction log.')
}
/**
* Not supported: historical commit views predate the generational tx-log.
*/
public async readTxLogLines(): Promise<string[]> {
throw new Error(
'Transaction-log reads are not supported on a historical commit view. ' +
'Use the live store (or a Db from brain.now()/brain.asOf()).'
)
}
/**
* Not supported: snapshot a live store via db.persist(path) instead.
*/
public async snapshotToDirectory(_targetPath: string): Promise<void> {
throw new Error(
'snapshotToDirectory is not supported on a historical commit view. ' +
'Call db.persist(path) on the live store instead.'
)
}
/**
* WRITE BLOCKED: cannot restore into a read-only historical view.
*/
public async restoreFromDirectory(_sourcePath: string): Promise<void> {
throw new Error('Historical storage is read-only. Cannot restore from a snapshot.')
}
} }

View file

@ -3,6 +3,9 @@
* In-memory storage adapter for environments where persistent storage is not available or needed * In-memory storage adapter for environments where persistent storage is not available or needed
*/ */
import * as fs from 'node:fs'
import * as nodePath from 'node:path'
import * as zlib from 'node:zlib'
import { import {
GraphVerb, GraphVerb,
HNSWNoun, HNSWNoun,
@ -35,6 +38,11 @@ export class MemoryStorage extends BaseStorage {
// storage has no local file, so getBinaryBlobPath() returns null. // storage has no local file, so getBinaryBlobPath() returns null.
private blobStore: Map<string, Buffer> = new Map() private blobStore: Map<string, Buffer> = new Map()
// Transaction log (8.0 MVCC) — the in-memory equivalent of the filesystem
// adapter's `_system/tx-log.jsonl`. One JSON document per committed
// transact(); serialized to a real JSONL file by snapshotToDirectory().
private txLogLines: string[] = []
// Backward compatibility aliases // Backward compatibility aliases
private get metadata(): Map<string, any> { private get metadata(): Map<string, any> {
return this.objectStore return this.objectStore
@ -189,6 +197,132 @@ export class MemoryStorage extends BaseStorage {
return null return null
} }
// ===========================================================================
// Generational record layer (8.0 MVCC) — tx-log + snapshot primitives
// ===========================================================================
/**
* Append one line to the in-memory transaction log (mirror of the
* filesystem adapter's `_system/tx-log.jsonl` append).
*
* @param line - One complete JSON document, without trailing newline.
*/
public async appendTxLogLine(line: string): Promise<void> {
this.txLogLines.push(line)
}
/**
* Read all transaction-log lines, oldest first (a copy callers cannot
* mutate the log).
*/
public async readTxLogLines(): Promise<string[]> {
return [...this.txLogLines]
}
/**
* Serialize the entire in-memory store to a directory in the exact layout
* the filesystem adapter uses (uncompressed JSON objects, `_blobs/*.bin`
* payloads, `_system/tx-log.jsonl`). The resulting directory is a
* self-contained store openable via `Brainy.load(path)` persisting an
* in-memory brain produces a real, durable snapshot.
*
* @param targetPath - Absolute directory for the snapshot. Created if
* missing; must be empty or absent (refuses to overwrite).
*/
public async snapshotToDirectory(targetPath: string): Promise<void> {
try {
const existing = await fs.promises.readdir(targetPath)
if (existing.length > 0) {
throw new Error(
`snapshotToDirectory: target ${targetPath} already exists and is not empty. ` +
`Choose a fresh directory per snapshot.`
)
}
} catch (error: any) {
if (error.code !== 'ENOENT') throw error
}
await fs.promises.mkdir(targetPath, { recursive: true })
for (const [key, value] of this.objectStore.entries()) {
const filePath = nodePath.join(targetPath, ...key.split('/'))
await fs.promises.mkdir(nodePath.dirname(filePath), { recursive: true })
await fs.promises.writeFile(filePath, JSON.stringify(value, null, 2), 'utf-8')
}
for (const [key, data] of this.blobStore.entries()) {
const blobPath = nodePath.join(targetPath, '_blobs', ...key.split('/')) + '.bin'
await fs.promises.mkdir(nodePath.dirname(blobPath), { recursive: true })
await fs.promises.writeFile(blobPath, data)
}
if (this.txLogLines.length > 0) {
const logPath = nodePath.join(targetPath, '_system', 'tx-log.jsonl')
await fs.promises.mkdir(nodePath.dirname(logPath), { recursive: true })
await fs.promises.writeFile(logPath, this.txLogLines.map((l) => `${l}\n`).join(''), 'utf-8')
}
}
/**
* Replace the in-memory store's contents from a snapshot directory
* accepts snapshots produced by either adapter (handles the filesystem
* adapter's gzip-compressed `.json.gz` objects transparently), then reloads
* all derived state.
*
* @param sourcePath - Absolute path of a snapshot directory.
*/
public async restoreFromDirectory(sourcePath: string): Promise<void> {
const sourceStat = await fs.promises.stat(sourcePath).catch(() => null)
if (!sourceStat || !sourceStat.isDirectory()) {
throw new Error(`restoreFromDirectory: ${sourcePath} is not a directory`)
}
this.objectStore.clear()
this.blobStore.clear()
this.txLogLines = []
this.statistics = null
await this.loadDirectoryIntoStore(sourcePath, '')
await this.reloadDerivedState()
}
/**
* Recursively load a snapshot directory into the object/blob/tx-log stores.
* Lock directories from filesystem-adapter snapshots are skipped (process-
* local state, meaningless in memory).
*/
private async loadDirectoryIntoStore(dirAbs: string, relPrefix: string): Promise<void> {
const entries = await fs.promises.readdir(dirAbs, { withFileTypes: true })
for (const entry of entries) {
const rel = relPrefix ? `${relPrefix}/${entry.name}` : entry.name
const abs = nodePath.join(dirAbs, entry.name)
if (entry.isDirectory()) {
if (relPrefix === '' && entry.name === 'locks') continue
await this.loadDirectoryIntoStore(abs, rel)
continue
}
if (!entry.isFile()) continue
if (rel === '_system/tx-log.jsonl') {
const content = await fs.promises.readFile(abs, 'utf-8')
this.txLogLines = content.split('\n').filter((l) => l.length > 0)
} else if (relPrefix.startsWith('_blobs') && rel.endsWith('.bin')) {
const key = rel.slice('_blobs/'.length, -'.bin'.length)
this.blobStore.set(key, await fs.promises.readFile(abs))
} else if (rel.endsWith('.json.gz') || rel.endsWith('.gz')) {
const raw = await fs.promises.readFile(abs)
const key = rel.slice(0, -'.gz'.length)
this.objectStore.set(key, JSON.parse(zlib.gunzipSync(raw).toString('utf-8')))
} else if (entry.name.includes('.tmp.')) {
// In-flight atomic-write remnant from a crashed filesystem store — skip.
} else {
const content = await fs.promises.readFile(abs, 'utf-8')
this.objectStore.set(rel, JSON.parse(content))
}
}
}
/** /**
* Get multiple metadata objects in batches (CRITICAL: Prevents socket exhaustion) * Get multiple metadata objects in batches (CRITICAL: Prevents socket exhaustion)
* Memory storage implementation is simple since all data is already in memory * Memory storage implementation is simple since all data is already in memory
@ -217,6 +351,7 @@ export class MemoryStorage extends BaseStorage {
public async clear(): Promise<void> { public async clear(): Promise<void> {
this.objectStore.clear() this.objectStore.clear()
this.blobStore.clear() this.blobStore.clear()
this.txLogLines = []
this.statistics = null this.statistics = null
this.totalNounCount = 0 this.totalNounCount = 0
this.totalVerbCount = 0 this.totalVerbCount = 0

View file

@ -881,6 +881,271 @@ export abstract class BaseStorage extends BaseStorageAdapter {
return Array.from(pathsSet) return Array.from(pathsSet)
} }
// ============================================================================
// GENERATIONAL RECORD LAYER PRIMITIVES (8.0 MVCC)
//
// The narrow surface `GenerationStore` (src/db/generationStore.ts) needs from
// an adapter — see the `GenerationStorage` contract in src/db/types.ts.
//
// Raw-object methods operate on storage-root-relative paths and deliberately
// bypass branch scoping and the write cache: the record layer owns the
// `_system/` + `_generations/` areas outright and its files must never be
// shadowed by branch resolution. Entity-raw methods, by contrast, go through
// the branch-scoped, write-cache-coherent helpers so before-images capture
// exactly the bytes the live read paths see.
// ============================================================================
/**
* Hook invoked after every entity-visible single-operation write (noun/verb
* metadata save or delete). Registered by the generation store so
* `brain.generation()` advances on writes performed outside `transact()`.
* See {@link BaseStorage.setGenerationBumpHook}.
*/
protected generationBumpHook?: () => void
/**
* Register (or detach, with `undefined`) the generation-bump hook.
*
* The hook fires once per entity-visible metadata mutation noun/verb
* metadata saves and deletes, the one storage write every logical Brainy
* mutation (`add`, `update`, `delete`, `relate`, `updateRelation`,
* `unrelate`) performs exactly once per entity it touches. It does NOT fire
* for derived-index writes (HNSW node data, metadata-index chunks, LSM
* segments, statistics), so the generation counter tracks *data* mutations,
* not index maintenance. The counter is a monotonic watermark, not an
* operation count: a cascade delete bumps once per removed record.
*
* @param hook - Callback invoked synchronously after each qualifying write,
* or `undefined` to detach.
*/
public setGenerationBumpHook(hook: (() => void) | undefined): void {
this.generationBumpHook = hook
}
/**
* Read a raw object at a storage-root-relative path. Bypasses branch
* scoping and the write cache (record-layer files are written through
* {@link BaseStorage.writeRawObject} only).
*
* @param path - Storage-root-relative object path (e.g. `_system/manifest.json`).
* @returns The parsed object, or `null` if absent.
*/
public async readRawObject(path: string): Promise<any | null> {
await this.ensureInitialized()
return this.readObjectFromPath(path)
}
/**
* Write a raw object at a storage-root-relative path. On disk this is an
* atomic tmp+rename (the filesystem adapter's primitive), which is what
* makes the manifest rename a valid commit point.
*
* @param path - Storage-root-relative object path.
* @param data - JSON-serializable object to persist.
*/
public async writeRawObject(path: string, data: any): Promise<void> {
await this.ensureInitialized()
await this.writeObjectToPath(path, data)
}
/**
* Delete a raw object at a storage-root-relative path (no-op if absent).
*
* @param path - Storage-root-relative object path.
*/
public async deleteRawObject(path: string): Promise<void> {
await this.ensureInitialized()
await this.deleteObjectFromPath(path)
}
/**
* List raw object paths under a storage-root-relative prefix (normalized,
* `.gz`-stripped the adapter primitives already normalize).
*
* @param prefix - Storage-root-relative directory prefix.
* @returns Normalized object paths under the prefix (empty when none).
*/
public async listRawObjects(prefix: string): Promise<string[]> {
await this.ensureInitialized()
return this.listObjectsUnderPath(prefix)
}
/**
* Remove every object under a storage-root-relative prefix. The filesystem
* adapter overrides this with a recursive directory removal; this default
* lists and deletes individually (exactly what the in-memory adapter needs).
*
* @param prefix - Storage-root-relative directory prefix to remove.
*/
public async removeRawPrefix(prefix: string): Promise<void> {
await this.ensureInitialized()
const paths = await this.listObjectsUnderPath(prefix)
for (const p of paths) {
await this.deleteObjectFromPath(p)
}
}
/**
* Durability barrier for the commit protocol: ensure the listed raw-object
* paths are durable before the caller proceeds. The base implementation is
* a no-op (in-memory writes are durable-by-definition within the process);
* the filesystem adapter overrides it with real `fsync` of the files and
* their parent directories.
*
* @param paths - Storage-root-relative object paths previously written via
* {@link BaseStorage.writeRawObject}.
*/
public async syncRawObjects(paths: string[]): Promise<void> {
void paths
}
/**
* Read an entity's raw stored objects the exact bytes at its canonical
* metadata + vector paths (branch-scoped, write-cache coherent). Used by
* the generation store to capture before-images.
*
* @param id - The entity id.
* @returns The raw stored metadata and vector objects (`null` per part when
* the corresponding file is absent).
*/
public async readNounRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> {
await this.ensureInitialized()
const [metadata, vector] = await Promise.all([
this.readWithInheritance(getNounMetadataPath(id)),
this.readWithInheritance(getNounVectorPath(id))
])
return { metadata: metadata ?? null, vector: vector ?? null }
}
/**
* Restore an entity's raw stored objects byte-for-byte (a `null` part
* deletes that file). Used by crash recovery and transaction aborts to
* restore before-images.
*
* Bypasses the statistics/count bookkeeping of the normal save paths on
* purpose: restores must reproduce the exact prior bytes, and the count
* rollups are derived state with their own rebuild paths
* (`rebuildTypeCounts()` / `rebuildSubtypeCounts()`).
*
* @param id - The entity id.
* @param record - Raw stored objects as returned by {@link BaseStorage.readNounRaw}.
*/
public async writeNounRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise<void> {
await this.ensureInitialized()
if (record.metadata === null) {
await this.deleteObjectFromBranch(getNounMetadataPath(id))
} else {
await this.writeObjectToBranch(getNounMetadataPath(id), record.metadata)
}
if (record.vector === null) {
await this.deleteObjectFromBranch(getNounVectorPath(id))
} else {
await this.writeObjectToBranch(getNounVectorPath(id), record.vector)
}
}
/**
* Read a relationship's raw stored objects (verb-side mirror of
* {@link BaseStorage.readNounRaw}).
*
* @param id - The relationship id.
* @returns The raw stored metadata and vector objects (`null` per part when
* the corresponding file is absent).
*/
public async readVerbRaw(id: string): Promise<{ metadata: any | null; vector: any | null }> {
await this.ensureInitialized()
const [metadata, vector] = await Promise.all([
this.readWithInheritance(getVerbMetadataPath(id)),
this.readWithInheritance(getVerbVectorPath(id))
])
return { metadata: metadata ?? null, vector: vector ?? null }
}
/**
* Restore a relationship's raw stored objects byte-for-byte (verb-side
* mirror of {@link BaseStorage.writeNounRaw}; same bookkeeping caveats).
*
* @param id - The relationship id.
* @param record - Raw stored objects as returned by {@link BaseStorage.readVerbRaw}.
*/
public async writeVerbRaw(id: string, record: { metadata: any | null; vector: any | null }): Promise<void> {
await this.ensureInitialized()
if (record.metadata === null) {
await this.deleteObjectFromBranch(getVerbMetadataPath(id))
} else {
await this.writeObjectToBranch(getVerbMetadataPath(id), record.metadata)
}
if (record.vector === null) {
await this.deleteObjectFromBranch(getVerbVectorPath(id))
} else {
await this.writeObjectToBranch(getVerbVectorPath(id), record.vector)
}
}
/**
* Append one line to the transaction log (`_system/tx-log.jsonl`). The
* filesystem adapter appends to a real JSONL file; the in-memory adapter
* keeps a line array (serialized by `snapshotToDirectory`).
*
* @param line - One complete JSON document, without trailing newline.
*/
public abstract appendTxLogLine(line: string): Promise<void>
/**
* Read all transaction-log lines, oldest first (empty array when no log
* exists). A torn trailing line from a crashed append is returned as-is
* callers tolerate unparseable lines.
*/
public abstract readTxLogLines(): Promise<string[]>
/**
* Snapshot the entire store into `targetPath`. The filesystem adapter
* builds a hard-link farm (instant, space-shared safe because data files
* are immutable-by-rename); the in-memory adapter serializes its object
* store to a filesystem-storage-compatible directory. The result is a
* self-contained store openable via `Brainy.load(path)`.
*
* @param targetPath - Absolute directory path for the snapshot (created if
* missing; must be empty or absent).
*/
public abstract snapshotToDirectory(targetPath: string): Promise<void>
/**
* Replace the entire store's contents from a snapshot directory previously
* produced by {@link BaseStorage.snapshotToDirectory}. Implementations
* clear current contents (preserving live lock files), copy the snapshot
* in (byte copy never hard links, so the snapshot stays independent),
* and then call {@link BaseStorage.reloadDerivedState}.
*
* @param sourcePath - Absolute path of the snapshot directory.
*/
public abstract restoreFromDirectory(sourcePath: string): Promise<void>
/**
* Reset and reload every piece of adapter-internal derived state after the
* underlying objects changed wholesale (restore-from-snapshot): the
* write-through cache, the idtype/subtype caches, type/subtype statistics,
* total counts, and the graph-index singleton (invalidated so the next
* accessor rebuilds from the restored verbs).
*/
protected async reloadDerivedState(): Promise<void> {
this.clearWriteCache()
this.nounTypeByIdCache.clear()
this.nounSubtypeByIdCache.clear()
this.verbSubtypeByIdCache.clear()
this.nounCountsByType.fill(0)
this.verbCountsByType.fill(0)
this.subtypeCountsByType.clear()
this.verbSubtypeCountsByType.clear()
this.statisticsCache = null
this.statisticsModified = false
this.invalidateGraphIndex()
await this.loadTypeStatistics()
await this.loadSubtypeStatistics()
await this.loadVerbSubtypeStatistics()
await this.initializeCounts()
}
/** /**
* Save a noun to storage (vector only, metadata saved separately) * Save a noun to storage (vector only, metadata saved separately)
* @param noun Pure HNSW vector data (no metadata) * @param noun Pure HNSW vector data (no metadata)
@ -2315,6 +2580,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// Ignore persist errors - will retry on next operation // Ignore persist errors - will retry on next operation
}) })
} }
// 8.0 MVCC: entity-visible write — advance the generation watermark
// (suppressed inside transact batches by the generation store).
this.generationBumpHook?.()
} }
/** /**
@ -2717,6 +2986,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
this.decrementSubtypeCount(priorType, priorSubtype) this.decrementSubtypeCount(priorType, priorSubtype)
} }
} }
// 8.0 MVCC: entity-visible write — advance the generation watermark
// (suppressed inside transact batches by the generation store).
this.generationBumpHook?.()
} }
/** /**
@ -2750,6 +3023,8 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// Backward compatibility: fallback to old path if no verb type // Backward compatibility: fallback to old path if no verb type
const keyInfo = this.analyzeKey(id, 'verb-metadata') const keyInfo = this.analyzeKey(id, 'verb-metadata')
await this.writeObjectToBranch(keyInfo.fullPath, metadata) await this.writeObjectToBranch(keyInfo.fullPath, metadata)
// 8.0 MVCC: still an entity-visible write — advance the watermark.
this.generationBumpHook?.()
return return
} }
@ -2804,6 +3079,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// Ignore persist errors - will retry on next operation // Ignore persist errors - will retry on next operation
}) })
} }
// 8.0 MVCC: entity-visible write — advance the generation watermark
// (suppressed inside transact batches by the generation store).
this.generationBumpHook?.()
} }
/** /**
@ -2841,6 +3120,10 @@ export abstract class BaseStorage extends BaseStorageAdapter {
this.verbSubtypeByIdCache.delete(id) this.verbSubtypeByIdCache.delete(id)
this.decrementVerbSubtypeCount(priorEntry.verb, priorEntry.subtype) this.decrementVerbSubtypeCount(priorEntry.verb, priorEntry.subtype)
} }
// 8.0 MVCC: entity-visible write — advance the generation watermark
// (suppressed inside transact batches by the generation store).
this.generationBumpHook?.()
} }
// ============================================================================ // ============================================================================

View file

@ -0,0 +1,955 @@
/**
* @module tests/integration/db-mvcc
* @description Credibility-bar proof suite for Brainy 8.0's generational MVCC
* storage core + Datomic-style Db API. Each test PROVES a stated guarantee
* (not merely exercises the API):
*
* 1. Isolation a pinned Db reads its pinned state exactly, across 200
* subsequent transact mutations (updates + removes).
* 2. Atomicity a batch with a failing later operation applies NOTHING
* (generation unchanged, no partial records), for both plan-time and
* execution-phase failures (the latter via real fault injection into the
* storage adapter, exercising the production rollback path).
* 3. CAS `ifAtGeneration` commits when fresh, throws
* `GenerationConflictError` with correct expected/actual when stale.
* 4. Snapshot integrity persist mutate source heavily `Brainy.load()`
* is byte-identical to the pre-mutation state (hard-link safety), and an
* in-memory brain persists to a loadable directory.
* 5. Compaction safety pinned reads stay correct across `compactHistory()`;
* after release + compact the superseded record-sets are reclaimed.
* 6. with() speculative overlays are visible on the overlay Db only; the
* underlying brain and the original Db are untouched.
* 7. Generation monotonicity across close/reopen.
* 8. Crash consistency a simulated crash between record staging and the
* manifest rename (the commit point) recovers to the exact pre-transaction
* state on reopen, through the REAL production recovery path.
* 9. Versioned provider wiring a provider implementing the 4-method
* `VersionedIndexProvider` contract receives balanced pin/release (and a
* visibility consult per pin) as Db values are created and released.
*
* All entities carry explicit vectors so the suite never invokes the
* embedding engine (semantic search is exercised only for its documented
* historical-generation error).
*/
import { describe, it, expect, afterEach, vi } from 'vitest'
import * as fs from 'node:fs'
import * as os from 'node:os'
import * as path from 'node:path'
import { Brainy } from '../../src/brainy.js'
import { Db } from '../../src/db/db.js'
import {
GenerationCompactedError,
GenerationConflictError,
NotYetSupportedAtHistoricalGenerationError
} from '../../src/db/errors.js'
import type { GenerationStore } from '../../src/db/generationStore.js'
import { GraphAdjacencyIndex } from '../../src/graph/graphAdjacencyIndex.js'
import { RevisionConflictError } from '../../src/transaction/RevisionConflictError.js'
import type { StorageAdapter } from '../../src/coreTypes.js'
import { NounType, VerbType } from '../../src/types/graphTypes.js'
/** Deterministic 384-dim vector so no test ever invokes the embedder. */
function vec(seed: number): number[] {
return Array.from({ length: 384 }, (_, i) => ((seed * 31 + i * 7) % 100) / 100)
}
/**
* Map a readable label to a deterministic UUID-shaped id (entity ids must be
* UUIDs the sharded storage layout derives the shard from the UUID hex).
* Two independent FNV-style mixes keep 7+ distinct labels collision-free.
*/
function uid(label: string): string {
let h1 = 0x811c9dc5
for (let i = 0; i < label.length; i++) {
h1 = Math.imul(h1 ^ label.charCodeAt(i), 0x01000193) >>> 0
}
let h2 = 0xdeadbeef
for (let i = label.length - 1; i >= 0; i--) {
h2 = Math.imul(h2 ^ label.charCodeAt(i), 0x85ebca6b) >>> 0
}
const hex = h1.toString(16).padStart(8, '0') + h2.toString(16).padStart(8, '0')
return `00000000-0000-4000-8000-${hex.slice(0, 12)}`
}
/** Typed access to the brain's private generation store (test injection point). */
function generationStoreOf(brain: Brainy): GenerationStore {
return (brain as unknown as { generationStore: GenerationStore }).generationStore
}
describe('8.0 Db API — generational MVCC', () => {
const brains: Brainy[] = []
const tempDirs: string[] = []
/** Track a brain for afterEach teardown. */
async function openMemoryBrain(): Promise<Brainy> {
const brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' } })
await brain.init()
brains.push(brain)
return brain
}
/** Open (and track) a filesystem brain rooted at a fresh temp directory. */
async function openFsBrain(dir?: string): Promise<{ brain: Brainy; dir: string }> {
const rootDirectory = dir ?? makeTempDir()
const brain = new Brainy({
requireSubtype: false,
storage: { type: 'filesystem', rootDirectory }
})
await brain.init()
brains.push(brain)
return { brain, dir: rootDirectory }
}
function makeTempDir(): string {
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-db-mvcc-'))
tempDirs.push(dir)
return dir
}
afterEach(async () => {
for (const brain of brains.splice(0)) {
try {
await brain.close()
} catch {
// already closed by the test
}
}
for (const dir of tempDirs.splice(0)) {
await fs.promises.rm(dir, { recursive: true, force: true })
}
})
// ==========================================================================
// 1. Isolation
// ==========================================================================
it('proof 1 — a pinned Db reads its pinned state exactly across 200 transact mutations', async () => {
const brain = await openMemoryBrain()
const ids = Array.from({ length: 20 }, (_, i) => uid(`iso-e${i}`))
await brain.transact(
ids.map((id, i) => ({
op: 'add' as const,
id,
type: NounType.Document,
data: `doc-${i}`,
vector: vec(i),
metadata: { v: 0, slot: i }
}))
)
const db = brain.now()
const pinned = new Map<string, unknown>()
for (const id of ids) {
pinned.set(id, await db.get(id))
expect(pinned.get(id)).not.toBeNull()
}
const pinnedFind = await db.find({ type: NounType.Document, limit: 100 })
expect(pinnedFind.length).toBe(20)
// 200 mutations: 198 updates over the first 18 ids + 2 removes.
for (let i = 0; i < 200; i++) {
if (i === 100) {
await brain.transact([{ op: 'remove', id: uid('iso-e18') }])
} else if (i === 150) {
await brain.transact([{ op: 'remove', id: uid('iso-e19') }])
} else {
await brain.transact([
{ op: 'update', id: ids[i % 18], metadata: { v: i + 1, touchedAt: i } }
])
}
}
// Live state moved…
const liveUpdated = await brain.get(ids[0])
expect((liveUpdated?.metadata as { v: number }).v).toBeGreaterThan(0)
expect(await brain.get(uid('iso-e18'))).toBeNull()
expect(await brain.get(uid('iso-e19'))).toBeNull()
// …but EVERY read through the pinned Db is exactly the pinned state,
// including the two entities removed after the pin.
for (const id of ids) {
expect(await db.get(id)).toEqual(pinned.get(id))
}
// Set-level isolation: metadata find() at the pin still sees all 20.
const histFind = await db.find({ type: NounType.Document, limit: 100 })
expect(histFind.length).toBe(20)
for (const row of histFind) {
expect((row.metadata as { v: number }).v).toBe(0)
}
await db.release()
})
// ==========================================================================
// 2. Atomicity
// ==========================================================================
it('proof 2a — a plan-time failure in a later op applies nothing', async () => {
const brain = await openMemoryBrain()
const g0 = brain.generation()
await expect(
brain.transact([
{
op: 'add',
id: uid('atomic-a'),
type: NounType.Document,
data: 'a',
vector: vec(1),
metadata: { ok: true }
},
{ op: 'update', id: uid('never-created'), metadata: { boom: true } }
])
).rejects.toThrow(`Entity ${uid('never-created')} not found`)
expect(brain.generation()).toBe(g0)
expect(await brain.get(uid('atomic-a'))).toBeNull()
expect((await brain.find({ type: NounType.Document, limit: 10 })).length).toBe(0)
})
it('proof 2b — an execution-phase failure rolls back every applied op (generation unchanged, zero partial records)', async () => {
const brain = await openMemoryBrain()
await brain.transact([
{
op: 'add',
id: uid('atomic-base'),
type: NounType.Document,
data: 'base',
vector: vec(0),
metadata: { base: true }
}
])
const g0 = brain.generation()
const storage = brain.storageAdapter
// Real fault injection: the verb-metadata write of the LAST operation
// fails once, after the earlier add operations have already executed —
// exercising the production TransactionManager rollback path.
const spy = vi
.spyOn(storage, 'saveVerbMetadata')
.mockImplementationOnce(async () => {
throw new Error('disk full (injected)')
})
await expect(
brain.transact([
{
op: 'add',
id: uid('atomic-b'),
type: NounType.Person,
data: 'b',
vector: vec(2),
metadata: { name: 'b' }
},
{
op: 'relate',
from: uid('atomic-b'),
to: uid('atomic-base'),
type: VerbType.RelatedTo
}
])
).rejects.toThrow('disk full (injected)')
spy.mockRestore()
// ZERO ops applied: generation unchanged, the add rolled back, no edge.
expect(brain.generation()).toBe(g0)
expect(await brain.get(uid('atomic-b'))).toBeNull()
expect((await brain.getRelations({ from: uid('atomic-b') })).length).toBe(0)
expect((await brain.find({ type: NounType.Person, limit: 10 })).length).toBe(0)
// No partial generation records: the staging directory was removed.
const records = await storage.listRawObjects(`_generations/${g0 + 1}`)
expect(records).toEqual([])
// The store remains fully writable and consistent after the rollback.
const db = await brain.transact([
{
op: 'add',
id: uid('atomic-c'),
type: NounType.Person,
data: 'c',
vector: vec(3),
metadata: { name: 'c' }
}
])
expect(db.generation).toBe(g0 + 1)
await db.release()
})
it('proof 2c — an ifRev conflict on a later update rejects the whole batch', async () => {
const brain = await openMemoryBrain()
await brain.transact([
{
op: 'add',
id: uid('rev-e'),
type: NounType.Document,
data: 'rev',
vector: vec(4),
metadata: { v: 1 }
}
])
await brain.update({ id: uid('rev-e'), metadata: { v: 2 } }) // _rev is now 2
const g0 = brain.generation()
await expect(
brain.transact([
{
op: 'add',
id: uid('rev-new'),
type: NounType.Document,
data: 'new',
vector: vec(5),
metadata: {}
},
{ op: 'update', id: uid('rev-e'), metadata: { v: 3 }, ifRev: 1 } // stale
])
).rejects.toThrow(RevisionConflictError)
expect(brain.generation()).toBe(g0)
expect(await brain.get(uid('rev-new'))).toBeNull()
expect(((await brain.get(uid('rev-e')))?.metadata as { v: number }).v).toBe(2)
})
// ==========================================================================
// 3. CAS (ifAtGeneration)
// ==========================================================================
it('proof 3 — ifAtGeneration commits when fresh and conflicts when stale', async () => {
const brain = await openMemoryBrain()
const db = brain.now()
const committed = await brain.transact(
[
{
op: 'add',
id: uid('cas-e'),
type: NounType.Document,
data: 'cas',
vector: vec(6),
metadata: { n: 1 }
}
],
{ ifAtGeneration: db.generation }
)
expect(committed.generation).toBe(db.generation + 1)
try {
await brain.transact(
[{ op: 'update', id: uid('cas-e'), metadata: { n: 2 } }],
{ ifAtGeneration: db.generation } // stale — committed moved past it
)
expect.unreachable('should have thrown GenerationConflictError')
} catch (err) {
expect(err).toBeInstanceOf(GenerationConflictError)
expect((err as GenerationConflictError).expected).toBe(db.generation)
expect((err as GenerationConflictError).actual).toBe(committed.generation)
}
// Nothing committed by the conflicting attempt.
expect(((await brain.get(uid('cas-e')))?.metadata as { n: number }).n).toBe(1)
expect(brain.generation()).toBe(committed.generation)
await db.release()
await committed.release()
})
// ==========================================================================
// 4. Snapshot integrity (persist → mutate → load)
// ==========================================================================
it('proof 4a — a persisted snapshot is immune to heavy source mutation (hard-link safety)', async () => {
const { brain } = await openFsBrain()
const ids = Array.from({ length: 10 }, (_, i) => uid(`snap-e${i}`))
await brain.transact(
ids.map((id, i) => ({
op: 'add' as const,
id,
type: NounType.Document,
data: `snapshot-doc-${i}`,
vector: vec(i + 10),
metadata: { generation: 'pre', slot: i }
}))
)
const relDb = await brain.transact([
{ op: 'relate', from: ids[0], to: ids[1], type: VerbType.References },
{ op: 'relate', from: ids[1], to: ids[2], type: VerbType.References }
])
await relDb.release()
// Capture pre-mutation state, then snapshot.
const preState = new Map<string, unknown>()
for (const id of ids) {
preState.set(id, await brain.get(id))
}
const preRelations = await brain.getRelations({ from: ids[0] })
expect(preRelations.length).toBe(1)
const snapDir = path.join(makeTempDir(), 'snapshot')
const db = brain.now()
await db.persist(snapDir)
await db.release()
// Mutate the source heavily: rewrite every entity, delete some, add new.
for (const [i, id] of ids.entries()) {
await brain.transact([
{ op: 'update', id, data: `mutated-${i}`, metadata: { generation: 'post' } }
])
}
await brain.transact([{ op: 'remove', id: ids[0] }])
await brain.transact([
{
op: 'add',
id: uid('snap-after'),
type: NounType.Document,
data: 'added after snapshot',
vector: vec(99),
metadata: { generation: 'post' }
}
])
await brain.flush()
// The snapshot opens as a self-contained store with the PRE state.
const loaded = await Brainy.load(snapDir)
for (const id of ids) {
const entity = await loaded.get(id)
expect(entity).toEqual(preState.get(id))
}
expect(await loaded.get(uid('snap-after'))).toBeNull()
const loadedFind = await loaded.find({ type: NounType.Document, limit: 100 })
expect(loadedFind.length).toBe(10)
for (const row of loadedFind) {
expect((row.metadata as { generation: string }).generation).toBe('pre')
}
const loadedRelations = await loaded.related({ from: ids[0] })
expect(loadedRelations.length).toBe(1)
expect(loadedRelations[0].to).toBe(ids[1])
await loaded.release()
// And asOf(path) is the instance-method route to the same snapshot.
const viaAsOf = await brain.asOf(snapDir)
expect(((await viaAsOf.get(ids[3])) as { metadata: { generation: string } }).metadata.generation).toBe('pre')
await viaAsOf.release()
})
it('proof 4b — an in-memory brain persists to a directory loadable as a real store', async () => {
const brain = await openMemoryBrain()
await brain.transact([
{
op: 'add',
id: uid('mem-e1'),
type: NounType.Document,
data: 'memory-persisted',
vector: vec(21),
metadata: { from: 'memory' }
}
])
const snapDir = path.join(makeTempDir(), 'memory-snapshot')
const db = brain.now()
await db.persist(snapDir)
await db.release()
const loaded = await Brainy.load(snapDir)
const entity = await loaded.get(uid('mem-e1'))
expect(entity).not.toBeNull()
expect((entity?.metadata as { from: string }).from).toBe('memory')
await loaded.release()
})
it('proof 4c — persist() refuses a view that history has moved past (honest boundary)', async () => {
const brain = await openMemoryBrain()
const db = brain.now()
await brain.transact([
{
op: 'add',
id: uid('stale-persist'),
type: NounType.Document,
data: 'x',
vector: vec(30),
metadata: {}
}
])
const snapDir = path.join(makeTempDir(), 'stale-snapshot')
await expect(db.persist(snapDir)).rejects.toThrow(
NotYetSupportedAtHistoricalGenerationError
)
await db.release()
})
// ==========================================================================
// 5. Compaction safety
// ==========================================================================
it('proof 5 — compaction never breaks pinned reads; after release the records are reclaimed', async () => {
const { brain } = await openFsBrain()
const storage = brain.storageAdapter
// Every transact() returns a PINNED Db — release the ones this test
// does not keep, so compaction eligibility is determined solely by `db`.
await (
await brain.transact([
{
op: 'add',
id: uid('compact-e'),
type: NounType.Document,
data: 'v1',
vector: vec(40),
metadata: { v: 1 }
}
])
).release()
await (
await brain.transact([{ op: 'update', id: uid('compact-e'), metadata: { v: 2 } }])
).release()
const db = brain.now() // pinned at v2
const pinnedEntity = await db.get(uid('compact-e'))
await (
await brain.transact([{ op: 'update', id: uid('compact-e'), metadata: { v: 3 } }])
).release()
await (
await brain.transact([{ op: 'update', id: uid('compact-e'), metadata: { v: 4 } }])
).release()
const recordsBefore = (await storage.listRawObjects('_generations')).length
expect(recordsBefore).toBeGreaterThan(0)
// Compact while pinned: record-sets above the pin survive, pinned reads stay correct.
const first = await brain.compactHistory()
expect(await db.get(uid('compact-e'))).toEqual(pinnedEntity)
expect(((await db.get(uid('compact-e')))?.metadata as { v: number }).v).toBe(2)
// After release, everything is reclaimable.
await db.release()
const second = await brain.compactHistory()
expect(first.removedGenerations + second.removedGenerations).toBeGreaterThan(0)
const recordsAfter = (await storage.listRawObjects('_generations')).length
expect(recordsAfter).toBeLessThan(recordsBefore)
expect(recordsAfter).toBe(0)
// Compacted generations are honestly unreachable.
await expect(brain.asOf(1)).rejects.toThrow(GenerationCompactedError)
})
// ==========================================================================
// 6. with() — speculative overlays
// ==========================================================================
it('proof 6 — with() overlays are visible only on the overlay Db; brain and base Db untouched', async () => {
const brain = await openMemoryBrain()
await brain.transact([
{
op: 'add',
id: uid('with-a'),
type: NounType.Person,
data: 'Ada',
vector: vec(50),
metadata: { v: 1 }
},
{
op: 'add',
id: uid('with-b'),
type: NounType.Person,
data: 'Grace',
vector: vec(51),
metadata: { v: 1 }
}
])
const g0 = brain.generation()
const db = brain.now()
const spec = await db.with([
{ op: 'update', id: uid('with-a'), metadata: { v: 2 } },
{
op: 'add',
id: uid('with-new'),
type: NounType.Project,
data: 'Apollo',
metadata: { speculative: true }
},
{ op: 'remove', id: uid('with-b') },
{ op: 'relate', from: uid('with-a'), to: uid('with-new'), type: VerbType.WorksOn }
])
// The overlay view sees everything…
expect(spec.speculative).toBe(true)
expect(((await spec.get(uid('with-a')))?.metadata as { v: number }).v).toBe(2)
expect(await spec.get(uid('with-new'))).not.toBeNull()
expect(await spec.get(uid('with-b'))).toBeNull()
const specEdges = await spec.related({ from: uid('with-a') })
expect(specEdges.length).toBe(1)
expect(specEdges[0].to).toBe(uid('with-new'))
const specFind = await spec.find({ type: NounType.Project, limit: 10 })
expect(specFind.length).toBe(1)
// …the base Db sees none of it…
expect(db.speculative).toBe(false)
expect(((await db.get(uid('with-a')))?.metadata as { v: number }).v).toBe(1)
expect(await db.get(uid('with-new'))).toBeNull()
expect(await db.get(uid('with-b'))).not.toBeNull()
// …and the brain (disk, indexes, generation counter) is untouched.
expect(brain.generation()).toBe(g0)
expect(((await brain.get(uid('with-a')))?.metadata as { v: number }).v).toBe(1)
expect(await brain.get(uid('with-new'))).toBeNull()
expect((await brain.getRelations({ from: uid('with-a') })).length).toBe(0)
await spec.release()
await db.release()
})
// ==========================================================================
// 7. Generation monotonicity across close/reopen
// ==========================================================================
it('proof 7 — the generation counter never moves backwards across close/reopen', async () => {
const dir = makeTempDir()
const { brain: first } = await openFsBrain(dir)
await first.transact([
{
op: 'add',
id: uid('mono-e'),
type: NounType.Document,
data: 'v1',
vector: vec(60),
metadata: { v: 1 }
}
])
await first.transact([{ op: 'update', id: uid('mono-e'), metadata: { v: 2 } }])
// Single-operation write — bumps the counter outside transact.
await first.update({ id: uid('mono-e'), metadata: { v: 3 } })
const beforeClose = first.generation()
expect(beforeClose).toBeGreaterThanOrEqual(3)
await first.close()
const { brain: second } = await openFsBrain(dir)
expect(second.generation()).toBeGreaterThanOrEqual(beforeClose)
const db = await second.transact([{ op: 'update', id: uid('mono-e'), metadata: { v: 4 } }])
expect(db.generation).toBeGreaterThan(beforeClose)
await db.release()
})
// ==========================================================================
// 8. Crash consistency
// ==========================================================================
it('proof 8 — a crash before the manifest rename recovers to the exact pre-transaction state', async () => {
const dir = makeTempDir()
const { brain: first } = await openFsBrain(dir)
await first.transact([
{
op: 'add',
id: uid('crash-e'),
type: NounType.Document,
data: 'stable',
vector: vec(70),
metadata: { v: 1 }
}
])
const committedGen = first.generation()
// Simulated crash at the worst point of the REAL commit protocol: every
// record is staged and the batch has executed, but the atomic manifest
// rename — the commit point — never happens. The throwing injector skips
// the abort cleanup exactly as a dead process would.
generationStoreOf(first).setCommitFaultInjector((phase) => {
if (phase === 'before-manifest-rename') {
throw new Error('simulated process crash')
}
})
await expect(
first.transact([
{ op: 'update', id: uid('crash-e'), metadata: { v: 2 } },
{
op: 'add',
id: uid('crash-new'),
type: NounType.Document,
data: 'uncommitted',
vector: vec(71),
metadata: {}
}
])
).rejects.toThrow('simulated process crash')
generationStoreOf(first).setCommitFaultInjector(undefined)
// The in-process state is dirty (batch executed) — close flushes it all,
// the realistic worst case for the recovery path.
await first.close()
// Reopen: recovery rolls the uncommitted generation back and rebuilds
// the indexes from the repaired records.
const { brain: second } = await openFsBrain(dir)
const recovered = await second.get(uid('crash-e'))
expect((recovered?.metadata as { v: number }).v).toBe(1)
expect(await second.get(uid('crash-new'))).toBeNull()
const found = await second.find({ type: NounType.Document, limit: 10 })
expect(found.length).toBe(1)
expect((found[0].metadata as { v: number }).v).toBe(1)
// The crashed generation number is never reissued.
expect(second.generation()).toBeGreaterThanOrEqual(committedGen + 1)
const db = await second.transact([{ op: 'update', id: uid('crash-e'), metadata: { v: 5 } }])
expect(db.generation).toBeGreaterThan(committedGen + 1)
await db.release()
})
// ==========================================================================
// 9. Versioned provider wiring
// ==========================================================================
it('proof 9 — a VersionedIndexProvider receives balanced pin/release as Db values live and die', async () => {
const calls = {
pins: [] as bigint[],
releases: [] as bigint[],
visibility: [] as bigint[]
}
const brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' } })
brain.use({
name: 'test-versioned-graph-index',
activate: async (ctx) => {
ctx.registerProvider('graphIndex', (storage: StorageAdapter) =>
Object.assign(new GraphAdjacencyIndex(storage), {
generation: () => 0n,
isGenerationVisible: (g: bigint) => {
calls.visibility.push(g)
return true
},
pin: (g: bigint) => {
calls.pins.push(g)
},
release: (g: bigint) => {
calls.releases.push(g)
}
})
)
return true
}
})
await brain.init()
brains.push(brain)
const base = brain.generation()
const db1 = brain.now()
expect(calls.pins).toEqual([BigInt(base)])
expect(calls.visibility).toEqual([BigInt(base)]) // consulted at pin time
const db2 = await brain.transact([
{
op: 'add',
id: uid('vp-e'),
type: NounType.Document,
data: 'pinned',
vector: vec(80),
metadata: {}
}
])
expect(calls.pins).toEqual([BigInt(base), BigInt(base + 1)])
const db3 = await brain.asOf(base)
expect(calls.pins).toEqual([BigInt(base), BigInt(base + 1), BigInt(base)])
// A with() overlay takes its own pin on the same generation.
const spec = await db2.with([{ op: 'update', id: uid('vp-e'), metadata: { spec: true } }])
expect(calls.pins.length).toBe(4)
await db1.release()
await db2.release()
await db3.release()
await spec.release()
await spec.release() // idempotent — must NOT double-release the pin
expect(calls.releases.length).toBe(calls.pins.length)
expect([...calls.releases].sort()).toEqual([...calls.pins].sort())
})
// ==========================================================================
// Supporting guarantees: asOf, since, receipts, tx-log, honest errors
// ==========================================================================
it('asOf(generation) and asOf(Date) resolve historical state; future/negative are rejected', async () => {
const brain = await openMemoryBrain()
const tx1 = await brain.transact(
[
{
op: 'add',
id: uid('asof-e'),
type: NounType.Document,
data: 'v1',
vector: vec(90),
metadata: { v: 1 }
}
],
{ meta: { reason: 'first commit' } }
)
await new Promise((resolve) => setTimeout(resolve, 5))
const betweenCommits = new Date()
await new Promise((resolve) => setTimeout(resolve, 5))
await brain.transact([{ op: 'update', id: uid('asof-e'), metadata: { v: 2 } }])
// By generation.
const atFirst = await brain.asOf(tx1.generation)
expect(((await atFirst.get(uid('asof-e')))?.metadata as { v: number }).v).toBe(1)
expect(atFirst.timestamp).toBe(tx1.timestamp)
await atFirst.release()
// By timestamp: resolves to the newest commit at-or-before the instant.
const atInstant = await brain.asOf(betweenCommits)
expect(atInstant.generation).toBe(tx1.generation)
expect(((await atInstant.get(uid('asof-e')))?.metadata as { v: number }).v).toBe(1)
await atInstant.release()
// Honest range errors.
await expect(brain.asOf(brain.generation() + 100)).rejects.toThrow(RangeError)
await expect(brain.asOf(-1)).rejects.toThrow(RangeError)
await expect(brain.asOf('/no/such/snapshot/dir')).rejects.toThrow('not a snapshot directory')
// The transact meta was reified into the tx-log.
const lines = await brain.storageAdapter.readTxLogLines()
const entries = lines.map((line) => JSON.parse(line))
const first = entries.find((entry) => entry.generation === tx1.generation)
expect(first?.meta).toEqual({ reason: 'first commit' })
await tx1.release()
})
it('receipts resolve ids in input order, including intra-batch references and relate dedupe', async () => {
const brain = await openMemoryBrain()
const db = await brain.transact([
{ op: 'add', id: uid('rcpt-aa'), type: NounType.Person, data: 'a', vector: vec(91), metadata: {} },
{ op: 'add', id: uid('rcpt-bb'), type: NounType.Person, data: 'b', vector: vec(92), metadata: {} },
{ op: 'relate', from: uid('rcpt-aa'), to: uid('rcpt-bb'), type: VerbType.Knows },
{ op: 'update', id: uid('rcpt-bb'), metadata: { seen: true } }
])
expect(db.receipt).toBeDefined()
expect(db.receipt!.generation).toBe(db.generation)
expect(db.receipt!.ids.length).toBe(4)
expect(db.receipt!.ids[0]).toBe(uid('rcpt-aa'))
expect(db.receipt!.ids[1]).toBe(uid('rcpt-bb'))
expect(db.receipt!.ids[3]).toBe(uid('rcpt-bb'))
const relationId = db.receipt!.ids[2]
expect((await brain.getRelations({ from: uid('rcpt-aa') }))[0].id).toBe(relationId)
// Duplicate relate (same from/to/type) dedupes to the SAME id — both
// within a batch and against committed state, mirroring relate().
const dup = await brain.transact([
{ op: 'relate', from: uid('rcpt-aa'), to: uid('rcpt-bb'), type: VerbType.Knows }
])
expect(dup.receipt!.ids[0]).toBe(relationId)
expect((await brain.getRelations({ from: uid('rcpt-aa') })).length).toBe(1)
await db.release()
await dup.release()
})
it('since() reports the ids changed between two pinned views', async () => {
const brain = await openMemoryBrain()
await brain.transact([
{ op: 'add', id: uid('since-a'), type: NounType.Document, data: 'a', vector: vec(93), metadata: {} }
])
const before = brain.now()
await brain.transact([
{ op: 'add', id: uid('since-b'), type: NounType.Document, data: 'b', vector: vec(94), metadata: {} },
{ op: 'relate', from: uid('since-a'), to: uid('since-b'), type: VerbType.References }
])
await brain.transact([{ op: 'update', id: uid('since-a'), metadata: { touched: true } }])
const after = brain.now()
const changed = await after.since(before)
expect([...changed.nouns].sort()).toEqual([uid('since-a'), uid('since-b')].sort())
expect(changed.verbs.length).toBe(1)
expect(changed.fromGeneration).toBe(before.generation)
expect(changed.toGeneration).toBe(after.generation)
// Direction is enforced.
await expect(before.since(after)).rejects.toThrow(RangeError)
await before.release()
await after.release()
})
it('index-accelerated queries at a historical generation throw the documented error — never silently-wrong results', async () => {
const brain = await openMemoryBrain()
await brain.transact([
{ op: 'add', id: uid('hist-e'), type: NounType.Document, data: 'x', vector: vec(95), metadata: { v: 1 } }
])
const db = brain.now()
await brain.transact([{ op: 'update', id: uid('hist-e'), metadata: { v: 2 } }])
// Storage-record reads remain fully supported…
expect(((await db.get(uid('hist-e')))?.metadata as { v: number }).v).toBe(1)
expect((await db.find({ type: NounType.Document, where: { v: 1 } })).length).toBe(1)
// …index-accelerated dimensions throw the documented, named error.
await expect(db.search('anything')).rejects.toThrow(
NotYetSupportedAtHistoricalGenerationError
)
await expect(db.find({ vector: vec(95) })).rejects.toThrow(
NotYetSupportedAtHistoricalGenerationError
)
await expect(db.find({ connected: { from: uid('hist-e') } })).rejects.toThrow(
NotYetSupportedAtHistoricalGenerationError
)
// Released views refuse all reads.
await db.release()
await expect(db.get(uid('hist-e'))).rejects.toThrow('released Db')
})
it('restore() requires confirmation and replaces state from a snapshot (generation floor preserved)', async () => {
const { brain } = await openFsBrain()
await brain.transact([
{ op: 'add', id: uid('restore-e'), type: NounType.Document, data: 'keep', vector: vec(96), metadata: { v: 1 } }
])
const snapDir = path.join(makeTempDir(), 'restore-snapshot')
const db = brain.now()
await db.persist(snapDir)
await db.release()
await brain.transact([{ op: 'update', id: uid('restore-e'), metadata: { v: 2 } }])
await brain.transact([
{ op: 'add', id: uid('restore-extra'), type: NounType.Document, data: 'extra', vector: vec(97), metadata: {} }
])
const beforeRestore = brain.generation()
await expect(
brain.restore(snapDir, { confirm: false })
).rejects.toThrow('confirm')
await brain.restore(snapDir, { confirm: true })
expect(((await brain.get(uid('restore-e')))?.metadata as { v: number }).v).toBe(1)
expect(await brain.get(uid('restore-extra'))).toBeNull()
expect((await brain.find({ type: NounType.Document, limit: 10 })).length).toBe(1)
// The counter never moves backwards — pre-restore generations are not reissued.
expect(brain.generation()).toBeGreaterThanOrEqual(beforeRestore)
const after = await brain.transact([{ op: 'update', id: uid('restore-e'), metadata: { v: 9 } }])
expect(after.generation).toBeGreaterThan(beforeRestore)
await after.release()
})
it('Db values are first-class: Brainy.open() + chained with() overlays compose', async () => {
const brain = await Brainy.open({ requireSubtype: false, storage: { type: 'memory' } })
brains.push(brain)
expect(brain.isInitialized).toBe(true)
await brain.transact([
{ op: 'add', id: uid('chain-e'), type: NounType.Document, data: 'x', vector: vec(98), metadata: { v: 1 } }
])
const db = brain.now()
const spec1 = await db.with([{ op: 'update', id: uid('chain-e'), metadata: { v: 2 } }])
const spec2 = await spec1.with([{ op: 'update', id: uid('chain-e'), metadata: { v: 3 } }])
expect(((await db.get(uid('chain-e')))?.metadata as { v: number }).v).toBe(1)
expect(((await spec1.get(uid('chain-e')))?.metadata as { v: number }).v).toBe(2)
expect(((await spec2.get(uid('chain-e')))?.metadata as { v: number }).v).toBe(3)
expect(spec2).toBeInstanceOf(Db)
await spec2.release()
await spec1.release()
await db.release()
})
})

View file

@ -0,0 +1,333 @@
/**
* @module tests/unit/db/generationStore
* @description Unit tests for the generational MVCC record layer
* (`src/db/generationStore.ts`) against a real `MemoryStorage` adapter
* counter monotonicity, the commit protocol (staging execute manifest
* rename tx-log), point-in-time resolution from before-images, pin
* refcounting, compaction retention rules, CAS conflicts, timestamp
* resolution, and crash rollback of uncommitted generations.
*/
import { describe, it, expect, beforeEach } from 'vitest'
import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js'
import {
GenerationStore,
GENERATIONS_PREFIX,
MANIFEST_PATH
} from '../../../src/db/generationStore.js'
import {
GenerationCompactedError,
GenerationConflictError
} from '../../../src/db/errors.js'
import { NounType } from '../../../src/types/graphTypes.js'
/** Stored-metadata fixture in the canonical shape the live write paths use. */
function metadataFixture(version: number): Record<string, unknown> {
return {
noun: NounType.Document,
subtype: 'note',
data: `payload-v${version}`,
version,
createdAt: 1000,
updatedAt: 1000 + version,
_rev: version
}
}
// Entity ids must be UUID-shaped (the sharded storage layout derives the
// shard from the UUID hex). Fixed values keep assertions deterministic;
// suffix ordering (…aa < …bb < …cc) matches `changedBetween`'s sorted output.
const ID_A = '00000000-0000-4000-8000-0000000000aa'
const ID_B = '00000000-0000-4000-8000-0000000000bb'
const ID_C = '00000000-0000-4000-8000-0000000000cc'
const ID_SOLO = '00000000-0000-4000-8000-0000000000d0'
const ID_LATER = '00000000-0000-4000-8000-0000000000e0'
describe('db/GenerationStore', () => {
let storage: MemoryStorage
let store: GenerationStore
beforeEach(async () => {
storage = new MemoryStorage()
await storage.init()
store = new GenerationStore(storage)
await store.open()
})
/** Commit one transaction that writes `metadata` for `id`. */
async function commitWrite(
id: string,
version: number,
options?: { meta?: Record<string, unknown>; ifAtGeneration?: number }
): Promise<{ generation: number; timestamp: number }> {
return store.commitTransaction({
touched: { nouns: [id], verbs: [] },
meta: options?.meta,
ifAtGeneration: options?.ifAtGeneration,
execute: async () => {
await storage.saveNounMetadata(id, metadataFixture(version))
}
})
}
describe('generation counter', () => {
it('starts at 0 on a fresh store', () => {
expect(store.generation()).toBe(0)
expect(store.committedGeneration()).toBe(0)
})
it('advances by exactly one per committed transaction', async () => {
await commitWrite(ID_A, 1)
expect(store.generation()).toBe(1)
await commitWrite(ID_A, 2)
expect(store.generation()).toBe(2)
expect(store.committedGeneration()).toBe(2)
})
it('bumps on single-operation writes via the storage hook (suppressed inside transact)', async () => {
// Single-op write outside any transaction → hook bumps.
await storage.saveNounMetadata(ID_SOLO, metadataFixture(1))
expect(store.generation()).toBe(1)
expect(store.committedGeneration()).toBe(0) // no transact committed
// A transact batch writing twice bumps ONCE (the batch is one generation).
await store.commitTransaction({
touched: { nouns: [ID_A, ID_B], verbs: [] },
execute: async () => {
await storage.saveNounMetadata(ID_A, metadataFixture(1))
await storage.saveNounMetadata(ID_B, metadataFixture(1))
}
})
expect(store.generation()).toBe(2)
})
it('survives close + reopen monotonically', async () => {
await commitWrite(ID_A, 1)
await storage.saveNounMetadata(ID_SOLO, metadataFixture(1)) // single-op bump
const before = store.generation()
await store.close()
const reopened = new GenerationStore(storage)
await reopened.open()
expect(reopened.generation()).toBeGreaterThanOrEqual(before)
expect(reopened.committedGeneration()).toBe(1)
})
})
describe('CAS (ifAtGeneration)', () => {
it('commits when the expectation matches and rejects with expected/actual when stale', async () => {
await commitWrite(ID_A, 1)
const pinned = store.generation()
await commitWrite(ID_A, 2, { ifAtGeneration: pinned }) // fresh → commits
expect(store.generation()).toBe(pinned + 1)
try {
await commitWrite(ID_A, 3, { ifAtGeneration: pinned }) // stale → rejects
expect.unreachable('should have thrown GenerationConflictError')
} catch (err) {
expect(err).toBeInstanceOf(GenerationConflictError)
expect((err as GenerationConflictError).expected).toBe(pinned)
expect((err as GenerationConflictError).actual).toBe(pinned + 1)
}
expect(store.generation()).toBe(pinned + 1) // nothing moved
})
})
describe('failed transactions', () => {
it('removes the staging directory and returns the generation reservation', async () => {
await commitWrite(ID_A, 1)
const before = store.generation()
await expect(
store.commitTransaction({
touched: { nouns: [ID_A], verbs: [] },
execute: async () => {
throw new Error('boom mid-batch')
}
})
).rejects.toThrow('boom mid-batch')
expect(store.generation()).toBe(before)
const leftovers = await storage.listRawObjects(`${GENERATIONS_PREFIX}/${before + 1}`)
expect(leftovers).toEqual([])
})
})
describe('point-in-time resolution', () => {
it('resolves untouched ids to the live fast path', async () => {
await commitWrite(ID_A, 1)
const pinned = store.generation()
await commitWrite(ID_B, 1) // later commit touches a DIFFERENT id
expect(await store.resolveAt('noun', ID_A, pinned)).toEqual({ source: 'current' })
})
it('resolves changed ids from the first before-image after the pin', async () => {
await commitWrite(ID_A, 1)
const pinned = store.generation()
await commitWrite(ID_A, 2)
await commitWrite(ID_A, 3)
const resolved = await store.resolveAt('noun', ID_A, pinned)
expect(resolved.source).toBe('record')
if (resolved.source === 'record') {
expect(resolved.metadata.version).toBe(1) // state as of the pin
}
})
it('resolves ids created after the pin as absent', async () => {
await commitWrite(ID_A, 1)
const pinned = store.generation()
await commitWrite(ID_LATER, 1)
expect(await store.resolveAt('noun', ID_LATER, pinned)).toEqual({ source: 'absent' })
})
it('changedBetween reports the ids touched in the interval', async () => {
await commitWrite(ID_A, 1)
const from = store.generation()
await commitWrite(ID_B, 1)
await commitWrite(ID_C, 1)
const to = store.generation()
const changed = await store.changedBetween(from, to)
expect(changed.nouns).toEqual([ID_B, ID_C])
expect(changed.verbs).toEqual([])
expect(changed.fromGeneration).toBe(from)
expect(changed.toGeneration).toBe(to)
})
})
describe('timestamp resolution + tx-log', () => {
it('records meta in the tx-log and resolves timestamps to generations', async () => {
const t0 = Date.now() - 10
const first = await commitWrite(ID_A, 1, { meta: { author: 'unit-test' } })
// Distinct commit timestamps: resolution picks the NEWEST generation
// committed at-or-before the instant, so same-millisecond commits tie.
await new Promise((resolve) => setTimeout(resolve, 5))
const second = await commitWrite(ID_A, 2)
const lines = await storage.readTxLogLines()
expect(lines.length).toBe(2)
const firstEntry = JSON.parse(lines[0])
expect(firstEntry.generation).toBe(first.generation)
expect(firstEntry.meta).toEqual({ author: 'unit-test' })
expect((await store.resolveTimestamp(t0)).generation).toBe(0)
expect((await store.resolveTimestamp(first.timestamp)).generation).toBe(first.generation)
expect((await store.resolveTimestamp(Date.now() + 1000)).generation).toBe(
second.generation
)
})
})
describe('pins + compaction', () => {
it('refcounts pins and never reclaims records a live pin needs', async () => {
await commitWrite(ID_A, 1)
const pinned = store.generation()
store.pin(pinned)
store.pin(pinned) // refcount 2
await commitWrite(ID_A, 2)
await commitWrite(ID_A, 3)
// Pin at G protects every record-set ABOVE G (resolution reads
// before-images from generations strictly greater than the pin).
await store.compact()
const resolved = await store.resolveAt('noun', ID_A, pinned)
expect(resolved.source).toBe('record')
if (resolved.source === 'record') {
expect(resolved.metadata.version).toBe(1)
}
store.release(pinned)
await store.compact()
// Still one live pin — records above it still resolve correctly.
const stillResolved = await store.resolveAt('noun', ID_A, pinned)
expect(stillResolved.source).toBe('record')
store.release(pinned)
const result = await store.compact()
expect(result.removedGenerations).toBeGreaterThan(0)
const remaining = await storage.listRawObjects(GENERATIONS_PREFIX)
expect(remaining).toEqual([])
})
it('retainGenerations keeps the most recent record-sets', async () => {
await commitWrite(ID_A, 1)
await commitWrite(ID_A, 2)
await commitWrite(ID_A, 3)
const result = await store.compact({ retainGenerations: 2 })
expect(result.removedGenerations).toBe(1)
expect(result.horizon).toBe(1)
// Removing record-set N breaks pins BELOW N (their reads need N's
// before-images) — the horizon itself stays reachable: state at the
// horizon resolves from the retained record-sets above it.
expect(() => store.assertReachable(0)).toThrow(GenerationCompactedError)
expect(store.assertReachable(1)).toBe(1)
expect(store.assertReachable(2)).toBe(2)
})
it('rejects future and negative generations', async () => {
await commitWrite(ID_A, 1)
expect(() => store.assertReachable(99)).toThrow(RangeError)
expect(() => store.assertReachable(-1)).toThrow(RangeError)
expect(() => store.assertReachable(1.5)).toThrow(RangeError)
})
})
describe('crash recovery', () => {
it('rolls back an uncommitted generation on open (manifest rename is the commit point)', async () => {
await commitWrite(ID_A, 1)
// Simulate a crash between record staging and the manifest rename: a
// throwing fault injector skips the abort cleanup, exactly as a dead
// process would, leaving canonical files mutated and the staging
// directory present — but the manifest still at the prior generation.
store.setCommitFaultInjector((phase) => {
if (phase === 'before-manifest-rename') {
throw new Error('simulated crash')
}
})
await expect(commitWrite(ID_A, 2)).rejects.toThrow('simulated crash')
store.setCommitFaultInjector(undefined)
// The canonical file currently holds the EXECUTED (uncommitted) state.
const dirty = await storage.readNounRaw(ID_A)
expect((dirty.metadata as { version: number }).version).toBe(2)
// Recovery on the next open restores the before-image byte-identically.
const reopened = new GenerationStore(storage)
const openResult = await reopened.open()
expect(openResult.rolledBackGenerations).toBe(1)
const repaired = await storage.readNounRaw(ID_A)
expect((repaired.metadata as { version: number }).version).toBe(1)
expect(reopened.committedGeneration()).toBe(1)
// The crashed generation number is never reused.
expect(reopened.generation()).toBeGreaterThanOrEqual(2)
})
it('keeps a transaction whose manifest rename landed (tx-log append is advisory)', async () => {
await commitWrite(ID_A, 1)
store.setCommitFaultInjector((phase) => {
if (phase === 'after-manifest-rename') {
throw new Error('simulated crash after commit point')
}
})
await expect(commitWrite(ID_A, 2)).rejects.toThrow('simulated crash after commit point')
store.setCommitFaultInjector(undefined)
const reopened = new GenerationStore(storage)
const openResult = await reopened.open()
expect(openResult.rolledBackGenerations).toBe(0)
expect(reopened.committedGeneration()).toBe(2)
const kept = await storage.readNounRaw(ID_A)
expect((kept.metadata as { version: number }).version).toBe(2)
const manifest = await storage.readRawObject(MANIFEST_PATH)
expect(manifest.generation).toBe(2)
})
})
})

View file

@ -0,0 +1,223 @@
/**
* @module tests/unit/db/whereMatcher
* @description Unit tests for the in-memory `find()` filter evaluator behind
* historical and speculative `Db` reads (`src/db/whereMatcher.ts`). The
* evaluator must mirror the metadata index's operator semantics exactly an
* entity matches in-memory if and only if it would have matched through the
* index and must throw `UnsupportedWhereOperatorError` for anything it
* does not recognize (never guess on a historical read).
*/
import { describe, it, expect } from 'vitest'
import {
entityMatchesFind,
resolveEntityField,
whereMatches,
UnsupportedWhereOperatorError
} from '../../../src/db/whereMatcher.js'
import type { Entity } from '../../../src/types/brainy.types.js'
import { NounType } from '../../../src/types/graphTypes.js'
/** Build a minimal entity for matcher tests. */
function entity(overrides: Partial<Entity> = {}): Entity {
return {
id: 'e-1',
vector: [],
type: NounType.Document,
createdAt: 1000,
updatedAt: 2000,
metadata: {},
...overrides
}
}
describe('db/whereMatcher — resolveEntityField', () => {
it('resolves standard top-level fields', () => {
const e = entity({
subtype: 'invoice',
service: 'billing',
confidence: 0.9,
weight: 0.5,
_rev: 3,
data: 'payload'
})
expect(resolveEntityField(e, 'id')).toBe('e-1')
expect(resolveEntityField(e, 'type')).toBe(NounType.Document)
expect(resolveEntityField(e, 'noun')).toBe(NounType.Document) // alias
expect(resolveEntityField(e, 'subtype')).toBe('invoice')
expect(resolveEntityField(e, 'service')).toBe('billing')
expect(resolveEntityField(e, 'confidence')).toBe(0.9)
expect(resolveEntityField(e, 'weight')).toBe(0.5)
expect(resolveEntityField(e, '_rev')).toBe(3)
expect(resolveEntityField(e, 'createdAt')).toBe(1000)
expect(resolveEntityField(e, 'updatedAt')).toBe(2000)
expect(resolveEntityField(e, 'data')).toBe('payload')
})
it('resolves custom fields from the metadata bag', () => {
const e = entity({ metadata: { status: 'open', priority: 2 } })
expect(resolveEntityField(e, 'status')).toBe('open')
expect(resolveEntityField(e, 'priority')).toBe(2)
expect(resolveEntityField(e, 'absent')).toBeUndefined()
})
it('resolves dotted paths against the entity and the metadata bag', () => {
const e = entity({ metadata: { address: { city: 'Lyon' }, priority: 7 } })
expect(resolveEntityField(e, 'metadata.priority')).toBe(7)
expect(resolveEntityField(e, 'address.city')).toBe('Lyon')
expect(resolveEntityField(e, 'address.zip')).toBeUndefined()
})
})
describe('db/whereMatcher — operators', () => {
const e = entity({
subtype: 'invoice',
metadata: { amount: 250, status: 'open', tags: ['urgent', 'q3'], city: 'Lyon' }
})
it('shorthand equality', () => {
expect(whereMatches(e, { status: 'open' })).toBe(true)
expect(whereMatches(e, { status: 'closed' })).toBe(false)
})
it('eq / equals / is aliases', () => {
expect(whereMatches(e, { amount: { eq: 250 } })).toBe(true)
expect(whereMatches(e, { amount: { equals: 250 } })).toBe(true)
expect(whereMatches(e, { amount: { is: 250 } })).toBe(true)
expect(whereMatches(e, { amount: { eq: 99 } })).toBe(false)
})
it('array-membership equality (index posting semantics)', () => {
expect(whereMatches(e, { tags: 'urgent' })).toBe(true)
expect(whereMatches(e, { tags: { contains: 'q3' } })).toBe(true)
expect(whereMatches(e, { tags: 'missing-tag' })).toBe(false)
})
it('ne / notEquals / isNot aliases', () => {
expect(whereMatches(e, { status: { ne: 'closed' } })).toBe(true)
expect(whereMatches(e, { status: { notEquals: 'open' } })).toBe(false)
expect(whereMatches(e, { status: { isNot: 'open' } })).toBe(false)
})
it('in / oneOf set membership', () => {
expect(whereMatches(e, { status: { in: ['open', 'closed'] } })).toBe(true)
expect(whereMatches(e, { status: { oneOf: ['archived'] } })).toBe(false)
})
it('numeric range operators with aliases', () => {
expect(whereMatches(e, { amount: { gt: 200 } })).toBe(true)
expect(whereMatches(e, { amount: { greaterThan: 250 } })).toBe(false)
expect(whereMatches(e, { amount: { gte: 250 } })).toBe(true)
expect(whereMatches(e, { amount: { greaterThanOrEqual: 251 } })).toBe(false)
expect(whereMatches(e, { amount: { greaterEqual: 250 } })).toBe(true)
expect(whereMatches(e, { amount: { lt: 251 } })).toBe(true)
expect(whereMatches(e, { amount: { lessThan: 250 } })).toBe(false)
expect(whereMatches(e, { amount: { lte: 250 } })).toBe(true)
expect(whereMatches(e, { amount: { lessThanOrEqual: 249 } })).toBe(false)
expect(whereMatches(e, { amount: { lessEqual: 250 } })).toBe(true)
})
it('string range comparison is lexicographic', () => {
expect(whereMatches(e, { city: { gt: 'Aix' } })).toBe(true)
expect(whereMatches(e, { city: { lt: 'Aix' } })).toBe(false)
})
it('mixed-type range comparison never matches', () => {
expect(whereMatches(e, { city: { gt: 5 } })).toBe(false)
expect(whereMatches(e, { amount: { lt: 'zzz' } })).toBe(false)
})
it('between', () => {
expect(whereMatches(e, { amount: { between: [200, 300] } })).toBe(true)
expect(whereMatches(e, { amount: { between: [251, 300] } })).toBe(false)
expect(whereMatches(e, { amount: { between: [200] } })).toBe(false) // malformed operand
})
it('exists / missing', () => {
expect(whereMatches(e, { status: { exists: true } })).toBe(true)
expect(whereMatches(e, { nope: { exists: true } })).toBe(false)
expect(whereMatches(e, { nope: { exists: false } })).toBe(true)
expect(whereMatches(e, { nope: { missing: true } })).toBe(true)
expect(whereMatches(e, { status: { missing: true } })).toBe(false)
expect(whereMatches(e, { status: { missing: false } })).toBe(true)
})
it('multiple operators on one field AND together', () => {
expect(whereMatches(e, { amount: { gte: 200, lte: 300 } })).toBe(true)
expect(whereMatches(e, { amount: { gte: 200, lte: 249 } })).toBe(false)
})
it('allOf / anyOf / not logical composition', () => {
expect(whereMatches(e, { allOf: [{ status: 'open' }, { amount: { gt: 100 } }] })).toBe(true)
expect(whereMatches(e, { allOf: [{ status: 'open' }, { amount: { gt: 999 } }] })).toBe(false)
expect(whereMatches(e, { anyOf: [{ status: 'closed' }, { amount: 250 }] })).toBe(true)
expect(whereMatches(e, { anyOf: [{ status: 'closed' }, { amount: 9 }] })).toBe(false)
expect(whereMatches(e, { not: { status: 'closed' } })).toBe(true)
expect(whereMatches(e, { not: { status: 'open' } })).toBe(false)
})
it('throws UnsupportedWhereOperatorError for unknown operators — never guesses', () => {
expect(() => whereMatches(e, { amount: { approximately: 250 } })).toThrow(
UnsupportedWhereOperatorError
)
try {
whereMatches(e, { amount: { approximately: 250 } })
expect.unreachable('should have thrown')
} catch (err) {
expect(err).toBeInstanceOf(UnsupportedWhereOperatorError)
expect((err as UnsupportedWhereOperatorError).operator).toBe('approximately')
}
})
})
describe('db/whereMatcher — entityMatchesFind', () => {
const e = entity({
subtype: 'invoice',
service: 'billing',
metadata: { amount: 250 }
})
it('filters by type (single + array)', () => {
expect(entityMatchesFind(e, { type: NounType.Document })).toBe(true)
expect(entityMatchesFind(e, { type: [NounType.Person, NounType.Document] })).toBe(true)
expect(entityMatchesFind(e, { type: NounType.Person })).toBe(false)
})
it('filters by subtype (single + array; entities without subtype excluded)', () => {
expect(entityMatchesFind(e, { subtype: 'invoice' })).toBe(true)
expect(entityMatchesFind(e, { subtype: ['invoice', 'receipt'] })).toBe(true)
expect(entityMatchesFind(e, { subtype: 'receipt' })).toBe(false)
expect(entityMatchesFind(entity(), { subtype: 'invoice' })).toBe(false)
})
it('filters by service', () => {
expect(entityMatchesFind(e, { service: 'billing' })).toBe(true)
expect(entityMatchesFind(e, { service: 'crm' })).toBe(false)
})
it('excludeVFS drops VFS-marked entities', () => {
const vfsEntity = entity({ metadata: { vfsType: 'file' } })
const markedEntity = entity({ metadata: { isVFSEntity: true } })
expect(entityMatchesFind(vfsEntity, { excludeVFS: true })).toBe(false)
expect(entityMatchesFind(markedEntity, { excludeVFS: true })).toBe(false)
expect(entityMatchesFind(e, { excludeVFS: true })).toBe(true)
expect(entityMatchesFind(vfsEntity, {})).toBe(true)
})
it('composes type + subtype + where', () => {
expect(
entityMatchesFind(e, {
type: NounType.Document,
subtype: 'invoice',
where: { amount: { gte: 200 } }
})
).toBe(true)
expect(
entityMatchesFind(e, {
type: NounType.Document,
subtype: 'invoice',
where: { amount: { gte: 999 } }
})
).toBe(false)
})
})