feat(8.0): generational MVCC storage + Datomic-style Db API (now/transact/asOf/with/persist)
One mechanism replaces the COW and versioning subsystems: immutable
generation-stamped records behind a Datomic-style database value.
Record layer (src/db/generationStore.ts):
- Monotonic generation counter in _system/generation.json; bumped once per
transact() commit and once per single-operation write (storage hook), so
brain.generation() is always a meaningful watermark.
- Commit protocol: stage before-images + tx.json delta -> fsync -> execute
batch via TransactionManager -> atomic tmp+rename of _system/manifest.json
(the rename IS the commit point) -> append _system/tx-log.jsonl.
- Crash recovery on open rolls uncommitted generations back byte-identically
and forces an index rebuild; refcounted pins gate compactHistory(), which
records a horizon (asOf below it throws GenerationCompactedError).
Db API:
- Db: get/find/search/related pinned at a generation, with() speculative
overlays, since() diffs, persist() hard-link-farm snapshots, timestamp,
generation, release() + FinalizationRegistry backstop.
- Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch
(GenerationConflictError CAS), asOf(generation|Date|path), restore(path,
{confirm}), compactHistory(), generation(), static open() + load().
- get()/metadata find()/related() are fully correct at any reachable pinned
generation; index-accelerated queries at historical generations throw
NotYetSupportedAtHistoricalGenerationError - never silently-wrong results.
- VersionedIndexProvider (generation/isGenerationVisible/pin/release) in
plugin.ts: feature-detected, balanced pin/release in lockstep with Db
lifecycle, post-commit applier + replay-gap model documented.
Storage primitives (BaseStorage + filesystem/memory adapters): raw-object
read/write/list/remove, fsync barrier, noun/verb raw before-image capture,
tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and
append-in-place exceptions), restoreFromDirectory + derived-state reload.
Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation
across 200 mutations, batch atomicity under injected execution failure,
ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety
under pins, with() overlay isolation, generation monotonicity across
reopen, crash consistency through the real recovery path, and versioned
provider pin/release balance. Design record in
docs/ADR-001-generational-mvcc.md.
This commit is contained in:
parent
49e49483d1
commit
431cd64406
16 changed files with 6244 additions and 5 deletions
102
src/plugin.ts
102
src/plugin.ts
|
|
@ -259,6 +259,108 @@ export interface GraphIndexProvider {
|
|||
getAllRelationshipCounts(): Map<string, number>
|
||||
}
|
||||
|
||||
/**
|
||||
* Optional capability interface for index providers that maintain versioned
|
||||
* (generation-aware) internal state — the provider-side half of Brainy 8.0's
|
||||
* generational MVCC contract. Implemented by native providers whose storage
|
||||
* engines keep immutable versions (e.g. LSM snapshots); Brainy's own JS
|
||||
* indexes do not implement it (they are rebuilt from the storage records,
|
||||
* which carry the versioning).
|
||||
*
|
||||
* **Detection.** Brainy feature-detects this interface on every registered
|
||||
* index provider (`'vector'`, `'metadataIndex'`, `'graphIndex'`): when a
|
||||
* provider exposes `pin`/`release` functions, Brainy calls them in lockstep
|
||||
* with `Db` lifecycle — `pin(g)` when a `Db` value pins generation `g`
|
||||
* (`brain.now()`, `brain.transact()`, `brain.asOf()`), `release(g)` when that
|
||||
* `Db` is released (explicitly or via the GC backstop). Pins are refcounted
|
||||
* on the Brainy side; a provider may receive multiple `pin(g)` calls for the
|
||||
* same generation and will receive exactly one matching `release(g)` per pin.
|
||||
*
|
||||
* **Consistency model (locked cross-team design).** Index providers are
|
||||
* *post-commit appliers*: the storage-record commit (atomic manifest rename)
|
||||
* is the source of truth, and provider index state is derived, applied after
|
||||
* the commit point. On open, a provider compares its own persisted
|
||||
* `generation()` against the store's committed generation and replays the
|
||||
* gap from the storage records (or requests a rebuild) — there are no
|
||||
* provider rollback hooks, because an uncommitted transaction is repaired at
|
||||
* the storage layer before any index is opened. The explicit pin/release
|
||||
* lifetime OVERRIDES any time-based snapshot retention the provider has
|
||||
* (e.g. an LSM snapshot TTL): a pinned generation must stay readable until
|
||||
* released, regardless of age.
|
||||
*
|
||||
* **Speculative reads.** `db.with(txData)` overlays are Brainy-side only —
|
||||
* providers are never asked to read uncommitted/speculative state; they
|
||||
* always serve committed generations.
|
||||
*/
|
||||
export interface VersionedIndexProvider {
|
||||
/**
|
||||
* @description The newest generation this provider's persisted index state
|
||||
* reflects. Brainy compares it against the storage layer's committed
|
||||
* generation on open to detect a replay gap (index behind storage after a
|
||||
* crash between commit and index apply).
|
||||
* @returns The provider's current generation as a `bigint` (u64 at the
|
||||
* native boundary; Brainy's generation counter is a safe integer today).
|
||||
*/
|
||||
generation(): bigint
|
||||
|
||||
/**
|
||||
* @description True ⇒ the provider can serve consistent reads at
|
||||
* `generation` (segments retained). Combined with pin: pinning a VISIBLE
|
||||
* generation guarantees it stays servable until release. Pinning an
|
||||
* invisible generation is permitted (refcount-only) — Brainy serves that
|
||||
* generation from canonical storage instead.
|
||||
*
|
||||
* Brainy consults this at pin time (the read-routing rule: a `Db` at
|
||||
* generation `g` uses provider-accelerated reads when
|
||||
* `isGenerationVisible(g)` was true at pin time; otherwise canonical
|
||||
* generation records).
|
||||
* @param generation - The generation a `Db` value is about to pin.
|
||||
*/
|
||||
isGenerationVisible(generation: bigint): boolean
|
||||
|
||||
/**
|
||||
* @description Pin a generation: the provider must keep index state for
|
||||
* `generation` readable until the matching {@link VersionedIndexProvider.release}
|
||||
* call, overriding any time-based snapshot retention. Called once per
|
||||
* Brainy-side pin (refcounted upstream — expect balanced pin/release pairs).
|
||||
* @param generation - The generation a live `Db` value just pinned.
|
||||
*/
|
||||
pin(generation: bigint): void
|
||||
|
||||
/**
|
||||
* @description Release one pin on `generation`. After the last release the
|
||||
* provider may reclaim resources for that generation at its discretion.
|
||||
* @param generation - The generation being released (matches a prior
|
||||
* {@link VersionedIndexProvider.pin} call).
|
||||
*/
|
||||
release(generation: bigint): void
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Feature-detection guard for {@link VersionedIndexProvider}.
|
||||
* Brainy applies it to every registered index provider (vector, metadata,
|
||||
* graph) when a `Db` value pins or releases a generation: providers exposing
|
||||
* all four capability methods receive lockstep `pin`/`release` calls;
|
||||
* everything else (including Brainy's own JS indexes) is skipped.
|
||||
*
|
||||
* @param candidate - A registered index provider instance.
|
||||
* @returns Whether the candidate implements the versioned capability.
|
||||
*/
|
||||
export function isVersionedIndexProvider(
|
||||
candidate: unknown
|
||||
): candidate is VersionedIndexProvider {
|
||||
if (candidate === null || typeof candidate !== 'object') {
|
||||
return false
|
||||
}
|
||||
const c = candidate as Record<string, unknown>
|
||||
return (
|
||||
typeof c.generation === 'function' &&
|
||||
typeof c.isGenerationVisible === 'function' &&
|
||||
typeof c.pin === 'function' &&
|
||||
typeof c.release === 'function'
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* The object returned by the `'vector'` provider factory — Brainy's vector
|
||||
* index contract. Implementations include Brainy's own JS HNSW index and any
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue