committedGens was a number[] with one element per committed generation. Every
single-op write reserves a distinct generation, so an insert-built 1B corpus
held a ~1B-element array resident regardless of provider — the MVCC sibling of
the storage-cache gate.
Replace it with a sorted disjoint interval set (committedRanges). With no
compaction gaps the whole ledger collapses to a single [start,end] pair, so
resident size is O(number-of-gaps) not O(writes). reservedGens becomes a
generator; point-resolution binary-searches the ranges; compaction trims the
oldest prefix with a defensive prefix-invariant guard. Behaviour is identical
across asOf/diff/since/history/transactionLog — unit 1718 + integration 607 +
the focused MVCC suite green.
8.0 keeps the canonical operators (eq/ne/gt/gte/lt/lte) and their clean
long-form aliases (equals/notEquals/greaterThan/greaterThanOrEqual/lessThan/
lessThanOrEqual), and drops the four redundant deprecated spellings:
is → eq, isNot → ne, greaterEqual → gte, lessEqual → lte
Removed from every evaluator (metadataIndex criteria + range switches,
metadataFilter, the db whereMatcher egress path) and from the
BrainyFieldOperators type, the unsupported-operator error message, the docs
(QUERY_OPERATORS / api README / VFS projection + semantic guides), and the
whereMatcher alias tests. Also migrated Brainy's own internal use — the VFS
TemporalProjection queried with greaterEqual/lessEqual, which would have
silently broken — to gte/lte.
Full gate green: build, unit 1512, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- find() search-mode: collapsed the two overlapping options to one canonical
`searchMode: SearchMode` (the redundant `mode` alias was silently ignored on
the primary find() path while honored on the historical path — a footgun) and
removed the unwired `explain?` FindParams field (never read by find()).
- Documentation accuracy on the public type surface: entity ids documented as
UUID v7 (auto) / v5 (natural key), not v4; dropped the stale "Brainy 3.0"
banners and "backward compatibility" hedging on the fresh-8.0 Result type;
documented the via/type alias; removed the dead GraphConstraints.bidirectional;
fixed the no-op `EmbeddedGraphVerb = Omit<GraphVerb,'source'>` to omit the real
sourceId; corrected the GraphIndexProvider.isInitialized JSDoc; documented
removeMany's per-chunk generation granularity; JSDoc'd the public VFS surface.
- Honest perf comments in source: removed unmeasured billion-scale figures from
the LSMTree / graphAdjacencyIndex headers (the JS fallback is in-memory after
open; native is the scale path).
- Robustness: getVerbMetadata propagates read errors symmetrically with
getNounMetadata; the find() egress integrity-guard keeps a row when the JS
matcher doesn't implement an operator the provider already matched on; hardened
the boundary-no-native CI guard to catch side-effect imports + re-exports.
- Tests: de-theatricalized a relateMany test to assert the real contract; fixed
a stale Model-B header.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A filtered semantic read at a historical generation (db.asOf(g).find({ query, where }))
no longer unconditionally rebuilds an ephemeral JS-HNSW over every at-g vector
(O(n@G), the OOM ceiling the 2026-06-24 scaling audit flagged). When the vector
index is a VersionedIndexProvider that advertises isGenerationVisible(g), Brainy:
1. resolves the at-g metadata∩graph universe from the record-overlay path (no
materialization — it's the metadata-only historical find),
2. routes the vector leg to the provider with { allowedIds, generation }, and
3. composes the at-g entities ranked by the provider's at-g vector distance.
Without a versioned provider (the JS index, or a native one that refuses the
generation) it falls through to the existing materialization — unchanged.
- Seam (A): VectorIndexProvider.search gains an optional `generation?: bigint`
("omitted = now"), the vector mirror of the graph provider's trailing-gen + the
#46 allowedIds field. JsHnswVectorIndex accepts and ignores it (no per-gen
segments → refuse/fall-back posture; Brainy never routes a historical read there).
- Gate: Db.find tries the native at-gen vector path before materialize(); host
exposes canServeVectorAtGeneration + vectorSearchAtGeneration.
- generation is Brainy's u64 commit counter — the same value handed to the graph
index on writes (graphWriteGeneration), so it maps 1:1 to the native side.
Decided lockstep with the cor team (handoff #35 thread: (A) + vector-only). The
gen-g candidate-vector supply for cor's exact-rerank (part-3) is a separate,
non-blocking seam still being confirmed; the honesty guard keeps this path inactive
(falls through to materialize) until cor's native at-gen rerank is live.
Tested: provider routing + at-gen universe correctness (excludes future-born,
applies the filter) + page window via a mock versioned provider; seam-ignore on the
JS index; 38 db-mvcc/db-temporal historical-read tests green (materialize fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every write — transact() AND single-op add/update/remove/relate — is now its
own immutable generation (Model-B), so a now() pin always freezes and
asOf/since/diff/history/transactionLog reflect single-ops exactly like
transacts. Closes the Model-A hole where pins did not freeze against single-op
writes.
Generation-stamping:
- GenerationStore.commitSingleOp: a one-operation commitTransaction with
deferred durability. Wired into add/update/remove/relate/updateRelation/
unrelate + removeMany (the *Many and VFS paths delegate to these).
- Async group-commit (flushPendingSingleOps): the live write is acknowledged
immediately; its before-image is buffered in an in-memory pending tier that
resolveAt/chains/changedBetween/tx-log read like on-disk generations, so the
synchronous now() freezes with no forced flush. One fsync per window
(triggers: size / 50ms timer / flush / close / transact / compactHistory).
- Crash recovery is drop-without-restore for group-commit generations (marked
groupCommit:true): a crash mid-flush discards the partial generation and
never restores its before-images, which would otherwise revert the
already-acknowledged live write.
- Init-time infrastructure (the VFS root) is the un-versioned generation-0
baseline: a fresh brain reports generation()===0 and an empty
transactionLog(); the first user write is generation 1.
- Historical find()/related() overlay bound is the full reserved watermark
(generation()), so un-flushed single-op writes are overlaid too.
Retention knob:
- config `history` -> `retention`: 'all' | 'adaptive' |
{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset ->
adaptive (disk/RAM pressure, zero-config). CompactHistoryOptions floors ->
caps (retainGenerations->maxGenerations, retainMs->maxAge, +maxBytes):
reclaim oldest-unpinned while ANY cap is exceeded; pins always exempt.
- brain.setRetentionBudget(bytes) drives the adaptive byte budget at runtime
(a coordinator's fair-share input). Per-generation bytes recorded in each
delta enable historyBytes() introspection without a storage size API.
Tests: per-write generation resolution, pin freeze vs add/update/remove,
drop-without-restore corruption-trap (fault injector), clean-reopen replay,
maxBytes/maxAge/no-cap reclamation, retention-then-reopen. 107 db/generation/
temporal tests green, tsc clean. Docs (ADR-001, consistency-model, snapshots
guide, api reference, RELEASES) updated to per-write granularity + retention.
Historical reads (asOf/get) scanned the global committedGens list linearly,
making them O(database-age): a read of an unchanged entity at an old pin scaled
~12x for 10x history depth. Add per-id inverted history chains (nounChains/
verbChains) so resolveAt binary-searches the id's own generation chain instead —
O(log) and flat with depth. Chains build lazily under the commit mutex,
maintain incrementally on commit, and invalidate on compaction.
Also bound deltaCache (LRU cap 4096; getDelta re-reads evicted deltas) so a
long-lived high-write process's heap is O(cap) not O(generations) on the
disk-backed path, and binary-search commitTimestampAtOrBefore (O(log)).
Verified: 373 unit tests green; new tests/unit/db/generation-chain.test.ts
covers chain resolution, eviction re-read, and chain rebuild after compaction.
related({ from }) / related({ to }) each return one direction; related({ node })
unions both (every edge where the node is source OR target), deduped, in a single
O(degree) call — the indexed equivalent of merging two related() calls by hand.
The 'all edges on a noun' query the graph-viz path needs.
- RelatedParams.node: mutually exclusive with from/to (throws if combined);
composes with type/subtype/visibility filters; natural-key ids resolved.
- Implemented as a parallel out-edge (sourceId) + in-edge (targetId) fetch through
the O(degree) graph-index fast paths, merged + deduped by id (a self-loop appears
once), sorted for a stable page. Full incidence is the intended O(degree) cost;
deep offset paging of a very-high-degree node is best-effort (use the native
edgesForNode / a graph cursor for exhaustive paging).
- db.related() parity: node resolved + honored in relationMatchesParams (matches an
edge incident in either direction) for the historical/overlay paths.
Test: related-node-bidirectional.test.ts — both directions, equals from∪to, self-loop
dedupe, type-filter composition, default-hides-internal, node+from/to throws.
Three additive ergonomics from the API-simplification audit (no behavior change
to existing call sites):
- AddParams.upsert: create-or-update in one call. With a custom id, an existing
entity is MERGED via the update path (merges metadata, re-embeds changed data,
bumps _rev, PRESERVES createdAt) instead of the destructive full overwrite a
plain add() does. Mutually exclusive with ifAbsent (throws if both set);
ignored when no id is supplied. Wired into add(), addMany (per-item flag
propagation), and the transact add op (routes to planTxUpdate). Kills the
get()-then-add() round-trip for idempotent writes.
- FindParams.includeVectors: mirror of GetOptions.includeVectors — find() returns
stored vectors when set; default stays empty (the perf contract is preserved).
Honored on both the query and metadata-only where paths, and in db.find().
- removeMany adaptive chunking: replaced the hardcoded chunkSize=10 with
params.chunkSize ?? storageConfig.maxBatchSize, matching addMany/relateMany —
one storage-adaptive batch policy across all *Many methods (no thrash on
high-latency backends).
Tests: tests/unit/brainy/upsert.test.ts (insert/merge/createdAt-preserved/
re-embed/ifAbsent-conflict/no-id/addMany/transact), find-include-vectors.test.ts
(true/default/where-path), batch-operations.test.ts (removeMany >10 items).
- id-normalization (#18): `brain.newId()` (UUID v7) + v7 default ids; non-UUID string ids are
transparently normalized to a stable UUID v5 on creation AND every lookup — get/update/remove,
relate(from,to), related, find({connected}), getMany, removeMany, the transact op-handler, and
db.with() overlays — with the caller's original key preserved under `_originalId`. The engine only
ever sees a UUID; real UUIDs pass through untouched. universal/uuid.ts gains v5/v7/isUUID +
BRAINY_ID_NAMESPACE; new utils/idNormalization.ts (coerceNewEntityId / resolveEntityId).
- aggregation min/max delete-safety: `queryAggregate` no longer leaves a stale min/max after
deleting the current extreme — min/max now track a value multiset and recompute in-memory (no
entity scan; that scan was the prior delete-then-hang a consumer reported). Other metrics were
already exact across deletes. Regression test proves resolve-after-delete + correct new min/max.
- release.sh RC-safe: explicit version arg, prerelease detection (npm `--tag rc` + GitHub
`--prerelease`), full `test:ci` gate (unit + integration), current-branch push, npm-access
verification after publish.
- native-engine references point at `@soulcraft/cor` (8.0's partner) in user-facing strings/docs.
Unit 1461/0 · integration 599/0 · tsc clean.
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
asOf() answers "state AT a point"; these answer "what happened BETWEEN two
points" and "one entity's whole history", all on the existing generation records.
- Extract resolveAsOfGeneration() as the shared gen|Date→{generation,timestamp}
chokepoint (reachability + Date semantics identical everywhere). asOf()'s
inclusive path is byte-identical — db-mvcc still 25/25.
- asOf(target, { exclusive }) — strict-before (resolved−1, clamped at gen 0,
never a RangeError).
- db.since(Db | generation | Date) — overload; number/Date resolve via the shared
resolver (new DbHost.resolveGeneration); EXCLUSIVE lower bound;
since(db) === since(db.generation). Same-store guard via Db.belongsToStore.
- brain.diff(a, b) → { added, removed, modified } split by nouns/verbs. EARNS its
name: candidate set is ONLY changedBetween(gLow,gHigh), each classified by
existence at both endpoints + a key-order-insensitive value compare
(new src/db/stableEqual.ts) — a touched-but-reverted / born-and-died id is in
no bucket.
- brain.history(id, { from, to }) → every distinct version oldest→newest, each
value === asOf(version.generation).get(id); null = removal; kind auto-detected
(throws on UUID-space collision). New store helper generationsTouching().
- brain.transactionLog({ from, to, limit }) — INCLUSIVE generation/Date window
(contrast since's exclusive lower bound); limit applied last.
- Compaction policy locked: diff/since THROW GenerationCompactedError below the
horizon; history TRUNCATES to it.
Types DiffResult/HistoryVersion/EntityHistory exported from the package root.
Tests: tests/integration/db-temporal.test.ts (8 proofs incl. diff-earns-its-name,
history↔asOf cross-check, the composition proof, granularity, compaction contrast)
+ tests/unit/db/stableEqual.test.ts (6). docs/guides/snapshots-and-time-travel.md
+ RELEASES updated. 1477 unit + db-temporal 8 + db-mvcc 25 green.
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
Re-adds the portable graph round-trip on 8.0 (deleted with DataAPI), now on the
immutable Db so it composes with the generational model.
- src/db/backup.ts: BackupData v1 engine (identical wire format to the 7.32.0 line).
exportGraph() reads through a Db at its pinned generation; importGraph() applies the
whole backup as ONE atomic transact() (a single generation).
- Db.export(selector?, options?) — read at this view's generation, so
brain.asOf(g).export() is a time-travel export and brain.now().with(ops).export() a
what-if export.
- brain.export(selector?, options?) — sugar for now().export().
- brain.import() is polymorphic: a BackupData document -> graph round-trip; a
file/buffer -> existing foreign-file ingestion (dispatched on the 'brainy-backup'
tag; zero migration for ingestion callers).
- Selectors (ids/collection/connected/vfsPath/predicate/whole) + edge policy
(induced/incident/none) + includeVectors/includeContent/includeSystem; system
entities excluded by visibility. onConflict merge/replace/skip, reembed auto/never,
remapIds. DbHost gains storage (VFS blob bytes).
- Types exported from the package root: BackupData/BackupEntity/BackupRelation/
ExportSelector/ExportOptions/ImportOptions/ImportResult + isBackupData.
Tests: tests/unit/db/db-backup.test.ts (10, green) — round-trip, selectors, edges,
vectors+re-embed, merge, asOf/with composition, validation. Full unit suite green.
Follow-ups: docs (8.0 guide + RELEASES + api/README), completeness adds
(since(prior).export() delta, exportStream/importStream, validate()), includeContent
filesystem test, ids-at-asOf get() refinement, @soulcraft/formats Zod schema mirror.
Brainy-owned field names (noun/verb, subtype, createdAt, updatedAt,
confidence, weight, service, data, createdBy, _rev) now have exactly one
home — top level — enforced by three layers driven from a single source of
truth, src/types/reservedFields.ts (RESERVED_ENTITY_FIELDS /
RESERVED_RELATION_FIELDS, exported):
1. Compile time — AddParams/UpdateParams/RelateParams/UpdateRelationParams
metadata (and the transact() ops that extend them) reject a literal
reserved key as a TypeScript error while keeping generic T ergonomics
(typed bags, untyped brains, index-signature shapes, and a documented
exemption for T-declared reserved keys). Pinned by @ts-expect-error
type tests run under vitest typecheck mode on every unit run.
2. Write time — the 7.x update() remap is ported to 8.0 and extended to
every write path: add/update/relate/updateRelation, their transact()
mirrors, and db.with() overlays. User-settable fields lift to their
dedicated param (top-level wins when both are supplied — closes the 7.x
trap where update({metadata:{confidence}}) silently no-oped), and
system-managed fields drop with a one-shot warning naming the right
path. A remapped subtype satisfies subtype-pairing enforcement exactly
like a top-level one.
3. Read time — every storage combine goes through one canonical hydration
helper (hydrateNounWithMetadata / hydrateVerbWithMetadata over
splitNoun/VerbMetadataRecord), so reserved fields surface ONLY top-level
and entity/relation.metadata carry ONLY custom fields on live reads,
batch reads, paginated listings, getRelations by source/target, streamed
verbs, and historical asOf() materialization alike.
Read-path echoes found and fixed (previously the full stored record —
including the verb type key — leaked inside metadata): noun pagination,
verb pagination, getVerbsBySource/ByTarget (adjacency + shard fallback),
getVerbsBySourceBatch (which also dropped subtype/data), and the
filesystem verb stream. getRelations() results now surface
confidence/updatedAt top-level via verbsToRelations, updateRelation() no
longer erases service/createdBy, relate() persists its top-level
confidence/service params, and the dead convertHNSWVerbToGraphVerb echo
path is deleted. Import paths (CLI extract, deduplicator, coordinators,
neural import) write confidence through the dedicated param instead of the
bag. UpdateRelationParams is now exported from the package root.
Documented for consumers in docs/concepts/consistency-model.md ("Reserved
fields") and RELEASES.md. Regression tests ported from the 7.x fix and
extended to the full 8.0 contract (17 runtime tests + 41 type-level
assertions); full unit suite 1427/1427, db-mvcc integration 24/24.
Historical Db values (now()/asOf() pins that history has moved past) now
serve the COMPLETE query surface - vector/hybrid search, graph traversal,
cursor pagination, and aggregation - by materializing ephemeral in-memory
indexes over the exact at-generation record set. The historical-query
throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted.
Materializer (Brainy.materializeAtGeneration):
- Copies the at-G record set (live bytes for ids untouched since the pin,
immutable before-images otherwise) into a fresh MemoryStorage; a final
reconciliation pass under the commit mutex makes the copy exact even
when transactions commit mid-build.
- Opens a read-only Brainy over the copy: init rebuilds the metadata and
graph-adjacency indexes from the records; the vector index is built by
inserting every at-G vector (the at-G HNSW graph never existed on disk,
so there is nothing to restore). Host embedder and aggregate definitions
are shared - no second model load, aggregates backfill at-G values.
- Cost is the documented contract: O(n at G) time and memory, ONCE per Db
(handle cached; freed by release(), with a FinalizationRegistry backstop
that also closes leaked readers). A native VersionedIndexProvider serves
the same reads from retained segments with no rebuild.
Db routing (src/db/db.ts): metadata-level find()/related() keep the free
record path; index-only dimensions (query/vector/near/connected/cursor/
aggregate/includeRelations/non-metadata modes) route to the cached
materialization; unsupported where-operators on the record path re-route
there too instead of erroring. Speculative with() overlays keep the one
honest boundary - SpeculativeOverlayError (overlay entities carry no
embeddings, so index reads over them would be silently incomplete);
metadata find()/get()/filter related() work on overlays.
UpdateParams.vector contract now honored: an explicit pre-computed vector
applies directly (with dimension validation) in update() and transact
update ops, re-indexing HNSW - previously it was silently ignored unless
data also changed.
GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees
filtered through the live-verb tombstone set (entity->entity edge trees
deleted - they carried no verb ids, so removeVerb could never tombstone
them and traversal served stale neighbors forever). Neighbor reads batch-
load live verbs via the unified cache; addVerb seeds the cache.
Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector
search finds old vector placement including since-deleted entities;
historical graph traversal walks the old wiring after a rewire; historical
aggregation computes at-G group values; asOf() pins get the same surface;
the materialization builds once per Db and release() closes the ephemeral
reader (it refuses reads afterwards); overlays throw the documented error.
ADR-001 updated to the no-throws historical model.
The COW version-control surface (fork, branches, checkout, commit,
getHistory/streamHistory, asOfCommit, brain.versions) is gone, along with
its subsystems: src/versioning/, the COW object store (CommitLog,
CommitObject, RefManager, TreeObject), HistoricalStorageAdapter, and the
TypeAwareHNSWIndex path it kept alive. The Db API (now/transact/asOf/
with/persist/restore) is the one versioning model in 8.0.
Survivors and replacements:
- BlobStorage survives (the VFS stores file content through it), relocated
to src/storage/blobStorage.ts with binaryDataCodec.ts; its adapter
interface is now BlobStoreAdapter, slimmed to the consumed surface
(write/read/has/delete/getMetadata + MIME-aware compression policy).
- brain.migrate() backup branches are replaced by persist-before-migrate:
MigrateOptions.backupTo persists a hard-link snapshot of the current
generation before any transform runs; MigrationResult.backupPath reports
it, and brain.restore(path) brings it back wholesale.
- CLI: cow.ts (fork/branch/checkout/history/migrate) is replaced by
snapshot.ts — snapshot <path>, restore <path>, history (tx-log),
generation.
- New public read API: brain.transactionLog({limit}) exposes the reified
tx-log (generation/timestamp/meta, newest first) that backs the CLI
history command; TxLogEntry is exported.
Tests: superseded suites deleted; fork/commit blocks excised from shared
suites; BlobStorage tests relocated + reworked against the slimmed store;
migration tests now prove the backupTo snapshot/restore round trip; new
transactionLog coverage in db-mvcc.
One mechanism replaces the COW and versioning subsystems: immutable
generation-stamped records behind a Datomic-style database value.
Record layer (src/db/generationStore.ts):
- Monotonic generation counter in _system/generation.json; bumped once per
transact() commit and once per single-operation write (storage hook), so
brain.generation() is always a meaningful watermark.
- Commit protocol: stage before-images + tx.json delta -> fsync -> execute
batch via TransactionManager -> atomic tmp+rename of _system/manifest.json
(the rename IS the commit point) -> append _system/tx-log.jsonl.
- Crash recovery on open rolls uncommitted generations back byte-identically
and forces an index rebuild; refcounted pins gate compactHistory(), which
records a horizon (asOf below it throws GenerationCompactedError).
Db API:
- Db: get/find/search/related pinned at a generation, with() speculative
overlays, since() diffs, persist() hard-link-farm snapshots, timestamp,
generation, release() + FinalizationRegistry backstop.
- Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch
(GenerationConflictError CAS), asOf(generation|Date|path), restore(path,
{confirm}), compactHistory(), generation(), static open() + load().
- get()/metadata find()/related() are fully correct at any reachable pinned
generation; index-accelerated queries at historical generations throw
NotYetSupportedAtHistoricalGenerationError - never silently-wrong results.
- VersionedIndexProvider (generation/isGenerationVisible/pin/release) in
plugin.ts: feature-detected, balanced pin/release in lockstep with Db
lifecycle, post-commit applier + replay-gap model documented.
Storage primitives (BaseStorage + filesystem/memory adapters): raw-object
read/write/list/remove, fsync barrier, noun/verb raw before-image capture,
tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and
append-in-place exceptions), restoreFromDirectory + derived-state reload.
Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation
across 200 mutations, batch atomicity under injected execution failure,
ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety
under pins, with() overlay isolation, generation monotonicity across
reopen, crash consistency through the real recovery path, and versioned
provider pin/release balance. Design record in
docs/ADR-001-generational-mvcc.md.
