asOf() answers "state AT a point"; these answer "what happened BETWEEN two
points" and "one entity's whole history", all on the existing generation records.
- Extract resolveAsOfGeneration() as the shared gen|Date→{generation,timestamp}
chokepoint (reachability + Date semantics identical everywhere). asOf()'s
inclusive path is byte-identical — db-mvcc still 25/25.
- asOf(target, { exclusive }) — strict-before (resolved−1, clamped at gen 0,
never a RangeError).
- db.since(Db | generation | Date) — overload; number/Date resolve via the shared
resolver (new DbHost.resolveGeneration); EXCLUSIVE lower bound;
since(db) === since(db.generation). Same-store guard via Db.belongsToStore.
- brain.diff(a, b) → { added, removed, modified } split by nouns/verbs. EARNS its
name: candidate set is ONLY changedBetween(gLow,gHigh), each classified by
existence at both endpoints + a key-order-insensitive value compare
(new src/db/stableEqual.ts) — a touched-but-reverted / born-and-died id is in
no bucket.
- brain.history(id, { from, to }) → every distinct version oldest→newest, each
value === asOf(version.generation).get(id); null = removal; kind auto-detected
(throws on UUID-space collision). New store helper generationsTouching().
- brain.transactionLog({ from, to, limit }) — INCLUSIVE generation/Date window
(contrast since's exclusive lower bound); limit applied last.
- Compaction policy locked: diff/since THROW GenerationCompactedError below the
horizon; history TRUNCATES to it.
Types DiffResult/HistoryVersion/EntityHistory exported from the package root.
Tests: tests/integration/db-temporal.test.ts (8 proofs incl. diff-earns-its-name,
history↔asOf cross-check, the composition proof, granularity, compaction contrast)
+ tests/unit/db/stableEqual.test.ts (6). docs/guides/snapshots-and-time-travel.md
+ RELEASES updated. 1477 unit + db-temporal 8 + db-mvcc 25 green.
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
Re-adds the portable graph round-trip on 8.0 (deleted with DataAPI), now on the
immutable Db so it composes with the generational model.
- src/db/backup.ts: BackupData v1 engine (identical wire format to the 7.32.0 line).
exportGraph() reads through a Db at its pinned generation; importGraph() applies the
whole backup as ONE atomic transact() (a single generation).
- Db.export(selector?, options?) — read at this view's generation, so
brain.asOf(g).export() is a time-travel export and brain.now().with(ops).export() a
what-if export.
- brain.export(selector?, options?) — sugar for now().export().
- brain.import() is polymorphic: a BackupData document -> graph round-trip; a
file/buffer -> existing foreign-file ingestion (dispatched on the 'brainy-backup'
tag; zero migration for ingestion callers).
- Selectors (ids/collection/connected/vfsPath/predicate/whole) + edge policy
(induced/incident/none) + includeVectors/includeContent/includeSystem; system
entities excluded by visibility. onConflict merge/replace/skip, reembed auto/never,
remapIds. DbHost gains storage (VFS blob bytes).
- Types exported from the package root: BackupData/BackupEntity/BackupRelation/
ExportSelector/ExportOptions/ImportOptions/ImportResult + isBackupData.
Tests: tests/unit/db/db-backup.test.ts (10, green) — round-trip, selectors, edges,
vectors+re-embed, merge, asOf/with composition, validation. Full unit suite green.
Follow-ups: docs (8.0 guide + RELEASES + api/README), completeness adds
(since(prior).export() delta, exportStream/importStream, validate()), includeContent
filesystem test, ids-at-asOf get() refinement, @soulcraft/formats Zod schema mirror.
Brainy-owned field names (noun/verb, subtype, createdAt, updatedAt,
confidence, weight, service, data, createdBy, _rev) now have exactly one
home — top level — enforced by three layers driven from a single source of
truth, src/types/reservedFields.ts (RESERVED_ENTITY_FIELDS /
RESERVED_RELATION_FIELDS, exported):
1. Compile time — AddParams/UpdateParams/RelateParams/UpdateRelationParams
metadata (and the transact() ops that extend them) reject a literal
reserved key as a TypeScript error while keeping generic T ergonomics
(typed bags, untyped brains, index-signature shapes, and a documented
exemption for T-declared reserved keys). Pinned by @ts-expect-error
type tests run under vitest typecheck mode on every unit run.
2. Write time — the 7.x update() remap is ported to 8.0 and extended to
every write path: add/update/relate/updateRelation, their transact()
mirrors, and db.with() overlays. User-settable fields lift to their
dedicated param (top-level wins when both are supplied — closes the 7.x
trap where update({metadata:{confidence}}) silently no-oped), and
system-managed fields drop with a one-shot warning naming the right
path. A remapped subtype satisfies subtype-pairing enforcement exactly
like a top-level one.
3. Read time — every storage combine goes through one canonical hydration
helper (hydrateNounWithMetadata / hydrateVerbWithMetadata over
splitNoun/VerbMetadataRecord), so reserved fields surface ONLY top-level
and entity/relation.metadata carry ONLY custom fields on live reads,
batch reads, paginated listings, getRelations by source/target, streamed
verbs, and historical asOf() materialization alike.
Read-path echoes found and fixed (previously the full stored record —
including the verb type key — leaked inside metadata): noun pagination,
verb pagination, getVerbsBySource/ByTarget (adjacency + shard fallback),
getVerbsBySourceBatch (which also dropped subtype/data), and the
filesystem verb stream. getRelations() results now surface
confidence/updatedAt top-level via verbsToRelations, updateRelation() no
longer erases service/createdBy, relate() persists its top-level
confidence/service params, and the dead convertHNSWVerbToGraphVerb echo
path is deleted. Import paths (CLI extract, deduplicator, coordinators,
neural import) write confidence through the dedicated param instead of the
bag. UpdateRelationParams is now exported from the package root.
Documented for consumers in docs/concepts/consistency-model.md ("Reserved
fields") and RELEASES.md. Regression tests ported from the 7.x fix and
extended to the full 8.0 contract (17 runtime tests + 41 type-level
assertions); full unit suite 1427/1427, db-mvcc integration 24/24.
Historical Db values (now()/asOf() pins that history has moved past) now
serve the COMPLETE query surface - vector/hybrid search, graph traversal,
cursor pagination, and aggregation - by materializing ephemeral in-memory
indexes over the exact at-generation record set. The historical-query
throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted.
Materializer (Brainy.materializeAtGeneration):
- Copies the at-G record set (live bytes for ids untouched since the pin,
immutable before-images otherwise) into a fresh MemoryStorage; a final
reconciliation pass under the commit mutex makes the copy exact even
when transactions commit mid-build.
- Opens a read-only Brainy over the copy: init rebuilds the metadata and
graph-adjacency indexes from the records; the vector index is built by
inserting every at-G vector (the at-G HNSW graph never existed on disk,
so there is nothing to restore). Host embedder and aggregate definitions
are shared - no second model load, aggregates backfill at-G values.
- Cost is the documented contract: O(n at G) time and memory, ONCE per Db
(handle cached; freed by release(), with a FinalizationRegistry backstop
that also closes leaked readers). A native VersionedIndexProvider serves
the same reads from retained segments with no rebuild.
Db routing (src/db/db.ts): metadata-level find()/related() keep the free
record path; index-only dimensions (query/vector/near/connected/cursor/
aggregate/includeRelations/non-metadata modes) route to the cached
materialization; unsupported where-operators on the record path re-route
there too instead of erroring. Speculative with() overlays keep the one
honest boundary - SpeculativeOverlayError (overlay entities carry no
embeddings, so index reads over them would be silently incomplete);
metadata find()/get()/filter related() work on overlays.
UpdateParams.vector contract now honored: an explicit pre-computed vector
applies directly (with dimension validation) in update() and transact
update ops, re-indexing HNSW - previously it was silently ignored unless
data also changed.
GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees
filtered through the live-verb tombstone set (entity->entity edge trees
deleted - they carried no verb ids, so removeVerb could never tombstone
them and traversal served stale neighbors forever). Neighbor reads batch-
load live verbs via the unified cache; addVerb seeds the cache.
Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector
search finds old vector placement including since-deleted entities;
historical graph traversal walks the old wiring after a rewire; historical
aggregation computes at-G group values; asOf() pins get the same surface;
the materialization builds once per Db and release() closes the ephemeral
reader (it refuses reads afterwards); overlays throw the documented error.
ADR-001 updated to the no-throws historical model.
The COW version-control surface (fork, branches, checkout, commit,
getHistory/streamHistory, asOfCommit, brain.versions) is gone, along with
its subsystems: src/versioning/, the COW object store (CommitLog,
CommitObject, RefManager, TreeObject), HistoricalStorageAdapter, and the
TypeAwareHNSWIndex path it kept alive. The Db API (now/transact/asOf/
with/persist/restore) is the one versioning model in 8.0.
Survivors and replacements:
- BlobStorage survives (the VFS stores file content through it), relocated
to src/storage/blobStorage.ts with binaryDataCodec.ts; its adapter
interface is now BlobStoreAdapter, slimmed to the consumed surface
(write/read/has/delete/getMetadata + MIME-aware compression policy).
- brain.migrate() backup branches are replaced by persist-before-migrate:
MigrateOptions.backupTo persists a hard-link snapshot of the current
generation before any transform runs; MigrationResult.backupPath reports
it, and brain.restore(path) brings it back wholesale.
- CLI: cow.ts (fork/branch/checkout/history/migrate) is replaced by
snapshot.ts — snapshot <path>, restore <path>, history (tx-log),
generation.
- New public read API: brain.transactionLog({limit}) exposes the reified
tx-log (generation/timestamp/meta, newest first) that backs the CLI
history command; TxLogEntry is exported.
Tests: superseded suites deleted; fork/commit blocks excised from shared
suites; BlobStorage tests relocated + reworked against the slimmed store;
migration tests now prove the backupTo snapshot/restore round trip; new
transactionLog coverage in db-mvcc.
One mechanism replaces the COW and versioning subsystems: immutable
generation-stamped records behind a Datomic-style database value.
Record layer (src/db/generationStore.ts):
- Monotonic generation counter in _system/generation.json; bumped once per
transact() commit and once per single-operation write (storage hook), so
brain.generation() is always a meaningful watermark.
- Commit protocol: stage before-images + tx.json delta -> fsync -> execute
batch via TransactionManager -> atomic tmp+rename of _system/manifest.json
(the rename IS the commit point) -> append _system/tx-log.jsonl.
- Crash recovery on open rolls uncommitted generations back byte-identically
and forces an index rebuild; refcounted pins gate compactHistory(), which
records a horizon (asOf below it throws GenerationCompactedError).
Db API:
- Db: get/find/search/related pinned at a generation, with() speculative
overlays, since() diffs, persist() hard-link-farm snapshots, timestamp,
generation, release() + FinalizationRegistry backstop.
- Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch
(GenerationConflictError CAS), asOf(generation|Date|path), restore(path,
{confirm}), compactHistory(), generation(), static open() + load().
- get()/metadata find()/related() are fully correct at any reachable pinned
generation; index-accelerated queries at historical generations throw
NotYetSupportedAtHistoricalGenerationError - never silently-wrong results.
- VersionedIndexProvider (generation/isGenerationVisible/pin/release) in
plugin.ts: feature-detected, balanced pin/release in lockstep with Db
lifecycle, post-commit applier + replay-gap model documented.
Storage primitives (BaseStorage + filesystem/memory adapters): raw-object
read/write/list/remove, fsync barrier, noun/verb raw before-image capture,
tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and
append-in-place exceptions), restoreFromDirectory + derived-state reload.
Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation
across 200 mutations, batch atomicity under injected execution failure,
ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety
under pins, with() overlay isolation, generation monotonicity across
reopen, crash consistency through the real recovery path, and versioned
provider pin/release balance. Design record in
docs/ADR-001-generational-mvcc.md.
