An adopter's full suite found two v2 write-path defects on fresh brains,
reproduced with stacks; both cured and both pinned with their exact
production shapes:
1. PAD-FRAME CONSTRUCTIBILITY: a single msgpack bin filler steps its
header by one byte at each size class (bin8→bin16→bin32), leaving one
unreachable payload size per boundary — the sealer requested a
291-byte pad, the encoder threw 'not constructible', and sync() died
whole. Construction is now TOTAL: the class-boundary holes bridge with
a trailing fixint beside the bin ({bin(n)} ∪ {bin(n)+fixint} covers
every size ≥ minimum). Pinned exhaustively: every size from the
minimum through a full sector plus boundary spill constructs
byte-exact and decodes as reader-invisible filler.
2. THE NON-MONOTONIC REFUSAL LOOP: the append-failure compensation
rewound the generation counter on ANY throw — including a covering
SYNC failure after a SUCCESSFUL append. The log carried generation N
while the counter re-minted N, and every later append refused
'non-monotonic (N ≤ head N)' — the write path wedged in a refusal
loop through deferred-embed retries and flush backoff. The
compensation now splits by phase: an append failure (log never took
the fact) fully compensates — un-buffer and rewind; a sync failure
after append earns the rewind ONLY if the appended fact is provably
dropped, otherwise the generation stays consumed and buffered — the
counter never re-mints a number the log may carry. Pinned: an
injected one-shot sync failure fails its write loudly and the very
next write mints fresh and succeeds, with the log scanning strictly
ascending end to end.
Also probed against the adopter's carried report: the 9.0 vfs.rename
stale-ghost shape does NOT reproduce on this head (old path cleanly
unresolvable on exists/stat/readdir after rename).
Gates: unit 2067/2067 (160 files) · integration 833 (97 files) ·
conformance 36/36.
THE DEFAULT FLIP (ruled on proven evidence — at-ack survived 301/301
acked-writes-through-power-cut in block-layer fault injection; deferred
tree authority demonstrably loses flush-covered acks): a brain with NO
stored authority artifact now ADOPTS LOG AUTHORITY AT OPEN. The oracle
gates the flip exactly as the guarded adoption path always did — curable
divergences baseline-backfilled, the flip lands ONLY on a green verdict —
and a brain that cannot verify STAYS tree-authoritative loudly, with the
refusal recorded on the switch artifact so subsequent opens are cheap.
config logAuthority: 'defer' is the explicit documented opt-out (no
automatic adoption; declared flush-window loss; adoptLogAuthority() flips
later). A stored artifact always wins. RELEASES.md carries the posture.
Two standing .fails debt pins FLIP TO HOLDING under the default: the
at-ack crash-survival gap and the ack-at-log durability target — both now
permanent asserted truths, not aspirations.
POWER-CUT THROW SITES (fault-injection findings, brainy-alone config):
- A manifest-listed-but-unloadable column segment QUARANTINES at
discovery (loud once, counted always, quarantinedSegments() exposed for
the heal) and the field serves its remaining segments DEGRADED — never
a raw throw killing every query on the field. Real storage faults still
propagate untouched.
- Torn generation artifacts (NaN/garbage in manifest or counter) DISCARD
with narration at the store's open and recovery re-derives — plus a
defensive finite-integer guard at the init consumer. Never a RangeError
killing an open.
THE LOUD TORN-RECORD CONTRACT: an existing-but-unparseable stored record
now surfaces as a typed, counted TornRecordError on every entity-read
surface (including fifteen previously-blind per-item batch catches);
ENOENT stays clean-absent; artifact readers with designed absent-recovery
keep null-tolerance behind the loud floor. Disk corruption can no longer
read as silent data invisibility.
Suite migration: the default's pins inverted deliberately, generation
baselines made relative, quarantine-contract pins rewritten to the ruled
behavior.
Gates: tsc 0 · unit 2065/2065 (159 files) · integration 826 (93 files) ·
conformance 31/31 · kill-matrix 15/15 · torn-open guards 2/2.
- Watermark stamping fans out at flush: all three projections stamped
with the committed generation before their flushes persist.
- waitForIndexed(path?, {generation, timeoutMs}) — the one honest read
barrier for write-then-recall consumers; typed timeout error carries
the pending count and names the gauge; getIndexStatus() gains
per-projection gauges. awaitPendingEmbeds() unchanged underneath.
- adoptLogAuthority() self-backfills curable divergences (pre-log
records, witness drift) by identity re-commit before flipping — a
fresh brain flips clean; log-ahead divergences still refuse loudly.
- The verification oracle gains VERB legs (all four divergence classes;
unwired = honest verbsChecked: 0, never a scope claim).
- find({where: {}}) match-all serves (was silent-empty, warm AND cold;
same fix in count/streaming/subgraph seeding); removeMany({where:{}})
refuses typed — a match-all bulk delete must be explicit.
- Aggregation native envelope stamped via noteSourceGeneration before
serializeState; the native-blob restore gates through the same
adoption verdict as caller-side state (the unconditional adopt dies).
- LC8 pinned: a wholesale directory move opens and serves identically
across all three intelligences, with history traveling.
Gates: unit 2031/2031 (156 files) · integration 812 (91 files) ·
conformance 27/27.
The cutover: new tail segments write format v2 (per-record [type, version,
cipherFlag, keyId] envelope; noun/verb after-images carry dense ints
MINTED AT APPEND from the id mapper — a rebuilt mapper reproduces
assignments exactly; log.genesis opens every new log with the id-space
width + a minted brain id; sync() seals to the header-declared sector
boundary with reader-invisible pad frames). Existing v1 segments are
never rewritten — per-segment decoder dispatch reads both formats and v2
facts map to the exact CommitFact shape all consumers already read.
Cutover on a live v1 log: an empty v1 tail re-heads in place; a non-empty
one is sealed by rotation, byte-identical. Records reserve the encryption
fields (cipherFlag 0 / keyId nil are the only legal values; anything else
refuses typed naming the needed newer reader) — crypto-ready with no
future bump on the compat surface. Empty-records facts are legal (an
all-deduped batch is a real generation — v1 semantics preserved; the
refusal there tore a column-store flush mid-commit in the full suite, the
consistency guard caught it loudly, and the root is fixed).
Golden byte vectors pinned for the second (native) reader implementation.
Pins: cutover 5/5 · codec 54 · kill-matrix stays 11/11.
Two release-blocking findings from the durability kill-matrix, both fixed
in the owning layer:
1. LOG-AUTHORITY REPLAY AT OPEN: durable-at-ack fsynced the fact before
the ack, but open() truncated every fact above the manifest — after a
power loss that takes the un-fsynced tmp+rename canonical bytes, the
acked write's ONLY durable copy was discarded. Now: under 'log'
authority, open() REPLAYS intact facts above the manifest into
canonical (FactLog.peekFactsAbove — CRC-gated, order-sorted) and
advances the manifest to cover them; tree-authority brains keep the
truncate contract they were promised. Pinned end to end: the power-loss
row constructs the exact disk state (fsynced log, vanished canonical
rename) and the acked write lives.
2. NO SILENT COMMIT: commitSingleOp buffered the generation BEFORE the
fact append; an append failure (ENOSPC) rejected the caller but the
next flush durably committed the generation with NO fact — a permanent
silent log gap. Now the failure path un-buffers and returns the counter
reservation: nothing commits, the log stays gap-free, and the canonical
execute-residue orphan is the documented crash-equivalent.
Plus: the kill-matrix itself (11 rows — every commit-path fault point ×
reopen-as-crash recovery contract, at-ack variants, disk-full row; five
new zero-cost faultPoint sites), the log-authority pin suite (oracle
green/red/state-differs, flip refusal, switch survives reopen, 9/9), and
the group-commit covering pins (5/5).
Gates: unit 2002/2002 (152 files) · integration 785 · conformance 27/27.
The two-implementation contract surface as one pure module (no I/O):
segment header v2 (formatVersion 2 + sealSize in the reserved bytes),
per-record [type u8, version u8] envelope killing the unknown-kind
misclassification trap, the 12-type registry (after-images with minted
ints, tombstones, batch.meta, embed.pending/landed, blob.manifest,
projection.note, bootstrap.baseline, log.genesis with id-space width and
TYPED width-mismatch refusal), vectorLeg inline|{sameAsGeneration} with
writer-enforced single-hop, sector-sealed groups with pad frames, torn-tail
discipline, and GOLDEN BYTE VECTORS pinned so a second (native) reader
implementation can conform byte-for-byte. 50 format pins + a
fault-injecting storage wrapper (tear/drop-sync/fail-append) with 13
self-tests. v1 segments remain readable; nothing writes v2 yet — the
live-format cutover is its own commit.
The write side of the law, ruled 2026-08-03: data is either in main space
where developers can use anything, or it is in system.*.
- The reserved-name write door DIES: add/update/relate/updateRelation
metadata bags accept EVERY name (confidence, type, id, data, level,
content, ...) as ordinary user fields — indexed, filterable, sortable,
aggregatable, identical to any other field. The remap/enforce/warn
machinery, the reservedFieldPolicy config (now a typed init refusal),
and the compile-time metadata key bans are all removed. The one write
refusal left: keys spelled 'system.*' (namespace forgery), now enforced
on all four write doors.
- STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag
nested verbatim under 'metadata', sealed by a format stamp — by-name
storage discrimination is unsound once colliders are admitted. Legacy
flat records stay readable forever through the shape-aware splitters
(sound for them: the old door refused colliders). Time travel rides the
same split (generation store snapshots whole records).
- Name-based index exclusions DIE: user frame indexes every name; the
excludeFields/indexedFields knobs and their silent-[] holes are gone;
bulk-payload protection is value-shape only, uniform across names.
- Consumer-sweep findings fixed in the same wave: per-type counts read
the frozen 'system.type' column (addToIndex sort, affinity tracking,
cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for
pre-rebuild reads); resolveHiddenIds addresses 'system.visibility'
(bare 'visibility' was a silent no-op under the law — VFS/system
entities leaked into default reads).
- Fidelity fallout fixed in the owning layers: readEntityFieldAddress
reads the bag first (colliders were absent-shadowed by its own guard)
and never serves system addresses from the bag; blob history refs read
the bag shape-aware; migration transforms now receive ONE normalized
view (engine fields + nested bag) regardless of stored era, and stray
flat-habit keys refuse with the fix in the message.
- THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as
gates-green): all ten collider names + plumbing names written as user
fields, verified verbatim + queryable across live reads, flush+reopen,
a forced epoch rebuild, and asOf time travel; relation mirror; forgery
refusals; legacy flat-record compat. 8/8 green.
Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance
27/27 (exit 0) · consumer test sweep migrated (10 files).
First stage of the co-frozen D1+D3+repacking unit: the format core,
self-contained under _generations/segments/.
- seg-<firstGen,20pad>.bgs: append-once packs of consecutive
generations (magic BGS1; frame = u32 len + u32 crc32c + msgpack
[generation, timestamp, delta, records, flags]; flags reserves
compressed-payload evolution without a format break). Sealed
segments are immutable — fold refuses overlap with sealed ranges.
- seg-<firstGen>.idx: DERIVED sidecar (per-generation frame offsets +
per-id generation postings + checksums); lost/corrupt sidecars
rebuild from their segment loudly; a damaged segment (frame CRC
mismatch) fails loudly, never serves wrong bytes.
- manifest.json: the one discovery path — open() reads it and never
lists the packed backlog (the scan-wedge class's cure); refuses a
newer manifest version rather than serving partial history.
- D3 semantics: dropSegmentsBelow reclaims WHOLE segments at
boundaries only and bumps compactedBelow durably; archival-profile
enforcement stays with the caller per the co-freeze.
- D8 rider: digestThroughPacked(g) — deterministic crc32c chain over
sealed-segment checksums (+ frame-level prefix mid-segment),
O(segments), reopen-stable.
Also fixes a cross-adapter contract bug the suite caught: memory
storage's deleteObjectFromPath ignored the raw-bytes store, so
deleteRawObject on a raw-bytes path (fact-log or segment files)
silently no-op'd — deletes now match filesystem unlink semantics.
Six pins. Wiring into GenerationStore (two-tier reads, the repacker,
cold-open manifest path) lands with the rest of the unit before its
release; cortex's fact-record/stamp shapes reconcile the sidecar
keying when they post.
Stage-2 D1 contract item (co-frozen): a fact scan may be slow, never
silent. batches() now races its FIRST pull against
SCANFACTS_FIRST_BATCH_MS (10s, exported; test-overridable) — a wedged
or unreadably slow store produces a loud abort naming the contract
instead of a consumer hanging indistinguishably from progress (the
production shape: a heal against a generations-backlogged brain
wedged silently on the first segment read).
Only the first pull is raced: the bound is time-to-first-batch (proof
the producer is alive), not per-batch pacing, and it runs only while
a pull is pending — consumer think-time between pulls never counts
against the producer (pinned).
Three pins: wedged-store loud failure within the bound, healthy scan
untouched end-to-end, slow-consumer immunity.
flush() is durability work: it must cost what the current window's
deltas cost, never what the history backlog costs. Under adaptive
retention the byte budget derives from free memory, so bulk-load
pressure shrank the budget exactly at peak write volume and flush paid
actual reclaim inline — a production deployment measured single writes
blocked 25-191s behind reclaim-on-flush.
- flush() no longer calls autoCompactHistory(); close() is THE
auto-compaction site (already ran there; now alone).
- Every auto pass is time-bounded (CLOSE_COMPACTION_BUDGET_MS = 5s):
reclamation is oldest-first, so an early stop is a consistent prefix
and the next pass resumes. Explicit compactHistory() gains an
optional timeBudgetMs for caller-chosen maintenance windows.
- Documented trade stated where operators read: a long-lived writer
that never closes accumulates history until its next explicit
compactHistory() — predictable writes, explicit maintenance.
Pins: flush-never-reclaims + close-reclaims-durably (db-mvcc), bounded
pass stops-then-resumes as a consistent prefix (generationStore unit).
Under default adaptive retention, every flush() recomputed total history
bytes by walking EVERY committed generation's delta — O(all generations)
with disk re-reads past the 4096-entry delta-cache bound. On a production
brain with 70,000+ accumulated generations this turned every write into a
full-tail scan (60-100s writes, escalating with history growth), even
though the free-RAM budget never tripped and nothing was ever reclaimed
(SELF-GENERATIONS-GROWTH).
historyBytes() now maintains a running total: seeded by one walk on first
use, then updated incrementally at both commit paths (+bytes) and the
compaction reclaim loop (−bytes), dropped on reopenAfterRestore. The
adaptive retention check on every flush is O(1). Invariant regression-
pinned: running total ≡ fresh walk through transact commits, single-op
group commits, and compaction.
New brain.historyStats() (exported HistoryStats): read-only generation
count / bytes / generation+timestamp range / horizon / retention mode /
effective budget — the one-call per-brain fleet audit for retention
exposure.
Every committed generation now also appends a FACT — an after-image
commit record (what each touched entity/relationship became, or a
body-less tombstone for a removal) — to an append-only, crc32c-framed
segment log under _generations/facts/. The before-image history and the
canonical tree remain authoritative; the fact log gives consumers ONE
sequential, self-verifying stream (index heals, incremental replays)
in place of a per-entity directory walk.
- Wire format: positional msgpack facts [generation, timestamp, ops,
meta, blobHashes]; op = [kind u8, id bin16, record | nil tombstone];
32-byte segment header (magic, formatVersion, firstGeneration,
zeroed+verified reserved); length+crc32c frame per fact; zero-padded
segment names so lexicographic order == generation order; JSON
manifest with an atomic rename flip, manifest-first rotation.
- Commit protocol: facts append+fsync BEFORE the commit point inside
the existing durability window, so a crash can only leave the log
AHEAD of committed truth — open() truncates back (torn tails detected
by CRC). Absent generation = never committed; a scan can never see an
uncommitted fact. transact() facts are durable-on-return; single-op
facts ride the group-commit flush exactly like buffered history. A
fact-append failure fails the write, loudly — a silent gap would be a
lie a later replay discovers.
- New public surface: brain.scanFacts() (sequential batches with heal
telemetry: head/segments/approx up front, per-batch generation range
+ bytes + segment id, loud abort on gaps, summary cross-check) and
brain.factSegmentPaths() (immutable sealed segments for zero-copy
consumers; the mutable tail excluded). Exported types CommitFact,
FactOp, FactScanBatch, FactScanHandle.
- Storage: optional binary raw-byte primitives (appendRawBytes,
readRawBytes, writeRawBytes, rawByteSize) on StorageAdapter —
feature-detected; filesystem + memory adapters implement them; an
adapter without them hosts no fact log. Fact segments are byte-copied
(never hard-linked) into snapshots. The _generations/facts/ namespace
is registered as a protected family (rebuildable: false): no sweeper
or GC may delete under it.
- New crc32c (Castagnoli) utility with RFC known-answer tests.
The async group-commit flush that persists single-op generation history
swallowed persist failures as a bare warn: writes kept succeeding while their
before-images piled up in memory, never durable and unbounded, with no signal.
The generation store now accounts for every failed flush at one place
(flushPendingSingleOps), tolerates a transient blip (retry with capped
exponential backoff), and after PENDING_FLUSH_FAILURE_THRESHOLD consecutive
failures LATCHES a durability failure and refuses further single-op and transact
writes with a typed, exported PendingFlushDurabilityError — rather than promise a
durability it cannot deliver. Live canonical data is untouched; only the
immutable history is stuck. The latch self-heals: the moment a flush succeeds
(a retry, or an explicit flush()/close()) it lifts and writes resume.
Loud errors, never quiet losses.
The MVCC time-travel layer kept nounChains/verbChains as unbounded
Map<id, number[]> — one resident chain per id ever touched across retained
history (O(N) RAM, defeating billion-scale time travel). Replace with a hot-tail
window + bounded cold LRU + a mutex-free bulk resolver, keeping resolveAt exactly
correct.
The most-recent W generations stay resident as full chains (the common recent-pin
read is O(log), zero scan); deeper pins reconstruct one id's chain on demand into
an L-bounded LRU. Reconstruction is LOCK-LIGHT — it holds no commit mutex, so a
historical read never stalls writers; correctness holds because a live pin's
answer is always > minPinnedGeneration, which compaction can never reclaim, and a
concurrently-reclaimed gen below that is skipped. materializeAtGeneration routes
through a new mutex-free resolveManyAt (one forward pass, O(R + |ids|)) — without
it a deep-pin materialize both regresses to O(N*R) and deadlocks on snapshotWith's
mutex. The cold cache is invalidated on re-touch to stay coherent. Resident RAM is
O(W*d + L), independent of N. 11-case test (oracle-vs-bruteforce, held-Db across
eviction + concurrent compaction, no-deadlock, write-path I/O-free); unit 1735 +
integration 613 (db-mvcc/db-temporal/db-asof green).
8.0 keeps the canonical operators (eq/ne/gt/gte/lt/lte) and their clean
long-form aliases (equals/notEquals/greaterThan/greaterThanOrEqual/lessThan/
lessThanOrEqual), and drops the four redundant deprecated spellings:
is → eq, isNot → ne, greaterEqual → gte, lessEqual → lte
Removed from every evaluator (metadataIndex criteria + range switches,
metadataFilter, the db whereMatcher egress path) and from the
BrainyFieldOperators type, the unsupported-operator error message, the docs
(QUERY_OPERATORS / api README / VFS projection + semantic guides), and the
whereMatcher alias tests. Also migrated Brainy's own internal use — the VFS
TemporalProjection queried with greaterEqual/lessEqual, which would have
silently broken — to gte/lte.
Full gate green: build, unit 1512, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the two coverage gaps a temporal-completeness audit surfaced (the code
was already correct, just untested):
- changedBetween / generationsTouching include un-flushed PENDING generations
and stay identical across the flush — the building blocks of since/diff/history
see single-op writes with no forced flush.
- setRetentionBudget() drives adaptive auto-compaction on flush() down to the
byte budget: history is reclaimed, the live record is untouched (the cor #65
retention-consume wiring, proven end-to-end).
65 generation/temporal tests green; 1528 unit green.
Every write — transact() AND single-op add/update/remove/relate — is now its
own immutable generation (Model-B), so a now() pin always freezes and
asOf/since/diff/history/transactionLog reflect single-ops exactly like
transacts. Closes the Model-A hole where pins did not freeze against single-op
writes.
Generation-stamping:
- GenerationStore.commitSingleOp: a one-operation commitTransaction with
deferred durability. Wired into add/update/remove/relate/updateRelation/
unrelate + removeMany (the *Many and VFS paths delegate to these).
- Async group-commit (flushPendingSingleOps): the live write is acknowledged
immediately; its before-image is buffered in an in-memory pending tier that
resolveAt/chains/changedBetween/tx-log read like on-disk generations, so the
synchronous now() freezes with no forced flush. One fsync per window
(triggers: size / 50ms timer / flush / close / transact / compactHistory).
- Crash recovery is drop-without-restore for group-commit generations (marked
groupCommit:true): a crash mid-flush discards the partial generation and
never restores its before-images, which would otherwise revert the
already-acknowledged live write.
- Init-time infrastructure (the VFS root) is the un-versioned generation-0
baseline: a fresh brain reports generation()===0 and an empty
transactionLog(); the first user write is generation 1.
- Historical find()/related() overlay bound is the full reserved watermark
(generation()), so un-flushed single-op writes are overlaid too.
Retention knob:
- config `history` -> `retention`: 'all' | 'adaptive' |
{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset ->
adaptive (disk/RAM pressure, zero-config). CompactHistoryOptions floors ->
caps (retainGenerations->maxGenerations, retainMs->maxAge, +maxBytes):
reclaim oldest-unpinned while ANY cap is exceeded; pins always exempt.
- brain.setRetentionBudget(bytes) drives the adaptive byte budget at runtime
(a coordinator's fair-share input). Per-generation bytes recorded in each
delta enable historyBytes() introspection without a storage size API.
Tests: per-write generation resolution, pin freeze vs add/update/remove,
drop-without-restore corruption-trap (fault injector), clean-reopen replay,
maxBytes/maxAge/no-cap reclamation, retention-then-reopen. 107 db/generation/
temporal tests green, tsc clean. Docs (ADR-001, consistency-model, snapshots
guide, api reference, RELEASES) updated to per-write granularity + retention.
Historical reads (asOf/get) scanned the global committedGens list linearly,
making them O(database-age): a read of an unchanged entity at an old pin scaled
~12x for 10x history depth. Add per-id inverted history chains (nounChains/
verbChains) so resolveAt binary-searches the id's own generation chain instead —
O(log) and flat with depth. Chains build lazily under the commit mutex,
maintain incrementally on commit, and invalidate on compaction.
Also bound deltaCache (LRU cap 4096; getDelta re-reads evicted deltas) so a
long-lived high-write process's heap is O(cap) not O(generations) on the
disk-backed path, and binary-search commitTimestampAtOrBefore (O(log)).
Verified: 373 unit tests green; new tests/unit/db/generation-chain.test.ts
covers chain resolution, eviction re-read, and chain rebuild after compaction.
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
asOf() answers "state AT a point"; these answer "what happened BETWEEN two
points" and "one entity's whole history", all on the existing generation records.
- Extract resolveAsOfGeneration() as the shared gen|Date→{generation,timestamp}
chokepoint (reachability + Date semantics identical everywhere). asOf()'s
inclusive path is byte-identical — db-mvcc still 25/25.
- asOf(target, { exclusive }) — strict-before (resolved−1, clamped at gen 0,
never a RangeError).
- db.since(Db | generation | Date) — overload; number/Date resolve via the shared
resolver (new DbHost.resolveGeneration); EXCLUSIVE lower bound;
since(db) === since(db.generation). Same-store guard via Db.belongsToStore.
- brain.diff(a, b) → { added, removed, modified } split by nouns/verbs. EARNS its
name: candidate set is ONLY changedBetween(gLow,gHigh), each classified by
existence at both endpoints + a key-order-insensitive value compare
(new src/db/stableEqual.ts) — a touched-but-reverted / born-and-died id is in
no bucket.
- brain.history(id, { from, to }) → every distinct version oldest→newest, each
value === asOf(version.generation).get(id); null = removal; kind auto-detected
(throws on UUID-space collision). New store helper generationsTouching().
- brain.transactionLog({ from, to, limit }) — INCLUSIVE generation/Date window
(contrast since's exclusive lower bound); limit applied last.
- Compaction policy locked: diff/since THROW GenerationCompactedError below the
horizon; history TRUNCATES to it.
Types DiffResult/HistoryVersion/EntityHistory exported from the package root.
Tests: tests/integration/db-temporal.test.ts (8 proofs incl. diff-earns-its-name,
history↔asOf cross-check, the composition proof, granularity, compaction contrast)
+ tests/unit/db/stableEqual.test.ts (6). docs/guides/snapshots-and-time-travel.md
+ RELEASES updated. 1477 unit + db-temporal 8 + db-mvcc 25 green.
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
Re-adds the portable graph round-trip on 8.0 (deleted with DataAPI), now on the
immutable Db so it composes with the generational model.
- src/db/backup.ts: BackupData v1 engine (identical wire format to the 7.32.0 line).
exportGraph() reads through a Db at its pinned generation; importGraph() applies the
whole backup as ONE atomic transact() (a single generation).
- Db.export(selector?, options?) — read at this view's generation, so
brain.asOf(g).export() is a time-travel export and brain.now().with(ops).export() a
what-if export.
- brain.export(selector?, options?) — sugar for now().export().
- brain.import() is polymorphic: a BackupData document -> graph round-trip; a
file/buffer -> existing foreign-file ingestion (dispatched on the 'brainy-backup'
tag; zero migration for ingestion callers).
- Selectors (ids/collection/connected/vfsPath/predicate/whole) + edge policy
(induced/incident/none) + includeVectors/includeContent/includeSystem; system
entities excluded by visibility. onConflict merge/replace/skip, reembed auto/never,
remapIds. DbHost gains storage (VFS blob bytes).
- Types exported from the package root: BackupData/BackupEntity/BackupRelation/
ExportSelector/ExportOptions/ImportOptions/ImportResult + isBackupData.
Tests: tests/unit/db/db-backup.test.ts (10, green) — round-trip, selectors, edges,
vectors+re-embed, merge, asOf/with composition, validation. Full unit suite green.
Follow-ups: docs (8.0 guide + RELEASES + api/README), completeness adds
(since(prior).export() delta, exportStream/importStream, validate()), includeContent
filesystem test, ids-at-asOf get() refinement, @soulcraft/formats Zod schema mirror.
One mechanism replaces the COW and versioning subsystems: immutable
generation-stamped records behind a Datomic-style database value.
Record layer (src/db/generationStore.ts):
- Monotonic generation counter in _system/generation.json; bumped once per
transact() commit and once per single-operation write (storage hook), so
brain.generation() is always a meaningful watermark.
- Commit protocol: stage before-images + tx.json delta -> fsync -> execute
batch via TransactionManager -> atomic tmp+rename of _system/manifest.json
(the rename IS the commit point) -> append _system/tx-log.jsonl.
- Crash recovery on open rolls uncommitted generations back byte-identically
and forces an index rebuild; refcounted pins gate compactHistory(), which
records a horizon (asOf below it throws GenerationCompactedError).
Db API:
- Db: get/find/search/related pinned at a generation, with() speculative
overlays, since() diffs, persist() hard-link-farm snapshots, timestamp,
generation, release() + FinalizationRegistry backstop.
- Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch
(GenerationConflictError CAS), asOf(generation|Date|path), restore(path,
{confirm}), compactHistory(), generation(), static open() + load().
- get()/metadata find()/related() are fully correct at any reachable pinned
generation; index-accelerated queries at historical generations throw
NotYetSupportedAtHistoricalGenerationError - never silently-wrong results.
- VersionedIndexProvider (generation/isGenerationVisible/pin/release) in
plugin.ts: feature-detected, balanced pin/release in lockstep with Db
lifecycle, post-commit applier + replay-gap model documented.
Storage primitives (BaseStorage + filesystem/memory adapters): raw-object
read/write/list/remove, fsync barrier, noun/verb raw before-image capture,
tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and
append-in-place exceptions), restoreFromDirectory + derived-state reload.
Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation
across 200 mutations, batch atomicity under injected execution failure,
ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety
under pins, with() overlay isolation, generation monotonicity across
reopen, crash consistency through the real recovery path, and versioned
provider pin/release balance. Design record in
docs/ADR-001-generational-mvcc.md.
