The generic reprojection engine (pure TS; the twin of the native
implementation — same frozen contract, one shared conformance intent):
register any ProjectionAdapter; advance(family, {budgetMs}) folds facts
from the adapter's own watermark to the head in installments ≤50ms with
real macrotask yields; foreground door traffic bumps the DoorSignal and
an in-flight advance yields within one installment ('preempted');
advanceAll round-robins families fairly. swap(family, buildAdapter) is
the doors-open migration primitive: the OLD projection keeps serving
while the new one builds beside it, the flip is atomic at parity, and a
concurrent second swap refuses typed. A fact the fold cannot apply
(typed ProjectionApplyError) is QUARANTINED — skipped, ledgered,
narrated per-doubling, exposed for refuse-affected-reads — the service
class law's fourth answer: never a wedged rebuild, never a silent skip.
The engine never writes stamps: each adapter owns its durability and its
stamp-after-data discipline. Upgrade, heal, and rebuild are now the same
machinery behind open doors.
FactLogSource wires any host's fact scan in one line
(factSourceFromHost(brain)); window-contract violations are loud.
Pins: 23 unit (budget resume without refold · preemption within one
installment · round-robin fairness under a skewed backlog · build-beside
visibility mid-swap · atomic flip · single-flight refusal · quarantine
skip/ledger/doubling · non-typed throw aborts · losing adapter
discarded) + 3 integration on a real brain (fold matches ground truth ·
doors answer mid-fold with the preemption path exercised · crash
mid-fold resumes from the stamp, never refolds).
Gates: unit 2054/2054 (157 files) · integration 820 (93 files) ·
conformance 31/31.
The private recovery discipline, applied to its own machinery: pending-
embed markers stop being sidecar files and become first-class log records
riding the write's OWN commit fact — embed.pending lands in the same
atomic append as its after-image (a marker can never be orphaned from its
write, or vice versa; in durable-at-ack mode it shares the write's
covering fsync — zero extra syncs), and the worker's landing commit rides
embed.landed with the inline vector. Crash recovery is now a FOLD of the
log (pending without a matching landed = recovered), skipped wholesale on
brains with no v2 history; the one-time legacy bridge folds existing
sidecar files in, migrates them as one fact, and deletes them —
idempotent under a crash mid-bridge. No code path writes the sidecar
again.
Plus the ENTITY-TRUTH digest law, found by this train's own pins:
canonical vector wrappers denormalize HNSW residue (connections + the
randomly-assigned node level) that the log deliberately does not carry —
the verification oracle digested it and would have reported false
state-differs on ~any nonzero-level node (a ~15% flake in the cutover pin
was the symptom). Both sides of every oracle comparison now normalize to
entity truth (nounEntityTruth); index residue has its own rebuild path
and is not entity state.
Pins: embed-markers-in-log 5/5 (same-generation marker, landed+fold-to-
zero, crash recovery via the log with the sidecar prefix EMPTY on disk,
legacy bridge, VFS hung-embedder ack) · deferred-embedding 5/5 unchanged
(the contract outlived its mechanism) · kill-matrix 11/11 · cutover 5/5
×10 runs (flake dead) · unit 2031/2031.
- Watermark stamping fans out at flush: all three projections stamped
with the committed generation before their flushes persist.
- waitForIndexed(path?, {generation, timeoutMs}) — the one honest read
barrier for write-then-recall consumers; typed timeout error carries
the pending count and names the gauge; getIndexStatus() gains
per-projection gauges. awaitPendingEmbeds() unchanged underneath.
- adoptLogAuthority() self-backfills curable divergences (pre-log
records, witness drift) by identity re-commit before flipping — a
fresh brain flips clean; log-ahead divergences still refuse loudly.
- The verification oracle gains VERB legs (all four divergence classes;
unwired = honest verbsChecked: 0, never a scope claim).
- find({where: {}}) match-all serves (was silent-empty, warm AND cold;
same fix in count/streaming/subgraph seeding); removeMany({where:{}})
refuses typed — a match-all bulk delete must be explicit.
- Aggregation native envelope stamped via noteSourceGeneration before
serializeState; the native-blob restore gates through the same
adoption verdict as caller-side state (the unconditional adopt dies).
- LC8 pinned: a wholesale directory move opens and serves identically
across all three intelligences, with history traveling.
Gates: unit 2031/2031 (156 files) · integration 812 (91 files) ·
conformance 27/27.
Every persisted projection artifact (metadata field indexes + column
segments, HNSW node records, graph adjacency LSM trees) now carries a
stamp asserting 'this state reflects every committed generation ≤ W,
atomically' — written LAST in each owner's flush (stamp-after-data: a
crash between data and stamp = unstamped = rescan, never trust). At load,
each owner computes the three-way verdict: stamped==committed → adopt
(zero work) · behind → catchup (gap reported) · above/unstamped → RESCAN,
loudly. Legacy artifacts re-derive once, then are stamped forever. Shared
law in projectionWatermark.ts (the aggregation verdict machinery,
generalized); vector artifacts carry model dimensions. Verdicts are
computed and exposed (watermark()/watermarkVerdict()/watermarkGap());
rebuild triggers unchanged — acting on 'catchup' is the fold train.
Pins: 22 unit (7 metadata · 8 hnsw · 7 graph, incl. spy-order
stamp-after-data) + the end-to-end reopen-adopts pin.
The cutover: new tail segments write format v2 (per-record [type, version,
cipherFlag, keyId] envelope; noun/verb after-images carry dense ints
MINTED AT APPEND from the id mapper — a rebuilt mapper reproduces
assignments exactly; log.genesis opens every new log with the id-space
width + a minted brain id; sync() seals to the header-declared sector
boundary with reader-invisible pad frames). Existing v1 segments are
never rewritten — per-segment decoder dispatch reads both formats and v2
facts map to the exact CommitFact shape all consumers already read.
Cutover on a live v1 log: an empty v1 tail re-heads in place; a non-empty
one is sealed by rotation, byte-identical. Records reserve the encryption
fields (cipherFlag 0 / keyId nil are the only legal values; anything else
refuses typed naming the needed newer reader) — crypto-ready with no
future bump on the compat surface. Empty-records facts are legal (an
all-deduped batch is a real generation — v1 semantics preserved; the
refusal there tore a column-store flush mid-commit in the full suite, the
consistency guard caught it loudly, and the root is fixed).
Golden byte vectors pinned for the second (native) reader implementation.
Pins: cutover 5/5 · codec 54 · kill-matrix stays 11/11.
The time-travel recall row moves from envelope-note to contracted: vector
search at a pinned past generation serves the vectors AS THEY STOOD —
a later re-embed never leaks into an earlier pin (byte-exact), tombstones
mask, the deferred-embed pin serves the stub on the vector leg until the
landing generation (text/metadata legs unaffected — triple intelligence by
design), and beyond-head pins refuse typed. Brainy-alone leg = the
documented ephemeral at-generation materialization; the at-scale leg rides
the accelerated provider's as-of index. Registry row added (shared ID
pending the master table).
Two release-blocking findings from the durability kill-matrix, both fixed
in the owning layer:
1. LOG-AUTHORITY REPLAY AT OPEN: durable-at-ack fsynced the fact before
the ack, but open() truncated every fact above the manifest — after a
power loss that takes the un-fsynced tmp+rename canonical bytes, the
acked write's ONLY durable copy was discarded. Now: under 'log'
authority, open() REPLAYS intact facts above the manifest into
canonical (FactLog.peekFactsAbove — CRC-gated, order-sorted) and
advances the manifest to cover them; tree-authority brains keep the
truncate contract they were promised. Pinned end to end: the power-loss
row constructs the exact disk state (fsynced log, vanished canonical
rename) and the acked write lives.
2. NO SILENT COMMIT: commitSingleOp buffered the generation BEFORE the
fact append; an append failure (ENOSPC) rejected the caller but the
next flush durably committed the generation with NO fact — a permanent
silent log gap. Now the failure path un-buffers and returns the counter
reservation: nothing commits, the log stays gap-free, and the canonical
execute-residue orphan is the documented crash-equivalent.
Plus: the kill-matrix itself (11 rows — every commit-path fault point ×
reopen-as-crash recovery contract, at-ack variants, disk-full row; five
new zero-cost faultPoint sites), the log-authority pin suite (oracle
green/red/state-differs, flip refusal, switch survives reopen, 9/9), and
the group-commit covering pins (5/5).
Gates: unit 2002/2002 (152 files) · integration 785 · conformance 27/27.
A3 of the service-class pair (BRAINY-PROD-LATENCY-TRIAD): a VFS file write
ran the embedder synchronously while the caller waited — 5.6s p50 / 21.4s
p95 per small file on a production deployment, the dominant stage of every
capture write.
- add()/update() gain deferEmbedding: the write acks at durability (data +
metadata persisted, a DURABLE pending marker under
_system/pending_embeds/<id> written BEFORE the commit — orphan-safe
direction); the single-flight background worker embeds the CURRENT data
and swaps the vector in ATOMICALLY (ReplaceInVectorIndex — the row is
never absent from search; a deferred UPDATE keeps serving the OLD vector,
stale-beats-absent per the flicker law). Typed refusals: defer+vector,
defer-without-data.
- CRASH-SAFE: markers are recovered at open by a BOUNDED prefix listing
(never a store walk) and the worker resumes in the background — a crash
can delay a vector, never lose one. A wedged embedder trips a LOUD 60s
hang guard and the worker moves on (marker retained for retry).
- The honest gauges: getIndexStatus().pendingEmbeds + pendingEmbedCount();
awaitPendingEmbeds() is the eventual-vector-index BARRIER for callers
and tests that need searchability before proceeding.
- VFS adopts it everywhere a write path could wait on the embedder:
writeFile (both branches) and directory creation. Pinned in the
strongest form: writeFile resolves while the embedder HANGS FOREVER.
Pins: deferred-embedding 5/5 (ack law · stale-beats-absent · crash
recovery across sessions · VFS hung-embedder ack · typed refusals).
Gates: unit 1928/1928 · integration 765 · conformance 27/27.
SELF-ENGINE-LIFECYCLE-SPRINT + BRAINY-PROD-LATENCY-TRIAD, the four asks:
(a) brain.flush() persists aggregation state stamped at the committed
generation. The stamp used to advance only at close(), so a long-lived
writer that flushes but never closes — the primary production shape —
left every write window behind the stamp, and ANY unclean exit forced a
whole-store backfill walk (per-entity work, measured >60s and
door-starving on a 9k-row production brain) on the first stats call.
(b) BEHIND-stamp adoption becomes adopt + INCREMENTAL CATCH-UP: the exact
missing window (stamp, committed] resolves its affected-id set from the
fact log and reconciles each entity with time-travel before/after reads
(asOf at both window bounds) through the same delta algebra the live
hooks use — cost bounded by writes since the last flush, never store
size, and exact under interleaving because reconciliation targets the
FIXED window end while later writes chain through hooks. Oversized
windows (>5000 affected) and unreadable windows demote to the announced
rescan — never a silent partial serve.
(c) The native provider's parallel rebuildAggregate — on the contract since
8.x but never invoked anywhere — is now the backfill walk's preferred
door: one call per aggregate with source-matched entities, replacing
the per-entity FFI stream.
(d) A delete whose before-image is unavailable can no longer SKIP the
aggregation hook silently (counts drifted upward forever): both delete
paths (remove() and transact) flag an exact rescan, loudly.
Pins: integration (flush stamp; unclean-exit reopen → exact counts through
an add + group-move + delete window with the walk spy proving ZERO
whole-store walks) + unit (provider rebuild invoked once with filtered
entities; flagAllForRescan; reconcile delta algebra). Gates: unit 1913/1913
· integration 760 · conformance 27/27.
The write side of the law, ruled 2026-08-03: data is either in main space
where developers can use anything, or it is in system.*.
- The reserved-name write door DIES: add/update/relate/updateRelation
metadata bags accept EVERY name (confidence, type, id, data, level,
content, ...) as ordinary user fields — indexed, filterable, sortable,
aggregatable, identical to any other field. The remap/enforce/warn
machinery, the reservedFieldPolicy config (now a typed init refusal),
and the compile-time metadata key bans are all removed. The one write
refusal left: keys spelled 'system.*' (namespace forgery), now enforced
on all four write doors.
- STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag
nested verbatim under 'metadata', sealed by a format stamp — by-name
storage discrimination is unsound once colliders are admitted. Legacy
flat records stay readable forever through the shape-aware splitters
(sound for them: the old door refused colliders). Time travel rides the
same split (generation store snapshots whole records).
- Name-based index exclusions DIE: user frame indexes every name; the
excludeFields/indexedFields knobs and their silent-[] holes are gone;
bulk-payload protection is value-shape only, uniform across names.
- Consumer-sweep findings fixed in the same wave: per-type counts read
the frozen 'system.type' column (addToIndex sort, affinity tracking,
cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for
pre-rebuild reads); resolveHiddenIds addresses 'system.visibility'
(bare 'visibility' was a silent no-op under the law — VFS/system
entities leaked into default reads).
- Fidelity fallout fixed in the owning layers: readEntityFieldAddress
reads the bag first (colliders were absent-shadowed by its own guard)
and never serves system addresses from the bag; blob history refs read
the bag shape-aware; migration transforms now receive ONE normalized
view (engine fields + nested bag) regardless of stored era, and stray
flat-habit keys refuse with the fix in the message.
- THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as
gates-green): all ten collider names + plumbing names written as user
fields, verified verbatim + queryable across live reads, flush+reopen,
a forced epoch rebuild, and asOf time travel; relation mirror; forgery
refusals; legacy flat-record compat. 8/8 green.
Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance
27/27 (exit 0) · consumer test sweep migrated (10 files).
Also completes the v8.10.2 write-granularity law for the transact() plan
path: a metadata-only batch update never rewrites the vector-bearing noun
record (planUpdate staged the unconditional save the update() fix removed).
Seven pins in tests/integration/level-field-shadow.test.ts including the
reporting consumer's exact repro rows; orderBy JSDoc documents the ordering
contract and the announced field-addressing law.
Also: idle PathResolver stats tick no longer logs NaN% every minute (logs
only on new traffic, via prodLog); graph-lsm-* key family recognized as
system resources (kills the per-boot unknown-key warning on provider-backed
brains). Four regression pins in tests/integration/update-write-granularity.
The packed tier goes live inside GenerationStore:
- Two-tier reads: getDelta / readBeforeImage / readGenerationRecords
fall through live-tier → sealed segments (live-tier-wins: a crash
mid-fold leaves a duplicate representation, never a gap). Cold-open
seeds committedRanges from the segment manifest via interval merge —
packed generations resolve without their directories existing.
- repackHistory({timeBudgetMs, batchGenerations}): folds cold
generations (older than the newest 1024) oldest-first into sealed
segments, deleting per-generation directories only after segment +
manifest are durable. Public API + automatic time-bounded pass at
close() (before compaction, so reclaim can drop whole segments);
re-representation only — the sole history transform under the
archival profile. Early stop = consistent prefix, next pass resumes.
- compact(): packed generations reclaim logically in the loop and
physically at whole-segment boundaries via dropSegmentsBelow (the
frozen partial-segments-wait rule).
- generationDigest(g) (D8): deterministic content digest through g —
sealed-segment checksum chain + live-tier delta hashes; O(segments +
live window); RangeError out of range, GenerationCompactedError
below the horizon (a gate can never silently pin reclaimed history).
Four end-to-end pins: asOf answers byte-identical across fold + cold
reopen with folded dirs physically gone; repack+reclaim composition;
digest reopen-stability/divergence/loud-horizon; budget no-op+resume.
clearInterval() stops future heartbeat ticks but not one already in
flight: a straggler tick past its ownership guards could land its
atomic lock rewrite AFTER releaseWriterLock()'s unlink, re-creating
the lock file as a phantom that blocks the next writer until the
stale TTL expires (~60s) — the pool-eviction reopen case. Found as an
ENOENT heartbeat warning during benchmark teardown; the quiet variant
is the harmful one.
releaseWriterLock() now awaits the in-flight tick (tracked per tick,
self-clearing) before reading/unlinking, so a straggler's write always
lands BEFORE the unlink and gets removed with everything else. The
heartbeat's ENOENT is also now benign-by-contract (lock or directory
removed under us — the next acquire recreates it); other errors stay
loud.
Pin: straggler-past-guards simulation — lock file absent after close,
directory immediately claimable (fails on the undrained code).
flush() is durability work: it must cost what the current window's
deltas cost, never what the history backlog costs. Under adaptive
retention the byte budget derives from free memory, so bulk-load
pressure shrank the budget exactly at peak write volume and flush paid
actual reclaim inline — a production deployment measured single writes
blocked 25-191s behind reclaim-on-flush.
- flush() no longer calls autoCompactHistory(); close() is THE
auto-compaction site (already ran there; now alone).
- Every auto pass is time-bounded (CLOSE_COMPACTION_BUDGET_MS = 5s):
reclamation is oldest-first, so an early stop is a consistent prefix
and the next pass resumes. Explicit compactHistory() gains an
optional timeBudgetMs for caller-chosen maintenance windows.
- Documented trade stated where operators read: a long-lived writer
that never closes accumulates history until its next explicit
compactHistory() — predictable writes, explicit maintenance.
Pins: flush-never-reclaims + close-reclaims-durably (db-mvcc), bounded
pass stops-then-resumes as a consistent prefix (generationStore unit).
Four fixes from a consumer conformance report, one root disease — two
field-resolution regimes where there must be one:
- The delete/update aggregation hooks fed the engine a partial entity
view (type/service/data/metadata only), so a reserved-field groupBy
(subtype, visibility, ...) resolved to a nonexistent group on the way
down: counts drifted upward forever after deletes, and updates moving
an entity between reserved-field groups double-counted. The hooks now
pass the full-fidelity view via entityForAggFromRawRecord (every
reserved field top-level, mirroring the add path); the update sites
pass the full get() view instead of a hand-rolled subset.
- Aggregation source.where resolved fields only against the custom
metadata bag, so where on a reserved field silently matched nothing.
The matcher now resolves each filtered field through
resolveEntityField — the same single source of truth groupBy uses.
- removeMany() with no usable selector (bare array passed positionally,
empty params, ids: []) resolved successfully having deleted nothing.
All three now throw; the two legacy tests that pinned the silent
no-op as 'graceful' now pin the refusal.
- find() where keys accept both spellings: a metadata.-prefixed key
falls back to its flattened spelling when the prefixed one is not
indexed (metadata is flattened at index time). A literal nested custom
key named metadata still wins when indexed as spelled.
Five regression pins in aggregate-reserved-fields.test.ts (4 of 5 vary
red on the unfixed code).
The post-import background deduplication pass (a merge-DELETE writer,
debounced ~5 minutes after import) had four lifecycle defects:
- enableDeduplication: false did not gate the background schedule — an
import that explicitly opted out could still have entities auto-removed
minutes later. The flag now gates both the inline and background passes.
- Each import() constructed its own coordinator-owned deduplicator, so the
debounce never spanned imports (N imports = N delete timers). The brain
now owns a single lazy instance (getBackgroundDeduplicator).
- close() never cancelled pending dedup; a delete pass could fire against
a closed brain. close() now cancels it first.
- The 5-minute timer held the process open (exit-hang class); now unref'd.
Four regression tests pin the contract (background-dedup-lifecycle);
import guides document that false disables both passes.
- acquireWriterLock now CLAIMS with an atomic create-exclusive write (O_EXCL)
inside a bounded retry loop: two processes racing an absent lock can never
both succeed (the old read-then-tmp-rename flow let the loser keep running
unlocked, silently). An EEXIST loser re-evaluates and either throws loudly
with the winner's details or performs a verified stale-takeover (re-read
before unlink so a lock that changed hands mid-deliberation is never
clobbered). Exhausted contention fails loudly instead of degrading into a
lockless open.
- BRAINY_WRITER_LOCKED passes through init() unwrapped: the error documents a
machine-readable contract (err.code + err.lockInfo with the holder's
pid/host/heartbeat), but init's blanket error wrapping stripped both,
leaving consumers a message to regex against.
- Two contract tests added: stale-foreign takeover installs OUR lock via the
atomic claim; the conflict error carries code + lockInfo at the public
init() surface.
- New public API: walks every canonical relationship record, queries the same
read path applications use (related() with all visibility tiers included),
and classifies every discrepancy into its failure family: missingFromReads
(records the read path omits — stale adjacency), danglingEndpoints
(endpoint entity gone — the partial-delete scar class), readOnlyVerbIds
(read-path edges with no canonical record — ghosts). Design-hidden
internal/system edges are counted separately so intentional hiding is never
misclassified as loss. Counts exact; example lists capped with an explicit
truncatedExamples flag; loud narration on incoherence; mutates nothing.
- Classification core lives in src/graph/graphAudit.ts behind injected seams
(canonical walks + the end-to-end read) so the discrepancy logic is testable
in isolation; brainy wires the seams to storage walks and related().
- getNounIdsWithPagination now refuses an undecodable resume cursor loudly —
the third and final pagination walk brought under the 8.5.2 cursor contract.
- Types exported: GraphAuditReport, GraphAuditDiscrepancy. Docs: the
inspection guide gains the audit -> repair -> audit verification ritual.
- Aggregation backfill walks build into a STAGING map and swap in atomically
on completion. A mid-walk failure drops the staging map — the previous live
state keeps serving, the aggregate stays flagged pending, and the storage
error surfaces to the failing query. Previously the walk wiped live state
before a scan that could throw, never cleared the pending flag on failure,
and re-ran a full walk on every subsequent query: a silent wipe/walk/throw
loop at the caller's retry rate.
- Failed walks are latched: retries within a 30s cooldown rethrow the recorded
error instantly instead of re-walking, so a tight caller-side retry loop
costs one loud error per query, never a full store walk per query.
- Persisted aggregation state is stamped with the store's committed generation
at flush; reopen adoption requires stamp equality. Stale state (unclean
shutdown) or over-counting state (a log truncation on a copied store pulled
the watermark back) triggers exactly one loud rescan, never a silent adopt.
- The backfill/adoption path narrates: adoption decisions, walk start/finish
with counts and duration, and failures all log by default.
- getNouns/getVerbs refuse a supplied-but-undecodable pagination cursor with a
loud error instead of silently restarting the walk at offset 0 (which
re-served page 1 forever to any while(hasMore) caller).
- The graph cold-load verb walk aborts loudly on a missing or non-advancing
cursor with hasMore=true.
- A versioned index provider whose generation is AHEAD of the committed
watermark (torn copy / crash-recovery truncation) is now named loudly at
open, alongside the existing behind-direction message.
- AggregationIndex: defineAggregate before init no longer forces a backfill.
init reconciles instead of clobbering: the app definition wins, persisted
state is adopted on hash match, a write landing pre-adoption forces an exact
rescan, and a successful state load clears the backfill flag. New ready()
settles every adoption decision before query paths consult backfill state.
- brainy: backfills are single-flight and batched. Concurrent queries share one
store walk and every pending aggregate fills from that same walk; the old
behavior let each concurrent query wipe the others' partial state and start
its own full walk, so a store under steady aggregate traffic never converged.
queryAggregate also waits for persisted definitions before deciding an
aggregate does not exist.
- paramValidation: recordQuery is telemetry-only. The duration-based cap
ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored
at 1000 - below the documented 10000 auto floor, reported under a stale
basis label) is removed; the cap comes from its construction-time basis or
explicit overrides alone.
- storage: __aggregation_* and singleton system keys (brainy:entityIdMapper)
are recognized before the unknown-key warning fires; routing is unchanged.
- docs: find-limits cap-immutability note, aggregation reopen/adoption
semantics, RELEASES.md 8.5.1 entry.
- storage.committedGeneration(): the committed watermark exposed on the
fact-scan capability, so a provider compares its stamp's
sourceGeneration against the store's truth without parsing the
private manifest format (the capability source now carries both the
live fact log and the committed closure).
- Pinned contract tests: fsync-before-ack (transact passes today; the
single-op path is pinned via it.fails as the documented target —
group commit must become LATENCY batching, ack waiting on the shared
fsync, never durability skipping; the pin flips red when that lands
so there is no cliff to discover) and scan-stability-under-rotation
(a scan snapshot yields exactly its facts — no gaps, duplicates, or
bleed-in — while segments rotate beneath it; the reclaim-during-scan
variant lands with fact compaction, which does not exist yet).
- FactLog accepts a rotateBytes option so tests exercise real rotation.
Three additive surfaces for native index providers, which hold only
`storage` and must never construct their own fact-log reader (the log's
open path is writer-side — it reconciles by truncating/rewriting):
- Fact-scan capability on the storage adapter (the getBinaryBlobPath
pattern): the host brain wires a closure over its LIVE fact log at
init; providers call storage.scanFacts()/factLogHeadGeneration()/
factSegmentPaths() and fall back to the enumeration walk on null.
Restore/reopen swaps the underlying log transparently.
- The family-stamp trio (readFamilyStamp/writeFamilyStamp/
verifyFamilyStamp + types) exported from @soulcraft/brainy/internals,
so 'one verifier' is literally one function shared with the native
side, never two synchronized copies.
- Rollup invariants widened to number|string: content fingerprints
(e.g. a per-tree SHA-256) are valid invariant values; strict equality
either way, and a type mismatch reads as incoherence, never a pass.
The canonical entity tree now carries a FAMILY STAMP
(_system/family-stamps/entity-tree.json, JSON — forensics stay
terminal-readable): which committed generation the tree reflects
(sourceGeneration) plus the rollup invariants (entity/relationship
counts) that verify a millions-of-files projection whole where per-file
checks cannot scale. Written at flush/close boundaries; open-time
coherence is a COMPARISON, not a walk:
- coherent / absent (legacy store) → silent
- behind → benign for the tree (it is written BY the commit; only the
stamp is stale after a crash between commit and flush) — refreshes at
the next flush
- incoherent (counts diverged at equal generation, or a stamp AHEAD of
the log head) → loud, names the failing invariant; repairIndex()'s
unconditional recount heals and RE-STAMPS so repair leaves a coherent
stamp behind
- a fault reading the stamp is UNVERIFIABLE — never conflated with
absence
One verifier (verifyFamilyStamp, exported) reads both member modes:
enumerated (exact byte size per member, bounded families) and rollup
(invariants, unbounded families). New exports: readFamilyStamp,
verifyFamilyStamp, ENTITY_TREE_STAMP_PATH, FamilyStamp, StampMembers,
StampVerdict.
Every committed generation now also appends a FACT — an after-image
commit record (what each touched entity/relationship became, or a
body-less tombstone for a removal) — to an append-only, crc32c-framed
segment log under _generations/facts/. The before-image history and the
canonical tree remain authoritative; the fact log gives consumers ONE
sequential, self-verifying stream (index heals, incremental replays)
in place of a per-entity directory walk.
- Wire format: positional msgpack facts [generation, timestamp, ops,
meta, blobHashes]; op = [kind u8, id bin16, record | nil tombstone];
32-byte segment header (magic, formatVersion, firstGeneration,
zeroed+verified reserved); length+crc32c frame per fact; zero-padded
segment names so lexicographic order == generation order; JSON
manifest with an atomic rename flip, manifest-first rotation.
- Commit protocol: facts append+fsync BEFORE the commit point inside
the existing durability window, so a crash can only leave the log
AHEAD of committed truth — open() truncates back (torn tails detected
by CRC). Absent generation = never committed; a scan can never see an
uncommitted fact. transact() facts are durable-on-return; single-op
facts ride the group-commit flush exactly like buffered history. A
fact-append failure fails the write, loudly — a silent gap would be a
lie a later replay discovers.
- New public surface: brain.scanFacts() (sequential batches with heal
telemetry: head/segments/approx up front, per-batch generation range
+ bytes + segment id, loud abort on gaps, summary cross-check) and
brain.factSegmentPaths() (immutable sealed segments for zero-copy
consumers; the mutable tail excluded). Exported types CommitFact,
FactOp, FactScanBatch, FactScanHandle.
- Storage: optional binary raw-byte primitives (appendRawBytes,
readRawBytes, writeRawBytes, rawByteSize) on StorageAdapter —
feature-detected; filesystem + memory adapters implement them; an
adapter without them hosts no fact log. Fact segments are byte-copied
(never hard-linked) into snapshots. The _generations/facts/ namespace
is registered as a protected family (rebuildable: false): no sweeper
or GC may delete under it.
- New crc32c (Castagnoli) utility with RFC known-answer tests.
Ports the fresh-brain probe that closed the type+subtype lens-drop
investigation into the permanent suite: a 7-pair corpus seeded through the
real write API (never restored bytes), every lens checked id-for-id against
an unfiltered canonical scan, warm AND after a cold reopen, plus the
type+subtype update-flip leg. Guards the under-inclusion class the egress
integrity guard structurally cannot catch.
A cross-directory rename added the new parent's Contains edge but skipped
removing the old one (a 'not critical' shortcut), leaving the moved entity
a child of BOTH directories: readdir(oldDir) kept listing it, re-creating
the old path showed the name twice, and tree-walking consumers saw the
file in two places. Reported live by a production deployment (37 stale
containment ghosts; blocks tree-sync integrations).
- rename() now removes the old parent's vfs containment edge(s) by edge
id resolved from the graph's own adjacency (the removal law: removal
never requires reading the removed thing), and a move to the root gets
its containment edge (previously skipped -> orphaned from readdir('/')).
- New vfs.repairContainment(): reconciles every VFS entity's containment
edges against canonical metadata.path — removes stale/duplicate vfs
edges, restores missing ones, never touches user knowledge edges. Wired
into repairIndex() so the one operator ritual heals existing ghosts.
- Regression: tests/integration/vfs-rename-containment.test.ts (7).
Persisted entity totals inflated permanently: a delete's count decrement
was sourced from re-reading the record being removed, so a null read
(replace race, or a ghost left by a pre-8.3.1 partial delete) silently
skipped the decrement while the paired add had counted — and
Math.max(persistedTotal, scanned) meant no disk cleanup could ever
lower the reported total. Reported by a production deployment with a
write→delete→re-create repro minting drift within hours.
- The caller's pre-delete read now rides through the entire delete path
(remove()/removeMany()/plan → DeleteNoun/DeleteVerb operations →
deleteNoun/deleteVerb → deleteNounMetadata/deleteVerbMetadata): a
null internal read falls back to the known prior record instead of
skipping the decrement. StorageAdapter.deleteNoun/deleteVerb gain an
optional priorMetadata parameter (additive).
- rebuildTypeCounts() rebuilt only the type-statistics arrays and
computed the total just to LOG it — the persisted scalar
(counts.json) survived every rebuild. One canonical walk now rebuilds
every counter rollup (scalar totals + per-type maps + type
statistics) and persists them; repairIndex() runs the recount
unconditionally, since counters can be inflated over clean shelves.
Two production-reported spine fixes:
- Canonical delete is a FULL removal: remove()/removeMany() deleted the
metadata leg but never the canonical vectors.json leg or the <id>/
directory, leaving ghost rows that inflated enumerated counts forever,
read as damage scars, and caused duplicate readdir entries for
re-created VFS paths (the stale-unpost path). The delete operation now
routes through storage.deleteNoun — both legs + the entity container —
with a full two-leg before-image rollback; deleteVerb symmetric; the
blind 'no metadata file' catch that masked real faults is gone. The
generation log still holds the delete's before-image, so asOf() history
is unchanged. repairIndex() gains a conservative, loud orphan-prune
sweep (containers with no metadata content leg) + count recompute for
stores damaged by earlier versions.
- Family-scoped migration gate: every read blocked on the whole-brain
migration lock even when the migrating index was irrelevant. The gate
now scopes to the index families a read actually consults — canonical
reads never wait; find() waits only on its query shape's families;
graph traversals wait only on the graph family. Writes keep the
conservative whole-brain wait. A read that needs the migrating family
still blocks (bounded) with the retryable MigrationInProgressError.
Write/index-spine hardening, first batch of Pass 1. Each fix restores an
invariant the surrounding code already intended; every one has a
fail-before/pass-after test.
- Pattern C, finding 5 (baseStorage): delete now decrements the user-facing
scalar total symmetrically — deleteNounMetadata was decrementing only the
per-type bucket, deleteVerbMetadata neither the bucket nor the scalar, so
getNounCount()/getVerbCount() inflated permanently (the stale scalar wins
pagination via Math.max and is persisted). Invariant now holds:
scalar total === Σ per-type across add/update/delete and reopen.
- Pattern A, finding 3 (graph/lsm/LSMTree): a partial SSTable-load failure no
longer publishes the manifest's full relationship count as healthy. Any
per-SSTable load failure throws after the batch, which resets to honest-empty
and lets the existing size()===0 self-heal rebuild run — size()/isHealthy()
can no longer lie about a partial load.
- Pattern B, finding 6 (hnsw/hnswIndex): deferred flush() no longer clears
dirty nodes whose connections failed to persist — failed nodes stay in the
retry set, and flush() throws HnswFlushError instead of returning a lying
node count. The immediate-mode first-noun saveHNSWSystem is un-swallowed, so
addItem() rejects rather than returning an id for a rootless index.
- Pattern B, finding 11 (part — storage reads): new shared isAbsentError()
helper (utils/errorClassification, ENOENT-only absence) applied to
loadBinaryBlob and readObjectFromPath — a real IO fault (EIO/EACCES/EMFILE)
now propagates loudly instead of masquerading as "absent", which had driven
needless rebuilds / empty reads (loadBinaryBlob feeds the native provider).
Regression: 78 green across the 3 new suites + db-mvcc, generationStore,
temporal-vfs, rollback-trapdoor, restore-nondestructive. Full gate runs before
the Pass-1 release (David-gated). Remaining Pass 1: finding 11 getNoun/getVerb
legs, finding 4 (ColumnStore), finding 8 (pending-flush), finding 10 (degraded),
finding 7 (clear). Pattern A guards (1,2,9) as a follow-up release.
A returned write must be immediately readable by id, metadata filter, vector
search, and graph traversal — no await, delay, or retry between the write
returning and the read. This holds by construction because every projection
(canonical + HNSW + metadata index + graph index) commits inside the write's
transaction; the generation counter is the {seq} a caller can pin. The test
locks that guarantee so index maintenance can never quietly move off the commit
path (which would turn a returned write into a not-yet-queryable one).
First brainy chapter of the write/index-spine hardening program.
When a transaction failed and its rollback ALSO failed to undo a canonical
write (retries exhausted), the old path logged, continued, threw
TransactionRollbackError, and set state='rolled_back' — while the record it
could not undo stayed durable on disk. The caller got an error implying the
write was undone; a read-back showed the record. A failed rollback had no
truthful response (the post-commit response lie).
Give a failed rollback a two-branch honest contract, decided by observation:
both commit paths already hold byte-identical before-images, so after a failed
rollback the store compares current canonical state to them and classifies each
touched record as reconciled, an additive orphan (present when it should be
gone), or a restorative loss (gone/wrong when it should have been restored).
- Adopt-forward: a single-op write whose only damage is a durably-present
orphan — the record the caller asked for — is committed forward (its
generation buffered) and returns success with a loud warning that the derived
index may be incomplete for that id until the next rebuild/repairIndex(). No
error, no double-write; the record is durable and get-able immediately.
- Fail loud + quarantine: a multi-op batch, or ANY restorative loss, throws the
new StoreInconsistentError naming every unreconciled record and its
disposition, and puts the brain into write-quarantine (reads keep working,
writes refused via assertWritable) until repairIndex() reconciles the derived
indexes against canonical and lifts it. The counter is not advanced, and
Transaction.rollback now reports state 'inconsistent' instead of the
'rolled_back' lie when any undo failed.
New: StoreInconsistentError + UnreconciledRecord exported from the root;
repairIndex() forces a rebuild and clears the quarantine. Regression
tests/integration/rollback-trapdoor.test.ts injects index-add-throws +
canonical-delete-undo-fails and pins adopt-forward (durable, get-able, not
quarantined), fail-loud (StoreInconsistentError + quarantine + reads work +
repairIndex lifts it), and the error's record naming.
restore() removed the entire live brain directory and then fs.cp'd the
snapshot in, so any copy failure left the store destroyed with only a partial
copy. The sharpest edge: fs.cp materialized the holes of sparse mmap blob
files, so a snapshot that fits on disk could balloon and ENOSPC mid-copy — the
recovery tool destroying the brain it was asked to recover.
Rewrite restoreFromDirectory as stage → verify → atomic swap:
- Copy the snapshot into a _restore_staging area BEFORE touching live data,
sparse-aware (copyTreeSparse/copyFileSparse skip all-zero 4 MiB chunks, so a
mostly-hole store restores at its true allocated size, not its apparent size).
- On any copy failure (ENOSPC included) remove only the half-written staging
area and throw — the live store is left exactly as it was.
- Only after the copy succeeds, fsync a completion marker naming the staged
entries, then swapStagedRestoreIn(): an idempotent per-entry
rm-old + rename-staged-in (same-filesystem renames that cannot ENOSPC),
removing stale live entries the snapshot lacks.
- completeInterruptedRestore(), wired into init() right after the root dir is
ensured, finishes a crash mid-swap FORWARD (resume a committed swap) or
discards an uncommitted staging area (live still authoritative).
_restore_staging is excluded from snapshots. persist() (hard-link snapshot)
was already safe and is unchanged; no public API change.
Regression (tests/integration/restore-nondestructive.test.ts): a forced copy
failure leaves live data fully intact and cleans staging; a normal restore
round-trips to the snapshot; an interrupted-but-committed restore completes on
reopen; an uncommitted staging area is discarded on open; the sparse copy is
byte-identical with allocated blocks far below apparent size.
A committed transact() reported success while its canonical entity writes were
still only in the OS page cache (tmp+rename, not fsync'd), even though the
generation counter and manifest were fsync'd. A hard kill in that window could
leave the durable counter ahead of the persisted entity bytes — phantom
progress for any generation-based consumer resuming from the counter. Reported
from a downstream migration's crash-lifecycle forensics.
Add an optional transaction durability barrier to GenerationStorage
(beginWriteBarrier / flushWriteBarrier). FileSystemStorage implements it:
writeObjectToPath records each successful canonical write and
deleteObjectFromPath records each delete's parent dir, between begin and flush;
flush fsyncs every recorded write (contents + rename dir entries, via
syncRawObjects) and the parent dir of every delete. commitTransaction opens the
barrier before running the planned operations and flushes it after, BEFORE
persisting the counter and manifest — so the batch's entire canonical footprint
is durable before the generation stamp advances. The barrier is optional: the
generation store calls it through optional chaining, so in-memory and
durable-per-call (cloud object-PUT) adapters treat it as a no-op.
The single-op group-commit path is unchanged (deferred durability is the
Model-B design that avoids a 3-5x per-write fsync regression), but its
durability contract is now documented explicitly on commitSingleOp: transact =
durable on return; single-op = durable at the next flush()/close(), with the
counter buffered alongside the data so a crash loses both together (never a
torn counter-ahead-of-state store).
Regression (tests/integration/transact-durability-barrier.test.ts): the entity
writes fsync in an earlier syncRawObjects batch than the manifest for single-op
and multi-op (add+relate) transactions; a precommit-rejected batch opens no
barrier and advances nothing; MemoryStorage exposes no barrier (optional-chain
no-op).
transact() promises atomic forward references — add an entity and relate to
it in one batch — but the planner resolved relationship endpoint ints at
PLAN time, before the batch's add operations had applied. Asking the id
mapper about an entity that does not exist yet made the (correctly strict)
native mapper throw in EntityIdMapper.getOrAssign, so
transact([{op:'add', id:X}, {op:'relate', to:X}]) failed on native
deployments; the permissive JS mapper masked the bug and silently leaked an
id assignment whenever a batch was later rejected by a commit precondition
(normal control flow since the conditional-commit CAS landed).
Make endpoint resolution lazy: AddToGraphIndexOperation and
RemoveFromGraphIndexOperation take VerbEndpointInts — an eager
{sourceInt, targetInt} or a thunk evaluated when the operation EXECUTES,
mirroring the constructor's already-lazy generationFn. The four
transact-planner sites (relate, its bidirectional reverse edge, remove's
relationship cascade, unrelate) pass thunks, so resolution happens inside
the commit after same-batch adds have applied; the seven single-operation
sites keep eager objects (their endpoints provably pre-exist — relate
validates both, unrelate/updateRelation/remove read the existing verb).
The remove operation's rollback captures the ints resolved at execute so
its re-add uses the same mappings.
Byproduct fix: a rejected batch no longer pollutes the id mapper — the
regression suite pins this (mapper has no assignment for the phantom
entity after a precondition-rejected forward-ref batch), alongside the
exact reported shape, both-endpoints-in-batch, bidirectional,
add+relate+remove in one batch, and the split-transact control
(tests/integration/transact-forward-ref-graph.test.ts).
The temporal model had a hole exactly where files were concerned: every
entity write is an immutable generation with before-images, but VFS content
BYTES lived under an eager refCount GC left over from the pre-8.0 design —
unlink could physically destroy bytes that in-window history still
referenced, and overwrite never released the old hash at all (an unbounded
silent leak whose accidental byproduct was the only thing "preserving"
history). Reading the past could therefore return a stale field, a dangling
hash, or nothing, depending on luck.
Fix: blob reclamation becomes a HISTORY decision instead of a LIVENESS
decision. Each blob's metadata now carries historyRefCount alongside the
live refCount:
- The commit seam counts one history reference per persisted before-image
record carrying a content hash (commitTransaction staging and the
group-commit flush), recorded BEFORE the record-set persists and carried
in the generation delta (blobHashes — always present on new deltas, so
compaction only falls back to reading records for pre-contract
generations). An aborted transaction compensates best-effort.
- unlink/rmdir/overwrite drop ONLY the live reference (BlobStorage.delete →
release; overwrite finally releases the superseded hash — cancelling the
dedup increment on same-content rewrites and closing the leak), and only
AFTER the canonical mutation commits, so a failed delete can never leave a
live file whose bytes compaction might reclaim.
- History compaction is the ONE reclamation point: after deleting a
generation's record-set it releases that set's references and physically
reclaims any hash at zero live AND zero history references. Pins are
exempt automatically. Crash ordering is over-count-only in every path
(record before persist, release after delete), so a crash can leak until
the scrub recounts but can never reclaim bytes a retained generation
needs. scrubBlobHistoryRefCounts() restores exactness; existing stores get
a one-time marker-gated backfill on open, failing into leak-safe mode
(reclamation disabled) rather than guessing.
On top of the protected history, the temporal API the generational model
always implied:
- vfs.readFile(path, { asOf }) — the exact bytes as of a generation or Date,
materialized from the history (pinned view released so compaction is
never blocked by a read).
- vfs.history(path) — FileVersion[] ascending ({ generation, timestamp,
hash, size, mimeType? }), the newest entry being the live state.
- Overwrites now refresh the file entity's data/embedding text — semantic
search and the data field previously served the FIRST version's text
forever (the stale-field defect a consumer's incident recovery depended
on by luck).
Integration suite (temporal-vfs.test.ts): per-version exact reads +
history listing, leak-fix + history protection on overwrite, rm keeps bytes
readable, compaction reclaims past-window bytes and preserves in-window
(including the cross-file dedup case where an old file's history and a
newer file's removal share one hash), data freshness, and scrub exactness.
Subscribe once and receive one post-commit event per affected record for
EVERY canonical write, regardless of origin: direct calls, batch methods,
transact(), imports, and Virtual Filesystem writes all funnel through the
same commit points the feed is emitted from. This is the authoritative
in-process signal for live UIs, cache invalidation, and realtime sync layers
that forward it over their own transports.
Architecture — the post-commit dual of the ifRev precommit hook: mutation
methods hand lightweight event descriptors to the commit seam
(persistSingleOp / transact's plan), which stamps the committed
{generation, timestamp}, enriches entity deletes with the record's LAST
committed state from the commit's own before-images (free — Model B reads
them anyway; removeMany's id-only deletes gain full payloads this way), and
emits only after the commit succeeds — a losing CAS or rejected batch never
announces anything. Dispatch is a microtask FIFO after the mutex releases:
commit-ordered, a slow listener never delays a write, a throwing listener is
logged and isolated, and with no subscribers the write path constructs no
events at all. Single-point emission was chosen over per-method hooks
because the aggregation-hook pattern demonstrably drifted (relation ops and
removeMany were silently missing from it).
Coverage: add/update/remove (+ cascade unrelate per deleted relationship),
relate (both edges when bidirectional)/unrelate/updateRelation, per-item
events for addMany/updateMany/relateMany/removeMany, per-item events sharing
one generation for transact(), and transitively imports + VFS. clear() and
restore() — wholesale raw-state operations outside the per-record commit
path — emit a single store-level event meaning "refetch everything".
brain.close() drops all listeners.
New module src/events/changeFeed.ts (BrainyChangeEvent + ChangeFeed,
exported from the package root); guide docs/guides/reacting-to-changes.md.
Integration suite pins the contract: per-op payload fidelity, delete
last-state payloads, batch per-item emission, one-generation transact
batches, VFS-origin events, CAS-loser silence, ordering, listener isolation,
unsubscribe, and store-level events.
The fast-follow to the ifRev CAS fix: the sweep for the same check-then-act
class found two more instances, both now closed with the same discipline —
the decision runs at the serialization point that guards the apply.
add({ifAbsent}) / add({upsert}): the absence check ran before the commit
mutex, so N concurrent same-id creates could all pass it and all write — the
second silently overwriting the first, violating ifAbsent's "return the
existing id WITHOUT writing" contract and upsert's merge-never-clobber
contract. The insert leg now carries a must-be-absent commit precondition
(the same conditional-commit primitive ifRev uses), verified under the commit
mutex against the authoritative before-image. A losing caller takes its
documented resolution instead of overwriting: ifAbsent returns the existing
id with zero writes; upsert merges into the now-existing entity via update()
(shared param mapping in upsertMergeParams so the planning-time branch and
the conflict retry can never drift), with a bounded retry if a concurrent
delete lands between the conflict and the merge. The planning-time checks
remain as fast-fails. transact()'s batch-level ifAbsent/upsert keep
planning-time semantics (documented, converges to a valid entity).
BlobStorage: write()'s dedup decision (exists → increment refCount, absent →
create at 1) and delete()'s decrement-then-remove were unserialized
read-modify-writes over blob-meta:<hash>. Concurrent writes of identical
content could lose references, so a later delete removed bytes another file
still referenced (data loss), or leaked unreferenced blobs. All
reference-count-bearing mutations now serialize through a per-hash
InMemoryMutex — distinct content never contends; incrementRefCount/
decrementRefCount are documented lock-assumed internals.
Both fixes are complete by construction in-process: storage enforces
single-writer-per-directory, so the process is the whole concurrency domain.
Tests (tests/integration/ifabsent-upsert-blob-concurrency.test.ts): 8-way
ifAbsent storm advances the store by exactly one generation with _rev 1;
8-way upsert storm on an absent id yields one create + seven merges
(_rev === 8); sequential ifAbsent/upsert semantics pinned unchanged; N
identical concurrent blob writes → refCount === N; the blob survives until
the true last reference drops; interleaved write/delete storm never loses a
landed reference.
N concurrent update({ ifRev }) calls carrying the same expected revision all
fulfilled (zero RevisionConflictErrors, last-writer-wins) — the check ran
before the commit mutex, so interleaved callers all passed it before any apply
landed. Sequential calls conflicted correctly, which hid the race. A production
cutover rehearsal caught it: 8 parallel ifRev-guarded ledger writes all
"succeeded" and 7 were silently lost. Advisory locks built on exactly-one-winner
semantics could hand the same lock to two workers.
Fix: conditional commit. commitSingleOp and commitTransaction accept an
optional precommit(beforeImages) precondition, invoked under the commit mutex
against the just-read authoritative before-images and before anything is staged
or applied — the per-record analogue of ifAtGeneration, which always ran there.
A throw aborts the commit atomically (generation reservation returned, zero
staging I/O in the transaction path). The store stays domain-ignorant; update()
and transact() supply the ifRev predicate:
- update(): the planning-time check remains as a fast-fail (avoids embedding
cost on an obviously stale expectation); the authoritative check re-verifies
the before-image's _rev and re-stamps the update's _rev from it, so the
counter is monotonic even for concurrent non-CAS updates (N plain updates
advance _rev by N). CAS against a concurrently-removed entity now throws
EntityNotFoundError instead of silently resurrecting it; plain updates keep
their last-writer-wins re-create semantics.
- transact(): per-op ifRev expectations registered during planning
(PlannedTransact.casUpdates) are re-verified in the batch's precommit; any
conflict rejects the whole batch before staging. Multiple updates to one
entity sequence through a running rev; ids the batch itself adds
(PlannedTransact.createdNouns — add resets the rev baseline even on
overwrite) keep their exact plan-time sequencing.
Regression suite (tests/integration/ifrev-concurrent-cas.test.ts): 8 parallel
same-rev updates → exactly 1 winner + 7 conflicts; same for transact batches;
_rev monotonicity; the documented read→CAS→retry ledger loop converging exactly
(8 workers, 8 decrements, 0 lost); sequential behavior unchanged; forward-ref
add+update in one batch unchanged.
The one-time 7→8 layout migration rescues the head branch's entities
(branches/<head>/entities/* → entities/*), then drained the entire
branches/<head>/ directory. Any durable state a 7.x engine wrote under the head
branch OUTSIDE entities/ — a branch-scoped index, blob area, or field registry —
would be silently deleted by that drain, the same failure class as the VFS
content blobs a 7.x store kept in _cow/.
Guard it: after the entity move, list what remains under branches/<head>/ and
exclude entities/. If anything survives, it is non-entity durable state, so
PRESERVE the branch (skip removeRawPrefix) and warn loudly with the leftover
keys — recoverable, not lost. A clean branch (only the moved entities) still
drains exactly as before. Adds a PARITY GUARD test to the 7→8 migration suite.
The 7.x branch system stored Virtual Filesystem content as blobs in its
copy-on-write area (`_cow/`). 8.0 removed that system and stores content blobs
in the content-addressed store (`_cas/`), but the one-time layout migration only
moves entities — it never adopted the `_cow/` content blobs. A store that used
the VFS was left with every VFS-backed read throwing "Blob metadata not found"
and its pages 500ing, with no first-party recovery and the pre-upgrade backup
already removed on entity-migration "success".
Add an on-open recovery pass (`autoAdoptLegacyVfsBlobsIfNeeded`) that runs right
after the layout migration and adopts every orphaned `_cow/` blob into `_cas/` —
copying both the bytes and the metadata via the raw object primitives,
idempotently and non-destructively (the `_cow/` originals are never deleted). It
is a separate phase, not folded into the migration, so it also heals a store
already upgraded by an earlier 8.0.x that stranded the blobs (whose layout-
migration marker is stamped): it is gated on the presence of `_cow/` and its own
`_system/vfs-blob-adoption.json` marker. Filesystem-only; native-8.0 and non-VFS
stores no-op on a cheap existence check.
- `BaseStorage.adoptLegacyCowBlobs()` — the scan/copy primitive, returning
`{ cowBlobs, adopted, alreadyPresent, incomplete }`; skips (does not half-adopt)
a blob missing its bytes or metadata.
- `brain.vfs.adoptOrphanedBlobs()` — the explicit force path; self-initializes.
- Backup retention: the automatic pre-upgrade backup is now kept if any blob
can't be fully adopted (`incomplete > 0`), instead of being removed on
entity-migration success alone — a blob-parity gate the migration signal
cannot provide.
Adds an end-to-end integration test (storage-level adopt with idempotency and
incomplete handling; self-heal on reopen; the explicit API) and an upgrade guide.
cor confirmed the migration write-new / tmp+rename invariant the hard-link backup
relies on — with one exception: the native shared mmap `_id_mapper/*` is mutated
IN PLACE (msync + a rebuild truncate+re-inject), the same category as the tx-log.
A hard-linked snapshot of it would be corrupted by a live in-place mutation
reaching through the shared inode, so the backup would not be a faithful
pre-upgrade copy.
Add a SNAPSHOT_BYTE_COPY_DIRS set (`_id_mapper`) so snapshotToDirectory byte-copies
every file under it (the directory analogue of SNAPSHOT_BYTE_COPY_PATHS); the id
map is bounded, so the copy cost is small. Everything else (SSTables / .dkann /
manifests / objects, all tmp+rename) still hard-links. Test: `_id_mapper/*` gets
an independent inode + faithful content; a regular file stays hard-linked.
David's ask: back up the brain before the one-time upgrade, drop it after. On
open, when the on-disk format is stale (a 7.x→8.0 rebuild will run) and the store
holds data, brainy hard-link-snapshots the brain dir to a sibling
`<brainDir>.migration-backup` BEFORE any provider rebuilds — near-zero cost/space
(shared inodes; the store is immutable-by-rename), instant even at scale. Removed
automatically once the upgrade verifies + stamps the marker; retained on failure
for rollback. Default-on; opt out with `migrationBackup: false`.
- Reuses the existing snapshotToDirectory() hard-link farm via new optional
adapter methods createMigrationBackup()/removeMigrationBackup() (filesystem
only — feature-detected; memory + non-fs are a graceful no-op).
- Taken before the native provider inits, so it captures true pre-migration
state; a prior failed upgrade's backup is reused, not overwritten.
- Best-effort: a backup failure never blocks the upgrade (the migration is itself
safe — reads canonical, reconstructable, self-heals). Catastrophe-insurance
against a migration bug, not a data-loss guard or an off-device backup.
4 lifecycle tests (adapter hard-link / reuse / remove-without-touching-live /
empty→null; stale-epoch reopen create+remove; opt-out; memory no-op). Gates:
typecheck 0, build 0, test:unit 1753/1753, layout-migration suite 5/5.
Consumer-invisible modernization pass — no public API or runtime-behavior
change; dist for the override-only files is byte-identical.
Toolchain:
- CI: GitHub Actions matrix — Node 22/24 (test:unit) + Bun latest (test:bun).
- engines: node ">=22" (was "22.x"), bun ">=1.1.0".
- tsconfig: isolatedModules + noImplicitOverride; add the 33 `override`
modifiers the flag requires across storage/integrations/vfs/transaction.
- deps: @types/node ^22; add prettier; drop dead standard-version,
@rollup/plugin-* and the redundant embedded eslintConfig (flat
eslint.config.js is the active config — verified identical lint output).
paramValidation: replace the top-level `await import('node:os'/'node:fs')`
with static ESM imports. The top-level-await form poisoned the module graph;
static imports also drop the browser/edge fallback branches no supported
runtime reaches (8.0 is Node/Bun/Deno-only).
Bun positioning: recommend Bun as a runtime (`bun add` / `bun run`), which is
green (test:bun 8/8). Drop single-binary `bun build --compile` as a target —
native addons cannot embed into it, and Bun 1.3.10 has a `--compile` codegen
regression around top-level await. Rename the Bun test to bun-runtime-test.ts
and correct docs that overclaimed single-binary support.
Gates: typecheck 0, build 0, test:unit 1743/1743, test:bun 8/8.
8.0 shipped with no cold-graph guard (the 7.33.4 fix was never ported), so on a
cold open of a large brain where the native adjacency reports membership but its
source->target edges did not load, find({connected})/neighbors()/related() could
silently return [] for persisted edges.
Add the converged isReady() contract: GraphIndexProvider.isReady?(): boolean is
the honest cold-load readiness signal (true ONLY when edges are loaded), so brainy
gates on it instead of the lying membership-size() proxy. verifyGraphAdjacencyLive()
checks it — Strategy 1: false -> hydrate the id-mapper, rebuild from storage, re-check,
throw GraphIndexNotReadyError if still not ready; providers without isReady() fall
back to a global known-edge-sample probe (Strategy 2, not the queried anchor, so a
genuinely edgeless node still returns []). rebuildIndexesIfNeeded gates the graph
rebuild on it too, with the id-mapper hydrated before the adjacency rebuild on the
lazy cold-open path (the CTX-BR-RESTORE-REBUILD ordering, shared with restore()).
executeGraphSearch re-verifies before trusting an empty connected result and
re-collects on a heal. 6-case integration test + 1718 unit green.
- find() search-mode: collapsed the two overlapping options to one canonical
`searchMode: SearchMode` (the redundant `mode` alias was silently ignored on
the primary find() path while honored on the historical path — a footgun) and
removed the unwired `explain?` FindParams field (never read by find()).
- Documentation accuracy on the public type surface: entity ids documented as
UUID v7 (auto) / v5 (natural key), not v4; dropped the stale "Brainy 3.0"
banners and "backward compatibility" hedging on the fresh-8.0 Result type;
documented the via/type alias; removed the dead GraphConstraints.bidirectional;
fixed the no-op `EmbeddedGraphVerb = Omit<GraphVerb,'source'>` to omit the real
sourceId; corrected the GraphIndexProvider.isInitialized JSDoc; documented
removeMany's per-chunk generation granularity; JSDoc'd the public VFS surface.
- Honest perf comments in source: removed unmeasured billion-scale figures from
the LSMTree / graphAdjacencyIndex headers (the JS fallback is in-memory after
open; native is the scale path).
- Robustness: getVerbMetadata propagates read errors symmetrically with
getNounMetadata; the find() egress integrity-guard keeps a row when the JS
matcher doesn't implement an operator the provider already matched on; hardened
the boundary-no-native CI guard to catch side-effect imports + re-exports.
- Tests: de-theatricalized a relateMany test to assert the real contract; fixed
a stale Model-B header.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Completes brainy's #35 side. When the at-gen vector gate serves a filtered
semantic read, Brainy now ships the historically-correct candidate vectors so the
native provider reranks against them with zero per-vector FFI crossing — the
provider need not duplicate the per-generation retention Brainy already does.
- New `AtGenerationVectors { ids: BigInt64Array; vectors: Float32Array; dim }` on
the VectorIndexProvider.search options bag (row-major: vectors[i*dim..] == ids[i];
invariant vectors.length === ids.length*dim; ids = the candidate set; dim must
match the index dim). The built-in JS index ignores it.
- buildAtGenerationVectors resolves each universe id's vector AS OF the generation
(the record before-image, or live getNoun when untouched since the pin — not
readNounRaw, whose canonical path is empty under lazy-vector eviction), interns
the id via the shared mapper, and packs row-major. Absent/vectorless/wrong-dim
ids are dropped (not vector-rankable). Bounded by the filtered universe.
Shape + the four clarifications (row-major, ids-are-the-candidate-set, dim-asserted,
filtered-path-only) confirmed with cor in the handoff #35 thread. Inert until cor's
native at-gen rerank advertises isGenerationVisible (honesty guard).
Tested: the mock versioned provider now asserts it receives atGenerationVectors with
the row-major invariant, correct dim, and ids resolving back to the at-gen universe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A filtered semantic read at a historical generation (db.asOf(g).find({ query, where }))
no longer unconditionally rebuilds an ephemeral JS-HNSW over every at-g vector
(O(n@G), the OOM ceiling the 2026-06-24 scaling audit flagged). When the vector
index is a VersionedIndexProvider that advertises isGenerationVisible(g), Brainy:
1. resolves the at-g metadata∩graph universe from the record-overlay path (no
materialization — it's the metadata-only historical find),
2. routes the vector leg to the provider with { allowedIds, generation }, and
3. composes the at-g entities ranked by the provider's at-g vector distance.
Without a versioned provider (the JS index, or a native one that refuses the
generation) it falls through to the existing materialization — unchanged.
- Seam (A): VectorIndexProvider.search gains an optional `generation?: bigint`
("omitted = now"), the vector mirror of the graph provider's trailing-gen + the
#46 allowedIds field. JsHnswVectorIndex accepts and ignores it (no per-gen
segments → refuse/fall-back posture; Brainy never routes a historical read there).
- Gate: Db.find tries the native at-gen vector path before materialize(); host
exposes canServeVectorAtGeneration + vectorSearchAtGeneration.
- generation is Brainy's u64 commit counter — the same value handed to the graph
index on writes (graphWriteGeneration), so it maps 1:1 to the native side.
Decided lockstep with the cor team (handoff #35 thread: (A) + vector-only). The
gen-g candidate-vector supply for cor's exact-rerank (part-3) is a separate,
non-blocking seam still being confirmed; the honesty guard keeps this path inactive
(falls through to materialize) until cor's native at-gen rerank is live.
Tested: provider routing + at-gen universe correctness (excludes future-born,
applies the filter) + page window via a mock versioned provider; seam-ignore on the
JS index; 38 db-mvcc/db-temporal historical-read tests green (materialize fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
