Follower to the self-rebuild deference. The open gate's consistency check —
"metadata index has 0 entries but storage has N entities" → CRITICAL + a forced
second rebuild — knew two states, migrating and not. A provider whose rebuild()
returns once the rebuild is OWNED AND RUNNING ONLINE (its doors refusing by
name while other families serve) legitimately reports 0 entries there, so every
first contact printed a false CRITICAL and kicked a redundant second rebuild.
The exemption rides the rebuild-progress hook, NOT isMigrating() — widening
that would hold every write and 503 the whole brain through the migration
snapshot, which is worse than the false alarm. The check's real class is
untouched: a provider reporting 0 entries with no rebuild in progress still
trips it.
The crash-recovery rebuild kick gets the same deference: a provider already
rebuilding itself from canonical is doing exactly that work, and the fold ran
in the generation store's open before any provider existed, so what it is
reading is the repaired canonical.
Pin: a provider stub reporting a rebuild and 0 entries opens with no CRITICAL
line and no second rebuild; the vacuous-stub case fails loudly.
MEASURED on a production store: a metadata provider that had to rebuild made
init() pay the ENTIRE rebuild on the foreground — 641 seconds — with every
other family idle behind it. The cause is a missing distinction: a provider
reporting serving:false because it is BUSY BUILDING ITSELF and one reporting
serving:false because it is BROKEN looked identical through healthReport(),
and both were answered the same way — call rebuild(), and wait for it.
The contract that tells them apart is one optional, synchronous, O(1) hook:
`rebuildInProgress(): ProviderRebuildProgress | null`, reporting a phase name
and whatever the provider actually measures (done/total/startedAt) — never an
estimate dressed as a fact. A provider without the hook behaves exactly as
before.
With it, a provider owns its own rebuild:
- the open gate neither starts a second rebuild nor waits for the provider's,
and narrates that it is not waiting and what will refuse meanwhile;
- init() returns and every other family serves;
- that family's doors refuse BY NAME, carrying the provider's own progress,
and say plainly that the door opens by itself and no action is needed —
distinct from a broken index, which names repairIndex();
- the epoch stamp does not advance while any family is still being built.
Nothing is ever served empty: a not-serving family refuses, as it already did.
Pins: tests/integration/open-does-not-wait-for-a-rebuilding-provider.test.ts —
init() returns in milliseconds against a provider claiming a 6s rebuild, brainy
starts no rebuild of its own, a filtered read refuses naming the phase and the
4,096/14,056 progress, and the door answers once the provider reports serving.
The pin fails loudly rather than vacuously if its stub never installs.
REPORTED from the field: a process holding many stores, with no writes for ten
minutes, printed "All indexes flushed to disk in 216-601ms" per store every
~35 seconds and burned over a core at idle. Every one of those flushes re-persisted
state identical to what was already on disk — the provider flushes, the
watermark stamps, the generation counter, the entity-tree stamp — because
flush() never asked whether anything had changed.
- flush() over a clean brain is now O(1) and silent: a dirty witness is set by
every committed write (both commit paths end at noteWriteForPersistence, and
the deferred-embed worker lands through the single-op path) and cleared by a
flush that runs. A write landing DURING a flush sets it again, so no write's
work is ever skipped — it is done by the next flush. Set before the policy
check, so a `'manual'` consumer's explicit flush is never a no-op it didn't
ask for.
- An explicit flush now tells the cadence it happened. It didn't, so the very
next write saw "30s since the last flush" and kicked a background flush with
nothing to do, and the idle timer fired two seconds later over writes the
explicit flush had already persisted.
- The graph adjacency index's auto-flush asks before it acts: two O(1) reads
of the LSM MemTables, and a tick over a quiet index returns without calling
into the trees at all.
assessProviderHealth is NOT timer-driven — it is a synchronous O(1) read of a
provider's own healthReport(), called on the read gate, so it costs nothing on
an idle brain. No change needed there.
Pins: tests/integration/idle-costs-nothing.test.ts — 90 idle seconds produce
zero flushes, zero provider calls and zero log lines; three explicit flushes
over a clean brain call no provider; one write earns exactly one flush.
Rule by rule, with the line of code that proves each: confirmed, corrected, or
changed-under-you. Written against 10.4.3 plus this branch.
Three rules moved since the draft was written against 10.4.1, and a reader
built to the draft would be wrong about them: the identity-record count fix
landed in 10.4.2 (the draft's open question 6 was already half-answered), the
legacy denominator now heals itself in the background instead of lying for the
life of the store, and counts.json is written atomically.
Answers the six open questions, states which scalar a consumer may subtract an
enumeration against (only the all-tier pair carrying the identity-record stamp
with suspect false; the counted-tier scalars never), ratifies the torn-row and
timestamp divergences — the timestamp one AGAINST this engine, which invents a
Date.now() where a stored value is unreadable — and files six findings back,
including that the spec's §4 code claim and §8.1 cause 2 are both out of date.
Internal document: no frontmatter, not published.
On a production store (14,647 nouns / 73,070 verbs) a repairIndex() ran for
more than thirty minutes at roughly a full core with ZERO log lines between
its start and its end, while the read doors kept serving. The operator could
tell it was alive only from `top`, and could not tell which of its
single-threaded walks it was inside.
Same law as the open, applied to the repair:
- every phase announces itself BEFORE it works, naming what it is about to
walk (each canonical walk, the VFS containment reconciliation, each
provider's invariant pass);
- an unref'd heartbeat names the phase still running every 5s, for as long as
it runs;
- every phase reports its own wall, and that wall is carried in the TYPED
receipt as RepairFamilyReport.durationMs — a receipt that cannot say where
the time went is not a receipt;
- the whole repair's narration moves to the always-visible channel, so a
production log level cannot silence it.
The phases move into runRepairIndexPhases() so the heartbeat can live in a
finally around them; the public door and its report shape are unchanged apart
from the added durationMs.
Pins: tests/integration/repair-narration.test.ts — every checked family has a
start line, a finish line with its wall, and a numeric durationMs in the
receipt; a phase slowed to 6.5s produces a heartbeat naming it, with the
logger clamped to ERROR.
MEASURED on a real store: the ALL-visibility ledger read 14,231 nouns against
14,056 identity records and 72,729 verbs against 72,679 — exactly that store's
25 noun and 50 verb SCAR directories. Two copies of the same archive derived
different numbers (14,231 and 14,081), because each had been persisted at a
different moment under the old rule that counted one entity per id DIRECTORY.
A downstream index heal subtracted against that denominator and reported
remaining work that did not exist.
The scan already applies the right predicate — one entity per IDENTITY RECORD
(the metadata content leg), shared with pruneOrphanedEntities so the two agree
by construction. What was missing is that a ledger persisted under the old rule
was only FLAGGED suspect and then went on serving its wrong numbers for the
life of the store, waiting for an operator to run repairIndex.
- The ledger now derives itself honestly in the BACKGROUND after the open,
narrating start and finish with the correction it made. Background because
these scalars are denominators — no read is served from them — and because
walks exactly like these are how a 24,898-id store spent minutes of a
restart in silence. Observable via whenCountLedgerSettled(); nothing in the
read path waits on it.
- A derivation that raced a write refuses to stamp its number "exact": one
retry on a quiet store, then the ledger stays SUSPECT and says so, naming
repairIndex as the door that recounts under a barrier.
- The one derivation that CANNOT leave the foreground says why it cannot:
getNounCount()/getVerbCount() are served from it, and a background walk
would make a populated store answer "0 entities" — a wrong answer, not a
slow one. It narrates its start and its wall instead.
- counts.json is written temp+rename. A truncating write left a window —
measured at roughly 750ms after a flush or close — in which a concurrent
reader saw the file EMPTY; an unparseable ledger sends the next open down
the full-rescan path, so the cheapest file in the store was buying the most
expensive recovery.
- The writer lock's clean-close record is now consulted before the
same-process branch too: a restart reported "Re-acquiring writer lock ...
this is a bug" immediately after a clean close, sending an operator after a
leak that did not exist.
Pins: tests/integration/count-ledger-identity-record.test.ts (background
correction with scar and ghost fixtures, two copies of one archive agreeing,
counts.json never observed unparseable across 40 persists);
tests/integration/ledger-derivation-identity.test.ts updated to the new law —
the OPEN still never walks (proved by slowing the walk 1.2s and timing the
open), and the ledger heals behind it.
An operator watched a production service open a 16 GB store and print nothing
for three minutes before its first line of work. Two defects, both fixed here.
The narration was written to `prodLog.warn`, and every environment that looks
like production clamps the logger to ERROR — so the phase breakdown that would
have named the slow phase was composed and thrown away. `prodLog.narrate` is
always visible, like `error`: it carries the two things an operator is
entitled to hear from a database regardless of a cost setting — why it is slow
and what it is doing about it. `silent: true` still silences it; that is a
request, not a default.
And nothing spoke DURING a phase, only after the whole open. init() now runs
an unref'd heartbeat that every 5s names the phase currently running, its
elapsed wall and what it is paying for, plus one line per phase as it ends for
any phase over 2s. The generation-log fold's own progress and completion lines
move to the same channel and now carry their wall — they were invisible in
production, which is how an operator came to restart a converging fold three
times.
Pins: tests/integration/open-narration.test.ts — narrate() survives the clamp
that silences warn(); a 6.5s storage-init produces a heartbeat naming the
phase and a completion line naming its wall, with the logger clamped to ERROR.
A production restart made this necessary: a service stopped with exit code 0,
having awaited close() on every pooled brain, and its next boot announced
"Overwriting stale writer lock ... appears dead" for every store it owned.
Nothing had crashed. "The recorded pid is gone" is equally true of an orderly
restart and of a crash, so the verdict could not tell an operator which one
they had — and when the OS recycles a pid it fails the other way, refusing to
open a store whose writer died days ago.
Three changes, all at the law:
- close() is two parts, and the second is unconditional. The durable steps
(flush, markers, component close, plugin deactivate, buffer drain) move to
closeDurableSteps(); the terminal releases — the flush-request watcher, the
WRITER LOCK, the VFS timers, the terminal `closed` flag — always run. The
original failure is narrated with what it costs the next open, then rethrown.
- releaseWriterLock() writes a CLEAN-CLOSE RECORD (`locks/_writer.close`)
naming the lock generation it released; the next claim consumes it, so a
record can never vouch for a later crash. An open reads the record instead
of guessing: recorded → nothing to recover; absent → say so, and name the
crash recovery this open will now run.
- The signal path stops failing in a batch. It was one try around a loop over
every open brain, so the first instance whose flush rejected stranded every
remaining brain's lock and markers — at exit code 0. Now: per-instance
isolation, the generation store's close (the clean-shutdown marker, without
which the next open folds the whole log) is part of shutdown, the lock is
given up in a finally, and the handler no longer calls process.exit() when
the host application has its own signal handler — that race truncated the
host's own close() mid-flight.
Pins: tests/integration/writer-lock-clean-close.test.ts — completed close
leaves no lock and a consumed-once record with a silent reopen; a failing
durable step still releases and still rethrows; SIGKILL leaves the lock with
no record and the reopen names the crash; a host SIGTERM handler runs to
completion.
Branch plan (10 lines):
1. writer lock: clean-close record + always-release close [this commit]
2. open narration: an always-on channel; production clamps prodLog to ERROR,
which is why a three-minute open printed nothing
3. open narration: per-phase lines as each phase ENDS, with progress cadence
4. measure both real-store fixtures on the box, before/after
5. move the generation-log fold out of the foreground where the serving law
allows; durable resumable progress marker
6. same for the VFS bootstrap
7. counts: a legacy container-rule ledger must not keep serving wrong
denominators; counts.json written atomically
8. counts pin with scar directories; two copies of one archive agree
9. docs/canonical-layout-ratification.md — 12 facts confirmed/corrected
10. report: MEASURED before/after, findings, and whether this is 10.4.4
Prepares the repo for its new home at soulcraftlabs/open-brainy ahead
of the Forgejo transfer: package name, publish registry, release
script, and every install/import reference across docs, src, tests,
examples, and integrations now point at @soulcraftlabs/brainy on
The Source. The npmjs storefront leg and byte-identity pair
verification are stripped from the release script — The Source is
now the only publish target. README gains an Open Brainy explainer
and a registry note for consumers.
@soulcraft/brainy 10.4.2 was the last release under the old name.
The engine-pair seam law: a zero-norm vector never crosses an engine
boundary. The index belt already refused to insert one, but the canonical
write and the vectored-noun ledger still counted it, so a near-empty store
whose only vectored row was zero-norm read "1 canonical vectored vs 0
indexed" and threw a not-ready error at open, and a store's own legacy
zero-norm VFS root could trip the same gate before its VFS-init-time cure
ever ran.
- add()/update() (single and transact()) now normalize an explicit
real all-zero vector to the unvectored [] shape before the dimension
pin, the ledger flag, and the index ops ever see it (loud, one warn per
write, canonical write still succeeds).
- The legacy counts.json derivation walk (scanVectoredNounCount) excludes
a persisted zero-norm row, matching the live ledger's definition.
- A legacy zero-norm VFS root now migrates at open, before the vector-leg
gate evaluates, via one O(1) fixed-path read (torn-tolerant — skips
rather than aborting init on a torn root, letting the recovery walk
heal it) — independent of whether a VirtualFileSystem is ever
constructed this session.
- update({ id, vector: [] }) (and the same op inside transact()) is now
the sanctioned, idempotent unvector door: index removal, exactly-once
ledger decrement, no re-embed, and it clears a pending deferred-embed
marker rather than leaving it to re-vectorize the row later. The
combination with deferEmbedding is a typed refusal.
- JsHnswVectorIndex.rebuild() now skips a zero-norm/empty persisted
vector when repopulating from canonical (the same belt the live
add/replace paths already had), and health()'s index-parity check now
compares HNSW size against the vectored-noun ledger rather than the
raw metadata-entry count, since a store's VFS root is permanently
unvectored by design.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A canonical row persisted with vector: [] (a system row, a deferred embed
not yet landed, or any other legitimately-unvectored record) is a normal,
enumerable row -- but rebuild()'s storage walk had no guard against it.
storage.getVectorIndexData() derives its answer from the row's own record,
so it returns non-null for any existing noun whether or not that noun was
ever actually indexed -- rebuild() admitted such rows into the live graph
with a length-0 vector. A vector-less node could become the entry point (or
occupy any graph position); the next real insert then ran a distance
calculation against it and blew up with a dimension mismatch.
Fix at two layers in src/hnsw/hnswIndex.ts:
- rebuild() now skips any row whose vector.length === 0 before it ever
becomes a graph node (one summary count line, never per-row spam), and
restores the pinned dimension from the first real vector it loads --
previously the pin stayed null across a restart, since addItem/updateItem
are the only sites that set it and rebuild() never goes through either.
- addItem/updateItem now refuse a length-0 vector with a typed
EmptyVectorIndexError instead of ever pinning dimension to 0 or storing a
vector-less node, so no future fill/rebuild/load path can poison the index
silently. getVectorSafe's lazy-load "not found" check also missed that an
empty array is truthy -- tightened to catch it.
IndexOperations.ts's ReplaceInVectorIndexOperation rollback paths now skip
re-adding an oldVector of length 0 (never a legal index member) instead of
attempting an illegal empty re-insert on rollback.
biography.test.ts's final ledger-exactness assertion assumed every noun the
lane creates is vectored, including the VFS root counted in
vfsBaselineNouns -- but the root is deliberately persisted unvectored.
Corrected the expected formula to exclude it.
Adds tests/integration/index-skips-unvectored.test.ts pinning: rebuild()
indexes only vectored rows with the dimension pinned correctly; clear()
then real adds never trip a dimension mismatch; addItem/updateItem refuse a
length-0 vector; and a crash/repair cycle stays dimension-consistent.
The noun/verb pagination walks (getNounsWithPagination,
getNounIdsWithPagination, getVerbsWithPagination) listed shard contents by
filtering for vectors.json, while the canonical count ledger has always
counted a row by its metadata.json presence alone. A row with metadata and
no vector file was therefore counted by the ledger but never yielded by the
walk — a permanent "counted but invisible" phantom for any downstream
consumer that iterates the walk to account for the ledger's total.
Nouns now enumerate by metadata.json and hydrate the vector leg optionally,
yielding the sanctioned unvectored shape (vector: []) when it's absent.
Verbs enumerate the same way, but a metadata-only verb row can only be fully
reconstructed when sourceId/targetId happen to be recoverable from metadata
(never true for a current production write — those fields live only in the
vector leg); otherwise the row is counted but loudly skipped rather than
fabricated, since a phantom edge with fake endpoints would be worse than the
original defect.
Separately, GenerationStore's recovery-fold replay (replayFact) now applies
preserve-if-absent: a metadata-only after-image replayed over an already-
vectored row carries the existing vector forward instead of deleting it via
writeNounRaw/writeVerbRaw's exact-restore null-means-delete contract (which
must stay exact for transaction-abort rollback). A genuine tombstone still
removes both legs.
A zero-norm vector is lawful inside brainy (cosine distance scores it at
maximum, never a false top hit) but a false attractor for a downstream
engine serving squared-euclidean distance, which cannot tell a real
all-zero vector apart from a legitimate origin point.
- The VFS root now persists with vector [] (the existing "unvectored"
shape) instead of a real all-zero 384-dim placeholder, and is never
routed into the deferred-embed pipeline.
- A one-time migration in the root-init path detects a pre-fix store's
all-zero placeholder root (by norm, not length) and rewrites it to []
through a new sanctioned Brainy method that keeps the canonical
vectored-noun ledger honest and removes the row from the vector index.
- The vector-index write seam (AddToVectorIndexOperation,
ReplaceInVectorIndexOperation, and the generation materializer's direct
insert) now refuses any real all-zero vector before it reaches a
provider, loudly naming the entity, while the canonical write still
lands.
- add()'s dimension-pinning and HNSW-insert gates, and the add-params
validator, now treat any empty vector as carrying no dimension
information, closing a latent trap where an explicit `vector: []`
would have pinned dimensions to 0.
Two builds of the same source tree could publish different artifacts
because buildEmbeddedPatterns.ts and buildTypeEmbeddings.ts stamped
their generated output with new Date().toISOString(). Route both
generators' "Generated:" header and the generatedAt runtime field
through a shared resolver: newest git commit time among the
generator's inputs (script + source data), falling back to the stamp
already present in the previous output when git is unavailable (e.g.
a published tarball build), and finally to a fixed epoch value. Every
fallback logs to stderr so degradation is never silent.
Regenerated both committed output files once so the tree carries
deterministic stamps; no other content changed.
Two cures from the pair's first production adoption, both measured live.
THE READ GATE IS PER-FAMILY. The report-driven gate refused on ANY
provider's not-ready verdict at every read choke point — so a pure
metadata find({ where }) was refused because the VECTOR leg was not
serving, and a deployment's badge reads returned errors for a verdict that
had nothing to do with them. A read may only be refused by the family it
actually consults: metadata reads by the metadata leg (plus graph for a
`connected` filter), vector search by the vector leg, traversal by the
graph leg. Callers name what they need; the existing narration-once-per-
generation and typed-refusal laws are unchanged within a family.
NO RE-EMBED ON UNCHANGED DATA. update() — and its transact() planner —
treated any write that carried `data` as a data change: with
deferEmbedding it queued a landing, and the worker re-embedded and re-landed
a vector for content that had not changed. A host heartbeat re-writing an
unchanged row every few seconds therefore fed a live index-row loop on a
production store. A write carrying the row's current data (structural
compare, key order normalized) is now not a data change: no re-embed, no
deferred landing, no vector rewrite; the metadata write itself still
commits. A real change re-embeds exactly as before.
Pinned in tests/integration/read-gate-scope-and-no-reembed.test.ts — both
pins red-proved against the unfixed code with the production shapes
verbatim. Two health-gate pins that encoded the old brain-global scope are
re-pointed to the family their reads consult.
The coverage denominator the health-by-accounting ratification named for
the vector family — never built until now, and its absence was measured as
the exact outage class it existed to prevent: a migrated store with
canonical vectors and no derived index opened with the vector leg EMPTY,
served [] from vector search with no error, and the report-driven read gate
had nothing to refuse on (the provider's coverage invariant was honestly
unledgered — the denominator was ours to supply).
- getCanonicalCounts() gains vectors: { all } — the count of canonical
nouns holding a REAL vector. Incremented where a vector lands (the
isNew-gated metadata seam for explicit vectors — the same discipline that
keeps HNSW neighbor-link re-saves from inflating counts; a narrow
noteVectorLanded hook for the deferred-embed landing, gated on the
worker's own pre-embed read). Decremented on a proven delete of a
vectored noun; a vector-uncertain delete marks the ledger suspect rather
than guessing (no new reads on the delete path). Recounted by the
sanctioned recount; legacy counts.json derives it once (a deferred noun's
vector file exists with an empty vector, so presence requires one
content read at derivation — never on the hot path).
- The open gate's vector leg: when a health-reporting provider claims
serving while the index holds zero nodes and the ledger proves vectored
canonical rows exist, open BUILDS (narrated) — routed through the
provider's idempotent fillFromCanonical() when exposed (the joint door;
a partial shortfall stays repair()'s operator business), the JS rebuild
otherwise — or fails typed pre-serve. Scoped exactly: bare isReady()
providers, migrating providers, and white-box size stubs open as before.
Pinned end-to-end from the partner gate's probe shape (store with vectored
canonical rows, no derived index, reopen → search serves N, never []),
red-proved against the pre-fix path; the inverse (zero vectored rows) opens
without building and serves [] honestly.
resolveVerbEndpointInts mirrors the resolved u64 endpoint ints onto the
verb object itself as BigInt (verb.sourceInt/targetInt) for the graph legs'
own params. The live verb path's delete legs then reused that same object
as the metadata-index crossing — and the seam's metadata is JSON-safe by
contract (a native provider serializes it; u64 as Number corrupts above
2^53), so JSON.stringify threw and the whole transaction aborted. Found by
the first joint pair gate; four downstream suites red from one crossing.
The crossing now routes through a JSON-safe view that drops BigInt-valued
top-level keys — endpoint ints ride their own op params on the graph legs,
exactly as designed, and never the metadata crossing. Applied at the
retraction helper (cascade + unrelate + transact mirrors) and
updateRelation's remove leg.
Pinned by driving the exact shape (relate resolves ints, remove cascades
the same object) through a provider shim enforcing the JSON contract —
red-proved against the unfixed path (the joint gate's verbatim error),
green with the fix.
repairIndex() acted only on heal:'rebuild' — an invariant asking for the
INCREMENTAL heal (re-post exactly what the ledger names, O(missing), never
a store-sized rebuild) did nothing on brainy's side. A failing 'repair'
verdict now routes to the provider's feature-detected repair(); the
post-heal RE-READ of the report decides success (the acceptance meta-pin's
law — run the named heal once, re-read, nothing may still fail the same
way), and a repair that does not converge is recorded with the escalation
named: repairIndex({ rebuild: [family] }).
The factory loudly rejects every REMOVED pre-8.0 path key, but an unknown
nested `config` object (e.g. `storage: { config: { baseDir } }` — a shape
that was never supported) fell through SILENTLY to the zero-config default
directory. Every instance constructed with such a shape wrote to ONE shared
on-disk root while its caller believed each had its own — found live when
two integration tests' brains shared a store across an entire
single-process CI run and a health probe refused on the foreign edges it
sampled. A nested `config` carrying any path-shaped key now throws the same
loud migration error, naming the canonical `path` rename. The two tests are
repaired to the supported shape (and now actually test isolated stores, for
the first time since 8.0).
validateAddParams() treated '' as falsy and rejected it with "Missing
required field 'data'" — so a legitimate empty file's first write always
failed. Only null/undefined data (with no vector either) is genuinely
absent; '' is real content. Fixed the check, plus the identical bug in
validateUpdateParams() (truncating a file to empty via overwrite hit the
same falsy check) and in update()/transact()'s update planner, where a
plain `Boolean(params.data)`/truthy check on the resolved vector would have
silently skipped both the deferred-embed marker and the eager re-embed for
an emptied value — a stale vector with no path to ever correct itself.
Verified end-to-end: vfs.writeFile('/empty.txt', '') now succeeds,
readFile() returns '', the file lists, and stat() reports size 0; the
existing "should reject empty string as data" tests (unit + integration)
asserted the old buggy behavior and are updated to assert the fixed
contract instead.
vfs.readdir()'s ReaddirOptions.recursive was typed but silently ignored: a
recursive request behaved identically to a non-recursive one. Implemented
properly: recursive listing walks every descendant (files and directories,
any depth) via the same graph-traversal + one-batch-fetch path
getTreeStructure()/getDescendants() already use, and reports each entry as
a path relative to the queried directory (Node's fs.readdir(dir,
{ recursive: true }) convention) — 'sub/file.txt', not just 'file.txt'.
With withFileTypes: true, each VFSDirent.name carries that same relative
path when recursive (matching the string-array form byte for byte);
VFSDirent.path stays the absolute VFS path either way, so no information is
lost. Filter/sort/pagination compose unchanged, now over the full recursive
set. Non-recursive behavior (direct children, named by basename) is
unchanged.
A production restart storm measured 90,017ms for a single brain init vs
1,117ms quiet (~80x contention multiplier), traced to performInit() eagerly
awaiting the process-global WASM embedding engine before the VFS root even
existed. Every writer's open() queued on the one throttled model compile
(90-140s on throttled CPUs).
- VirtualFileSystem.doInitializeRoot() no longer embeds '/'. The root is
system-tier plumbing nothing ever searches; when the default WASM engine
is active it now gets an explicit all-zero placeholder vector
(cosineDistance returns max distance for a zero vector, so it never ranks
ahead of real content). deferEmbedding was considered and rejected: its
landing path kicks the embed worker synchronously right after commit,
which would still force the cold compile within milliseconds — just off
the awaited path, not avoided. A registered native 'embeddings' provider
(no cold-start cost, possibly a different dimension) still embeds the
root for real, via the new Brainy.usesDefaultWasmEmbedder() seam.
- performInit()'s eager-embedding step now only STARTS the WASM engine warm
in the background instead of awaiting it inline. embed()/embeddingManager
already serialize concurrent callers on one shared init promise, so the
first real embed() converges correctly either way; a failed warm narrates
loudly instead of surfacing as a silent latency spike or an unhandled
rejection. eagerEmbeddings: false still means no warm at all.
- FileSystemStorage.init() batches its ~8 independent bootstrap mkdirs
(each creates its own full subtree via recursive:true, so none depend on
the others existing) into one Promise.all. The restore-completion step
and initializeCounts() stay strictly sequential — they have real order
dependencies on rootDir and systemDir respectively.
- performInit() now times five phases (storage init / generation-store
open+fold / index init+gate / VFS bootstrap / embedding-warm-started) and
logs one warning with the per-phase breakdown when total open exceeds
2000ms; silent otherwise.
The auto-detection "not installed" heuristic accepted any resolution failure
whose message merely CONTAINED the package name, unterminated — so a missing
platform-binary sibling package (what a deploy replacing node_modules
mid-restart leaves behind) read as "the accelerator is not installed", and
brainy silently served the default WASM engines with zero journal lines. A
production restart storm paid 90 seconds of throttled WASM compile behind
exactly that hole. The name must now terminate where it ends (quote,
whitespace, punctuation, end) — a sibling package, an inner file path, or a
dependency failure is a broken install and init() throws, as the guard's own
law always stated.
Second door: activate() returning false (the documented graceful decline)
warned on console.warn, which `silent: true` patches away — an invisible
degrade. The decline now narrates via the always-on channel.
Also exports CanonicalCounts from the public surface (the coverage-ledger
denominator type consumers read through getCanonicalCounts()).
Pinned in tests/unit/plugin-activation-loudness.test.ts (five error shapes;
the decline warn under silent: true).
Three cures on the JS metadata index, one seam:
- THE CATCHUP WIRING. The index computed its three-way watermark verdict at
open and nothing consumed it — after a crash + adopt reopen, find() served
the pre-crash index while canonical reads and counts recovered (caught by
the lifecycle lane's first run). The open path now consumes the verdict:
'adopt' is a no-op, 'catchup' folds the fact window (stamped, committed]
through the index legs — nouns and verbs, remove-then-add, one mechanism
for add and update — and 'rescan' runs the explicit rebuild, each narrated.
The lane's Ch4–6 release-blocking marker comes off: the contract holds.
Bonus root-cause: close() never stamped the projection watermarks (only
flush() did), so any close without a prior flush verdicted a needless
'rescan' on reopen — both doors now stamp.
- THE LIVE VERB PATH. Verb rows entered the metadata index only via rebuild
walks, so every rebuilt store minted phantom/stale verb postings from its
first live relate(). relate()/unrelate()/updateRelation() and remove()'s
cascade now post/retract the verb's row in the same commit as the graph
leg — transact() planners mirror identically — using the exact record
shape the rebuild walk uses, so live and rebuilt populations agree.
- THE ONLINE REBUILD. rebuild() was clear-then-walk — every metadata read
empty for the duration. rebuildMetadataIndexOnline builds a fresh manager
beside the serving one (shared identity, in-memory build, dual-write via
a shadow seam with zero call-site changes), atomically swaps the
reference, and persists exactly once post-swap. A find() polled ~200x
during a 2k-noun rebuild never dropped below its baseline.
repairIndex({ rebuild: ['metadata'] }) uses it automatically.
The read gate stops consulting the unnamed isReady() boolean: every provider
may expose healthReport() (sync, O(1), composed from exact ledgers —
HealthReport with a monotonic generation, per-invariant source
ledger|deep|unledgered, missing {count, sample}), and one readiness
authority (assessProviderHealth) derives the verdict. Unledgered families
are UNKNOWN — never healthy, never broken; a report that throws is a loud
not-ready, never a shrug. Reads at the four index choke points refuse with
the typed NotReady errors, narrated once per (provider, generation) — a
read NEVER starts a store walk:
- the first-read lazy build retires (open builds instead, regardless of
size — the ≥10k deferral and the "lazy loading on first query" branch go;
disableAutoRebuild is re-meant honestly in its docs);
- the verify*Live read-path rebuild triggers retire (refuse-or-serve);
- the read-time consistency probe that could launch a dark rebuild from an
ordinary find() retires;
- repairIndex({ rebuild: ['metadata'|'graph'|'vector'] | 'all' }) is the
one explicit door: rebuilds the named leg unconditionally and reports
rebuilt per family; bare repairIndex() stays report-driven.
test(lifecycle): the biography lane — a store's whole life, refereed
tests/lifecycle/: an independent shadow model referees every read after
every chapter (founding, a working day, clean restart, crash, repair,
second life). Chapters 1-3 green. Chapters 4-6 assert the true contract and
are marked .fails as a release-blocking finding (the kill-matrix
convention): after a crash + adopt reopen the metadata index computes its
'catchup' watermark verdict and nothing consumes it — find() serves the
pre-crash index while canonical and counts recover. The catchup wiring is
the cure; a passing .fails will force the marker's removal. The lane runs
in the integration gate (config + coverage guard).
