`readonly-close-no-marker` closed the clean-shutdown-marker half of this law and
named the rest as a known residual. This is that residual, closed.
MEASURED on the base: a read-only open → read → close rewrote FOUR files —
`_system/__metadata_field_registry__.json.gz`, `type-statistics.json.gz`,
`subtype-statistics.json.gz` and `verb-subtype-statistics.json.gz`. An IDLE
reader that only opened and closed rewrote all four as well.
The cause was not the closes the marker fix guarded. It was Phase 1 of
closeDurableSteps, where every component flush ran unconditionally. A flush is a
write by definition: MetadataIndexManager#flush() saves the field registry "even
with no dirty fields" (its own comment), and the storage adapter's count flush
re-stamps the three statistics files. A session that committed nothing re-stamped
all four. Phase 2's closes were ungated too — the graph index's close drains both
LSM MemTables to SSTables and stamps its watermark, and the optional
vector/metadata `close` hooks (unimplemented in the reference engine, filled in
by a native provider) persist buffered state.
Every one of those calls now carries the same `!isReadOnly` guard the generation
store already had.
A reader still RELEASES what it holds, so Phase 2 is a branch rather than a skip:
GraphAdjacencyIndex gains `stopBackgroundFlush()`, the non-writing half of its
close, which clears the auto-flush interval that would otherwise outlive the
session. `close()` now calls it too, so there is one place that owns the timer.
Why this matters beyond tidiness: `_system/` is where a store keeps its evidence
about itself — what the writer committed, what the projections have seen. A
reader that rewrites any of it vouches for a state it only observed, and on
shared or snapshot storage it mutates bytes another process owns.
The pin hashes every file under `_system/` (and, in one case, the whole store)
across a reader's open → read → close, names the four paths that used to move so
a regression says which subsystem did it, and asserts the asymmetry holds in the
other direction — a WRITER's close still persists.
An array-valued metadata field indexes one posting per element, so the index has
always carried a ceiling. It was 10, and it was applied by a bare `continue`
deep inside field extraction:
if (Array.isArray(value) && value.length > 10) continue
A row whose `tags` array held ELEVEN entries therefore had that field skipped
entirely — no posting, no error, no warning. The row then failed to match every
filtered search on `tags`, including a query for a tag it demonstrably held, and
the caller had no way to tell that from "no row matches". Eleven tags is not an
exotic shape; the eleventh tag made the row invisible. Measured on the pin here:
the where-clause returns [] on the base for all eleven values.
The ceiling is not the defect. The silence was.
THE LAW. MAX_INDEXED_ARRAY_LENGTH = 64, hardcoded (the zero-config law: no
knob), sitting far above every legitimate multi-value field — tags, authors,
categories, labels, participants — and far below any real embedding width, so
the two populations do not overlap and nobody has to tune it. Arrays of scalars
index in full up to the bound. Above it the WRITE IS REFUSED by name:
MetadataArrayTooLargeError carries the field (its full dotted address), the
length and the bound, and names the three cures. It fires at all four write
doors — add, update, relate, updateRelation — beside the existing forged-system-
key rejection, and walks nested bags because a nested field indexes under its
dotted address exactly like a top-level one.
THE ONE PLACE THE BOUND STILL SKIPS is a row already on disk, written by an
older engine under the old rule and read back by a rebuild, a catch-up fold or a
remove. extractIndexableFields serves all three, so refusing there would make an
existing store un-rebuildable — the row is admitted and the skipped field is
NARRATED with the field, the length and the bound. Never silent, either way.
tests/integration/metadata-vector-exclusion.test.ts carried the old law as a
green assertion ("should skip indexing large arrays (>10 elements)"). It is
rewritten to the new one, plus a case proving a 64-element array indexes in full
and its eleventh element is searchable. The original bug that suite exists for —
per-dimension numeric field explosion — is still asserted on both paths.
`getIdsForRange` routes two ways. The column store compares RAW values and is
correct. The legacy sparse chunk index — the pre-7.20.0 fallback, still read for
workspaces that have not been rebuilt — compared normalizeValue() output, and
normalizeValue carries an escape hatch that destroys order on purpose: a string
over 100 characters becomes a short hash so it can serve as a filesystem-safe
key. Ordering hashes ranks rows by digest.
Two shapes, both silent:
(a) A LONG BOUND against ordinary values. `{ gte: <a 120-character string> }`
collapsed the BOUND to `__HASH_…`, whose leading underscores sort below
every letter — so a bound that should have excluded everything matched the
entire field instead. Measured on the fixture here: 3 of 3 rows returned
where 0 is correct. This shape reaches a caller who never stored a long
value at all.
(b) LONG VALUES in the index. The field was persisted hashed, so its order is
not recoverable from this index. The old code compared the digests anyway
and returned a subset chosen by hash — 1 of 3 rows, the wrong one.
Bounds are now normalized WITHOUT the hash escape hatch, so a long bound stays
comparable and (a) is simply fixed. Where the persisted KEY is a hash the order
does not exist to be computed, and the query throws a typed
BrainyError('INVALID_QUERY') naming the field and the cure. The refusal is
checked before chunk SELECTION as well as during the scan: selection orders the
bounds against each chunk's zone map, and its failure mode is an empty answer —
the quietest wrong answer of all. Equality on a hashed field is untouched; only
ordering is refused.
KNOWN, NAMED DIVERGENCE, recorded in the doc comment rather than papered over:
the persisted keys are also lower-cased and trimmed, so this path's string
ranges are case-INSENSITIVE where the column store's are not. The raw values are
not in the index to compare — that is a property of the bytes a pre-7.20.0
engine wrote, and it ends when the column store adopts the field.
The pin builds a genuine legacy index through the same ChunkManager /
SparseIndex doors that engine wrote through, into a field the column store does
not serve. The chunk write path was removed in 11be039, so that is the only way
to build the shape this read path exists for.
`find({ where, orderBy })` answered in field order; `find({ query, where,
orderBy })` and `find({ vector, where, orderBy })` answered in SCORE order.
The vector/filter block ranks the fused candidates by score, cuts the page and
returns early — and the tail's orderBy sort sits below that early return, so on
those paths it never ran. Nothing threw and nothing warned: the ordering request
was dropped in silence, and the two paths disagreed about what "ordered by rank"
means.
Where `connected` or `fusion` kept the tail alive the defect changed shape
rather than disappearing. The block had already CUT the page by score, so the
tail ordered the rows relevance had chosen — a correctly sorted page of the
wrong rows.
The early cut fires only once the candidate set reaches offset+limit rows, which
is why it read green for so long: below that threshold the block falls through
and the tail's sort does apply. An ordering that is correct until there is
enough data to matter.
THE LAW: an explicit orderBy displaces score as the ordering key on every path.
The candidate set the path produced is ordered IN FULL and the page is cut from
that ordering — "page last", the graph-first law applied to ordering rather than
to filtering. Score-ranked early paging stays exactly as it was for the default
case, where score IS the requested order.
The pin is differential against the metadata-only path, the one path that always
honoured orderBy. It is sized so the hybrid legs (each bounded at limit*2)
provably cover the filter universe, and that covering is asserted from the leg's
own output rather than assumed — orderBy orders the candidate set, it does not
enlarge it, and the pin claims nothing about recall.
The seam hydrates the metadata and graph page paths; a vector or text leg builds
its own entities and is trimmed after the integrity guard instead. That is a
COST difference, and this pin exists so it can never quietly become an ANSWER
difference.
TypeScript's `private` is compile-time only, so the seam's helpers were real
prototype methods and the generated contract manifest listed them as public
DOORS — which would have obliged every other engine to implement an internal
detail. They are `#`-private now and the manifest is unchanged by this branch.
Found while checking that: docs/api-contract.json was ALREADY stale at v10.4.11
— promoteQueuedFlush and startFlushLeader are in src and absent from the
manifest, so they leaked the same way and were never re-emitted. Left alone
here rather than folded into this branch; it is someone's to fix deliberately,
and the fix is the same # conversion.
docs/FIND_SYSTEM.md gains the projection: the rules, why a missing field is
absent rather than an error, where the values come from and what a field the
column cannot serve costs.
A list view that shows a title and a slug hydrates the whole record for every
row, document bodies included, and discards almost all of it. find/get({ fields })
names what is wanted; the column store serves it; the canonical record is opened
only for fields the index cannot supply.
The provider grows an optional getScalarsForIds(ids, fields) door, batched: it
walks each column ONCE and picks out every requested id, rather than re-walking
per row. The column store grows the primitive that was missing — valuesForIds —
because every other read door there answers which entities have a value, and a
projection asks the opposite.
It reads the COLUMN store, never the sparse index: the column keeps raw values,
the sparse index keeps a bucketed form built for range queries, and a projection
served from the latter would return a value that differs from the record's. A
field the column cannot serve is omitted rather than approximated — omission
costs a read, a wrong value is a wrong answer nobody can see.
Two laws the pins hold: fields absent is byte-identical to today, and a missing
field is simply absent rather than an error — so this path deliberately avoids
the strict address resolver, whose UnresolvableFieldError is right for orderBy
and wrong here.
related() takes no fields: a Relation carries from/to as ids and hydrates no
record, so the param would be decorative.
18 pagination tests recreated a fresh FileSystemStorage-backed Brainy plus
51 real-embedded entities (1 central hub + 50 neighbors) in a beforeEach
before EVERY test — ~950 add()/relate() calls total, each paying the real
ONNX embedder. Measured before this change: 303.69s (fresh run, this
session). None of these tests exercise similarity search, only graph
pagination, so three changes cut the cost without touching an assertion:
- vector: [] on every add() — add()'s `params.vector || embed(...)` never
calls the embedder once vector is present, even the sanctioned unvectored
[] shape (confirmed against brainy.ts's zero-norm-law comment: the
dimension-pinning gate is `vector.length > 0`, so [] never poisons
dimensions for a later real embed).
- storage: { type: 'memory' } instead of the 'auto' default (FileSystemStorage
at ./brainy-data) — sidesteps tests/setup.ts's global per-test
`rm -rf brainy-data`, which would otherwise corrupt a brain shared across
a describe's beforeAll out from under it.
- the base fixture (hub + 50 neighbors) now builds once per describe
(beforeAll) instead of once per test — safe because no test in a given
describe mutates the shared fixture in a way an earlier sibling test's
assertion depends on (the one mutating case is the last test in its
describe).
Measured after: 416ms for all 18 tests (2.35s wall including vitest
startup), all 18 still passing.
vfs.unit.test.ts's 'Performance > should handle many files efficiently'
(100 writes + readdir, 5.5s write budget) is a wall-clock flake: 121ms
alone, 16.5s under the gate's sibling-file contention — the code never
caused it. Same pattern already used for storage-batch-operations.test.ts's
batch-vs-individual timing case: ctx.skip(!process.env.BRAINY_PERF_LANE,
reason) inside the test, and the file added to vitest.perf.config.ts's
include list (it stays in the unit gate's *.unit.test.ts match too, so
every other test in the file keeps running there).
tests/performance/triple-intelligence-scale.test.ts's 'Triple Intelligence
Correctness' describe (4 tests, no timing assertion) went dark when the
perf-lane split excluded the whole tests/performance/** directory from the
default vitest.config.ts gate — it ran nowhere since. Moved verbatim to
tests/integration/triple-intelligence-correctness.test.ts, which the gate
does collect.
Every expect() is byte-for-byte the original. Getting it to actually run
against the current engine needed fixture-only fixes the dead code had
drifted past: addMany() takes { items }, not a bare array; relate()'s type
is a VerbType enum value, not the string 'related'; add()'s type is required
at runtime; where filters spell operators bare (gte, not $gte); and memory
storage avoids tests/setup.ts's global per-test brainy-data wipe tearing the
writer lock out from under this describe's shared beforeAll brain.
Two of the four tests are it.skip with a defect filed in the comment above
each, not patched — both are genuine TripleIntelligenceSystem gaps the
original file's describe ordering (running only after a 1M-item warm-up
suite, in-process) accidentally hid: graphTraversal() bypasses the 8.0
id-normalization law for a natural-key `connected.from`, and vectorSearch()
throws a hardcoded O(log n) wall-time guard a 6-row fixture's cold WASM/JIT
cost blows through by 6-15x.
CI / Integration + conformance (Node 22) (push) Failing after 17m33s
10.4.11 gave shutdown one owner and one path — close() — and wired all three
process listeners to it. That is right for SIGTERM and SIGINT. It is wrong for
'beforeExit', which Node emits whenever the event loop has no REF'd work left:
not when the process is ending, and with no signal involved. A healthy script
reaches that state routinely, because this engine unref's its idle and cadence
timers ("an idle brain costs nothing"), so a script awaiting anything those
timers drive is, for that instant, a process with no ref'd work and an open
brain.
MEASURED on the 11.1 rehearsal lane against a copy of a real store: after the
heal phase the log printed "Shutdown signal received - flushing pending
data..." and "Flushed successfully (1 instance)" with no signal ever sent, and
the script's very next add() threw "Brainy instance is not initialized: it was
closed via close(). Create a new instance." The engine had closed a live brain
out from under a running script.
The beforeExit listener now runs its own pass, which closes nothing,
deregisters nothing, releases no writer lock, and never force-exits: it runs
flush() — the engine's own non-closing durability door — on each live brain and
leaves every one of them open and usable. flush() persists derived state only
(count ledger, projections, generation counter, aggregation, entity-tree
stamp); the clean-shutdown marker is generationStore.close()'s word about
itself, reached only from close(). Running it concurrently with live writes is
the engine's ordinary steady state — noteWriteForPersistence() kicks the same
call off an unref'd timer on every busy brain — and it is single-flight, so
there is no new race. A throw is reported per instance and the pass continues:
canonical data is durable at ack via the fact log, so a failed derived-state
flush costs the next open a rebuild, never the caller their brain.
The listener is no longer self-deregistered. It does not need to be: a flush on
a clean brain schedules no I/O, so the emit after it does no event-loop work
and the process exits on its own. A one-shot listener spent on a spurious
mid-script drain would leave the genuine end-of-script drain with nothing. The
drained-loop notice is printed once per registration cycle, because a
console.log to a pipe is itself event-loop work.
exitIfSoleShutdownOwner() stays on the signal path alone, and its contract now
says so: beforeExit suppresses no default behaviour, so exiting from it would
end a live script at code 0 mid-work.
THE NAMED TRADE: a script that opens a brain and never closes it now exits with
its writer lock still on disk and no clean-shutdown marker, so its next open
overwrites a stale lock and folds the log. That is the honest cost of never
closing, and the narration names the cure. Closing a live brain to avoid it was
the worse half of the trade.
Pins: tests/integration/beforeexit-never-closes.test.ts — a script that drains
the loop with a brain open keeps a working brain (add + find succeed, the lock
is still held, the process still exits 0), the pass flushed and wrote neither
of close()'s markers, and repeated drains are idempotent. Both cases fail on
10.4.11's handler with the exact production shape ("add() after the drain
failed", "pass 1 closed the brain"). Re-run green: shutdown-single-owner,
writer-lock-clean-close, idle-costs-nothing, shutdown-hooks-lifecycle.
docs/concepts/multi-process.md no longer claims beforeExit releases the lock.
CI / Integration + conformance (Node 22) (push) Failing after 17m12s
The single-flight gate queued its follow-up as
`leader.catch().then(() => this.flush())`. That waiter is settled ONLY by
resolving the very promise the leader is being awaited through, so the moment
anything inside a flush body awaits flush(), the promise graph closes on itself
and nobody resolves — an unbounded hang, not a slow flush, presenting exactly
like a bulk write timing out. No current call site awaits a flush from inside
one, so this is a latent cycle rather than an observed one; the gate should not
depend on that staying true.
The queue is now a bare deferred. The leader's finally opens the gate and
PROMOTES the waiter to a new leader, settling the deferred from that run; the
finally returns nothing, so the leader never awaits its own follower. Every
exit runs the same promotion — the leader resolving, the leader rejecting, the
promoted run rejecting — so a queued caller is settled exactly once on every
path, and a synchronous failure starting the promoted run is reported to the
waiter instead of thrown into the leader's finally. close() drains both handles.
tests/unit/brainy/flush-single-flight.test.ts pins the invariant on each path
that must settle a waiter: many callers during one flush all resolve within a
bound (one body, one follow-up, peak concurrency 1); a REJECTING leader still
runs and settles the queued waiter; a rejecting follow-up settles its waiter
and leaves the gate open; and the leader returns without waiting for a
deliberately slower follower.
Delta Gate / Delta gate — candidate vs control (push) Waiting to run
CI / Node 22 (push) Successful in 12m27s
CI / Bun (latest) (push) Has been cancelled
CI / Node 24 (push) Has been cancelled
CI / Integration + conformance (Node 22) (push) Has been cancelled
The wall-clock ratio (batch faster than N individual gets) started failing
under the exclusive release gate because individual gets got faster on
this candidate (open-path/hydration changes), not because batchGet
regressed — a perf assertion misclassified into a correctness file.
Skip it under the default gate via a BRAINY_PERF_LANE env marker the perf
config sets for itself; the file joins the perf config's include list so
the case still runs (with every other test in the file) under
`npm run test:perf`.
tests/configs/vitest.perf.config.ts (npm run test:perf) is a real gate,
not a manual-only slot, so inGate() now recognizes its include list
(tests/performance/** plus the four named files) directly. The 7 files
already correctly listed as perf move out of MANUAL_ONLY, which is now
reserved for files no automated lane covers.
That alone left the guard red: tests/vfs/vfs-search-path-scope.test.ts
was a genuine new orphan (added this cycle, named without the
.unit.test.ts suffix its siblings use) — it ran under the broad root
gate but silently missed test:unit. Renamed to match the sibling
convention in tests/vfs/, which puts it back in the unit gate.
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
Delta Gate / Delta gate — candidate vs control (push) Waiting to run
reservedGensAsc() documented but never enforced that pending single-op
generations must sort above every committed one. A direct
commitTransaction() call bypassing Brainy.transact()'s flush-first step
could commit a fresh generation into committedRanges above lower,
still-pending ones, unsorting the committed-then-pending walk
resolveManyAt relies on and returning a wrong before-image for a
point-in-time read — silently. commitTransaction() now refuses via a new
PendingSingleOpsUnflushedError when the pending tier is non-empty, before
any staging I/O. Behavior-neutral: both sanctioned callers
(Brainy.transact(), Brainy.compactHistory()) already flush first.
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
Delta Gate / Delta gate — candidate vs control (push) Waiting to run
Four pins in real child processes under real SIGTERM, following the
writer-lock-clean-close spawn pattern:
(a) A host owner registered on SIGTERM closes two brains while the engine's
hooks are live: exactly one close entered and one close body run per brain,
the writer lock given up exactly ONCE per brain, the handler announcing
that it stepped aside, no "Writer fence lost", no failed instance, both
durability markers written, exit 0, and both reopens adopting rather than
folding. The release count is the discriminating assertion — against the
old handler it reads {a: 2, b: 2}, one release from the owner's close and
one from the handler's own finally.
(b) No host owner: the engine's handler closes every instance by the same
path — one close each, markers written, clean exit, clean reopen.
(c) Two concurrent close() callers share one promise (by identity) and one
execution; a third call after they settle runs nothing.
(d) Eight kicks during a running flush — five through the cadence door, three
direct — arm exactly ONE follow-up: two flush bodies total, and the
concurrency high-water mark stays at 1.
The counts come out of the child through a file written synchronously on the
way out: the engine calls process.exit(0) when it is the sole shutdown owner,
and a console.log to a pipe can be dropped by that exit.
Delta Gate / Delta gate — candidate vs control (push) Waiting to run
CI / Node 22 (push) Failing after 1m29s
CI / Bun (latest) (push) Has been cancelled
CI / Node 24 (push) Failing after 1m29s
CI / Integration + conformance (Node 22) (push) Failing after 1m30s
`vfs.search({ path })` built its scope as `path: { $startsWith: path }`.
`$startsWith` is not in the filter vocabulary at all, and the `$`-less spelling
is REFUSED by the metadata index's served-operator law — an equality/range
posting index cannot evaluate a substring without reading every row, so it
refuses rather than answering an empty page. Every path-scoped VFS search threw
on this engine line; the two pins in tests/vfs/vfs.unit.test.ts that exercise it
have been red since the operator law landed.
The scope is now a half-open range over `metadata.path`:
`[dir + '/', dir + '0')`. Every descendant path begins with `dir + '/'`, and '0'
is the code point directly after '/', so membership in the range is EXACTLY
"carries that prefix" — and because the bounds differ at one ASCII position the
answer is identical under code-unit and code-point collation. Siblings fall out
correctly for the same reason: `/scope-sibling/x` sorts below the lower bound
and `/scope0` sits at the open upper bound. `recursive: false` narrows to the
directory's own identity instead — `parent`, an indexed equality. The root
adds no clause, because every VFS entity is under it.
`path` is the VFS's truth (write and rename maintain it; the `Contains` edges
are a projection of it), it is already indexed on every VFS entity, and
`explain()` reports the range as `column-store` — "O(log n) binary search +
roaring bitmap". So the scope narrows the search before it runs: no tree walk,
no migration, no backfill, and nothing fetched that the scope then discards.
Two other shapes were considered and rejected. A graph-scoped walk over
`Contains` reads the projection rather than the truth and costs O(subtree)
adjacency lookups per search, with the subtree's height as an unknown `depth`.
An indexed `ancestors: string[]` field cannot be implemented honestly today:
the index extractor skips arrays longer than ten elements, so a path more than
ten levels deep would silently drop out of every scoped search — and it needs
a backfill besides.
Pinned in tests/vfs/vfs-search-path-scope.test.ts (all eight red before this
change): descendants at three depths and never a sibling, including the
`/scope-sibling` and `/scope0` prefix traps; a trailing or doubled slash names
the same scope; the root scope equals the unscoped search; `recursive: false`
is the immediate children and refuses a missing directory by name; every
operator the search emits is ANSWERED by the index's own door rather than
refused; the id universe the index resolves for the search is already the
scope; and the range agrees with walking the tree.
Delta Gate / Delta gate — candidate vs control (push) Waiting to run
CI / Node 22 (push) Successful in 12m28s
CI / Node 24 (push) Successful in 12m15s
CI / Bun (latest) (push) Successful in 12m33s
CI / Integration + conformance (Node 22) (push) Failing after 17m16s
Four things, none of them a clock:
1. A brain with one permanently-stuck pending id, closed cleanly and
reopened, scans ONLY the facts after the checkpoint — read from the
fold's own accounting. The same fixture pins the DEFECT it cures: no
low-water mark exists on that brain, because it never drained, so nothing
could have shortened its fold. A second row proves the bound stays
O(delta) across repeated opens while the id is still stuck.
2. A crash matrix in a REAL child process (detached group, SIGKILL, no
close), following writer-lock-clean-close's pattern: killed before any
checkpoint was written, killed after one with an embed landed and flushed
above it, and killed after one with an UN-FLUSHED tail. The invariant in
every row is differential — the checkpoint-bounded fold the reopened
brain actually ran equals a full fold from generation 1 over the same
recovered log.
3. A torn checkpoint (bytes that are neither gzip nor JSON) falls back
loudly — the adapter's torn-record gauge and production error, plus the
fold's own narration of the bound it used — and still recovers the marker
from the log. A well-formed but shape-invalid checkpoint is refused
WHOLE: trusting its generation while ignoring its list is the one shape
that could bound a scan behind a set that was never recovered.
4. The existing low-water pins pass unchanged — the mark is still written
and still read, now as the fallback bound beneath the checkpoint.
Delta Gate / Delta gate — candidate vs control (push) Waiting to run
CI / Node 22 (push) Successful in 12m27s
CI / Node 24 (push) Successful in 12m29s
CI / Integration + conformance (Node 22) (push) Failing after 17m33s
CI / Bun (latest) (push) Successful in 12m28s
Every log-authority open asks the fact log one question — is there a fact
above the committed pointer? — and answered it by reading and CRC-decoding
EVERY segment file the manifest names. MEASURED in production on a 16k-row
brain at generation ~478,819: 34-37 seconds inside `generation-store-open-fold`
on every open, including the clean one where the answer is always "nothing".
The manifest already knows. A sealed segment's `lastGeneration` is written at
seal time, and the seal order has been the same since the log was introduced:
`rotate()` fsyncs the tail's bytes FIRST ("sealed segments are always fully
durable"), builds the entry from the content that fsync covered, and only then
flips the manifest — atomically, fsynced, and in the same write re-pointing
`tailSegment`, so a sealed file is never appended to again. A crash in that
order is safe in the pruning direction: before the manifest write the segment
is still the TAIL and is read whole; after it, the entry describes bytes that
were already durable. The only later mutation of a sealed segment is open()'s
straddle truncation, which removes facts and re-derives the entry from the
actual bytes — a recorded bound can drift DOWN with its file, never up.
So `lastGeneration = L` proves the file holds no fact above L, and both
manifest-direct passes (`peekFactsAbove` and its streaming twin, the recovery
fold) now read only the unsealed tail, entries with no numeric
`lastGeneration` — legacy or hand-repaired manifests, never prune what you
cannot prove — and entries whose recorded maximum is actually above the bound.
The open narrates what it read and what it pruned when the log holds more than
one segment.
Pinned in tests/integration/factlog-open-prune.test.ts, from the log's own
counters rather than a clock: a clean reopen over five sealed segments reads
exactly the tail (1 of 6) and finds nothing; a real SIGKILLed writer that
sealed segments holding facts above the committed pointer has those segments
READ, and its peek, its fold stream and its rollback all match the unpruned
full scan fact for fact; a manifest entry missing `lastGeneration` is read.
Delta Gate / Delta gate — candidate vs control (push) Waiting to run
CI / Node 24 (push) Successful in 12m34s
CI / Node 22 (push) Successful in 12m40s
CI / Integration + conformance (Node 22) (push) Failing after 17m14s
CI / Bun (latest) (push) Successful in 12m29s
`find({ query, connected, where, offset })` answered [] for every page but the
first. The metadata block ranks the fused candidates and CUTS the page itself
— rows [offset, offset+limit) — and then returns early. Two shapes do not take
that early return, `connected` and `fusion`, and they fell through to the tail,
which sliced the already-cut page by `offset` a second time: a five-row page
sliced at offset five is nothing at all. Every page after the first was empty,
and the caller had no way to tell that from "no more rows".
The block now records that it consumed the offset, and the tail returns the
page it was handed instead of re-cutting it. Nothing changes at offset 0, where
the second slice was the identity.
Pinned in tests/integration/find-hybrid-filter-before-hydrate.test.ts: page two
of a `connected` hybrid find matches the pipeline oracle row for row, paging
reaches every matching neighbour exactly once, and a `fusion` find's second
page is the same page the plain find returns.
A hybrid find fuses a text leg and a semantic leg. The semantic leg already
walked only the metadata filter's universe. The text leg did not: it ranked
the WHOLE store, took the top `limit * 4`, read every one of those rows from
canonical, and only then intersected with the filter. On a large store with a
selective filter that is hundreds of rows read to return a handful — and a row
matching both the query and the filter, but sitting outside the store-wide
text prefix, was silently dropped. The same defect `find({ connected })`
carried before the graph-first law, one leg over.
Both legs now rank ids inside the universe and neither reads canonical. The
text leg goes through a new optional `getIdsForTextQueryWithin` door on
MetadataIndexProvider — the text twin of `filterIdsWithin`, so a native index
can intersect its postings before any string crosses the boundary; the
reference index implements it from its own posting-list merge, so the two
doors can never disagree, and a provider without it is served by the
whole-store answer intersected here. The fusion ranks shells, the page is cut
from them, and canonical is read once for exactly that page — with the row
rebuilt in full, so a hydrated row is indistinguishable from an eagerly-built
one (same flattened fields, same entity, same match visibility, same key
order). The eager forms of both legs stay for the search modes whose leg
output IS the answer.
Measured on the production recall shape (query + type list + `missing`
negation + excludeVFS, limit 60) the old order read 241 rows in two batches to
return one; the new order reads the page.
Pinned in tests/integration/find-hybrid-filter-before-hydrate.test.ts. The
oracle there is the pre-change pipeline itself, replayed on the same brain
through the same doors: where the filter does not truncate the text leg the
answer is identical — rows, order, scores, match visibility and row shape —
across hybrid + where, + type list + excludeVFS + a `missing` negation, +
connected, with and without offset. Where it does truncate, the correction is
held by name: the old order's text leg contributed nothing at all, the new one
returns the matching rows and paging reaches every one of them. The cost pins
read the engine's own counters: one batchGet of `limit` ids, the whole-store
text door never called, and what the text leg marshals bounded by the universe.
CI / Integration + conformance (Node 22) (push) Failing after 17m24s
CI / Bun (latest) (push) Successful in 12m23s
The proximity search fetched its anchor through get(), which omits vectors
by default, then handed a zero-length vector to the index — every
find({ near }) refused with a dimension mismatch, for every caller. Found
by the Rust planner's first-contact pins comparing outcomes with and
without the planner on a refused shape. The anchor is now fetched with its
vector, and an anchor that has none refuses by name — a proximity search
around an unvectored row has no meaning and must not fail inside the index.
Pinned in tests/integration/find-near.test.ts.
CI / Integration + conformance (Node 22) (push) Failing after 17m18s
CI / Bun (latest) (push) Successful in 12m28s
The delta gate caught the backgrounded fold breaking six pinned
crash-recovery cases: a reopened brain must have its markers re-armed
when open() returns, and a background latch races every consumer of that
contract. The backgrounding is reverted; the low-water mark stays — it
is the part that kills the whole-history scan, and with it the
foreground fold costs the log's tail on any brain that has ever drained.
The unmarked first open after upgrade pays one full scan, once, and the
open narrates it as its own step.
CI / Integration + conformance (Node 22) (push) Failing after 17m4s
CI / Bun (latest) (push) Successful in 12m19s
The provider's read doors each serve one stage, so a find that consults three of
them crosses into the index three times and marshals a result set at every
crossing: a filter matching a hundred thousand rows builds a hundred thousand id
strings to return a page of twenty-five. An index able to decide the stage order
itself can answer the page in one call and build ids only for the page.
planFindPage is optional and additive, in the shape filterIdsWithin and
getIdSetForFilter already set. The hook sits above the branch selection, because
the branches are what decide stage order per call site and an index that plans
has to be asked before that choice is made. Absent — as it is on this engine's
own index — every find is served by the stage doors exactly as before, which is
what keeps this engine the ordering oracle for any index that implements one.
The contract the door must keep, written where an implementer will read it:
identical rows in identical order to what the stage doors would produce; the
graph-first law (neighbours are the candidate universe, the filter runs over
those ids, orderBy sorts the whole set, the page is cut last); null returned
BEFORE any work rather than instead of an answer; and emptyAt naming the stage
that produced an empty page, so the serving law is applied to the right index —
an empty graph answer is re-verified against the adjacency before it is
believed, and a filter-empty is not.
Pinned in tests/integration/find-planner-door.test.ts: absent changes nothing;
present it is asked first with normalized params, the hidden ids and the graph
provider; its page is used and hydrated in its order; a declining door leaves
the result identical to the no-door path; and the two emptyAt branches verify
the adjacency, or correctly do not.
CI / Integration + conformance (Node 22) (push) Failing after 16m58s
CI / Bun (latest) (push) Successful in 12m24s
Pass 2 issued one awaited related({ to }) per VFS entity — O(entities)
serialized graph calls, measured in whole minutes on large brains. Now a
single paged walk over every Contains edge (type-only, 1,000 per page)
feeds an in-memory group-by-target, and only actual defects mutate. The
verdicts are unchanged: a stale parent's edge is removed, a missing edge
is restored, duplicates cannot survive, and user knowledge edges are
never touched.
Pinned in tests/integration/vfs-containment-batched.test.ts: exact
removed/restored counts on a seeded defect tree, tree correctness after
the repair, user edges untouched, and the cost shape — related() call
count independent of the entity count.
CI / Integration + conformance (Node 22) (push) Failing after 16m55s
related() with a verb-type ARRAY returned edges for only the first type —
the storage fast paths collapsed `verbType` (and, in their sibling blocks,
`sourceId` and `targetId`) arrays to their first element, silently
dropping the rest of the ask. Every consumer passing a verb list
under-traversed with no error and no narration: the same quiet-loss class
as the graph-first paging defect, one seam over.
All four fast paths now union over the full requested set, deduped by
edge id, before the metadata filters and pagination run. Pinned in
tests/integration/related-verb-array.test.ts: the second requested type's
edge returns in both array orders, on the anchor side, the target side,
and the type-only path; a one-element array equals the scalar; no
duplicates on overlap; pagination walks the union consistently.
CI / Integration + conformance (Node 22) (push) Failing after 17m7s
CI / Bun (latest) (push) Successful in 12m24s
The recovery fold scanned the generation log from generation 1 at every
open, on the open's foreground — O(whole history) on long-lived brains
(measured at two minutes of a large brain's open). Now an advisory mark
records the log's head whenever the pending set drains to empty (and at
clean close when empty); recovery scans from the mark + 1. The mark is
advisory and monotone-safe: stale-low costs a longer scan, never a
marker. The fold itself moves behind the doors as a latched background
task — the embed worker starts when it settles, and awaitPendingEmbeds()
and close() wait on the latch first, so no caller can observe a
half-recovered set. A pending embed's outcome was always eventual;
moving its recovery off the foreground changes when the worker starts,
never whether a marker is honored.
Pinned in tests/integration/pending-embed-low-water.test.ts: the drain
writes the mark and the next open scans from mark + 1; a pending embed
enqueued after the mark survives an unclean stop; open arms the fold as
a background latch the barrier waits on; a clean close writes the mark
even without a drain.
CI / Integration + conformance (Node 22) (push) Failing after 16m56s
CI / Bun (latest) (push) Successful in 12m24s
With `connected` present, find() materialized the whole-store filtered id
list, paged it, hydrated the page, and only then intersected with the
neighbour set. Every such call paid O(store) for the filter and the
hydration of rows that were never neighbours, and a neighbour outside the
first page of the filtered STORE was silently dropped — the answer depended
on the store's order and the page size.
The neighbour set is now the candidate universe: resolved first from the
adjacency, the metadata filter evaluated over those ids only through the
provider's own evaluation (a new optional `filterIdsWithin` door on
MetadataIndexProvider; the reference index implements it from its own
getIdsForFilter so the two can never disagree; a provider without it is
served by the whole-store answer intersected here), `orderBy` sorts the
whole neighbour set before the page is cut, and the vector leg walks the
neighbours as its candidate set. The text leg of a hybrid find keeps its
post-intersection — it has no candidate door.
Pinned in tests/integration/find-connected-order.test.ts: paging reaches
every matching neighbour and never a non-neighbour; a `missing` negation is
evaluated over the neighbours; the index is asked about the neighbour ids
only and hydration is one page; orderBy sorts the whole set; the vector leg
stays inside the neighbours; an edgeless anchor answers [] before the
filter is asked.
CI / Integration + conformance (Node 22) (push) Failing after 17m25s
CI / Bun (latest) (push) Successful in 12m19s
persistCounts() was write-through on every count change with no
serialization, and the atomic writer named its temp file with millisecond
granularity. Two persists inside one millisecond shared the temp path: both
wrote it, the first rename consumed it, the second rename found nothing —
ENOENT, roughly 1,500 times a day on a busy production brain, with a full
ledger write per change behind it. No data was lost (the surviving rename
carried a complete ledger and the next change re-persisted), but the race
was real and the write rate absurd.
flushCounts() now runs exactly one persist at a time; requests arriving
during it collapse into one trailing pass that carries the burst's final
state — N changes cost at most two writes. writeFileAtomic() adds a
per-process sequence to the temp name so no two writes can share a path.
Pinned: a 25-change burst → ≤2 ledger writes, zero errors, ledger equal to
memory; parallel real writes land complete; three same-instant atomic
writes own three distinct temp paths.
CI / Integration + conformance (Node 22) (push) Failing after 16m54s
CI / Bun (latest) (push) Successful in 12m26s
transact()'s delete legs (direct unrelate and the noun-remove cascade) hand
the SAME verb object to the graph-retraction op and the metadata-retraction
op. The metadata leg sanitized at PLAN time, when the verb was still clean,
so the wrap returned the same reference — then the graph op's execute-time
endpoint resolution (deliberately deferred for same-batch forward refs)
mirrored BigInt sourceInt/targetInt onto the shared object, and the metadata
op crossed the seam with them. A strict provider rightly refuses that
crossing, so every transact-wrapped edge delete aborted; direct unrelate()
resolves ints at build time, before its sanitize, which is why no existing
gate saw it.
The JSON-safe view now lives in a shared leaf (utils/jsonSafeIndexMetadata)
and is applied INSIDE AddToMetadataIndexOperation and
RemoveFromMetadataIndexOperation at execute and rollback time — the one
place no plan-vs-execute ordering can bypass. Pins: the fleet repro, the
cascade shape, a mixed batch, and unit pins that mutate the entity after
construction against a strict seam (5 red before, 5 green after).
Diagnosis of the "packed history is damaged" narration that fires on every
run of the affected stores. It is a WRITER defect, and the reader's refusal
was the symptom rather than the cause.
A sealed segment declares one contiguous range [firstGeneration,
lastGeneration], and every reader treats that range as containment:
coveringSegment is an interval test, hasGeneration returns true for anything
inside it, and open() seeds committedRanges from it.
repackHistory handed fold() a SPARSE batch. Three filters punch holes in its
candidate list mid-run — a generation absent from committedRanges never
appears, one still in the pending buffer is skipped, one whose tx.json will
not read is skipped — and fold() then computed the range from the first and
last survivor, claiming every generation in between. The next open merged
that mis-declared range back into committedRanges, re-admitting the hole as
committed history, so the following auto-compaction pass asked the packed
tier for a frame that was never written and failed. Re-merged at every open,
which is why it repeated on every run.
Confirmed against a forensic fixture: generation directories 1..2503 present
except exactly one, 1416; and its fact-log segment already showed the tell —
seg-...1410.bfl declaring 1410..1940 (531 generations) while recording 530
facts.
Three changes:
- repackHistory folds each contiguous RUN as its own segment
(`contiguousRuns`), so ranges describe exactly what the segments contain.
- fold() REFUSES a non-contiguous batch, naming the gap and its width. The
density law is now mechanical, so no future caller can reintroduce it. A
refusal loses nothing: the generations stay live and readable.
- Stores already carrying the damage heal instead of wedging. A segment
whose declared span exceeds its frame count is SPARSE; `actualRanges()`
reads the real generation list from its sidecar so open() never re-admits
the holes, and readFrame reports such a hole as unpacked with a narration
naming the segment, rather than throwing. A DENSE segment missing a frame
is still loud damage — that one means the manifest and sidecar disagree.
Pins: nine unit cases (refusal and its message, honest ranges for separately
folded runs, a reconstructed pre-fix sparse segment serving its real frames
while reporting holes as unpacked, holes excluded from actualRanges, and the
dense-segment damage path still throwing) plus an end-to-end case that
deletes a generation directory and drives the real sequence — ordinary
close()-time repacking folds over the hole, then reopen and compact must both
complete. Verified red without the fix: the segment declared an
11-generation span while holding 10 frames.
(cherry picked from commit 9a888c37e9)
Two halves of one defect, found by a seeded-SIGKILL crash lane.
THE FALSE POSITIVE. stampEntityTree() recorded generationStore.generation()
— the ALLOCATED counter, a number a write in flight has claimed and may
never commit — while the JSDoc beside it already said the source is the
committed generation. Every crash inside a write window therefore produced
a spurious verdict at the next open: either 'sourceGeneration N is ahead of
the log head N-1' (the allocated generation died with the process) or
'rollup invariant nounCount: stamped X, observed Y' (the recovery fold
folded facts the stamp's counts predate). Both told the operator to run
repairIndex() — a whole-store recount — for a store that was coherent.
Measured before this commit: 4 of 11 SIGKILL cycles on a healthy store
raised one of the two. The stamp and the open now both read
committedGeneration(), which is what every other open-time watermark in the
class already reasons about.
THE TERMINAL VERDICT. A stamp still ahead of committed truth after the
recovery fold witnesses a generation that is not in the log — the stamp's
fsync outlived the tail's, and there is nothing to arrive. That is its own
verdict state now ('torn'), never folded in with 'incoherent': the two have
opposite cures. A writer open demotes it — the unusable stamped surface is
re-derived at the committed generation from the live counters, O(1),
straight-line, no loop and no await on external progress, narrated with
both count sets, the stamp's path and its committedAt. A read-only open
cannot re-stamp, so it says so and names the cure instead of guessing, and
still serves. Neither branch waits, and neither locks an owner out of a
canonical tree the stamp only describes.
Pins: the verifier returns the torn verdict with both generations; a
fabricated head-behind-source store narrates precisely, demotes inside a
bounded open, serves its rows, and is quiet at the next open (the demotion
converges); a read-only open narrates the same verdict and leaves the bytes
untouched.
(cherry picked from commit 298cb6daca)
The release gate's own output caught this: every test brain printed
"[VFS] old-root sweep complete in 1ms and recorded" — hundreds of lines — and
a consumer would get two of them on the first open of every store.
They were emitted on the always-visible channel, which a production log level
deliberately CANNOT silence. That channel exists so an operator can always
learn why a database is slow; a 0ms no-op on a fresh store is not that, and
announcing it there trains people to ignore the one channel built to be
impossible to ignore. It was also inconsistent with every other narration in
this work, all of which is silent under a threshold.
The sweep now speaks when it has something to say — duplicate roots removed, or
a wall over a second that a person watching a slow first open deserves
explained — and otherwise does its work, records its marker, and stays quiet.
cleanupOldRoots() reports what it removed so the decision rests on a fact
rather than on a guess.
Pin: a fresh store's sweep emits nothing on the channel and still records its
marker, so the silence can never be mistaken for the work being skipped.
Two gate failures on main, one real and one long-hidden.
THE HEALTH-GATE PIN encoded the old law — "narrates once per generation, twice
across a generation bump" — which the content-keyed dedupe deliberately
replaced. A provider's `generation` bumps on every ledger mutation and every
rebuild boundary, so keying narration on it re-printed an unchanged health line
on every read that consulted a busy provider, and let a provider that never
bumped suppress a line whose reasons had genuinely changed. The pin now asserts
BOTH directions: an unchanged verdict stays silent however the counter moves,
and a changed verdict is always heard.
THE VFS HYBRID-SEARCH SUITE configured its store with `options.basePath`, an
alias removed at the 8.0 major that configures nothing. The suite was therefore
never using its temp directory — it opened the DEFAULT store, shared with every
other run on the machine, and accumulated tens of thousands of rows until it
failed on that shared store's graph adjacency instead of on anything it tests.
It now passes `storage.path`. The suite drops from 6.5s to 0.3s, which is the
measure of how much foreign data it had been opening.
Neither failure was caused by the release branch; the first is the branch's own
behaviour change meeting its outdated pin, the second predates it.
Three idle-burn items from the steady-state audit, and one correction.
THE FLUSH-REQUEST WATCH (the strongest of them). It readdir'd the request
directory every 500 ms, per brain, for the life of every writer — armed on
every non-reader brain whether or not any inspector process existed. In a process holding many stores that is tens of directory reads per second
on a completely idle service, plus a stale-request GC on every one of them. It now
uses fs.watch, so the arrival itself wakes it and a request is seen SOONER
than the poll saw it. Two concessions ride along, both stated in the code: a
30s safety sweep (fs.watch drops events on some network and fuse filesystems,
and the GC needs a tick of its own — two orders of magnitude fewer reads than
the poll made), and a fall back to the original 500 ms poll, narrated, on a
filesystem that cannot watch at all, because an inspector whose request is
never seen waits forever.
THE WRITER HEARTBEAT goes 10s → 60s. It is observability ONLY — staleness is
decided by pid liveness and the fence compares pid + hostname, so no decision
anywhere reads the timestamp — and at 10s it was a lock-file write every ten
seconds per brain forever, for a value nothing computes with. An operator
still sees a heartbeat inside the minute.
THE HEALTH NARRATION dedupes by CONTENT, not by the provider's generation
counter. That counter bumps on every ledger mutation and rebuild boundary, so
a provider bumping it on routine work re-emitted the same unchanged line on
every read, while one that never bumped could suppress a line whose reasons
had genuinely changed. The generation is still reported; it no longer decides
whether the line is worth saying.
CORRECTION, and it is against my own earlier claim: the idle-flush commit read
a reported idle-CPU observation (many stores, no writes, a flush every ~35s,
over a core burned) as caused by
the flush path. That does not follow — this engine's cadence is write-driven
(every trigger runs through noteWriteForPersistence, which only a committed
write calls), so something was CALLING flush() on those brains and the caller
is still unidentified. The clean-flush gate makes such a call free; it does not
account for it. The code comments and the idle lane now say exactly that.
Pins: tests/integration/flush-watcher-event-driven.test.ts — an idle writer
makes at most one request-directory read in 8 seconds (the old poll made ~16),
and a dropped request is still acked well inside the safety sweep.
MEASURED on a 14,056-noun / 72,679-verb production-shaped store, measured
solo under an exclusive lock: the vfs-bootstrap phase cost 43,021 ms of a cold open and 52,696 ms of
a WARM REOPEN. What dominates it is a migration sweep — a filtered find() over
the whole store hunting for root directories created before the fixed root id
existed. A store either carries such duplicates or never will, and the sweep
ran on every open, forever, in the foreground.
It is now caused by the store's state instead of by the open count: a durable
marker under _system/ records that the sweep has run, and a store carrying it
never sweeps again. A store without one sweeps in the BACKGROUND — the sweep
only removes duplicate roots, nothing serves from them, and it was already
declared non-critical — narrated at both ends, with whenRootSweepSettled() for
anyone who needs to observe rather than race it. An adapter with no raw-object
door keeps the old behaviour: correctness over cost, never a silent skip.
Pins: tests/integration/vfs-root-sweep-once.test.ts — the sweep runs on the
first open and never on the second or third; a sweep slowed to 4s does not
delay the open.
Open Brainy's side of the API contract the accelerated engine published.
DECLARED: package.json carries "brainyContract": 1 and the engine states its
own via contractVersion() / BRAINY_CONTRACT_VERSION — two engines compare an
integer instead of probing prototypes, and a tool reads the package field
without importing the engine. Pinned so the two can never drift apart.
SERVED: hasAll, noneOf and excludes now work on the index path. The defect
underneath was worse than the reported divergence — the metadata index's
operator switch had NO DEFAULT CASE, so any operator without a case left the
field's match set at its initial [] and find() returned an empty page.
Documented, validator-accepted, matcher-implemented operators answering
silently wrong. hasAll intersects each element's posting set (an empty operand
is vacuously true of every row that has the field), noneOf complements their
union, excludes complements contains.
REFUSED BY NAME: startsWith, endsWith, matches and length raise
INVALID_QUERY naming the operator, the field and the reason. An equality/range
posting index cannot evaluate a substring, a pattern or an array length without
reading every row — which is the cost this path exists to avoid — so it refuses
rather than answering an empty page. Both engines now agree on all 25 tokens
and contract 1 has no remaining operator divergence. This is a visible change
for a consumer calling those four through find({ where }): an empty page
becomes a typed refusal.
EMITTED: scripts/emit-contract-manifest.mjs generates docs/api-contract.json
from the BUILT surface — prototype doors, exported error classes, the operator
sets read out of their single definitions, the field-addressing vocabulary, the
health verdicts. Nothing hand-maintained, so a diff between two manifests is a
diff between two engines. `--check` fails on a stale manifest, which makes the
announce-every-addition duty mechanical rather than remembered.
RATIFIED in docs/contract-1-ratification.md: the 41-of-57 required split with
the promise spelled out (a refusal is part of a door; deprecation is not
removal), the serving-withholding list confirmed exhaustive and identical, the
minor/major rule adopted with the announcement duty, the 30 storage seam
methods committed as supported surface until Stage 2, and a finding filed
against the spec — is / isNot / greaterEqual / lessEqual are listed there as
served aliases and have never existed in this engine, which throws
INVALID_QUERY on all four.
Follower to the self-rebuild deference. The open gate's consistency check —
"metadata index has 0 entries but storage has N entities" → CRITICAL + a forced
second rebuild — knew two states, migrating and not. A provider whose rebuild()
returns once the rebuild is OWNED AND RUNNING ONLINE (its doors refusing by
name while other families serve) legitimately reports 0 entries there, so every
first contact printed a false CRITICAL and kicked a redundant second rebuild.
The exemption rides the rebuild-progress hook, NOT isMigrating() — widening
that would hold every write and 503 the whole brain through the migration
snapshot, which is worse than the false alarm. The check's real class is
untouched: a provider reporting 0 entries with no rebuild in progress still
trips it.
The crash-recovery rebuild kick gets the same deference: a provider already
rebuilding itself from canonical is doing exactly that work, and the fold ran
in the generation store's open before any provider existed, so what it is
reading is the repaired canonical.
Pin: a provider stub reporting a rebuild and 0 entries opens with no CRITICAL
line and no second rebuild; the vacuous-stub case fails loudly.
MEASURED on a production store: a metadata provider that had to rebuild made
init() pay the ENTIRE rebuild on the foreground — 641 seconds — with every
other family idle behind it. The cause is a missing distinction: a provider
reporting serving:false because it is BUSY BUILDING ITSELF and one reporting
serving:false because it is BROKEN looked identical through healthReport(),
and both were answered the same way — call rebuild(), and wait for it.
The contract that tells them apart is one optional, synchronous, O(1) hook:
`rebuildInProgress(): ProviderRebuildProgress | null`, reporting a phase name
and whatever the provider actually measures (done/total/startedAt) — never an
estimate dressed as a fact. A provider without the hook behaves exactly as
before.
With it, a provider owns its own rebuild:
- the open gate neither starts a second rebuild nor waits for the provider's,
and narrates that it is not waiting and what will refuse meanwhile;
- init() returns and every other family serves;
- that family's doors refuse BY NAME, carrying the provider's own progress,
and say plainly that the door opens by itself and no action is needed —
distinct from a broken index, which names repairIndex();
- the epoch stamp does not advance while any family is still being built.
Nothing is ever served empty: a not-serving family refuses, as it already did.
Pins: tests/integration/open-does-not-wait-for-a-rebuilding-provider.test.ts —
init() returns in milliseconds against a provider claiming a 6s rebuild, brainy
starts no rebuild of its own, a filtered read refuses naming the phase and the
4,096/14,056 progress, and the door answers once the provider reports serving.
The pin fails loudly rather than vacuously if its stub never installs.
REPORTED from the field: a process holding many stores, with no writes for ten
minutes, printed "All indexes flushed to disk in 216-601ms" per store every
~35 seconds and burned over a core at idle. Every one of those flushes re-persisted
state identical to what was already on disk — the provider flushes, the
watermark stamps, the generation counter, the entity-tree stamp — because
flush() never asked whether anything had changed.
- flush() over a clean brain is now O(1) and silent: a dirty witness is set by
every committed write (both commit paths end at noteWriteForPersistence, and
the deferred-embed worker lands through the single-op path) and cleared by a
flush that runs. A write landing DURING a flush sets it again, so no write's
work is ever skipped — it is done by the next flush. Set before the policy
check, so a `'manual'` consumer's explicit flush is never a no-op it didn't
ask for.
- An explicit flush now tells the cadence it happened. It didn't, so the very
next write saw "30s since the last flush" and kicked a background flush with
nothing to do, and the idle timer fired two seconds later over writes the
explicit flush had already persisted.
- The graph adjacency index's auto-flush asks before it acts: two O(1) reads
of the LSM MemTables, and a tick over a quiet index returns without calling
into the trees at all.
assessProviderHealth is NOT timer-driven — it is a synchronous O(1) read of a
provider's own healthReport(), called on the read gate, so it costs nothing on
an idle brain. No change needed there.
Pins: tests/integration/idle-costs-nothing.test.ts — 90 idle seconds produce
zero flushes, zero provider calls and zero log lines; three explicit flushes
over a clean brain call no provider; one write earns exactly one flush.
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
On a production store (14,647 nouns / 73,070 verbs) a repairIndex() ran for
more than thirty minutes at roughly a full core with ZERO log lines between
its start and its end, while the read doors kept serving. The operator could
tell it was alive only from `top`, and could not tell which of its
single-threaded walks it was inside.
Same law as the open, applied to the repair:
- every phase announces itself BEFORE it works, naming what it is about to
walk (each canonical walk, the VFS containment reconciliation, each
provider's invariant pass);
- an unref'd heartbeat names the phase still running every 5s, for as long as
it runs;
- every phase reports its own wall, and that wall is carried in the TYPED
receipt as RepairFamilyReport.durationMs — a receipt that cannot say where
the time went is not a receipt;
- the whole repair's narration moves to the always-visible channel, so a
production log level cannot silence it.
The phases move into runRepairIndexPhases() so the heartbeat can live in a
finally around them; the public door and its report shape are unchanged apart
from the added durationMs.
Pins: tests/integration/repair-narration.test.ts — every checked family has a
start line, a finish line with its wall, and a numeric durationMs in the
receipt; a phase slowed to 6.5s produces a heartbeat naming it, with the
logger clamped to ERROR.
CI / Integration + conformance (Node 22) (push) Waiting to run
CI / Bun (latest) (push) Waiting to run
MEASURED on a real store: the ALL-visibility ledger read 14,231 nouns against
14,056 identity records and 72,729 verbs against 72,679 — exactly that store's
25 noun and 50 verb SCAR directories. Two copies of the same archive derived
different numbers (14,231 and 14,081), because each had been persisted at a
different moment under the old rule that counted one entity per id DIRECTORY.
A downstream index heal subtracted against that denominator and reported
remaining work that did not exist.
The scan already applies the right predicate — one entity per IDENTITY RECORD
(the metadata content leg), shared with pruneOrphanedEntities so the two agree
by construction. What was missing is that a ledger persisted under the old rule
was only FLAGGED suspect and then went on serving its wrong numbers for the
life of the store, waiting for an operator to run repairIndex.
- The ledger now derives itself honestly in the BACKGROUND after the open,
narrating start and finish with the correction it made. Background because
these scalars are denominators — no read is served from them — and because
walks exactly like these are how a 24,898-id store spent minutes of a
restart in silence. Observable via whenCountLedgerSettled(); nothing in the
read path waits on it.
- A derivation that raced a write refuses to stamp its number "exact": one
retry on a quiet store, then the ledger stays SUSPECT and says so, naming
repairIndex as the door that recounts under a barrier.
- The one derivation that CANNOT leave the foreground says why it cannot:
getNounCount()/getVerbCount() are served from it, and a background walk
would make a populated store answer "0 entities" — a wrong answer, not a
slow one. It narrates its start and its wall instead.
- counts.json is written temp+rename. A truncating write left a window —
measured at roughly 750ms after a flush or close — in which a concurrent
reader saw the file EMPTY; an unparseable ledger sends the next open down
the full-rescan path, so the cheapest file in the store was buying the most
expensive recovery.
- The writer lock's clean-close record is now consulted before the
same-process branch too: a restart reported "Re-acquiring writer lock ...
this is a bug" immediately after a clean close, sending an operator after a
leak that did not exist.
Pins: tests/integration/count-ledger-identity-record.test.ts (background
correction with scar and ghost fixtures, two copies of one archive agreeing,
counts.json never observed unparseable across 40 persists);
tests/integration/ledger-derivation-identity.test.ts updated to the new law —
the OPEN still never walks (proved by slowing the walk 1.2s and timing the
open), and the ledger heals behind it.
CI / Integration + conformance (Node 22) (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled
An operator watched a production service open a 16 GB store and print nothing
for three minutes before its first line of work. Two defects, both fixed here.
The narration was written to `prodLog.warn`, and every environment that looks
like production clamps the logger to ERROR — so the phase breakdown that would
have named the slow phase was composed and thrown away. `prodLog.narrate` is
always visible, like `error`: it carries the two things an operator is
entitled to hear from a database regardless of a cost setting — why it is slow
and what it is doing about it. `silent: true` still silences it; that is a
request, not a default.
And nothing spoke DURING a phase, only after the whole open. init() now runs
an unref'd heartbeat that every 5s names the phase currently running, its
elapsed wall and what it is paying for, plus one line per phase as it ends for
any phase over 2s. The generation-log fold's own progress and completion lines
move to the same channel and now carry their wall — they were invisible in
production, which is how an operator came to restart a converging fold three
times.
Pins: tests/integration/open-narration.test.ts — narrate() survives the clamp
that silences warn(); a 6.5s storage-init produces a heartbeat naming the
phase and a completion line naming its wall, with the logger clamped to ERROR.
