workflow_dispatch, runs-on gate-functional — a host-mode, Bun-only lane
with no Node.js runtime, so every step is plain git + bun in shell
rather than a JS-based action. Clones candidate and control, runs the
full vitest suite on each, enforces a >=3,000-collected guard per side,
and diffs the two fail lists for genuinely new reds. The lane's own
tripwire marker (host pressure — never our own red or green) is checked
before the verdict is printed, and the job cleans up its own checkouts
so repeat runs don't feed the lane's disk-budget trip.
CI / Integration + conformance (Node 22) (push) Failing after 17m24s
CI / Bun (latest) (push) Successful in 12m23s
The proximity search fetched its anchor through get(), which omits vectors
by default, then handed a zero-length vector to the index — every
find({ near }) refused with a dimension mismatch, for every caller. Found
by the Rust planner's first-contact pins comparing outcomes with and
without the planner on a refused shape. The anchor is now fetched with its
vector, and an anchor that has none refuses by name — a proximity search
around an unvectored row has no meaning and must not fail inside the index.
Pinned in tests/integration/find-near.test.ts.
CI / Integration + conformance (Node 22) (push) Failing after 17m18s
CI / Bun (latest) (push) Successful in 12m28s
The delta gate caught the backgrounded fold breaking six pinned
crash-recovery cases: a reopened brain must have its markers re-armed
when open() returns, and a background latch races every consumer of that
contract. The backgrounding is reverted; the low-water mark stays — it
is the part that kills the whole-history scan, and with it the
foreground fold costs the log's tail on any brain that has ever drained.
The unmarked first open after upgrade pays one full scan, once, and the
open narrates it as its own step.
CI / Integration + conformance (Node 22) (push) Failing after 17m4s
CI / Bun (latest) (push) Successful in 12m19s
The provider's read doors each serve one stage, so a find that consults three of
them crosses into the index three times and marshals a result set at every
crossing: a filter matching a hundred thousand rows builds a hundred thousand id
strings to return a page of twenty-five. An index able to decide the stage order
itself can answer the page in one call and build ids only for the page.
planFindPage is optional and additive, in the shape filterIdsWithin and
getIdSetForFilter already set. The hook sits above the branch selection, because
the branches are what decide stage order per call site and an index that plans
has to be asked before that choice is made. Absent — as it is on this engine's
own index — every find is served by the stage doors exactly as before, which is
what keeps this engine the ordering oracle for any index that implements one.
The contract the door must keep, written where an implementer will read it:
identical rows in identical order to what the stage doors would produce; the
graph-first law (neighbours are the candidate universe, the filter runs over
those ids, orderBy sorts the whole set, the page is cut last); null returned
BEFORE any work rather than instead of an answer; and emptyAt naming the stage
that produced an empty page, so the serving law is applied to the right index —
an empty graph answer is re-verified against the adjacency before it is
believed, and a filter-empty is not.
Pinned in tests/integration/find-planner-door.test.ts: absent changes nothing;
present it is asked first with normalized params, the hidden ids and the graph
provider; its page is used and hydrated in its order; a declining door leaves
the result identical to the no-door path; and the two emptyAt branches verify
the adjacency, or correctly do not.
CI / Integration + conformance (Node 22) (push) Failing after 16m58s
CI / Bun (latest) (push) Successful in 12m24s
Pass 2 issued one awaited related({ to }) per VFS entity — O(entities)
serialized graph calls, measured in whole minutes on large brains. Now a
single paged walk over every Contains edge (type-only, 1,000 per page)
feeds an in-memory group-by-target, and only actual defects mutate. The
verdicts are unchanged: a stale parent's edge is removed, a missing edge
is restored, duplicates cannot survive, and user knowledge edges are
never touched.
Pinned in tests/integration/vfs-containment-batched.test.ts: exact
removed/restored counts on a seeded defect tree, tree correctness after
the repair, user edges untouched, and the cost shape — related() call
count independent of the entity count.
CI / Integration + conformance (Node 22) (push) Failing after 16m55s
related() with a verb-type ARRAY returned edges for only the first type —
the storage fast paths collapsed `verbType` (and, in their sibling blocks,
`sourceId` and `targetId`) arrays to their first element, silently
dropping the rest of the ask. Every consumer passing a verb list
under-traversed with no error and no narration: the same quiet-loss class
as the graph-first paging defect, one seam over.
All four fast paths now union over the full requested set, deduped by
edge id, before the metadata filters and pagination run. Pinned in
tests/integration/related-verb-array.test.ts: the second requested type's
edge returns in both array orders, on the anchor side, the target side,
and the type-only path; a one-element array equals the scalar; no
duplicates on overlap; pagination walks the union consistently.
CI / Integration + conformance (Node 22) (push) Failing after 17m7s
CI / Bun (latest) (push) Successful in 12m24s
The recovery fold scanned the generation log from generation 1 at every
open, on the open's foreground — O(whole history) on long-lived brains
(measured at two minutes of a large brain's open). Now an advisory mark
records the log's head whenever the pending set drains to empty (and at
clean close when empty); recovery scans from the mark + 1. The mark is
advisory and monotone-safe: stale-low costs a longer scan, never a
marker. The fold itself moves behind the doors as a latched background
task — the embed worker starts when it settles, and awaitPendingEmbeds()
and close() wait on the latch first, so no caller can observe a
half-recovered set. A pending embed's outcome was always eventual;
moving its recovery off the foreground changes when the worker starts,
never whether a marker is honored.
Pinned in tests/integration/pending-embed-low-water.test.ts: the drain
writes the mark and the next open scans from mark + 1; a pending embed
enqueued after the mark survives an unclean stop; open arms the fold as
a background latch the barrier waits on; a clean close writes the mark
even without a drain.
CI / Integration + conformance (Node 22) (push) Failing after 17m3s
CI / Bun (latest) (push) Successful in 12m20s
The fleet's releases wall reads one public URL per product. These files are
that door for Brainy and Open Brainy: newest first, honest history from the
changelog, one entry appended by every release from here on.
CI / Integration + conformance (Node 22) (push) Failing after 16m56s
CI / Bun (latest) (push) Successful in 12m24s
With `connected` present, find() materialized the whole-store filtered id
list, paged it, hydrated the page, and only then intersected with the
neighbour set. Every such call paid O(store) for the filter and the
hydration of rows that were never neighbours, and a neighbour outside the
first page of the filtered STORE was silently dropped — the answer depended
on the store's order and the page size.
The neighbour set is now the candidate universe: resolved first from the
adjacency, the metadata filter evaluated over those ids only through the
provider's own evaluation (a new optional `filterIdsWithin` door on
MetadataIndexProvider; the reference index implements it from its own
getIdsForFilter so the two can never disagree; a provider without it is
served by the whole-store answer intersected here), `orderBy` sorts the
whole neighbour set before the page is cut, and the vector leg walks the
neighbours as its candidate set. The text leg of a hybrid find keeps its
post-intersection — it has no candidate door.
Pinned in tests/integration/find-connected-order.test.ts: paging reaches
every matching neighbour and never a non-neighbour; a `missing` negation is
evaluated over the neighbours; the index is asked about the neighbour ids
only and hydration is one page; orderBy sorts the whole set; the vector leg
stays inside the neighbours; an edgeless anchor answers [] before the
filter is asked.
CI / Integration + conformance (Node 22) (push) Failing after 17m25s
CI / Bun (latest) (push) Successful in 12m19s
persistCounts() was write-through on every count change with no
serialization, and the atomic writer named its temp file with millisecond
granularity. Two persists inside one millisecond shared the temp path: both
wrote it, the first rename consumed it, the second rename found nothing —
ENOENT, roughly 1,500 times a day on a busy production brain, with a full
ledger write per change behind it. No data was lost (the surviving rename
carried a complete ledger and the next change re-persisted), but the race
was real and the write rate absurd.
flushCounts() now runs exactly one persist at a time; requests arriving
during it collapse into one trailing pass that carries the burst's final
state — N changes cost at most two writes. writeFileAtomic() adds a
per-process sequence to the temp name so no two writes can share a path.
Pinned: a 25-change burst → ≤2 ledger writes, zero errors, ledger equal to
memory; parallel real writes land complete; three same-instant atomic
writes own three distinct temp paths.
CI / Integration + conformance (Node 22) (push) Failing after 16m54s
CI / Bun (latest) (push) Successful in 12m26s
transact()'s delete legs (direct unrelate and the noun-remove cascade) hand
the SAME verb object to the graph-retraction op and the metadata-retraction
op. The metadata leg sanitized at PLAN time, when the verb was still clean,
so the wrap returned the same reference — then the graph op's execute-time
endpoint resolution (deliberately deferred for same-batch forward refs)
mirrored BigInt sourceInt/targetInt onto the shared object, and the metadata
op crossed the seam with them. A strict provider rightly refuses that
crossing, so every transact-wrapped edge delete aborted; direct unrelate()
resolves ints at build time, before its sanitize, which is why no existing
gate saw it.
The JSON-safe view now lives in a shared leaf (utils/jsonSafeIndexMetadata)
and is applied INSIDE AddToMetadataIndexOperation and
RemoveFromMetadataIndexOperation at execute and rollback time — the one
place no plan-vs-execute ordering can bypass. Pins: the fleet repro, the
cascade shape, a mixed batch, and unit pins that mutate the entity after
construction against a strict seam (5 red before, 5 green after).
CI / Integration + conformance (Node 22) (push) Failing after 17m1s
CI / Bun (latest) (push) Successful in 12m26s
The one-doc-set ruling (2026-08-31) gives soulcraft.com/docs to the paid
product alone; the site serves redirects for the slugs this rail used to
push. The push script stays in the tree as history; the rail stops calling
it.
(cherry picked from commit 655aa13ea7)
Diagnosis of the "packed history is damaged" narration that fires on every
run of the affected stores. It is a WRITER defect, and the reader's refusal
was the symptom rather than the cause.
A sealed segment declares one contiguous range [firstGeneration,
lastGeneration], and every reader treats that range as containment:
coveringSegment is an interval test, hasGeneration returns true for anything
inside it, and open() seeds committedRanges from it.
repackHistory handed fold() a SPARSE batch. Three filters punch holes in its
candidate list mid-run — a generation absent from committedRanges never
appears, one still in the pending buffer is skipped, one whose tx.json will
not read is skipped — and fold() then computed the range from the first and
last survivor, claiming every generation in between. The next open merged
that mis-declared range back into committedRanges, re-admitting the hole as
committed history, so the following auto-compaction pass asked the packed
tier for a frame that was never written and failed. Re-merged at every open,
which is why it repeated on every run.
Confirmed against a forensic fixture: generation directories 1..2503 present
except exactly one, 1416; and its fact-log segment already showed the tell —
seg-...1410.bfl declaring 1410..1940 (531 generations) while recording 530
facts.
Three changes:
- repackHistory folds each contiguous RUN as its own segment
(`contiguousRuns`), so ranges describe exactly what the segments contain.
- fold() REFUSES a non-contiguous batch, naming the gap and its width. The
density law is now mechanical, so no future caller can reintroduce it. A
refusal loses nothing: the generations stay live and readable.
- Stores already carrying the damage heal instead of wedging. A segment
whose declared span exceeds its frame count is SPARSE; `actualRanges()`
reads the real generation list from its sidecar so open() never re-admits
the holes, and readFrame reports such a hole as unpacked with a narration
naming the segment, rather than throwing. A DENSE segment missing a frame
is still loud damage — that one means the manifest and sidecar disagree.
Pins: nine unit cases (refusal and its message, honest ranges for separately
folded runs, a reconstructed pre-fix sparse segment serving its real frames
while reporting holes as unpacked, holes excluded from actualRanges, and the
dense-segment damage path still throwing) plus an end-to-end case that
deletes a generation directory and drives the real sequence — ordinary
close()-time repacking folds over the hole, then reopen and compact must both
complete. Verified red without the fix: the segment declared an
11-generation span while holding 10 frames.
(cherry picked from commit 9a888c37e9)
Two halves of one defect, found by a seeded-SIGKILL crash lane.
THE FALSE POSITIVE. stampEntityTree() recorded generationStore.generation()
— the ALLOCATED counter, a number a write in flight has claimed and may
never commit — while the JSDoc beside it already said the source is the
committed generation. Every crash inside a write window therefore produced
a spurious verdict at the next open: either 'sourceGeneration N is ahead of
the log head N-1' (the allocated generation died with the process) or
'rollup invariant nounCount: stamped X, observed Y' (the recovery fold
folded facts the stamp's counts predate). Both told the operator to run
repairIndex() — a whole-store recount — for a store that was coherent.
Measured before this commit: 4 of 11 SIGKILL cycles on a healthy store
raised one of the two. The stamp and the open now both read
committedGeneration(), which is what every other open-time watermark in the
class already reasons about.
THE TERMINAL VERDICT. A stamp still ahead of committed truth after the
recovery fold witnesses a generation that is not in the log — the stamp's
fsync outlived the tail's, and there is nothing to arrive. That is its own
verdict state now ('torn'), never folded in with 'incoherent': the two have
opposite cures. A writer open demotes it — the unusable stamped surface is
re-derived at the committed generation from the live counters, O(1),
straight-line, no loop and no await on external progress, narrated with
both count sets, the stamp's path and its committedAt. A read-only open
cannot re-stamp, so it says so and names the cure instead of guessing, and
still serves. Neither branch waits, and neither locks an owner out of a
canonical tree the stamp only describes.
Pins: the verifier returns the torn verdict with both generations; a
fabricated head-behind-source store narrates precisely, demotes inside a
bounded open, serves its rows, and is quiet at the next open (the demotion
converges); a read-only open narrates the same verdict and leaves the bytes
untouched.
(cherry picked from commit 298cb6daca)
CI / Integration + conformance (Node 22) (push) Failing after 16m55s
CI / Bun (latest) (push) Successful in 12m28s
The one-doc-set ruling (2026-08-31) gives soulcraft.com/docs to the paid
product alone; the site serves redirects for the slugs this rail used to
push. The push script stays in the tree as history; the rail stops calling
it.
CI / Integration + conformance (Node 22) (push) Failing after 16m55s
Diagnosis of the "packed history is damaged" narration that fires on every
run of the affected stores. It is a WRITER defect, and the reader's refusal
was the symptom rather than the cause.
A sealed segment declares one contiguous range [firstGeneration,
lastGeneration], and every reader treats that range as containment:
coveringSegment is an interval test, hasGeneration returns true for anything
inside it, and open() seeds committedRanges from it.
repackHistory handed fold() a SPARSE batch. Three filters punch holes in its
candidate list mid-run — a generation absent from committedRanges never
appears, one still in the pending buffer is skipped, one whose tx.json will
not read is skipped — and fold() then computed the range from the first and
last survivor, claiming every generation in between. The next open merged
that mis-declared range back into committedRanges, re-admitting the hole as
committed history, so the following auto-compaction pass asked the packed
tier for a frame that was never written and failed. Re-merged at every open,
which is why it repeated on every run.
Confirmed against a forensic fixture: generation directories 1..2503 present
except exactly one, 1416; and its fact-log segment already showed the tell —
seg-...1410.bfl declaring 1410..1940 (531 generations) while recording 530
facts.
Three changes:
- repackHistory folds each contiguous RUN as its own segment
(`contiguousRuns`), so ranges describe exactly what the segments contain.
- fold() REFUSES a non-contiguous batch, naming the gap and its width. The
density law is now mechanical, so no future caller can reintroduce it. A
refusal loses nothing: the generations stay live and readable.
- Stores already carrying the damage heal instead of wedging. A segment
whose declared span exceeds its frame count is SPARSE; `actualRanges()`
reads the real generation list from its sidecar so open() never re-admits
the holes, and readFrame reports such a hole as unpacked with a narration
naming the segment, rather than throwing. A DENSE segment missing a frame
is still loud damage — that one means the manifest and sidecar disagree.
Pins: nine unit cases (refusal and its message, honest ranges for separately
folded runs, a reconstructed pre-fix sparse segment serving its real frames
while reporting holes as unpacked, holes excluded from actualRanges, and the
dense-segment damage path still throwing) plus an end-to-end case that
deletes a generation directory and drives the real sequence — ordinary
close()-time repacking folds over the hole, then reopen and compact must both
complete. Verified red without the fix: the segment declared an
11-generation span while holding 10 frames.
CI / Integration + conformance (Node 22) (push) Failing after 16m58s
CI / Bun (latest) (push) Successful in 12m23s
Two halves of one defect, found by a seeded-SIGKILL crash lane.
THE FALSE POSITIVE. stampEntityTree() recorded generationStore.generation()
— the ALLOCATED counter, a number a write in flight has claimed and may
never commit — while the JSDoc beside it already said the source is the
committed generation. Every crash inside a write window therefore produced
a spurious verdict at the next open: either 'sourceGeneration N is ahead of
the log head N-1' (the allocated generation died with the process) or
'rollup invariant nounCount: stamped X, observed Y' (the recovery fold
folded facts the stamp's counts predate). Both told the operator to run
repairIndex() — a whole-store recount — for a store that was coherent.
Measured before this commit: 4 of 11 SIGKILL cycles on a healthy store
raised one of the two. The stamp and the open now both read
committedGeneration(), which is what every other open-time watermark in the
class already reasons about.
THE TERMINAL VERDICT. A stamp still ahead of committed truth after the
recovery fold witnesses a generation that is not in the log — the stamp's
fsync outlived the tail's, and there is nothing to arrive. That is its own
verdict state now ('torn'), never folded in with 'incoherent': the two have
opposite cures. A writer open demotes it — the unusable stamped surface is
re-derived at the committed generation from the live counters, O(1),
straight-line, no loop and no await on external progress, narrated with
both count sets, the stamp's path and its committedAt. A read-only open
cannot re-stamp, so it says so and names the cure instead of guessing, and
still serves. Neither branch waits, and neither locks an owner out of a
canonical tree the stamp only describes.
Pins: the verifier returns the torn verdict with both generations; a
fabricated head-behind-source store narrates precisely, demotes inside a
bounded open, serves its rows, and is quiet at the next open (the demotion
converges); a read-only open narrates the same verdict and leaves the bytes
untouched.
CI / Integration + conformance (Node 22) (push) Failing after 17m2s
Step 11 POSTed to repos/soulcraft/brainy while printing the correct URL; dormant only because FORGEJO_RELEASE_TOKEN was unset. Found during the 10.4.4 cut verification.
The release gate's own output caught this: every test brain printed
"[VFS] old-root sweep complete in 1ms and recorded" — hundreds of lines — and
a consumer would get two of them on the first open of every store.
They were emitted on the always-visible channel, which a production log level
deliberately CANNOT silence. That channel exists so an operator can always
learn why a database is slow; a 0ms no-op on a fresh store is not that, and
announcing it there trains people to ignore the one channel built to be
impossible to ignore. It was also inconsistent with every other narration in
this work, all of which is silent under a threshold.
The sweep now speaks when it has something to say — duplicate roots removed, or
a wall over a second that a person watching a slow first open deserves
explained — and otherwise does its work, records its marker, and stays quiet.
cleanupOldRoots() reports what it removed so the decision rests on a fact
rather than on a guess.
Pin: a fresh store's sweep emits nothing on the channel and still records its
marker, so the silence can never be mistaken for the work being skipped.
Two gate failures on main, one real and one long-hidden.
THE HEALTH-GATE PIN encoded the old law — "narrates once per generation, twice
across a generation bump" — which the content-keyed dedupe deliberately
replaced. A provider's `generation` bumps on every ledger mutation and every
rebuild boundary, so keying narration on it re-printed an unchanged health line
on every read that consulted a busy provider, and let a provider that never
bumped suppress a line whose reasons had genuinely changed. The pin now asserts
BOTH directions: an unchanged verdict stays silent however the counter moves,
and a changed verdict is always heard.
THE VFS HYBRID-SEARCH SUITE configured its store with `options.basePath`, an
alias removed at the 8.0 major that configures nothing. The suite was therefore
never using its temp directory — it opened the DEFAULT store, shared with every
other run on the machine, and accumulated tens of thousands of rows until it
failed on that shared store's graph adjacency instead of on anything it tests.
It now passes `storage.path`. The suite drops from 6.5s to 0.3s, which is the
measure of how much foreign data it had been opening.
Neither failure was caused by the release branch; the first is the branch's own
behaviour change meeting its outdated pin, the second predates it.
Correctness and observability for 10.4.4: the writer-lock clean-close record
and the always-run terminal releases, the self-healing count ledger and an
atomic counts.json, always-visible open and repair narration, the non-blocking
open for a provider rebuilding itself, an event-driven flush-request watch, and
the operator-visible operator set (three served, four refused by name).
CI / Integration + conformance (Node 22) (push) Failing after 17m7s
CI / Bun (latest) (push) Successful in 12m19s
The manifest's prose pointer named a document that answers a confidential
specification, and such a document does not belong in a public repository even
in summary. The pointer is dropped — the manifest is generated from this
engine's own surface and is self-describing — and the requirement marking it
deliberately omits is recorded with the contract's owner rather than here.
The standard is written down so this is not relitigated per document.
A release audit found hostnames, store identities and operational anecdotes in
this branch's commit messages — not trade secrets, but nothing a public
repository's permanent history should carry either. The messages were rewritten
to keep every number and drop every provenance; the rule is written down here
so the next measurement does not have to be caught by an audit.
MEASURED on a 14,056-noun / 72,679-verb production-shaped store, measured
solo under an exclusive lock: the vfs-bootstrap
phase costs 37.8s on main and 38.0s on this branch — unchanged — and NO
"vfs.init" step line was emitted at all, meaning the VFS's own init fell under
the 2s narration threshold. The phase is therefore almost entirely NOT the VFS,
and the old-root sweep this branch moved to the background was never what made
it expensive.
What else lives in that span is now named: the log-authority artifact read, the
adoption ORACLE (which verifies the log against canonical before flipping a
brain to durable-at-ack), the legacy pending-embed sidecar bridge, and the
pending-embed recovery fold. One of those holds ~38 seconds of every open of
this store and the next measurement will say which, by name, instead of leaving
a phase label to be guessed at.
The safety sweep is armed alongside the watch, and startFlushRequestPolling()
declines to arm over an existing interval — so when a watch died mid-life the
fallback did nothing and the store quietly answered flush requests on a 30s
cadence instead of the 500ms one the door promises. The sweep is cleared first.
A degrade nobody asked for is still a degrade.
Arming is asynchronous — the request directory is created before it can be
watched — so during that window neither the watcher nor the sweep interval
exists yet and the guard let a second call through, leaving two watchers and
two sweeps for the life of the store. The callback is the flag that covers the
window.
Three idle-burn items from the steady-state audit, and one correction.
THE FLUSH-REQUEST WATCH (the strongest of them). It readdir'd the request
directory every 500 ms, per brain, for the life of every writer — armed on
every non-reader brain whether or not any inspector process existed. In a process holding many stores that is tens of directory reads per second
on a completely idle service, plus a stale-request GC on every one of them. It now
uses fs.watch, so the arrival itself wakes it and a request is seen SOONER
than the poll saw it. Two concessions ride along, both stated in the code: a
30s safety sweep (fs.watch drops events on some network and fuse filesystems,
and the GC needs a tick of its own — two orders of magnitude fewer reads than
the poll made), and a fall back to the original 500 ms poll, narrated, on a
filesystem that cannot watch at all, because an inspector whose request is
never seen waits forever.
THE WRITER HEARTBEAT goes 10s → 60s. It is observability ONLY — staleness is
decided by pid liveness and the fence compares pid + hostname, so no decision
anywhere reads the timestamp — and at 10s it was a lock-file write every ten
seconds per brain forever, for a value nothing computes with. An operator
still sees a heartbeat inside the minute.
THE HEALTH NARRATION dedupes by CONTENT, not by the provider's generation
counter. That counter bumps on every ledger mutation and rebuild boundary, so
a provider bumping it on routine work re-emitted the same unchanged line on
every read, while one that never bumped could suppress a line whose reasons
had genuinely changed. The generation is still reported; it no longer decides
whether the line is worth saying.
CORRECTION, and it is against my own earlier claim: the idle-flush commit read
a reported idle-CPU observation (many stores, no writes, a flush every ~35s,
over a core burned) as caused by
the flush path. That does not follow — this engine's cadence is write-driven
(every trigger runs through noteWriteForPersistence, which only a committed
write calls), so something was CALLING flush() on those brains and the caller
is still unidentified. The clean-flush gate makes such a call free; it does not
account for it. The code comments and the idle lane now say exactly that.
Pins: tests/integration/flush-watcher-event-driven.test.ts — an idle writer
makes at most one request-directory read in 8 seconds (the old poll made ~16),
and a dropped request is still acked well inside the safety sweep.
The 7.x-to-8.0 layout probe runs on the open path of every store that does not
yet carry its completion marker — a restore, a store built by an older release
— and asked whether the canonical tree holds anything by LISTING it: a
recursive walk of every file in every entity directory, to learn a boolean. It
now asks the one-level door added for generation discovery, falling back to the
listing on an adapter that lacks it.
Also files a defect found while ratifying the operator set: the VFS builds its
path-prefix filter as `$startsWith`, an operator no engine spelling accepts,
so vfs.searchFiles({ path }) throws INVALID_QUERY on every call that passes a
path. Pre-existing, unrelated to the operator work, and left as a filing —
a path-prefix search needs a design answer, not a spelling correction.
MEASURED on a production-shaped store (14,056 nouns / 72,679 verbs, an 11 GB
generation history), measured solo under an exclusive lock: the generation-store phase cost 55,538 ms of a WARM
REOPEN after a clean close — with the fold correctly skipped, so nothing in
that phase's name explained it.
This is what it was doing. Discovering which generations exist on disk called
listRawObjects('_generations'), which RECURSES the whole tree and returns
every file in every generation directory — to extract a set of integers that
the top-level directory NAMES already spell out. The cost scales with the
entire history, is paid on every open, warm or cold, and grows for the life of
the store.
A one-level door — listRawPrefixes(prefix), the immediate child directory
names — is added to the storage seam. The filesystem adapter answers it with a
single readdir; BaseStorage derives it from the recursive listing, so an
adapter without a cheap implementation is never wrong, only never faster; and
the generation store falls back to the old listing when the door is absent.
One behavioural difference, stated: an EMPTY generation directory is now
discovered where the file listing could not see it. Above the committed
watermark that is a crash scar, and recovery already has an explicit branch
for it ("indeterminate partial dir" — dropped, narrated). Below it, it becomes
a resolvable generation holding no records, which is what an empty generation
means.
Suites: the durability kill matrix (15), db-mvcc (30), history repacking (4),
rollback trapdoor (3), entity-tree stamp (4) and the full unit suite (2,105)
all green.
Both mutate durable state outside the two commit paths, so neither was seen by
the flush witness added with the idle-flush law. A clear() followed by a
flush() would have found the brain "clean" and skipped the entity-tree stamp,
leaving a stamp describing the population the clear had just removed — a false
divergence warning at the next open. Closing the gap where it is, rather than
widening the witness to guess.
A phase that costs a minute and names only itself tells an operator where to
look but not what to look at. MEASURED on a real 14,056-noun / 72,679-verb
store, the warm reopen's generation-store phase cost 55,538 ms with nothing
inside it named — the fold was skipped (the close was clean), so the cost was
somewhere else entirely and the breakdown could not say where.
Six steps inside the open now report their own wall with their own cause when
they exceed the phase threshold: the generation store's open (manifest,
committed ranges, fact log, packed tier, crash replay), the entity-tree stamp
verification, the brain-format read, the pre-upgrade backup, the derived-index
gate, and the VFS init. Silent under the threshold, so a fast open says nothing
extra. Same always-visible channel as the phase lines.
MEASURED on a 14,056-noun / 72,679-verb production-shaped store, measured
solo under an exclusive lock: the vfs-bootstrap phase cost 43,021 ms of a cold open and 52,696 ms of
a WARM REOPEN. What dominates it is a migration sweep — a filtered find() over
the whole store hunting for root directories created before the fixed root id
existed. A store either carries such duplicates or never will, and the sweep
ran on every open, forever, in the foreground.
It is now caused by the store's state instead of by the open count: a durable
marker under _system/ records that the sweep has run, and a store carrying it
never sweeps again. A store without one sweeps in the BACKGROUND — the sweep
only removes duplicate roots, nothing serves from them, and it was already
declared non-critical — narrated at both ends, with whenRootSweepSettled() for
anyone who needs to observe rather than race it. An adapter with no raw-object
door keeps the old behaviour: correctness over cost, never a silent skip.
Pins: tests/integration/vfs-root-sweep-once.test.ts — the sweep runs on the
first open and never on the second or third; a sweep slowed to 4s does not
delay the open.
Open Brainy's side of the API contract the accelerated engine published.
DECLARED: package.json carries "brainyContract": 1 and the engine states its
own via contractVersion() / BRAINY_CONTRACT_VERSION — two engines compare an
integer instead of probing prototypes, and a tool reads the package field
without importing the engine. Pinned so the two can never drift apart.
SERVED: hasAll, noneOf and excludes now work on the index path. The defect
underneath was worse than the reported divergence — the metadata index's
operator switch had NO DEFAULT CASE, so any operator without a case left the
field's match set at its initial [] and find() returned an empty page.
Documented, validator-accepted, matcher-implemented operators answering
silently wrong. hasAll intersects each element's posting set (an empty operand
is vacuously true of every row that has the field), noneOf complements their
union, excludes complements contains.
REFUSED BY NAME: startsWith, endsWith, matches and length raise
INVALID_QUERY naming the operator, the field and the reason. An equality/range
posting index cannot evaluate a substring, a pattern or an array length without
reading every row — which is the cost this path exists to avoid — so it refuses
rather than answering an empty page. Both engines now agree on all 25 tokens
and contract 1 has no remaining operator divergence. This is a visible change
for a consumer calling those four through find({ where }): an empty page
becomes a typed refusal.
EMITTED: scripts/emit-contract-manifest.mjs generates docs/api-contract.json
from the BUILT surface — prototype doors, exported error classes, the operator
sets read out of their single definitions, the field-addressing vocabulary, the
health verdicts. Nothing hand-maintained, so a diff between two manifests is a
diff between two engines. `--check` fails on a stale manifest, which makes the
announce-every-addition duty mechanical rather than remembered.
RATIFIED in docs/contract-1-ratification.md: the 41-of-57 required split with
the promise spelled out (a refusal is part of a door; deprecation is not
removal), the serving-withholding list confirmed exhaustive and identical, the
minor/major rule adopted with the announcement duty, the 30 storage seam
methods committed as supported surface until Stage 2, and a finding filed
against the spec — is / isNot / greaterEqual / lessEqual are listed there as
served aliases and have never existed in this engine, which throws
INVALID_QUERY on all four.
