Four things, none of them a clock:
1. A brain with one permanently-stuck pending id, closed cleanly and
reopened, scans ONLY the facts after the checkpoint — read from the
fold's own accounting. The same fixture pins the DEFECT it cures: no
low-water mark exists on that brain, because it never drained, so nothing
could have shortened its fold. A second row proves the bound stays
O(delta) across repeated opens while the id is still stuck.
2. A crash matrix in a REAL child process (detached group, SIGKILL, no
close), following writer-lock-clean-close's pattern: killed before any
checkpoint was written, killed after one with an embed landed and flushed
above it, and killed after one with an UN-FLUSHED tail. The invariant in
every row is differential — the checkpoint-bounded fold the reopened
brain actually ran equals a full fold from generation 1 over the same
recovered log.
3. A torn checkpoint (bytes that are neither gzip nor JSON) falls back
loudly — the adapter's torn-record gauge and production error, plus the
fold's own narration of the bound it used — and still recovers the marker
from the log. A well-formed but shape-invalid checkpoint is refused
WHOLE: trusting its generation while ignoring its list is the one shape
that could bound a scan behind a set that was never recovered.
4. The existing low-water pins pass unchanged — the mark is still written
and still read, now as the fallback bound beneath the checkpoint.
The low-water mark shipped in 10.4.9 can only be written when the pending
set is EMPTY, because it carries no set — it means "everything at or below
G is consumed". A brain holding even one id that never lands (an embed that
keeps failing, a data-less row reaped in memory only and re-folded every
open) never drains, so it never writes a mark, so the bound never engaged on
exactly the brains whose fold is expensive: `recover-pending-embeds` re-read
the WHOLE fact log at every open, on the open's foreground.
_system/pending_embeds_checkpoint.json carries the set: { generation,
pending, writtenAt } = "as of durable generation G the pending set was
exactly this list". Open seeds the set from the list and scans from G + 1,
so the fold is O(facts since G) whether or not the set ever drains. Measured
on a 301-row brain with one stuck id: 302 facts read before, 0 after; at 601
rows, 602 before, 0 after — same pending set both ways.
THE DURABILITY LAW, by construction. A checkpoint at head H taken while the
facts up to H are still buffered would be read back after a crash that
truncated the tail: an `embed.landed` in a truncated fact would be gone from
the log while the checkpoint still recorded its id as landed, and its
landing vector went with the fact — a LOST VECTOR. So a capture is refused
unless `0 < head <= committed`, the manifest watermark below which
FactLog.open() never truncates and which the group-commit flush only
advances after fsyncing the log. The (generation, set) pair is taken in one
synchronous instant with no await between reading the generations and
snapshotting the set. The one remaining asymmetry runs the safe way: an id
enqueued in memory whose marker lands at G+1 is captured as pending at G —
one idempotent re-embed, never a loss.
Written at clean close (inside closeDurableSteps, after the generation
store's own close flushed the log and advanced the manifest), at
drain-to-empty, and on a cadence of max(64, ceil(|pending| / 64))
transitions while open — an interval that holds the mechanism's amortized
cost at <= 64 ids written per transition however large the backlog grows, so
the cure cannot reintroduce the defect class it fixes. No timer, no knob.
The debt stays armed across attempts the durability law refuses, so a write
burst does not skip a checkpoint, it defers it.
Degradation is loud and always toward a LONGER scan: a torn checkpoint
throws typed on read (the adapter's tmp+rename write means it can never
parse into a partial list) and a malformed one is refused whole, both
falling back to the low-water mark — still written, still read — and then to
generation 1. The fold narrates which bound applied and how many facts it
read, on every open, so a bound that stops engaging is visible instead of
silent.
The worker's orphan reap splits: a row that is GONE clears durably (its
tombstone is in the log, or its create never was), while a present-but
data-less row keeps clearing in memory only and is carried in the checkpoint
list, so the bounded fold and a full fold from generation 1 agree exactly.
The crash-recovery contract is unchanged: the fold stays on the open's
foreground, markers re-armed when open() returns.
Every log-authority open asks the fact log one question — is there a fact
above the committed pointer? — and answered it by reading and CRC-decoding
EVERY segment file the manifest names. MEASURED in production on a 16k-row
brain at generation ~478,819: 34-37 seconds inside `generation-store-open-fold`
on every open, including the clean one where the answer is always "nothing".
The manifest already knows. A sealed segment's `lastGeneration` is written at
seal time, and the seal order has been the same since the log was introduced:
`rotate()` fsyncs the tail's bytes FIRST ("sealed segments are always fully
durable"), builds the entry from the content that fsync covered, and only then
flips the manifest — atomically, fsynced, and in the same write re-pointing
`tailSegment`, so a sealed file is never appended to again. A crash in that
order is safe in the pruning direction: before the manifest write the segment
is still the TAIL and is read whole; after it, the entry describes bytes that
were already durable. The only later mutation of a sealed segment is open()'s
straddle truncation, which removes facts and re-derives the entry from the
actual bytes — a recorded bound can drift DOWN with its file, never up.
So `lastGeneration = L` proves the file holds no fact above L, and both
manifest-direct passes (`peekFactsAbove` and its streaming twin, the recovery
fold) now read only the unsealed tail, entries with no numeric
`lastGeneration` — legacy or hand-repaired manifests, never prune what you
cannot prove — and entries whose recorded maximum is actually above the bound.
The open narrates what it read and what it pruned when the log holds more than
one segment.
Pinned in tests/integration/factlog-open-prune.test.ts, from the log's own
counters rather than a clock: a clean reopen over five sealed segments reads
exactly the tail (1 of 6) and finds nothing; a real SIGKILLed writer that
sealed segments holding facts above the committed pointer has those segments
READ, and its peek, its fold stream and its rollback all match the unpruned
full scan fact for fact; a manifest entry missing `lastGeneration` is read.
`find({ query, connected, where, offset })` answered [] for every page but the
first. The metadata block ranks the fused candidates and CUTS the page itself
— rows [offset, offset+limit) — and then returns early. Two shapes do not take
that early return, `connected` and `fusion`, and they fell through to the tail,
which sliced the already-cut page by `offset` a second time: a five-row page
sliced at offset five is nothing at all. Every page after the first was empty,
and the caller had no way to tell that from "no more rows".
The block now records that it consumed the offset, and the tail returns the
page it was handed instead of re-cutting it. Nothing changes at offset 0, where
the second slice was the identity.
Pinned in tests/integration/find-hybrid-filter-before-hydrate.test.ts: page two
of a `connected` hybrid find matches the pipeline oracle row for row, paging
reaches every matching neighbour exactly once, and a `fusion` find's second
page is the same page the plain find returns.
A hybrid find fuses a text leg and a semantic leg. The semantic leg already
walked only the metadata filter's universe. The text leg did not: it ranked
the WHOLE store, took the top `limit * 4`, read every one of those rows from
canonical, and only then intersected with the filter. On a large store with a
selective filter that is hundreds of rows read to return a handful — and a row
matching both the query and the filter, but sitting outside the store-wide
text prefix, was silently dropped. The same defect `find({ connected })`
carried before the graph-first law, one leg over.
Both legs now rank ids inside the universe and neither reads canonical. The
text leg goes through a new optional `getIdsForTextQueryWithin` door on
MetadataIndexProvider — the text twin of `filterIdsWithin`, so a native index
can intersect its postings before any string crosses the boundary; the
reference index implements it from its own posting-list merge, so the two
doors can never disagree, and a provider without it is served by the
whole-store answer intersected here. The fusion ranks shells, the page is cut
from them, and canonical is read once for exactly that page — with the row
rebuilt in full, so a hydrated row is indistinguishable from an eagerly-built
one (same flattened fields, same entity, same match visibility, same key
order). The eager forms of both legs stay for the search modes whose leg
output IS the answer.
Measured on the production recall shape (query + type list + `missing`
negation + excludeVFS, limit 60) the old order read 241 rows in two batches to
return one; the new order reads the page.
Pinned in tests/integration/find-hybrid-filter-before-hydrate.test.ts. The
oracle there is the pre-change pipeline itself, replayed on the same brain
through the same doors: where the filter does not truncate the text leg the
answer is identical — rows, order, scores, match visibility and row shape —
across hybrid + where, + type list + excludeVFS + a `missing` negation, +
connected, with and without offset. Where it does truncate, the correction is
held by name: the old order's text leg contributed nothing at all, the new one
returns the matching rows and paging reaches every one of them. The cost pins
read the engine's own counters: one batchGet of `limit` ids, the whole-store
text door never called, and what the text leg marshals bounded by the universe.
workflow_dispatch needs Actions-unit write on the dispatching credential;
push does not, since Forgejo runs the workflow straight from the pushed
ref's tree. A plain push to a rel/** or ci/** branch now also fires the
gate, resolving candidate to the pushed commit and control to the last
released, known-good tip (10.4.9) when the workflow_dispatch inputs
aren't present.
workflow_dispatch, runs-on gate-functional — a host-mode, Bun-only lane
with no Node.js runtime, so every step is plain git + bun in shell
rather than a JS-based action. Clones candidate and control, runs the
full vitest suite on each, enforces a >=3,000-collected guard per side,
and diffs the two fail lists for genuinely new reds. The lane's own
tripwire marker (host pressure — never our own red or green) is checked
before the verdict is printed, and the job cleans up its own checkouts
so repeat runs don't feed the lane's disk-budget trip.
The proximity search fetched its anchor through get(), which omits vectors
by default, then handed a zero-length vector to the index — every
find({ near }) refused with a dimension mismatch, for every caller. Found
by the Rust planner's first-contact pins comparing outcomes with and
without the planner on a refused shape. The anchor is now fetched with its
vector, and an anchor that has none refuses by name — a proximity search
around an unvectored row has no meaning and must not fail inside the index.
Pinned in tests/integration/find-near.test.ts.
The delta gate caught the backgrounded fold breaking six pinned
crash-recovery cases: a reopened brain must have its markers re-armed
when open() returns, and a background latch races every consumer of that
contract. The backgrounding is reverted; the low-water mark stays — it
is the part that kills the whole-history scan, and with it the
foreground fold costs the log's tail on any brain that has ever drained.
The unmarked first open after upgrade pays one full scan, once, and the
open narrates it as its own step.
The provider's read doors each serve one stage, so a find that consults three of
them crosses into the index three times and marshals a result set at every
crossing: a filter matching a hundred thousand rows builds a hundred thousand id
strings to return a page of twenty-five. An index able to decide the stage order
itself can answer the page in one call and build ids only for the page.
planFindPage is optional and additive, in the shape filterIdsWithin and
getIdSetForFilter already set. The hook sits above the branch selection, because
the branches are what decide stage order per call site and an index that plans
has to be asked before that choice is made. Absent — as it is on this engine's
own index — every find is served by the stage doors exactly as before, which is
what keeps this engine the ordering oracle for any index that implements one.
The contract the door must keep, written where an implementer will read it:
identical rows in identical order to what the stage doors would produce; the
graph-first law (neighbours are the candidate universe, the filter runs over
those ids, orderBy sorts the whole set, the page is cut last); null returned
BEFORE any work rather than instead of an answer; and emptyAt naming the stage
that produced an empty page, so the serving law is applied to the right index —
an empty graph answer is re-verified against the adjacency before it is
believed, and a filter-empty is not.
Pinned in tests/integration/find-planner-door.test.ts: absent changes nothing;
present it is asked first with normalized params, the hidden ids and the graph
provider; its page is used and hydrated in its order; a declining door leaves
the result identical to the no-door path; and the two emptyAt branches verify
the adjacency, or correctly do not.
Pass 2 issued one awaited related({ to }) per VFS entity — O(entities)
serialized graph calls, measured in whole minutes on large brains. Now a
single paged walk over every Contains edge (type-only, 1,000 per page)
feeds an in-memory group-by-target, and only actual defects mutate. The
verdicts are unchanged: a stale parent's edge is removed, a missing edge
is restored, duplicates cannot survive, and user knowledge edges are
never touched.
Pinned in tests/integration/vfs-containment-batched.test.ts: exact
removed/restored counts on a seeded defect tree, tree correctness after
the repair, user edges untouched, and the cost shape — related() call
count independent of the entity count.
related() with a verb-type ARRAY returned edges for only the first type —
the storage fast paths collapsed `verbType` (and, in their sibling blocks,
`sourceId` and `targetId`) arrays to their first element, silently
dropping the rest of the ask. Every consumer passing a verb list
under-traversed with no error and no narration: the same quiet-loss class
as the graph-first paging defect, one seam over.
All four fast paths now union over the full requested set, deduped by
edge id, before the metadata filters and pagination run. Pinned in
tests/integration/related-verb-array.test.ts: the second requested type's
edge returns in both array orders, on the anchor side, the target side,
and the type-only path; a one-element array equals the scalar; no
duplicates on overlap; pagination walks the union consistently.
The recovery fold scanned the generation log from generation 1 at every
open, on the open's foreground — O(whole history) on long-lived brains
(measured at two minutes of a large brain's open). Now an advisory mark
records the log's head whenever the pending set drains to empty (and at
clean close when empty); recovery scans from the mark + 1. The mark is
advisory and monotone-safe: stale-low costs a longer scan, never a
marker. The fold itself moves behind the doors as a latched background
task — the embed worker starts when it settles, and awaitPendingEmbeds()
and close() wait on the latch first, so no caller can observe a
half-recovered set. A pending embed's outcome was always eventual;
moving its recovery off the foreground changes when the worker starts,
never whether a marker is honored.
Pinned in tests/integration/pending-embed-low-water.test.ts: the drain
writes the mark and the next open scans from mark + 1; a pending embed
enqueued after the mark survives an unclean stop; open arms the fold as
a background latch the barrier waits on; a clean close writes the mark
even without a drain.
With `connected` present, find() materialized the whole-store filtered id
list, paged it, hydrated the page, and only then intersected with the
neighbour set. Every such call paid O(store) for the filter and the
hydration of rows that were never neighbours, and a neighbour outside the
first page of the filtered STORE was silently dropped — the answer depended
on the store's order and the page size.
The neighbour set is now the candidate universe: resolved first from the
adjacency, the metadata filter evaluated over those ids only through the
provider's own evaluation (a new optional `filterIdsWithin` door on
MetadataIndexProvider; the reference index implements it from its own
getIdsForFilter so the two can never disagree; a provider without it is
served by the whole-store answer intersected here), `orderBy` sorts the
whole neighbour set before the page is cut, and the vector leg walks the
neighbours as its candidate set. The text leg of a hybrid find keeps its
post-intersection — it has no candidate door.
Pinned in tests/integration/find-connected-order.test.ts: paging reaches
every matching neighbour and never a non-neighbour; a `missing` negation is
evaluated over the neighbours; the index is asked about the neighbour ids
only and hydration is one page; orderBy sorts the whole set; the vector leg
stays inside the neighbours; an edgeless anchor answers [] before the
filter is asked.
persistCounts() was write-through on every count change with no
serialization, and the atomic writer named its temp file with millisecond
granularity. Two persists inside one millisecond shared the temp path: both
wrote it, the first rename consumed it, the second rename found nothing —
ENOENT, roughly 1,500 times a day on a busy production brain, with a full
ledger write per change behind it. No data was lost (the surviving rename
carried a complete ledger and the next change re-persisted), but the race
was real and the write rate absurd.
flushCounts() now runs exactly one persist at a time; requests arriving
during it collapse into one trailing pass that carries the burst's final
state — N changes cost at most two writes. writeFileAtomic() adds a
per-process sequence to the temp name so no two writes can share a path.
Pinned: a 25-change burst → ≤2 ledger writes, zero errors, ledger equal to
memory; parallel real writes land complete; three same-instant atomic
writes own three distinct temp paths.
transact()'s delete legs (direct unrelate and the noun-remove cascade) hand
the SAME verb object to the graph-retraction op and the metadata-retraction
op. The metadata leg sanitized at PLAN time, when the verb was still clean,
so the wrap returned the same reference — then the graph op's execute-time
endpoint resolution (deliberately deferred for same-batch forward refs)
mirrored BigInt sourceInt/targetInt onto the shared object, and the metadata
op crossed the seam with them. A strict provider rightly refuses that
crossing, so every transact-wrapped edge delete aborted; direct unrelate()
resolves ints at build time, before its sanitize, which is why no existing
gate saw it.
The JSON-safe view now lives in a shared leaf (utils/jsonSafeIndexMetadata)
and is applied INSIDE AddToMetadataIndexOperation and
RemoveFromMetadataIndexOperation at execute and rollback time — the one
place no plan-vs-execute ordering can bypass. Pins: the fleet repro, the
cascade shape, a mixed batch, and unit pins that mutate the entity after
construction against a strict seam (5 red before, 5 green after).
The one-doc-set ruling (2026-08-31) gives soulcraft.com/docs to the paid
product alone; the site serves redirects for the slugs this rail used to
push. The push script stays in the tree as history; the rail stops calling
it.
(cherry picked from commit 655aa13ea7)
Diagnosis of the "packed history is damaged" narration that fires on every
run of the affected stores. It is a WRITER defect, and the reader's refusal
was the symptom rather than the cause.
A sealed segment declares one contiguous range [firstGeneration,
lastGeneration], and every reader treats that range as containment:
coveringSegment is an interval test, hasGeneration returns true for anything
inside it, and open() seeds committedRanges from it.
repackHistory handed fold() a SPARSE batch. Three filters punch holes in its
candidate list mid-run — a generation absent from committedRanges never
appears, one still in the pending buffer is skipped, one whose tx.json will
not read is skipped — and fold() then computed the range from the first and
last survivor, claiming every generation in between. The next open merged
that mis-declared range back into committedRanges, re-admitting the hole as
committed history, so the following auto-compaction pass asked the packed
tier for a frame that was never written and failed. Re-merged at every open,
which is why it repeated on every run.
Confirmed against a forensic fixture: generation directories 1..2503 present
except exactly one, 1416; and its fact-log segment already showed the tell —
seg-...1410.bfl declaring 1410..1940 (531 generations) while recording 530
facts.
Three changes:
- repackHistory folds each contiguous RUN as its own segment
(`contiguousRuns`), so ranges describe exactly what the segments contain.
- fold() REFUSES a non-contiguous batch, naming the gap and its width. The
density law is now mechanical, so no future caller can reintroduce it. A
refusal loses nothing: the generations stay live and readable.
- Stores already carrying the damage heal instead of wedging. A segment
whose declared span exceeds its frame count is SPARSE; `actualRanges()`
reads the real generation list from its sidecar so open() never re-admits
the holes, and readFrame reports such a hole as unpacked with a narration
naming the segment, rather than throwing. A DENSE segment missing a frame
is still loud damage — that one means the manifest and sidecar disagree.
Pins: nine unit cases (refusal and its message, honest ranges for separately
folded runs, a reconstructed pre-fix sparse segment serving its real frames
while reporting holes as unpacked, holes excluded from actualRanges, and the
dense-segment damage path still throwing) plus an end-to-end case that
deletes a generation directory and drives the real sequence — ordinary
close()-time repacking folds over the hole, then reopen and compact must both
complete. Verified red without the fix: the segment declared an
11-generation span while holding 10 frames.
(cherry picked from commit 9a888c37e9)
Two halves of one defect, found by a seeded-SIGKILL crash lane.
THE FALSE POSITIVE. stampEntityTree() recorded generationStore.generation()
— the ALLOCATED counter, a number a write in flight has claimed and may
never commit — while the JSDoc beside it already said the source is the
committed generation. Every crash inside a write window therefore produced
a spurious verdict at the next open: either 'sourceGeneration N is ahead of
the log head N-1' (the allocated generation died with the process) or
'rollup invariant nounCount: stamped X, observed Y' (the recovery fold
folded facts the stamp's counts predate). Both told the operator to run
repairIndex() — a whole-store recount — for a store that was coherent.
Measured before this commit: 4 of 11 SIGKILL cycles on a healthy store
raised one of the two. The stamp and the open now both read
committedGeneration(), which is what every other open-time watermark in the
class already reasons about.
THE TERMINAL VERDICT. A stamp still ahead of committed truth after the
recovery fold witnesses a generation that is not in the log — the stamp's
fsync outlived the tail's, and there is nothing to arrive. That is its own
verdict state now ('torn'), never folded in with 'incoherent': the two have
opposite cures. A writer open demotes it — the unusable stamped surface is
re-derived at the committed generation from the live counters, O(1),
straight-line, no loop and no await on external progress, narrated with
both count sets, the stamp's path and its committedAt. A read-only open
cannot re-stamp, so it says so and names the cure instead of guessing, and
still serves. Neither branch waits, and neither locks an owner out of a
canonical tree the stamp only describes.
Pins: the verifier returns the torn verdict with both generations; a
fabricated head-behind-source store narrates precisely, demotes inside a
bounded open, serves its rows, and is quiet at the next open (the demotion
converges); a read-only open narrates the same verdict and leaves the bytes
untouched.
(cherry picked from commit 298cb6daca)
The release gate's own output caught this: every test brain printed
"[VFS] old-root sweep complete in 1ms and recorded" — hundreds of lines — and
a consumer would get two of them on the first open of every store.
They were emitted on the always-visible channel, which a production log level
deliberately CANNOT silence. That channel exists so an operator can always
learn why a database is slow; a 0ms no-op on a fresh store is not that, and
announcing it there trains people to ignore the one channel built to be
impossible to ignore. It was also inconsistent with every other narration in
this work, all of which is silent under a threshold.
The sweep now speaks when it has something to say — duplicate roots removed, or
a wall over a second that a person watching a slow first open deserves
explained — and otherwise does its work, records its marker, and stays quiet.
cleanupOldRoots() reports what it removed so the decision rests on a fact
rather than on a guess.
Pin: a fresh store's sweep emits nothing on the channel and still records its
marker, so the silence can never be mistaken for the work being skipped.
Two gate failures on main, one real and one long-hidden.
THE HEALTH-GATE PIN encoded the old law — "narrates once per generation, twice
across a generation bump" — which the content-keyed dedupe deliberately
replaced. A provider's `generation` bumps on every ledger mutation and every
rebuild boundary, so keying narration on it re-printed an unchanged health line
on every read that consulted a busy provider, and let a provider that never
bumped suppress a line whose reasons had genuinely changed. The pin now asserts
BOTH directions: an unchanged verdict stays silent however the counter moves,
and a changed verdict is always heard.
THE VFS HYBRID-SEARCH SUITE configured its store with `options.basePath`, an
alias removed at the 8.0 major that configures nothing. The suite was therefore
never using its temp directory — it opened the DEFAULT store, shared with every
other run on the machine, and accumulated tens of thousands of rows until it
failed on that shared store's graph adjacency instead of on anything it tests.
It now passes `storage.path`. The suite drops from 6.5s to 0.3s, which is the
measure of how much foreign data it had been opening.
Neither failure was caused by the release branch; the first is the branch's own
behaviour change meeting its outdated pin, the second predates it.
Correctness and observability for 10.4.4: the writer-lock clean-close record
and the always-run terminal releases, the self-healing count ledger and an
atomic counts.json, always-visible open and repair narration, the non-blocking
open for a provider rebuilding itself, an event-driven flush-request watch, and
the operator-visible operator set (three served, four refused by name).
The manifest's prose pointer named a document that answers a confidential
specification, and such a document does not belong in a public repository even
in summary. The pointer is dropped — the manifest is generated from this
engine's own surface and is self-describing — and the requirement marking it
deliberately omits is recorded with the contract's owner rather than here.
The standard is written down so this is not relitigated per document.
A release audit found hostnames, store identities and operational anecdotes in
this branch's commit messages — not trade secrets, but nothing a public
repository's permanent history should carry either. The messages were rewritten
to keep every number and drop every provenance; the rule is written down here
so the next measurement does not have to be caught by an audit.
MEASURED on a 14,056-noun / 72,679-verb production-shaped store, measured
solo under an exclusive lock: the vfs-bootstrap
phase costs 37.8s on main and 38.0s on this branch — unchanged — and NO
"vfs.init" step line was emitted at all, meaning the VFS's own init fell under
the 2s narration threshold. The phase is therefore almost entirely NOT the VFS,
and the old-root sweep this branch moved to the background was never what made
it expensive.
What else lives in that span is now named: the log-authority artifact read, the
adoption ORACLE (which verifies the log against canonical before flipping a
brain to durable-at-ack), the legacy pending-embed sidecar bridge, and the
pending-embed recovery fold. One of those holds ~38 seconds of every open of
this store and the next measurement will say which, by name, instead of leaving
a phase label to be guessed at.
The safety sweep is armed alongside the watch, and startFlushRequestPolling()
declines to arm over an existing interval — so when a watch died mid-life the
fallback did nothing and the store quietly answered flush requests on a 30s
cadence instead of the 500ms one the door promises. The sweep is cleared first.
A degrade nobody asked for is still a degrade.
Arming is asynchronous — the request directory is created before it can be
watched — so during that window neither the watcher nor the sweep interval
exists yet and the guard let a second call through, leaving two watchers and
two sweeps for the life of the store. The callback is the flag that covers the
window.
Three idle-burn items from the steady-state audit, and one correction.
THE FLUSH-REQUEST WATCH (the strongest of them). It readdir'd the request
directory every 500 ms, per brain, for the life of every writer — armed on
every non-reader brain whether or not any inspector process existed. In a process holding many stores that is tens of directory reads per second
on a completely idle service, plus a stale-request GC on every one of them. It now
uses fs.watch, so the arrival itself wakes it and a request is seen SOONER
than the poll saw it. Two concessions ride along, both stated in the code: a
30s safety sweep (fs.watch drops events on some network and fuse filesystems,
and the GC needs a tick of its own — two orders of magnitude fewer reads than
the poll made), and a fall back to the original 500 ms poll, narrated, on a
filesystem that cannot watch at all, because an inspector whose request is
never seen waits forever.
THE WRITER HEARTBEAT goes 10s → 60s. It is observability ONLY — staleness is
decided by pid liveness and the fence compares pid + hostname, so no decision
anywhere reads the timestamp — and at 10s it was a lock-file write every ten
seconds per brain forever, for a value nothing computes with. An operator
still sees a heartbeat inside the minute.
THE HEALTH NARRATION dedupes by CONTENT, not by the provider's generation
counter. That counter bumps on every ledger mutation and rebuild boundary, so
a provider bumping it on routine work re-emitted the same unchanged line on
every read, while one that never bumped could suppress a line whose reasons
had genuinely changed. The generation is still reported; it no longer decides
whether the line is worth saying.
CORRECTION, and it is against my own earlier claim: the idle-flush commit read
a reported idle-CPU observation (many stores, no writes, a flush every ~35s,
over a core burned) as caused by
the flush path. That does not follow — this engine's cadence is write-driven
(every trigger runs through noteWriteForPersistence, which only a committed
write calls), so something was CALLING flush() on those brains and the caller
is still unidentified. The clean-flush gate makes such a call free; it does not
account for it. The code comments and the idle lane now say exactly that.
Pins: tests/integration/flush-watcher-event-driven.test.ts — an idle writer
makes at most one request-directory read in 8 seconds (the old poll made ~16),
and a dropped request is still acked well inside the safety sweep.
The 7.x-to-8.0 layout probe runs on the open path of every store that does not
yet carry its completion marker — a restore, a store built by an older release
— and asked whether the canonical tree holds anything by LISTING it: a
recursive walk of every file in every entity directory, to learn a boolean. It
now asks the one-level door added for generation discovery, falling back to the
listing on an adapter that lacks it.
Also files a defect found while ratifying the operator set: the VFS builds its
path-prefix filter as `$startsWith`, an operator no engine spelling accepts,
so vfs.searchFiles({ path }) throws INVALID_QUERY on every call that passes a
path. Pre-existing, unrelated to the operator work, and left as a filing —
a path-prefix search needs a design answer, not a spelling correction.
MEASURED on a production-shaped store (14,056 nouns / 72,679 verbs, an 11 GB
generation history), measured solo under an exclusive lock: the generation-store phase cost 55,538 ms of a WARM
REOPEN after a clean close — with the fold correctly skipped, so nothing in
that phase's name explained it.
This is what it was doing. Discovering which generations exist on disk called
listRawObjects('_generations'), which RECURSES the whole tree and returns
every file in every generation directory — to extract a set of integers that
the top-level directory NAMES already spell out. The cost scales with the
entire history, is paid on every open, warm or cold, and grows for the life of
the store.
A one-level door — listRawPrefixes(prefix), the immediate child directory
names — is added to the storage seam. The filesystem adapter answers it with a
single readdir; BaseStorage derives it from the recursive listing, so an
adapter without a cheap implementation is never wrong, only never faster; and
the generation store falls back to the old listing when the door is absent.
One behavioural difference, stated: an EMPTY generation directory is now
discovered where the file listing could not see it. Above the committed
watermark that is a crash scar, and recovery already has an explicit branch
for it ("indeterminate partial dir" — dropped, narrated). Below it, it becomes
a resolvable generation holding no records, which is what an empty generation
means.
Suites: the durability kill matrix (15), db-mvcc (30), history repacking (4),
rollback trapdoor (3), entity-tree stamp (4) and the full unit suite (2,105)
all green.
Both mutate durable state outside the two commit paths, so neither was seen by
the flush witness added with the idle-flush law. A clear() followed by a
flush() would have found the brain "clean" and skipped the entity-tree stamp,
leaving a stamp describing the population the clear had just removed — a false
divergence warning at the next open. Closing the gap where it is, rather than
widening the witness to guess.
A phase that costs a minute and names only itself tells an operator where to
look but not what to look at. MEASURED on a real 14,056-noun / 72,679-verb
store, the warm reopen's generation-store phase cost 55,538 ms with nothing
inside it named — the fold was skipped (the close was clean), so the cost was
somewhere else entirely and the breakdown could not say where.
Six steps inside the open now report their own wall with their own cause when
they exceed the phase threshold: the generation store's open (manifest,
committed ranges, fact log, packed tier, crash replay), the entity-tree stamp
verification, the brain-format read, the pre-upgrade backup, the derived-index
gate, and the VFS init. Silent under the threshold, so a fast open says nothing
extra. Same always-visible channel as the phase lines.
MEASURED on a 14,056-noun / 72,679-verb production-shaped store, measured
solo under an exclusive lock: the vfs-bootstrap phase cost 43,021 ms of a cold open and 52,696 ms of
a WARM REOPEN. What dominates it is a migration sweep — a filtered find() over
the whole store hunting for root directories created before the fixed root id
existed. A store either carries such duplicates or never will, and the sweep
ran on every open, forever, in the foreground.
It is now caused by the store's state instead of by the open count: a durable
marker under _system/ records that the sweep has run, and a store carrying it
never sweeps again. A store without one sweeps in the BACKGROUND — the sweep
only removes duplicate roots, nothing serves from them, and it was already
declared non-critical — narrated at both ends, with whenRootSweepSettled() for
anyone who needs to observe rather than race it. An adapter with no raw-object
door keeps the old behaviour: correctness over cost, never a silent skip.
Pins: tests/integration/vfs-root-sweep-once.test.ts — the sweep runs on the
first open and never on the second or third; a sweep slowed to 4s does not
delay the open.
Open Brainy's side of the API contract the accelerated engine published.
DECLARED: package.json carries "brainyContract": 1 and the engine states its
own via contractVersion() / BRAINY_CONTRACT_VERSION — two engines compare an
integer instead of probing prototypes, and a tool reads the package field
without importing the engine. Pinned so the two can never drift apart.
SERVED: hasAll, noneOf and excludes now work on the index path. The defect
underneath was worse than the reported divergence — the metadata index's
operator switch had NO DEFAULT CASE, so any operator without a case left the
field's match set at its initial [] and find() returned an empty page.
Documented, validator-accepted, matcher-implemented operators answering
silently wrong. hasAll intersects each element's posting set (an empty operand
is vacuously true of every row that has the field), noneOf complements their
union, excludes complements contains.
REFUSED BY NAME: startsWith, endsWith, matches and length raise
INVALID_QUERY naming the operator, the field and the reason. An equality/range
posting index cannot evaluate a substring, a pattern or an array length without
reading every row — which is the cost this path exists to avoid — so it refuses
rather than answering an empty page. Both engines now agree on all 25 tokens
and contract 1 has no remaining operator divergence. This is a visible change
for a consumer calling those four through find({ where }): an empty page
becomes a typed refusal.
EMITTED: scripts/emit-contract-manifest.mjs generates docs/api-contract.json
from the BUILT surface — prototype doors, exported error classes, the operator
sets read out of their single definitions, the field-addressing vocabulary, the
health verdicts. Nothing hand-maintained, so a diff between two manifests is a
diff between two engines. `--check` fails on a stale manifest, which makes the
announce-every-addition duty mechanical rather than remembered.
RATIFIED in docs/contract-1-ratification.md: the 41-of-57 required split with
the promise spelled out (a refusal is part of a door; deprecation is not
removal), the serving-withholding list confirmed exhaustive and identical, the
minor/major rule adopted with the announcement duty, the 30 storage seam
methods committed as supported surface until Stage 2, and a finding filed
against the spec — is / isNot / greaterEqual / lessEqual are listed there as
served aliases and have never existed in this engine, which throws
INVALID_QUERY on all four.
Follower to the self-rebuild deference. The open gate's consistency check —
"metadata index has 0 entries but storage has N entities" → CRITICAL + a forced
second rebuild — knew two states, migrating and not. A provider whose rebuild()
returns once the rebuild is OWNED AND RUNNING ONLINE (its doors refusing by
name while other families serve) legitimately reports 0 entries there, so every
first contact printed a false CRITICAL and kicked a redundant second rebuild.
The exemption rides the rebuild-progress hook, NOT isMigrating() — widening
that would hold every write and 503 the whole brain through the migration
snapshot, which is worse than the false alarm. The check's real class is
untouched: a provider reporting 0 entries with no rebuild in progress still
trips it.
The crash-recovery rebuild kick gets the same deference: a provider already
rebuilding itself from canonical is doing exactly that work, and the fold ran
in the generation store's open before any provider existed, so what it is
reading is the repaired canonical.
Pin: a provider stub reporting a rebuild and 0 entries opens with no CRITICAL
line and no second rebuild; the vacuous-stub case fails loudly.
MEASURED on a production store: a metadata provider that had to rebuild made
init() pay the ENTIRE rebuild on the foreground — 641 seconds — with every
other family idle behind it. The cause is a missing distinction: a provider
reporting serving:false because it is BUSY BUILDING ITSELF and one reporting
serving:false because it is BROKEN looked identical through healthReport(),
and both were answered the same way — call rebuild(), and wait for it.
The contract that tells them apart is one optional, synchronous, O(1) hook:
`rebuildInProgress(): ProviderRebuildProgress | null`, reporting a phase name
and whatever the provider actually measures (done/total/startedAt) — never an
estimate dressed as a fact. A provider without the hook behaves exactly as
before.
With it, a provider owns its own rebuild:
- the open gate neither starts a second rebuild nor waits for the provider's,
and narrates that it is not waiting and what will refuse meanwhile;
- init() returns and every other family serves;
- that family's doors refuse BY NAME, carrying the provider's own progress,
and say plainly that the door opens by itself and no action is needed —
distinct from a broken index, which names repairIndex();
- the epoch stamp does not advance while any family is still being built.
Nothing is ever served empty: a not-serving family refuses, as it already did.
Pins: tests/integration/open-does-not-wait-for-a-rebuilding-provider.test.ts —
init() returns in milliseconds against a provider claiming a 6s rebuild, brainy
starts no rebuild of its own, a filtered read refuses naming the phase and the
4,096/14,056 progress, and the door answers once the provider reports serving.
The pin fails loudly rather than vacuously if its stub never installs.
REPORTED from the field: a process holding many stores, with no writes for ten
minutes, printed "All indexes flushed to disk in 216-601ms" per store every
~35 seconds and burned over a core at idle. Every one of those flushes re-persisted
state identical to what was already on disk — the provider flushes, the
watermark stamps, the generation counter, the entity-tree stamp — because
flush() never asked whether anything had changed.
- flush() over a clean brain is now O(1) and silent: a dirty witness is set by
every committed write (both commit paths end at noteWriteForPersistence, and
the deferred-embed worker lands through the single-op path) and cleared by a
flush that runs. A write landing DURING a flush sets it again, so no write's
work is ever skipped — it is done by the next flush. Set before the policy
check, so a `'manual'` consumer's explicit flush is never a no-op it didn't
ask for.
- An explicit flush now tells the cadence it happened. It didn't, so the very
next write saw "30s since the last flush" and kicked a background flush with
nothing to do, and the idle timer fired two seconds later over writes the
explicit flush had already persisted.
- The graph adjacency index's auto-flush asks before it acts: two O(1) reads
of the LSM MemTables, and a tick over a quiet index returns without calling
into the trees at all.
assessProviderHealth is NOT timer-driven — it is a synchronous O(1) read of a
provider's own healthReport(), called on the read gate, so it costs nothing on
an idle brain. No change needed there.
Pins: tests/integration/idle-costs-nothing.test.ts — 90 idle seconds produce
zero flushes, zero provider calls and zero log lines; three explicit flushes
over a clean brain call no provider; one write earns exactly one flush.
On a production store (14,647 nouns / 73,070 verbs) a repairIndex() ran for
more than thirty minutes at roughly a full core with ZERO log lines between
its start and its end, while the read doors kept serving. The operator could
tell it was alive only from `top`, and could not tell which of its
single-threaded walks it was inside.
Same law as the open, applied to the repair:
- every phase announces itself BEFORE it works, naming what it is about to
walk (each canonical walk, the VFS containment reconciliation, each
provider's invariant pass);
- an unref'd heartbeat names the phase still running every 5s, for as long as
it runs;
- every phase reports its own wall, and that wall is carried in the TYPED
receipt as RepairFamilyReport.durationMs — a receipt that cannot say where
the time went is not a receipt;
- the whole repair's narration moves to the always-visible channel, so a
production log level cannot silence it.
The phases move into runRepairIndexPhases() so the heartbeat can live in a
finally around them; the public door and its report shape are unchanged apart
from the added durationMs.
Pins: tests/integration/repair-narration.test.ts — every checked family has a
start line, a finish line with its wall, and a numeric durationMs in the
receipt; a phase slowed to 6.5s produces a heartbeat naming it, with the
logger clamped to ERROR.
MEASURED on a real store: the ALL-visibility ledger read 14,231 nouns against
14,056 identity records and 72,729 verbs against 72,679 — exactly that store's
25 noun and 50 verb SCAR directories. Two copies of the same archive derived
different numbers (14,231 and 14,081), because each had been persisted at a
different moment under the old rule that counted one entity per id DIRECTORY.
A downstream index heal subtracted against that denominator and reported
remaining work that did not exist.
The scan already applies the right predicate — one entity per IDENTITY RECORD
(the metadata content leg), shared with pruneOrphanedEntities so the two agree
by construction. What was missing is that a ledger persisted under the old rule
was only FLAGGED suspect and then went on serving its wrong numbers for the
life of the store, waiting for an operator to run repairIndex.
- The ledger now derives itself honestly in the BACKGROUND after the open,
narrating start and finish with the correction it made. Background because
these scalars are denominators — no read is served from them — and because
walks exactly like these are how a 24,898-id store spent minutes of a
restart in silence. Observable via whenCountLedgerSettled(); nothing in the
read path waits on it.
- A derivation that raced a write refuses to stamp its number "exact": one
retry on a quiet store, then the ledger stays SUSPECT and says so, naming
repairIndex as the door that recounts under a barrier.
- The one derivation that CANNOT leave the foreground says why it cannot:
getNounCount()/getVerbCount() are served from it, and a background walk
would make a populated store answer "0 entities" — a wrong answer, not a
slow one. It narrates its start and its wall instead.
- counts.json is written temp+rename. A truncating write left a window —
measured at roughly 750ms after a flush or close — in which a concurrent
reader saw the file EMPTY; an unparseable ledger sends the next open down
the full-rescan path, so the cheapest file in the store was buying the most
expensive recovery.
- The writer lock's clean-close record is now consulted before the
same-process branch too: a restart reported "Re-acquiring writer lock ...
this is a bug" immediately after a clean close, sending an operator after a
leak that did not exist.
Pins: tests/integration/count-ledger-identity-record.test.ts (background
correction with scar and ghost fixtures, two copies of one archive agreeing,
counts.json never observed unparseable across 40 persists);
tests/integration/ledger-derivation-identity.test.ts updated to the new law —
the OPEN still never walks (proved by slowing the walk 1.2s and timing the
open), and the ledger heals behind it.
