Diagnosis of the "packed history is damaged" narration that fires on every
run of the affected stores. It is a WRITER defect, and the reader's refusal
was the symptom rather than the cause.
A sealed segment declares one contiguous range [firstGeneration,
lastGeneration], and every reader treats that range as containment:
coveringSegment is an interval test, hasGeneration returns true for anything
inside it, and open() seeds committedRanges from it.
repackHistory handed fold() a SPARSE batch. Three filters punch holes in its
candidate list mid-run — a generation absent from committedRanges never
appears, one still in the pending buffer is skipped, one whose tx.json will
not read is skipped — and fold() then computed the range from the first and
last survivor, claiming every generation in between. The next open merged
that mis-declared range back into committedRanges, re-admitting the hole as
committed history, so the following auto-compaction pass asked the packed
tier for a frame that was never written and failed. Re-merged at every open,
which is why it repeated on every run.
Confirmed against a forensic fixture: generation directories 1..2503 present
except exactly one, 1416; and its fact-log segment already showed the tell —
seg-...1410.bfl declaring 1410..1940 (531 generations) while recording 530
facts.
Three changes:
- repackHistory folds each contiguous RUN as its own segment
(`contiguousRuns`), so ranges describe exactly what the segments contain.
- fold() REFUSES a non-contiguous batch, naming the gap and its width. The
density law is now mechanical, so no future caller can reintroduce it. A
refusal loses nothing: the generations stay live and readable.
- Stores already carrying the damage heal instead of wedging. A segment
whose declared span exceeds its frame count is SPARSE; `actualRanges()`
reads the real generation list from its sidecar so open() never re-admits
the holes, and readFrame reports such a hole as unpacked with a narration
naming the segment, rather than throwing. A DENSE segment missing a frame
is still loud damage — that one means the manifest and sidecar disagree.
Pins: nine unit cases (refusal and its message, honest ranges for separately
folded runs, a reconstructed pre-fix sparse segment serving its real frames
while reporting holes as unpacked, holes excluded from actualRanges, and the
dense-segment damage path still throwing) plus an end-to-end case that
deletes a generation directory and drives the real sequence — ordinary
close()-time repacking folds over the hole, then reopen and compact must both
complete. Verified red without the fix: the segment declared an
11-generation span while holding 10 frames.
Two halves of one defect, found by a seeded-SIGKILL crash lane.
THE FALSE POSITIVE. stampEntityTree() recorded generationStore.generation()
— the ALLOCATED counter, a number a write in flight has claimed and may
never commit — while the JSDoc beside it already said the source is the
committed generation. Every crash inside a write window therefore produced
a spurious verdict at the next open: either 'sourceGeneration N is ahead of
the log head N-1' (the allocated generation died with the process) or
'rollup invariant nounCount: stamped X, observed Y' (the recovery fold
folded facts the stamp's counts predate). Both told the operator to run
repairIndex() — a whole-store recount — for a store that was coherent.
Measured before this commit: 4 of 11 SIGKILL cycles on a healthy store
raised one of the two. The stamp and the open now both read
committedGeneration(), which is what every other open-time watermark in the
class already reasons about.
THE TERMINAL VERDICT. A stamp still ahead of committed truth after the
recovery fold witnesses a generation that is not in the log — the stamp's
fsync outlived the tail's, and there is nothing to arrive. That is its own
verdict state now ('torn'), never folded in with 'incoherent': the two have
opposite cures. A writer open demotes it — the unusable stamped surface is
re-derived at the committed generation from the live counters, O(1),
straight-line, no loop and no await on external progress, narrated with
both count sets, the stamp's path and its committedAt. A read-only open
cannot re-stamp, so it says so and names the cure instead of guessing, and
still serves. Neither branch waits, and neither locks an owner out of a
canonical tree the stamp only describes.
Pins: the verifier returns the torn verdict with both generations; a
fabricated head-behind-source store narrates precisely, demotes inside a
bounded open, serves its rows, and is quiet at the next open (the demotion
converges); a read-only open narrates the same verdict and leaves the bytes
untouched.
MEASURED on a production-shaped store (14,056 nouns / 72,679 verbs, an 11 GB
generation history), measured solo under an exclusive lock: the generation-store phase cost 55,538 ms of a WARM
REOPEN after a clean close — with the fold correctly skipped, so nothing in
that phase's name explained it.
This is what it was doing. Discovering which generations exist on disk called
listRawObjects('_generations'), which RECURSES the whole tree and returns
every file in every generation directory — to extract a set of integers that
the top-level directory NAMES already spell out. The cost scales with the
entire history, is paid on every open, warm or cold, and grows for the life of
the store.
A one-level door — listRawPrefixes(prefix), the immediate child directory
names — is added to the storage seam. The filesystem adapter answers it with a
single readdir; BaseStorage derives it from the recursive listing, so an
adapter without a cheap implementation is never wrong, only never faster; and
the generation store falls back to the old listing when the door is absent.
One behavioural difference, stated: an EMPTY generation directory is now
discovered where the file listing could not see it. Above the committed
watermark that is a crash scar, and recovery already has an explicit branch
for it ("indeterminate partial dir" — dropped, narrated). Below it, it becomes
a resolvable generation holding no records, which is what an empty generation
means.
Suites: the durability kill matrix (15), db-mvcc (30), history repacking (4),
rollback trapdoor (3), entity-tree stamp (4) and the full unit suite (2,105)
all green.
An operator watched a production service open a 16 GB store and print nothing
for three minutes before its first line of work. Two defects, both fixed here.
The narration was written to `prodLog.warn`, and every environment that looks
like production clamps the logger to ERROR — so the phase breakdown that would
have named the slow phase was composed and thrown away. `prodLog.narrate` is
always visible, like `error`: it carries the two things an operator is
entitled to hear from a database regardless of a cost setting — why it is slow
and what it is doing about it. `silent: true` still silences it; that is a
request, not a default.
And nothing spoke DURING a phase, only after the whole open. init() now runs
an unref'd heartbeat that every 5s names the phase currently running, its
elapsed wall and what it is paying for, plus one line per phase as it ends for
any phase over 2s. The generation-log fold's own progress and completion lines
move to the same channel and now carry their wall — they were invisible in
production, which is how an operator came to restart a converging fold three
times.
Pins: tests/integration/open-narration.test.ts — narrate() survives the clamp
that silences warn(); a 6.5s storage-init produces a heartbeat naming the
phase and a completion line naming its wall, with the logger clamped to ERROR.
Prepares the repo for its new home at soulcraftlabs/open-brainy ahead
of the Forgejo transfer: package name, publish registry, release
script, and every install/import reference across docs, src, tests,
examples, and integrations now point at @soulcraftlabs/brainy on
The Source. The npmjs storefront leg and byte-identity pair
verification are stripped from the release script — The Source is
now the only publish target. README gains an Open Brainy explainer
and a registry note for consumers.
@soulcraft/brainy 10.4.2 was the last release under the old name.
The noun/verb pagination walks (getNounsWithPagination,
getNounIdsWithPagination, getVerbsWithPagination) listed shard contents by
filtering for vectors.json, while the canonical count ledger has always
counted a row by its metadata.json presence alone. A row with metadata and
no vector file was therefore counted by the ledger but never yielded by the
walk — a permanent "counted but invisible" phantom for any downstream
consumer that iterates the walk to account for the ledger's total.
Nouns now enumerate by metadata.json and hydrate the vector leg optionally,
yielding the sanctioned unvectored shape (vector: []) when it's absent.
Verbs enumerate the same way, but a metadata-only verb row can only be fully
reconstructed when sourceId/targetId happen to be recoverable from metadata
(never true for a current production write — those fields live only in the
vector leg); otherwise the row is counted but loudly skipped rather than
fabricated, since a phantom edge with fake endpoints would be worse than the
original defect.
Separately, GenerationStore's recovery-fold replay (replayFact) now applies
preserve-if-absent: a metadata-only after-image replayed over an already-
vectored row carries the existing vector forward instead of deleting it via
writeNounRaw/writeVerbRaw's exact-restore null-means-delete contract (which
must stay exact for transaction-abort rollback). A genuine tombstone still
removes both legs.
A production brain's first process boot after a live authority flip looked
hung and was restarted three times mid-recovery — three defects with one
scene. (1) THE FOLD MATERIALIZED THE LOG: peekFactsAbove(0) decoded every
fact into one array (GBs of after-images on a ~7k-fact log, a GC storm, a
starved write lane). The fold now STREAMS one segment-batch at a time —
memory is one segment at any log size — with structural ordering asserted
loudly. (2) THE FOLD WAS SILENT UNTIL DONE: minutes of boot work with zero
narration is what invited the restarts. It now announces itself BEFORE the
work ('do not restart, the fold is finite') and prints progress every
thousand facts. (3) THE CHAIN COULD ONLY ARM AT A CRASH: a live mid-session
flip left the fold checkpoint unfounded, so the brain's first unclean boot
paid a whole-log fold. Adoption now founds the checkpoint AT THE FLIP — one
paged full canonical barrier (bounded memory), then the stamp — so bounded
recovery holds from minute zero for every store that flips, at any size.
Pinned: a non-fresh flip stamps immediately; the first post-flip unclean
boot folds bounded (an unflushed at-ack fact above the checkpoint is
restored; a barrier-covered row below it is outside the fold). Kill matrix
and both adoption suites green alongside.
The production dev-store split-brain (two live writers alternating a store's
id-mapper between two internally-consistent truths), cured at all three of
its roots. (1) STALENESS REQUIRES PID-DEATH: the old rule evicted on
heartbeat age alone, so a >60s event-loop stall (debugger pause, GC, heavy
sync work) handed the lock to a second opener while the first kept writing;
a live process is now never auto-evicted — a wedged-but-alive holder is the
operator's call via {force:true}, and the heartbeat stays for observability.
(2) THE CLAIM IS ATOMIC: writeFile(wx)'s open→write→close left an empty-file
window a concurrent opener could read as torn, unlink a LIVE claim, and take
the lock; the claim is now tmp-write + hard-link — the lock appears with its
full contents in one step. (3) THE FENCE: every flush commit and transact
barrier verifies lock ownership first (one small read per window) — a
forced-out or lock-deleted writer fails typed (BRAINY_WRITER_FENCED) before
a single staged byte or manifest advance, instead of writing on unaware.
Pinned: live-with-ancient-heartbeat refuses typed; dead-PID self-clears
narrated; a forced-out writer's flush and transact both fence, advancing
nothing. Requested by a downstream team as single-writer guard or loud
lockout — this is both.
Two consumer-driven cures sharing one stamp. (1) TX-LOG ORIGIN: engine-
originated commits stamp an optional origin on their tx-log entry AND the
commit fact's meta — 'system:embed-landing' (the deferred vector landing),
'system:adoption-backfill' (baseline re-commits), 'system:reconcile'. A
downstream activity feed showed a double tick because the landing commit was
indistinguishable from a user save, and the consumer rightly refused a
time-window collapse as a quiet loss; feeds now filter on fact. User writes
stay unstamped — absent origin is the user shape, every existing consumer
unchanged. (2) reconcileLogDivergence(id, {attest}): the human's door for
log-live-canonical-absent, the one class adoption refuses by design because
a lost-tombstone deletion is indistinguishable from canonical loss.
'deleted' mints the missing tombstone (history keeps the earlier live
record); 'restore' folds the log's only copy back into canonical; wrong-
class calls refuse typed with nothing written. Loud, narrated, single-row,
origin-stamped. From a production adoption's one surviving divergence.
A production brain with a 12.7k-row pre-log baseline advanced exactly 800
rows per adoptLogAuthority() call (a five-pass ceiling × the oracle's
200-row listing cap), refused the flip, and sat tree-authoritative for
hours across restarts. The bound was sized for drift, never for a baseline.
Now: the adoption path runs the oracle uncapped so ONE scan yields the
ENTIRE curable set, every pass cures all of it, and the loop runs to
completion with the no-progress guard as its only stop. Pace rides the
write path (one full-brain scan amortizes over thousands of cures, not two
hundred): 1,000 drifted rows adopt green in one call in ~10s. Progress is
narrated for a live operator. The wire report keeps its 200-row cap.
Pinned: a baseline above the old ceiling adopts green in a single call.
Two defects with one root, found by the fold-checkpoint work's first
integration gate. (1) THE RACE: restore() never quiesced the generation
store, so a background flush could write into _system/ while the swap was
removing it — observed as ENOTEMPTY mid-swap when a checkpoint stamp landed
between readdir and rmdir. The swap now runs inside the store's exclusive
section (runStateReplacement): flush timer disarmed, pending tier and
checkpoint accumulator discarded BEFORE any directory moves. (2) THE
INHERITED ASSERTION: a snapshot carries its source brain's clean-shutdown
marker and fold checkpoint, but the restored files were bulk-copied without
per-file fsync — the inherited stamps would suppress exactly the recovery
fold that cures a post-restore power cut. reopenAfterRestore now deletes
both stamps before reopening: the open treats the store as uncleanly shut,
folds the restored log into canonical, barrier-syncs what it re-applied,
and stamps fresh — the restored state is durably founded at restore time
instead of borrowing assertions about bytes this disk never synced.
Pinned: restore under in-flight traffic completes; the pre-restore stamp
does not survive; the post-restore stamp is the reopen fold's own, at the
restored watermark.
The fold checkpoint (_system/fold-checkpoint.json) is stamped strictly after
a canonical-sync barrier over every live entity touched since the last stamp
(syncEntityCanonical: ids → canonical paths → fsync; an absent file fsyncs
its parent directory so deletes are as durable as writes). An unclean open
under log authority now folds only (checkpoint, head]; the chain bootstraps
at an empty brain's adoption (three-phase hooks around adoptLogAuthority) or
at a brain's first whole-log fold — existing brains converge at their first
crash with zero regression. Rollback restores sync immediately; abort paths
feed the barrier; a failed barrier retains the old bound (bigger fold later,
never a lost write). Five structural pins including boundedness itself.
Also: the production-shaped write-flow gate leg (mixed traffic racing
flushes, crash mid-traffic, every ack survives — from a consumer-reported
gate miss), and two release-ceremony cures (tag-first push so the publish
never queues behind the release commit's CI run; raw-curl npmjs shasum
probe with propagation grace instead of a one-shot false divergence).
An adopter's full suite found two v2 write-path defects on fresh brains,
reproduced with stacks; both cured and both pinned with their exact
production shapes:
1. PAD-FRAME CONSTRUCTIBILITY: a single msgpack bin filler steps its
header by one byte at each size class (bin8→bin16→bin32), leaving one
unreachable payload size per boundary — the sealer requested a
291-byte pad, the encoder threw 'not constructible', and sync() died
whole. Construction is now TOTAL: the class-boundary holes bridge with
a trailing fixint beside the bin ({bin(n)} ∪ {bin(n)+fixint} covers
every size ≥ minimum). Pinned exhaustively: every size from the
minimum through a full sector plus boundary spill constructs
byte-exact and decodes as reader-invisible filler.
2. THE NON-MONOTONIC REFUSAL LOOP: the append-failure compensation
rewound the generation counter on ANY throw — including a covering
SYNC failure after a SUCCESSFUL append. The log carried generation N
while the counter re-minted N, and every later append refused
'non-monotonic (N ≤ head N)' — the write path wedged in a refusal
loop through deferred-embed retries and flush backoff. The
compensation now splits by phase: an append failure (log never took
the fact) fully compensates — un-buffer and rewind; a sync failure
after append earns the rewind ONLY if the appended fact is provably
dropped, otherwise the generation stays consumed and buffered — the
counter never re-mints a number the log may carry. Pinned: an
injected one-shot sync failure fails its write loudly and the very
next write mints fresh and succeeds, with the log scanning strictly
ascending end to end.
Also probed against the adopter's carried report: the 9.0 vfs.rename
stale-ghost shape does NOT reproduce on this head (old path cleanly
unresolvable on exists/stat/readdir after rename).
Gates: unit 2067/2067 (160 files) · integration 833 (97 files) ·
conformance 36/36.
A first adopter's namespace migration went 341 red on one class: the
never-carried-field refusal firing on CORRECT filters against fresh and
sparse stores — a freshly provisioned tenant refused its own first
filtered read, with the did-you-mean built for typos firing hardest on
day-one stores where nothing is wrong.
The ruled cut: a WHERE filter naming a field no row carries is SERVED
OPERATOR-TRUTHFULLY — eq/in/range/contains answer [] (nothing carries
it, nothing matches); ne and exists:false answer ALL rows (the equally
true complement — a blanket empty here would be silently wrong, which is
why the simpler cut was rejected); exists:true answers []. Served from
the field registry, with the did-you-mean demoted to a once-per-field
WARN. orderBy and genuinely ambiguous addresses KEEP their hard typed
refusals: no truthful order exists over an uncarried field, and
ambiguity is a contract error while absence is data.
Mechanics: the negative operator absorbs the FIELD_NOT_INDEXED throw as
its empty exclude set (the clause-level catch correctly zeroes positive
operators only); the egress matcher already agreed. Plus the
provider-seam belt: a field refusal thrown by a replacement metadata
manager is normalized to THIS package's UnresolvableFieldError at every
filter call site — one class identity for consumers, instanceof works
(a first adopter's cross-package finding).
Conformance: tests/conformance/sparse-store-cut.test.ts — the shared
operator rows both engines run (positive-empty, negative-all,
fresh-tenant day-one, orderBy refusal kept, compound composition).
Gates: unit 2065/2065 · integration 832 · conformance 36/36.
The last rung of the default-flip ruling: with the sentinel exemption in,
real production-shaped brains still refused adoption over state-differs
mismatches the backfill could not cure — rows written before the
hydration law carry denormalized wrapper fields that disagree with their
own metadata leg, and the previous as-is identity re-commit PRESERVED
that drift, so the oracle re-flagged it every pass and the flip never
happened. In practice the crash-safe default reached zero existing
brains: the exact outcome the hold ruling forbade.
The cure: the backfill now rewrites canonical in the LAW SHAPE — exactly
the wrapper the log's reconstruction produces (denormalized enumeration
fields derived from the metadata leg, which is their authority under the
field-addressing law; the embedding floats ride through byte-identical;
adjacency residue keeps its own rebuild path). The oracle then verifies
the rewrite before the flip — the same safety, no operator chore.
Log-ahead divergence classes (a log the witness denies) still refuse
loudly, exactly as before.
Classification note for the record: the flagged uuid-v7 rows postdate the
fact log's introduction, so they classify as state-differs (in-log,
drift-shaped) rather than pre-log — both classes ride the same backfill.
Pins: a manufactured depot-shape drifted wrapper adopts green with floats
preserved and metadata intact; log-ahead still refuses typed.
Gates: unit 2065/2065 · integration 832 · conformance 31/31.
The release-holding finding from the joint gate's six real depot brains:
the adoption path's positive-int mint check false-flagged the reserved
VFS-root sentinel (the all-zeros UUID, minted int 0 BY CONSTRUCTION at
genesis on existing brains) as a corrupt mint — so every existing brain
refused log-authority adoption and stayed on the old lossy-under-power-cut
durability, defeating the release's headline crash-safety exactly where
it matters most.
The exemption, at both mint seams (the host's minter thunk and the fact
log's encoder guard): int 0 is legal iff the id is the reserved root;
zero for ANY other id remains a corrupt-mint refusal naming the reserved
exception. The codec's u64 layer already tolerated 0 — only the guards
over-refused.
Pins: adoption goes green on a brain whose VFS root carries int 0 (the
depot-brain shape, previously refused) · a non-root zero still refuses
typed at the mint seam — held at the seam itself because a full write
SELF-HEALS a poisoned zero (the index cycle re-mints before the fact is
written, which is the correct outcome and was verified in the pinning).
Gates: unit 2065/2065 · integration 830 · conformance 31/31.
The quiet-loss cure regressed recovery: the new typed torn-record error
was correct at identity-read time but threw inside init-time recovery
walks, killing opens that previously survived. The boundary, redrawn:
- IDENTITY READS (get-by-id of a specific record, CAS blob point-get):
typed TornRecordError, unchanged — a caller who asked for THAT record
can act on the answer.
- SET-SHAPED READS AND WALKS (enumeration, pagination, batch hydration —
the paths recovery rebuilds and finds page over): HEAL PAST the torn
victim. The adapter's loud floor (error log + counted gauge) fires at
the encounter; the walk serves the remaining rows. One crash casualty
can no longer kill every query on its shard — or the open itself.
- WRITES OVER TORN RECORDS ARE THE CURE: the save path's read-merge, the
commit path's before-image capture, and the operations' rollback
captures all treat a torn prior as the create sentinel, narrated — the
incoming bytes replace the unreadable ones, and history for the id
honestly restarts at that generation. Corruption can never block its
own heal.
- THE NaN SOURCE: torn mapper state (nextId/entries carrying garbage)
discards with narration and re-derives via the existing rebuild path;
the mint gains a source guard healing a non-integer counter from the
live map. The reopen and first-write RangeError shapes are dead at the
source, both authority branches.
Pinned with the exact fault-injection scenarios: a torn entity record
(including the VFS root) no longer kills the open — walks heal past it,
the keeper rows serve, and the identity read of the victim itself is
typed-or-healed; a torn mapper reopens and mints sanely on the first
post-recovery write.
Gates: tsc 0 · unit 2065/2065 · integration 828 · conformance 31/31.
THE DEFAULT FLIP (ruled on proven evidence — at-ack survived 301/301
acked-writes-through-power-cut in block-layer fault injection; deferred
tree authority demonstrably loses flush-covered acks): a brain with NO
stored authority artifact now ADOPTS LOG AUTHORITY AT OPEN. The oracle
gates the flip exactly as the guarded adoption path always did — curable
divergences baseline-backfilled, the flip lands ONLY on a green verdict —
and a brain that cannot verify STAYS tree-authoritative loudly, with the
refusal recorded on the switch artifact so subsequent opens are cheap.
config logAuthority: 'defer' is the explicit documented opt-out (no
automatic adoption; declared flush-window loss; adoptLogAuthority() flips
later). A stored artifact always wins. RELEASES.md carries the posture.
Two standing .fails debt pins FLIP TO HOLDING under the default: the
at-ack crash-survival gap and the ack-at-log durability target — both now
permanent asserted truths, not aspirations.
POWER-CUT THROW SITES (fault-injection findings, brainy-alone config):
- A manifest-listed-but-unloadable column segment QUARANTINES at
discovery (loud once, counted always, quarantinedSegments() exposed for
the heal) and the field serves its remaining segments DEGRADED — never
a raw throw killing every query on the field. Real storage faults still
propagate untouched.
- Torn generation artifacts (NaN/garbage in manifest or counter) DISCARD
with narration at the store's open and recovery re-derives — plus a
defensive finite-integer guard at the init consumer. Never a RangeError
killing an open.
THE LOUD TORN-RECORD CONTRACT: an existing-but-unparseable stored record
now surfaces as a typed, counted TornRecordError on every entity-read
surface (including fifteen previously-blind per-item batch catches);
ENOENT stays clean-absent; artifact readers with designed absent-recovery
keep null-tolerance behind the loud floor. Disk corruption can no longer
read as silent data invisibility.
Suite migration: the default's pins inverted deliberately, generation
baselines made relative, quarantine-contract pins rewritten to the ruled
behavior.
Gates: tsc 0 · unit 2065/2065 (159 files) · integration 826 (93 files) ·
conformance 31/31 · kill-matrix 15/15 · torn-open guards 2/2.
An internal cross-engine fault-injection run (frozen-platter power-loss
capture) surfaced three release-gating findings; each cured in its owning
layer, each pinned:
1. WHOLE-LOG REPLAY ON UNCLEAN OPEN (the big one): log-authority replay
only covered facts ABOVE the manifest — but live canonical entity
writes are tmp+rename without per-file fsync, and the group-commit
flush syncs staging + manifest, never the live tree. Power loss could
therefore vaporize acked canonical bytes BELOW the manifest while the
log held every fact scan-clean (measured: 299 of 301 acks lost).
Now: a clean close stamps a clean-shutdown marker (fsynced, written
last); every open consumes it; an UNCLEAN open under log authority
folds the ENTIRE log into canonical — whole-entity after-images make
the re-apply idempotent and byte-safe. Zero cost on the happy path;
crash recovery pays one narrated fold. Recovery is replay: a crash is
just bigger lag.
2. TORN WRITER LOCK: power loss legally leaves the lock file present but
empty; the parse failure read as 'no holder' while the O_EXCL claim
EEXISTed forever — a PERMANENT lockout no staleness check could clear.
An unparseable lock is stale by definition (no live holder has one):
unlink loudly and re-loop; a racer rewriting a valid lock first wins.
3. PAIR GUARD: flush() called metadataIndex.stampWatermark unguarded;
a replacement metadata provider without the method killed the pair at
first flush. All three stamp calls are optional-chained — a missing
stamp is a verdict-side rescan, never a flush crash.
Pins: whole-log fold restores rows vanished below the manifest ·
clean-shutdown marker lifecycle (stamp/consume/re-stamp) · torn-lock
recovery with a fresh write after · stampless-provider flush.
Gates: unit 2055/2055 · integration 824 · kill-matrix 15/15.
The private recovery discipline, applied to its own machinery: pending-
embed markers stop being sidecar files and become first-class log records
riding the write's OWN commit fact — embed.pending lands in the same
atomic append as its after-image (a marker can never be orphaned from its
write, or vice versa; in durable-at-ack mode it shares the write's
covering fsync — zero extra syncs), and the worker's landing commit rides
embed.landed with the inline vector. Crash recovery is now a FOLD of the
log (pending without a matching landed = recovered), skipped wholesale on
brains with no v2 history; the one-time legacy bridge folds existing
sidecar files in, migrates them as one fact, and deletes them —
idempotent under a crash mid-bridge. No code path writes the sidecar
again.
Plus the ENTITY-TRUTH digest law, found by this train's own pins:
canonical vector wrappers denormalize HNSW residue (connections + the
randomly-assigned node level) that the log deliberately does not carry —
the verification oracle digested it and would have reported false
state-differs on ~any nonzero-level node (a ~15% flake in the cutover pin
was the symptom). Both sides of every oracle comparison now normalize to
entity truth (nounEntityTruth); index residue has its own rebuild path
and is not entity state.
Pins: embed-markers-in-log 5/5 (same-generation marker, landed+fold-to-
zero, crash recovery via the log with the sidecar prefix EMPTY on disk,
legacy bridge, VFS hung-embedder ack) · deferred-embedding 5/5 unchanged
(the contract outlived its mechanism) · kill-matrix 11/11 · cutover 5/5
×10 runs (flake dead) · unit 2031/2031.
One deterministic v2 log (nine facts covering every fold-relevant behavior:
genesis, after-images with minted ints, a deferred embed pending→landed,
a sameAsGeneration vector ref, a verb, a tombstone, and an all-deduped
empty commit) whose ENCODED BYTES and FOLDED STATE are both pinned by
sha256 literals. The fixture (tests/fixtures/golden-log-v2.bin, 4128 B,
byte-verified against the encoder on every run) is the shared artifact a
second reader implementation consumes — it must reproduce the identical
fold digest; the pair is normative on disagreement. The fold law is
stated in prose beside the code: generation-ordered latest-per-id,
tombstone masking, embed.landed vector application, single-hop ref
resolution, key-sorted digest.
Also: decodeGroupV2 discriminated pad filler by RECORD COUNT, silently
swallowing legitimate empty commits (an all-deduped batch at a real
generation). Pads carry generation 0 — which writers can never mint — so
the generation is the honest discriminator; empty commits stay visible.
Pins: 4/4 (encode-exact, fixture-identical, fold-exact, human-readable
spot checks beside the hashes).
- Watermark stamping fans out at flush: all three projections stamped
with the committed generation before their flushes persist.
- waitForIndexed(path?, {generation, timeoutMs}) — the one honest read
barrier for write-then-recall consumers; typed timeout error carries
the pending count and names the gauge; getIndexStatus() gains
per-projection gauges. awaitPendingEmbeds() unchanged underneath.
- adoptLogAuthority() self-backfills curable divergences (pre-log
records, witness drift) by identity re-commit before flipping — a
fresh brain flips clean; log-ahead divergences still refuse loudly.
- The verification oracle gains VERB legs (all four divergence classes;
unwired = honest verbsChecked: 0, never a scope claim).
- find({where: {}}) match-all serves (was silent-empty, warm AND cold;
same fix in count/streaming/subgraph seeding); removeMany({where:{}})
refuses typed — a match-all bulk delete must be explicit.
- Aggregation native envelope stamped via noteSourceGeneration before
serializeState; the native-blob restore gates through the same
adoption verdict as caller-side state (the unconditional adopt dies).
- LC8 pinned: a wholesale directory move opens and serves identically
across all three intelligences, with history traveling.
Gates: unit 2031/2031 (156 files) · integration 812 (91 files) ·
conformance 27/27.
The cutover: new tail segments write format v2 (per-record [type, version,
cipherFlag, keyId] envelope; noun/verb after-images carry dense ints
MINTED AT APPEND from the id mapper — a rebuilt mapper reproduces
assignments exactly; log.genesis opens every new log with the id-space
width + a minted brain id; sync() seals to the header-declared sector
boundary with reader-invisible pad frames). Existing v1 segments are
never rewritten — per-segment decoder dispatch reads both formats and v2
facts map to the exact CommitFact shape all consumers already read.
Cutover on a live v1 log: an empty v1 tail re-heads in place; a non-empty
one is sealed by rotation, byte-identical. Records reserve the encryption
fields (cipherFlag 0 / keyId nil are the only legal values; anything else
refuses typed naming the needed newer reader) — crypto-ready with no
future bump on the compat surface. Empty-records facts are legal (an
all-deduped batch is a real generation — v1 semantics preserved; the
refusal there tore a column-store flush mid-commit in the full suite, the
consistency guard caught it loudly, and the root is fixed).
Golden byte vectors pinned for the second (native) reader implementation.
Pins: cutover 5/5 · codec 54 · kill-matrix stays 11/11.
Two release-blocking findings from the durability kill-matrix, both fixed
in the owning layer:
1. LOG-AUTHORITY REPLAY AT OPEN: durable-at-ack fsynced the fact before
the ack, but open() truncated every fact above the manifest — after a
power loss that takes the un-fsynced tmp+rename canonical bytes, the
acked write's ONLY durable copy was discarded. Now: under 'log'
authority, open() REPLAYS intact facts above the manifest into
canonical (FactLog.peekFactsAbove — CRC-gated, order-sorted) and
advances the manifest to cover them; tree-authority brains keep the
truncate contract they were promised. Pinned end to end: the power-loss
row constructs the exact disk state (fsynced log, vanished canonical
rename) and the acked write lives.
2. NO SILENT COMMIT: commitSingleOp buffered the generation BEFORE the
fact append; an append failure (ENOSPC) rejected the caller but the
next flush durably committed the generation with NO fact — a permanent
silent log gap. Now the failure path un-buffers and returns the counter
reservation: nothing commits, the log stays gap-free, and the canonical
execute-residue orphan is the documented crash-equivalent.
Plus: the kill-matrix itself (11 rows — every commit-path fault point ×
reopen-as-crash recovery contract, at-ack variants, disk-full row; five
new zero-cost faultPoint sites), the log-authority pin suite (oracle
green/red/state-differs, flip refusal, switch survives reopen, 9/9), and
the group-commit covering pins (5/5).
Gates: unit 2002/2002 (152 files) · integration 785 · conformance 27/27.
The two-implementation contract surface as one pure module (no I/O):
segment header v2 (formatVersion 2 + sealSize in the reserved bytes),
per-record [type u8, version u8] envelope killing the unknown-kind
misclassification trap, the 12-type registry (after-images with minted
ints, tombstones, batch.meta, embed.pending/landed, blob.manifest,
projection.note, bootstrap.baseline, log.genesis with id-space width and
TYPED width-mismatch refusal), vectorLeg inline|{sameAsGeneration} with
writer-enforced single-hop, sector-sealed groups with pad frames, torn-tail
discipline, and GOLDEN BYTE VECTORS pinned so a second (native) reader
implementation can conform byte-for-byte. 50 format pins + a
fault-injecting storage wrapper (tear/drop-sync/fail-append) with 13
self-tests. v1 segments remain readable; nothing writes v2 yet — the
live-format cutover is its own commit.
The storage-authority adoption path, guarded shape: the canonical tree
stays authoritative by default ('tree'); a brain flips to 'log' only
through the verification oracle, and the flip is stored, per-brain,
checked at open only.
- FactLog.ensureSynced(): classic group commit — concurrent writers
append, then join ONE covering fsync (running + queued slots give the
covering guarantee: the sync a caller awaits always starts after its
append landed). Solo writer = immediate sync.
- GenerationStore.logDurability 'deferred' (default, byte-identical to
today: fact durability rides the group-commit flush, ack latency
unchanged) | 'at-ack' (log-authority mode: every single-op ack awaits a
covering log fsync — an acked write's fact survives power loss, by
contract). transact() was already durable-at-return in both modes.
- src/db/logAuthority.ts: the stored switch artifact
(_system/log-authority.json, absent = tree), readLogAuthority, and the
VERIFICATION ORACLE — replay the fact log, fold latest state per id
(digests, never bodies — memory-bounded), diff against the canonical
tree paged; verdict green iff every canonical row is exactly reproduced
AND the log claims nothing canonical denies. Divergences are NAMED by
class (pre-log-record → needs baseline backfill; state-differs;
log-live-canonical-absent; log-tombstone-canonical-present). The flip
REFUSES on red with the first divergence and the cure in the message.
- Brainy: authority read at open (log → durable-at-ack enabled);
logAuthority() / verifyLogAuthority() / adoptLogAuthority() public API.
Nothing flips by itself; nothing changes for existing brains.
The write side of the law, ruled 2026-08-03: data is either in main space
where developers can use anything, or it is in system.*.
- The reserved-name write door DIES: add/update/relate/updateRelation
metadata bags accept EVERY name (confidence, type, id, data, level,
content, ...) as ordinary user fields — indexed, filterable, sortable,
aggregatable, identical to any other field. The remap/enforce/warn
machinery, the reservedFieldPolicy config (now a typed init refusal),
and the compile-time metadata key bans are all removed. The one write
refusal left: keys spelled 'system.*' (namespace forgery), now enforced
on all four write doors.
- STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag
nested verbatim under 'metadata', sealed by a format stamp — by-name
storage discrimination is unsound once colliders are admitted. Legacy
flat records stay readable forever through the shape-aware splitters
(sound for them: the old door refused colliders). Time travel rides the
same split (generation store snapshots whole records).
- Name-based index exclusions DIE: user frame indexes every name; the
excludeFields/indexedFields knobs and their silent-[] holes are gone;
bulk-payload protection is value-shape only, uniform across names.
- Consumer-sweep findings fixed in the same wave: per-type counts read
the frozen 'system.type' column (addToIndex sort, affinity tracking,
cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for
pre-rebuild reads); resolveHiddenIds addresses 'system.visibility'
(bare 'visibility' was a silent no-op under the law — VFS/system
entities leaked into default reads).
- Fidelity fallout fixed in the owning layers: readEntityFieldAddress
reads the bag first (colliders were absent-shadowed by its own guard)
and never serves system addresses from the bag; blob history refs read
the bag shape-aware; migration transforms now receive ONE normalized
view (engine fields + nested bag) regardless of stored era, and stray
flat-habit keys refuse with the fix in the message.
- THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as
gates-green): all ten collider names + plumbing names written as user
fields, verified verbatim + queryable across live reads, flush+reopen,
a forced epoch rebuild, and asOf time travel; relation mirror; forgery
refusals; legacy flat-record compat. 8/8 green.
Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance
27/27 (exit 0) · consumer test sweep migrated (10 files).
A production incident: a native-provider op ground 38-40s inside a transaction,
blew the apply-phase budget, rolled back, and a downstream pipeline hot-retried
the identical operation into a 6-minute CPU storm. Brainy itself never
auto-retried the timeout; the gap was that TransactionTimeoutError only said
"retryable" in prose, with nothing machine-readable for a caller to branch on.
- TransactionTimeoutError gains two typed, always-true fields: retryable
(a later attempt may succeed once the slowness resolves or the budget is
raised) and hotRetryUnsafe (an immediate identical retry re-pays the full
cost that just timed out and can cascade into a CPU storm -- callers must
latch and back off, never loop). context's existing telemetry fields
(timeoutMs, operationIndex, elapsedMs, totalOperations, operationName) are
now documented as the caller's backoff inputs.
- Updated the "retryable" doc-prose sites (transact()'s timeoutMs option,
transactionBudgetFloorMs, Transaction.execute()'s contract) to point at
the new fields instead of bare prose.
- Regression pin (tests/unit/transaction/timeout-never-internally-retried.test.ts):
an execution counter proves the engine never re-drives a timed-out
operation, through both the single-op engine TransactionManager/Transaction
drives for every single-record write, and add()'s upsert-race retry loop
(which must exit on the first TransactionTimeoutError, never treat it like
the lost-insert-race signal it retries on).
- Removed TransactionManager.executeTransactionWithResult -- zero callers
anywhere in the codebase.
The packed tier goes live inside GenerationStore:
- Two-tier reads: getDelta / readBeforeImage / readGenerationRecords
fall through live-tier → sealed segments (live-tier-wins: a crash
mid-fold leaves a duplicate representation, never a gap). Cold-open
seeds committedRanges from the segment manifest via interval merge —
packed generations resolve without their directories existing.
- repackHistory({timeBudgetMs, batchGenerations}): folds cold
generations (older than the newest 1024) oldest-first into sealed
segments, deleting per-generation directories only after segment +
manifest are durable. Public API + automatic time-bounded pass at
close() (before compaction, so reclaim can drop whole segments);
re-representation only — the sole history transform under the
archival profile. Early stop = consistent prefix, next pass resumes.
- compact(): packed generations reclaim logically in the loop and
physically at whole-segment boundaries via dropSegmentsBelow (the
frozen partial-segments-wait rule).
- generationDigest(g) (D8): deterministic content digest through g —
sealed-segment checksum chain + live-tier delta hashes; O(segments +
live window); RangeError out of range, GenerationCompactedError
below the horizon (a gate can never silently pin reclaimed history).
Four end-to-end pins: asOf answers byte-identical across fold + cold
reopen with folded dirs physically gone; repack+reclaim composition;
digest reopen-stability/divergence/loud-horizon; budget no-op+resume.
First stage of the co-frozen D1+D3+repacking unit: the format core,
self-contained under _generations/segments/.
- seg-<firstGen,20pad>.bgs: append-once packs of consecutive
generations (magic BGS1; frame = u32 len + u32 crc32c + msgpack
[generation, timestamp, delta, records, flags]; flags reserves
compressed-payload evolution without a format break). Sealed
segments are immutable — fold refuses overlap with sealed ranges.
- seg-<firstGen>.idx: DERIVED sidecar (per-generation frame offsets +
per-id generation postings + checksums); lost/corrupt sidecars
rebuild from their segment loudly; a damaged segment (frame CRC
mismatch) fails loudly, never serves wrong bytes.
- manifest.json: the one discovery path — open() reads it and never
lists the packed backlog (the scan-wedge class's cure); refuses a
newer manifest version rather than serving partial history.
- D3 semantics: dropSegmentsBelow reclaims WHOLE segments at
boundaries only and bumps compactedBelow durably; archival-profile
enforcement stays with the caller per the co-freeze.
- D8 rider: digestThroughPacked(g) — deterministic crc32c chain over
sealed-segment checksums (+ frame-level prefix mid-segment),
O(segments), reopen-stable.
Also fixes a cross-adapter contract bug the suite caught: memory
storage's deleteObjectFromPath ignored the raw-bytes store, so
deleteRawObject on a raw-bytes path (fact-log or segment files)
silently no-op'd — deletes now match filesystem unlink semantics.
Six pins. Wiring into GenerationStore (two-tier reads, the repacker,
cold-open manifest path) lands with the rest of the unit before its
release; cortex's fact-record/stamp shapes reconcile the sidecar
keying when they post.
Stage-2 D1 contract item (co-frozen): a fact scan may be slow, never
silent. batches() now races its FIRST pull against
SCANFACTS_FIRST_BATCH_MS (10s, exported; test-overridable) — a wedged
or unreadably slow store produces a loud abort naming the contract
instead of a consumer hanging indistinguishably from progress (the
production shape: a heal against a generations-backlogged brain
wedged silently on the first segment read).
Only the first pull is raced: the bound is time-to-first-batch (proof
the producer is alive), not per-batch pacing, and it runs only while
a pull is pending — consumer think-time between pulls never counts
against the producer (pinned).
Three pins: wedged-store loud failure within the bound, healthy scan
untouched end-to-end, slow-consumer immunity.
flush() is durability work: it must cost what the current window's
deltas cost, never what the history backlog costs. Under adaptive
retention the byte budget derives from free memory, so bulk-load
pressure shrank the budget exactly at peak write volume and flush paid
actual reclaim inline — a production deployment measured single writes
blocked 25-191s behind reclaim-on-flush.
- flush() no longer calls autoCompactHistory(); close() is THE
auto-compaction site (already ran there; now alone).
- Every auto pass is time-bounded (CLOSE_COMPACTION_BUDGET_MS = 5s):
reclamation is oldest-first, so an early stop is a consistent prefix
and the next pass resumes. Explicit compactHistory() gains an
optional timeBudgetMs for caller-chosen maintenance windows.
- Documented trade stated where operators read: a long-lived writer
that never closes accumulates history until its next explicit
compactHistory() — predictable writes, explicit maintenance.
Pins: flush-never-reclaims + close-reclaims-durably (db-mvcc), bounded
pass stops-then-resumes as a consistent prefix (generationStore unit).
Under default adaptive retention, every flush() recomputed total history
bytes by walking EVERY committed generation's delta — O(all generations)
with disk re-reads past the 4096-entry delta-cache bound. On a production
brain with 70,000+ accumulated generations this turned every write into a
full-tail scan (60-100s writes, escalating with history growth), even
though the free-RAM budget never tripped and nothing was ever reclaimed
(SELF-GENERATIONS-GROWTH).
historyBytes() now maintains a running total: seeded by one walk on first
use, then updated incrementally at both commit paths (+bytes) and the
compaction reclaim loop (−bytes), dropped on reopenAfterRestore. The
adaptive retention check on every flush is O(1). Invariant regression-
pinned: running total ≡ fresh walk through transact commits, single-op
group commits, and compaction.
New brain.historyStats() (exported HistoryStats): read-only generation
count / bytes / generation+timestamp range / horizon / retention mode /
effective budget — the one-call per-brain fleet audit for retention
exposure.
- The transact apply budget scales with the batch — max(30s, opCount x 2s) —
or is exactly the caller's new TransactOptions.timeoutMs. A flat 30s cap
silently limited honest bulk work to ~15 operations on network-attached
storage (~2s/op measured in a production import) while looking generous for
small batches. Internal batch paths (removeMany chunks) get the same
scaling via the shared transactTimeoutBudget helper.
- TransactionTimeoutError now reports the operation it stopped at as i/N with
the operation's name, elapsed vs budgeted time, and states the batch rolled
back atomically and is retryable; context carries the fields
programmatically.
- The transact operational envelope is documented (optimistic-concurrency
guide): budget math, chunking with ifAbsent idempotency, when to reach for
addMany vs transact, and the precompute pattern (embedBatch + per-op
vector) that keeps inference out of the commit path.
- Embedding claims made honest per an end-to-end probe of the built dist:
batch and single embedding paths produce bit-identical vectors and a
vector-supplied add is fully searchable, but batch throughput on the
default WASM engine measures comparable to sequential (~160ms/text) — the
5-10x speedup claim in the addMany JSDoc is replaced with the measured
reality and the actual win (inference outside the budgeted write path).
- New public guide (guides/external-backups): the sparse-mmap reality
for external backup tooling — why a brain directory can show 100+ GB
apparent size on a small disk, which files are sparse, tar czSf /
rsync --sparse / cp --sparse=always, live-store caveats, and what
persist()/restore() already handle natively.
- New public concept (concepts/generation-fact-log): what facts are
(after-image commit records, body-less tombstones), the crash-safety
model (checksummed frames, reconcile-to-committed, absent = never
committed), the scanFacts()/factSegmentPaths() surfaces with the
telemetry shape, family stamps + open-time coherence, and the
storage capability seam for plugin authors.
- Cross-link from the snapshots guide; sparse notes added to the
public persist()/restore() JSDoc (the operator-facing sites).
- storage.committedGeneration(): the committed watermark exposed on the
fact-scan capability, so a provider compares its stamp's
sourceGeneration against the store's truth without parsing the
private manifest format (the capability source now carries both the
live fact log and the committed closure).
- Pinned contract tests: fsync-before-ack (transact passes today; the
single-op path is pinned via it.fails as the documented target —
group commit must become LATENCY batching, ack waiting on the shared
fsync, never durability skipping; the pin flips red when that lands
so there is no cliff to discover) and scan-stability-under-rotation
(a scan snapshot yields exactly its facts — no gaps, duplicates, or
bleed-in — while segments rotate beneath it; the reclaim-during-scan
variant lands with fact compaction, which does not exist yet).
- FactLog accepts a rotateBytes option so tests exercise real rotation.
Three additive surfaces for native index providers, which hold only
`storage` and must never construct their own fact-log reader (the log's
open path is writer-side — it reconciles by truncating/rewriting):
- Fact-scan capability on the storage adapter (the getBinaryBlobPath
pattern): the host brain wires a closure over its LIVE fact log at
init; providers call storage.scanFacts()/factLogHeadGeneration()/
factSegmentPaths() and fall back to the enumeration walk on null.
Restore/reopen swaps the underlying log transparently.
- The family-stamp trio (readFamilyStamp/writeFamilyStamp/
verifyFamilyStamp + types) exported from @soulcraft/brainy/internals,
so 'one verifier' is literally one function shared with the native
side, never two synchronized copies.
- Rollup invariants widened to number|string: content fingerprints
(e.g. a per-tree SHA-256) are valid invariant values; strict equality
either way, and a type mismatch reads as incoherence, never a pass.
The canonical entity tree now carries a FAMILY STAMP
(_system/family-stamps/entity-tree.json, JSON — forensics stay
terminal-readable): which committed generation the tree reflects
(sourceGeneration) plus the rollup invariants (entity/relationship
counts) that verify a millions-of-files projection whole where per-file
checks cannot scale. Written at flush/close boundaries; open-time
coherence is a COMPARISON, not a walk:
- coherent / absent (legacy store) → silent
- behind → benign for the tree (it is written BY the commit; only the
stamp is stale after a crash between commit and flush) — refreshes at
the next flush
- incoherent (counts diverged at equal generation, or a stamp AHEAD of
the log head) → loud, names the failing invariant; repairIndex()'s
unconditional recount heals and RE-STAMPS so repair leaves a coherent
stamp behind
- a fault reading the stamp is UNVERIFIABLE — never conflated with
absence
One verifier (verifyFamilyStamp, exported) reads both member modes:
enumerated (exact byte size per member, bounded families) and rollup
(invariants, unbounded families). New exports: readFamilyStamp,
verifyFamilyStamp, ENTITY_TREE_STAMP_PATH, FamilyStamp, StampMembers,
StampVerdict.
Every committed generation now also appends a FACT — an after-image
commit record (what each touched entity/relationship became, or a
body-less tombstone for a removal) — to an append-only, crc32c-framed
segment log under _generations/facts/. The before-image history and the
canonical tree remain authoritative; the fact log gives consumers ONE
sequential, self-verifying stream (index heals, incremental replays)
in place of a per-entity directory walk.
- Wire format: positional msgpack facts [generation, timestamp, ops,
meta, blobHashes]; op = [kind u8, id bin16, record | nil tombstone];
32-byte segment header (magic, formatVersion, firstGeneration,
zeroed+verified reserved); length+crc32c frame per fact; zero-padded
segment names so lexicographic order == generation order; JSON
manifest with an atomic rename flip, manifest-first rotation.
- Commit protocol: facts append+fsync BEFORE the commit point inside
the existing durability window, so a crash can only leave the log
AHEAD of committed truth — open() truncates back (torn tails detected
by CRC). Absent generation = never committed; a scan can never see an
uncommitted fact. transact() facts are durable-on-return; single-op
facts ride the group-commit flush exactly like buffered history. A
fact-append failure fails the write, loudly — a silent gap would be a
lie a later replay discovers.
- New public surface: brain.scanFacts() (sequential batches with heal
telemetry: head/segments/approx up front, per-batch generation range
+ bytes + segment id, loud abort on gaps, summary cross-check) and
brain.factSegmentPaths() (immutable sealed segments for zero-copy
consumers; the mutable tail excluded). Exported types CommitFact,
FactOp, FactScanBatch, FactScanHandle.
- Storage: optional binary raw-byte primitives (appendRawBytes,
readRawBytes, writeRawBytes, rawByteSize) on StorageAdapter —
feature-detected; filesystem + memory adapters implement them; an
adapter without them hosts no fact log. Fact segments are byte-copied
(never hard-linked) into snapshots. The _generations/facts/ namespace
is registered as a protected family (rebuildable: false): no sweeper
or GC may delete under it.
- New crc32c (Castagnoli) utility with RFC known-answer tests.
