The safety sweep is armed alongside the watch, and startFlushRequestPolling()
declines to arm over an existing interval — so when a watch died mid-life the
fallback did nothing and the store quietly answered flush requests on a 30s
cadence instead of the 500ms one the door promises. The sweep is cleared first.
A degrade nobody asked for is still a degrade.
Arming is asynchronous — the request directory is created before it can be
watched — so during that window neither the watcher nor the sweep interval
exists yet and the guard let a second call through, leaving two watchers and
two sweeps for the life of the store. The callback is the flag that covers the
window.
Three idle-burn items from the steady-state audit, and one correction.
THE FLUSH-REQUEST WATCH (the strongest of them). It readdir'd the request
directory every 500 ms, per brain, for the life of every writer — armed on
every non-reader brain whether or not any inspector process existed. In a process holding many stores that is tens of directory reads per second
on a completely idle service, plus a stale-request GC on every one of them. It now
uses fs.watch, so the arrival itself wakes it and a request is seen SOONER
than the poll saw it. Two concessions ride along, both stated in the code: a
30s safety sweep (fs.watch drops events on some network and fuse filesystems,
and the GC needs a tick of its own — two orders of magnitude fewer reads than
the poll made), and a fall back to the original 500 ms poll, narrated, on a
filesystem that cannot watch at all, because an inspector whose request is
never seen waits forever.
THE WRITER HEARTBEAT goes 10s → 60s. It is observability ONLY — staleness is
decided by pid liveness and the fence compares pid + hostname, so no decision
anywhere reads the timestamp — and at 10s it was a lock-file write every ten
seconds per brain forever, for a value nothing computes with. An operator
still sees a heartbeat inside the minute.
THE HEALTH NARRATION dedupes by CONTENT, not by the provider's generation
counter. That counter bumps on every ledger mutation and rebuild boundary, so
a provider bumping it on routine work re-emitted the same unchanged line on
every read, while one that never bumped could suppress a line whose reasons
had genuinely changed. The generation is still reported; it no longer decides
whether the line is worth saying.
CORRECTION, and it is against my own earlier claim: the idle-flush commit read
a reported idle-CPU observation (many stores, no writes, a flush every ~35s,
over a core burned) as caused by
the flush path. That does not follow — this engine's cadence is write-driven
(every trigger runs through noteWriteForPersistence, which only a committed
write calls), so something was CALLING flush() on those brains and the caller
is still unidentified. The clean-flush gate makes such a call free; it does not
account for it. The code comments and the idle lane now say exactly that.
Pins: tests/integration/flush-watcher-event-driven.test.ts — an idle writer
makes at most one request-directory read in 8 seconds (the old poll made ~16),
and a dropped request is still acked well inside the safety sweep.
MEASURED on a production-shaped store (14,056 nouns / 72,679 verbs, an 11 GB
generation history), measured solo under an exclusive lock: the generation-store phase cost 55,538 ms of a WARM
REOPEN after a clean close — with the fold correctly skipped, so nothing in
that phase's name explained it.
This is what it was doing. Discovering which generations exist on disk called
listRawObjects('_generations'), which RECURSES the whole tree and returns
every file in every generation directory — to extract a set of integers that
the top-level directory NAMES already spell out. The cost scales with the
entire history, is paid on every open, warm or cold, and grows for the life of
the store.
A one-level door — listRawPrefixes(prefix), the immediate child directory
names — is added to the storage seam. The filesystem adapter answers it with a
single readdir; BaseStorage derives it from the recursive listing, so an
adapter without a cheap implementation is never wrong, only never faster; and
the generation store falls back to the old listing when the door is absent.
One behavioural difference, stated: an EMPTY generation directory is now
discovered where the file listing could not see it. Above the committed
watermark that is a crash scar, and recovery already has an explicit branch
for it ("indeterminate partial dir" — dropped, narrated). Below it, it becomes
a resolvable generation holding no records, which is what an empty generation
means.
Suites: the durability kill matrix (15), db-mvcc (30), history repacking (4),
rollback trapdoor (3), entity-tree stamp (4) and the full unit suite (2,105)
all green.
MEASURED on a real store: the ALL-visibility ledger read 14,231 nouns against
14,056 identity records and 72,729 verbs against 72,679 — exactly that store's
25 noun and 50 verb SCAR directories. Two copies of the same archive derived
different numbers (14,231 and 14,081), because each had been persisted at a
different moment under the old rule that counted one entity per id DIRECTORY.
A downstream index heal subtracted against that denominator and reported
remaining work that did not exist.
The scan already applies the right predicate — one entity per IDENTITY RECORD
(the metadata content leg), shared with pruneOrphanedEntities so the two agree
by construction. What was missing is that a ledger persisted under the old rule
was only FLAGGED suspect and then went on serving its wrong numbers for the
life of the store, waiting for an operator to run repairIndex.
- The ledger now derives itself honestly in the BACKGROUND after the open,
narrating start and finish with the correction it made. Background because
these scalars are denominators — no read is served from them — and because
walks exactly like these are how a 24,898-id store spent minutes of a
restart in silence. Observable via whenCountLedgerSettled(); nothing in the
read path waits on it.
- A derivation that raced a write refuses to stamp its number "exact": one
retry on a quiet store, then the ledger stays SUSPECT and says so, naming
repairIndex as the door that recounts under a barrier.
- The one derivation that CANNOT leave the foreground says why it cannot:
getNounCount()/getVerbCount() are served from it, and a background walk
would make a populated store answer "0 entities" — a wrong answer, not a
slow one. It narrates its start and its wall instead.
- counts.json is written temp+rename. A truncating write left a window —
measured at roughly 750ms after a flush or close — in which a concurrent
reader saw the file EMPTY; an unparseable ledger sends the next open down
the full-rescan path, so the cheapest file in the store was buying the most
expensive recovery.
- The writer lock's clean-close record is now consulted before the
same-process branch too: a restart reported "Re-acquiring writer lock ...
this is a bug" immediately after a clean close, sending an operator after a
leak that did not exist.
Pins: tests/integration/count-ledger-identity-record.test.ts (background
correction with scar and ghost fixtures, two copies of one archive agreeing,
counts.json never observed unparseable across 40 persists);
tests/integration/ledger-derivation-identity.test.ts updated to the new law —
the OPEN still never walks (proved by slowing the walk 1.2s and timing the
open), and the ledger heals behind it.
A production restart made this necessary: a service stopped with exit code 0,
having awaited close() on every pooled brain, and its next boot announced
"Overwriting stale writer lock ... appears dead" for every store it owned.
Nothing had crashed. "The recorded pid is gone" is equally true of an orderly
restart and of a crash, so the verdict could not tell an operator which one
they had — and when the OS recycles a pid it fails the other way, refusing to
open a store whose writer died days ago.
Three changes, all at the law:
- close() is two parts, and the second is unconditional. The durable steps
(flush, markers, component close, plugin deactivate, buffer drain) move to
closeDurableSteps(); the terminal releases — the flush-request watcher, the
WRITER LOCK, the VFS timers, the terminal `closed` flag — always run. The
original failure is narrated with what it costs the next open, then rethrown.
- releaseWriterLock() writes a CLEAN-CLOSE RECORD (`locks/_writer.close`)
naming the lock generation it released; the next claim consumes it, so a
record can never vouch for a later crash. An open reads the record instead
of guessing: recorded → nothing to recover; absent → say so, and name the
crash recovery this open will now run.
- The signal path stops failing in a batch. It was one try around a loop over
every open brain, so the first instance whose flush rejected stranded every
remaining brain's lock and markers — at exit code 0. Now: per-instance
isolation, the generation store's close (the clean-shutdown marker, without
which the next open folds the whole log) is part of shutdown, the lock is
given up in a finally, and the handler no longer calls process.exit() when
the host application has its own signal handler — that race truncated the
host's own close() mid-flight.
Pins: tests/integration/writer-lock-clean-close.test.ts — completed close
leaves no lock and a consumed-once record with a silent reopen; a failing
durable step still releases and still rethrows; SIGKILL leaves the lock with
no record and the reopen names the crash; a host SIGTERM handler runs to
completion.
Branch plan (10 lines):
1. writer lock: clean-close record + always-release close [this commit]
2. open narration: an always-on channel; production clamps prodLog to ERROR,
which is why a three-minute open printed nothing
3. open narration: per-phase lines as each phase ENDS, with progress cadence
4. measure both real-store fixtures on the box, before/after
5. move the generation-log fold out of the foreground where the serving law
allows; durable resumable progress marker
6. same for the VFS bootstrap
7. counts: a legacy container-rule ledger must not keep serving wrong
denominators; counts.json written atomically
8. counts pin with scar directories; two copies of one archive agree
9. docs/canonical-layout-ratification.md — 12 facts confirmed/corrected
10. report: MEASURED before/after, findings, and whether this is 10.4.4
The engine-pair seam law: a zero-norm vector never crosses an engine
boundary. The index belt already refused to insert one, but the canonical
write and the vectored-noun ledger still counted it, so a near-empty store
whose only vectored row was zero-norm read "1 canonical vectored vs 0
indexed" and threw a not-ready error at open, and a store's own legacy
zero-norm VFS root could trip the same gate before its VFS-init-time cure
ever ran.
- add()/update() (single and transact()) now normalize an explicit
real all-zero vector to the unvectored [] shape before the dimension
pin, the ledger flag, and the index ops ever see it (loud, one warn per
write, canonical write still succeeds).
- The legacy counts.json derivation walk (scanVectoredNounCount) excludes
a persisted zero-norm row, matching the live ledger's definition.
- A legacy zero-norm VFS root now migrates at open, before the vector-leg
gate evaluates, via one O(1) fixed-path read (torn-tolerant — skips
rather than aborting init on a torn root, letting the recovery walk
heal it) — independent of whether a VirtualFileSystem is ever
constructed this session.
- update({ id, vector: [] }) (and the same op inside transact()) is now
the sanctioned, idempotent unvector door: index removal, exactly-once
ledger decrement, no re-embed, and it clears a pending deferred-embed
marker rather than leaving it to re-vectorize the row later. The
combination with deferEmbedding is a typed refusal.
- JsHnswVectorIndex.rebuild() now skips a zero-norm/empty persisted
vector when repopulating from canonical (the same belt the live
add/replace paths already had), and health()'s index-parity check now
compares HNSW size against the vectored-noun ledger rather than the
raw metadata-entry count, since a store's VFS root is permanently
unvectored by design.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The noun/verb pagination walks (getNounsWithPagination,
getNounIdsWithPagination, getVerbsWithPagination) listed shard contents by
filtering for vectors.json, while the canonical count ledger has always
counted a row by its metadata.json presence alone. A row with metadata and
no vector file was therefore counted by the ledger but never yielded by the
walk — a permanent "counted but invisible" phantom for any downstream
consumer that iterates the walk to account for the ledger's total.
Nouns now enumerate by metadata.json and hydrate the vector leg optionally,
yielding the sanctioned unvectored shape (vector: []) when it's absent.
Verbs enumerate the same way, but a metadata-only verb row can only be fully
reconstructed when sourceId/targetId happen to be recoverable from metadata
(never true for a current production write — those fields live only in the
vector leg); otherwise the row is counted but loudly skipped rather than
fabricated, since a phantom edge with fake endpoints would be worse than the
original defect.
Separately, GenerationStore's recovery-fold replay (replayFact) now applies
preserve-if-absent: a metadata-only after-image replayed over an already-
vectored row carries the existing vector forward instead of deleting it via
writeNounRaw/writeVerbRaw's exact-restore null-means-delete contract (which
must stay exact for transaction-abort rollback). A genuine tombstone still
removes both legs.
A zero-norm vector is lawful inside brainy (cosine distance scores it at
maximum, never a false top hit) but a false attractor for a downstream
engine serving squared-euclidean distance, which cannot tell a real
all-zero vector apart from a legitimate origin point.
- The VFS root now persists with vector [] (the existing "unvectored"
shape) instead of a real all-zero 384-dim placeholder, and is never
routed into the deferred-embed pipeline.
- A one-time migration in the root-init path detects a pre-fix store's
all-zero placeholder root (by norm, not length) and rewrites it to []
through a new sanctioned Brainy method that keeps the canonical
vectored-noun ledger honest and removes the row from the vector index.
- The vector-index write seam (AddToVectorIndexOperation,
ReplaceInVectorIndexOperation, and the generation materializer's direct
insert) now refuses any real all-zero vector before it reaches a
provider, loudly naming the entity, while the canonical write still
lands.
- add()'s dimension-pinning and HNSW-insert gates, and the add-params
validator, now treat any empty vector as carrying no dimension
information, closing a latent trap where an explicit `vector: []`
would have pinned dimensions to 0.
The coverage denominator the health-by-accounting ratification named for
the vector family — never built until now, and its absence was measured as
the exact outage class it existed to prevent: a migrated store with
canonical vectors and no derived index opened with the vector leg EMPTY,
served [] from vector search with no error, and the report-driven read gate
had nothing to refuse on (the provider's coverage invariant was honestly
unledgered — the denominator was ours to supply).
- getCanonicalCounts() gains vectors: { all } — the count of canonical
nouns holding a REAL vector. Incremented where a vector lands (the
isNew-gated metadata seam for explicit vectors — the same discipline that
keeps HNSW neighbor-link re-saves from inflating counts; a narrow
noteVectorLanded hook for the deferred-embed landing, gated on the
worker's own pre-embed read). Decremented on a proven delete of a
vectored noun; a vector-uncertain delete marks the ledger suspect rather
than guessing (no new reads on the delete path). Recounted by the
sanctioned recount; legacy counts.json derives it once (a deferred noun's
vector file exists with an empty vector, so presence requires one
content read at derivation — never on the hot path).
- The open gate's vector leg: when a health-reporting provider claims
serving while the index holds zero nodes and the ledger proves vectored
canonical rows exist, open BUILDS (narrated) — routed through the
provider's idempotent fillFromCanonical() when exposed (the joint door;
a partial shortfall stays repair()'s operator business), the JS rebuild
otherwise — or fails typed pre-serve. Scoped exactly: bare isReady()
providers, migrating providers, and white-box size stubs open as before.
Pinned end-to-end from the partner gate's probe shape (store with vectored
canonical rows, no derived index, reopen → search serves N, never []),
red-proved against the pre-fix path; the inverse (zero vectored rows) opens
without building and serves [] honestly.
The factory loudly rejects every REMOVED pre-8.0 path key, but an unknown
nested `config` object (e.g. `storage: { config: { baseDir } }` — a shape
that was never supported) fell through SILENTLY to the zero-config default
directory. Every instance constructed with such a shape wrote to ONE shared
on-disk root while its caller believed each had its own — found live when
two integration tests' brains shared a store across an entire
single-process CI run and a health probe refused on the foreign edges it
sampled. A nested `config` carrying any path-shaped key now throws the same
loud migration error, naming the canonical `path` rename. The two tests are
repaired to the supported shape (and now actually test isolated stores, for
the first time since 8.0).
A production restart storm measured 90,017ms for a single brain init vs
1,117ms quiet (~80x contention multiplier), traced to performInit() eagerly
awaiting the process-global WASM embedding engine before the VFS root even
existed. Every writer's open() queued on the one throttled model compile
(90-140s on throttled CPUs).
- VirtualFileSystem.doInitializeRoot() no longer embeds '/'. The root is
system-tier plumbing nothing ever searches; when the default WASM engine
is active it now gets an explicit all-zero placeholder vector
(cosineDistance returns max distance for a zero vector, so it never ranks
ahead of real content). deferEmbedding was considered and rejected: its
landing path kicks the embed worker synchronously right after commit,
which would still force the cold compile within milliseconds — just off
the awaited path, not avoided. A registered native 'embeddings' provider
(no cold-start cost, possibly a different dimension) still embeds the
root for real, via the new Brainy.usesDefaultWasmEmbedder() seam.
- performInit()'s eager-embedding step now only STARTS the WASM engine warm
in the background instead of awaiting it inline. embed()/embeddingManager
already serialize concurrent callers on one shared init promise, so the
first real embed() converges correctly either way; a failed warm narrates
loudly instead of surfacing as a silent latency spike or an unhandled
rejection. eagerEmbeddings: false still means no warm at all.
- FileSystemStorage.init() batches its ~8 independent bootstrap mkdirs
(each creates its own full subtree via recursive:true, so none depend on
the others existing) into one Promise.all. The restore-completion step
and initializeCounts() stay strictly sequential — they have real order
dependencies on rootDir and systemDir respectively.
- performInit() now times five phases (storage init / generation-store
open+fold / index init+gate / VFS bootstrap / embedding-warm-started) and
logs one warning with the per-phase breakdown when total open exceeds
2000ms; silent otherwise.
The storage-level unfiltered getNouns()/getVerbs() walks enumerate every
tier, but their totalCount reported the user-facing scalar, which skips
system/internal records on the write path — so a derived-index coverage
ledger comparing its posted count against that total would read
"over-posted by N" on every store with a VFS. This adds the ledger's real
denominators:
- totalNounCountAll / totalVerbCountAll: +1 for every new canonical record
regardless of tier, −1 for every PROVEN delete (record read, or the
caller's prior image), persisted in counts.json beside the counted
scalars, recomputed by the sanctioned recount (rebuildTypeCounts).
- The unfiltered storage-level totalCount is now the ALL scalar and is
never clamped: Math.max(scalar, scanned) could only move a scalar up, so
an inflated counter hid forever; a divergence is now visible and healed
by repairIndex().
- A delete that cannot prove the record existed never decrements on faith:
it marks the ledger SUSPECT (persisted, narrated once per session) and
the recount clears the flag with proof.
- getCanonicalCounts() on StorageAdapter (optional) exposes {counted, all}
per family plus the suspect flag — O(1), no I/O.
- A counts.json written before the ledger existed derives both scalars
once from the canonical id tree at open and persists them; absent keys
are a legacy file, never a zero.
User-facing getNounCount()/getVerbCount() are unchanged.
Pinned in tests/integration/canonical-count-ledger.test.ts (5 laws).
The plant's integration lane caught it twice: the fence's startedAt-strict
comparison turned the documented same-process warn-and-take-over path (two
instances in one Node process — the server-restart test pattern, and the
shared-default-store pattern across test files) into a flush-killer: the
first instance's background flushes latched dead while its own process held
the lock ('PID N no longer holds the lock — it is now held by PID N').
Ownership is per-process: pid + hostname. startedAt stays in the lock for
observability but not in the fence — it protects nothing (a pid-recycled
successor's victim is a dead process that runs no fence checks) and it
convicted the innocent. Pinned: a same-process re-open leaves both
instances' flushes working; the cross-process eviction pins unchanged.
Verified under the lane's exact command: 102/102 files, 850 passed, exit 0.
The production dev-store split-brain (two live writers alternating a store's
id-mapper between two internally-consistent truths), cured at all three of
its roots. (1) STALENESS REQUIRES PID-DEATH: the old rule evicted on
heartbeat age alone, so a >60s event-loop stall (debugger pause, GC, heavy
sync work) handed the lock to a second opener while the first kept writing;
a live process is now never auto-evicted — a wedged-but-alive holder is the
operator's call via {force:true}, and the heartbeat stays for observability.
(2) THE CLAIM IS ATOMIC: writeFile(wx)'s open→write→close left an empty-file
window a concurrent opener could read as torn, unlink a LIVE claim, and take
the lock; the claim is now tmp-write + hard-link — the lock appears with its
full contents in one step. (3) THE FENCE: every flush commit and transact
barrier verifies lock ownership first (one small read per window) — a
forced-out or lock-deleted writer fails typed (BRAINY_WRITER_FENCED) before
a single staged byte or manifest advance, instead of writing on unaware.
Pinned: live-with-ancient-heartbeat refuses typed; dead-PID self-clears
narrated; a forced-out writer's flush and transact both fence, advancing
nothing. Requested by a downstream team as single-writer guard or loud
lockout — this is both.
The fold checkpoint (_system/fold-checkpoint.json) is stamped strictly after
a canonical-sync barrier over every live entity touched since the last stamp
(syncEntityCanonical: ids → canonical paths → fsync; an absent file fsyncs
its parent directory so deletes are as durable as writes). An unclean open
under log authority now folds only (checkpoint, head]; the chain bootstraps
at an empty brain's adoption (three-phase hooks around adoptLogAuthority) or
at a brain's first whole-log fold — existing brains converge at their first
crash with zero regression. Rollback restores sync immediately; abort paths
feed the barrier; a failed barrier retains the old bound (bigger fold later,
never a lost write). Five structural pins including boundedness itself.
Also: the production-shaped write-flow gate leg (mixed traffic racing
flushes, crash mid-traffic, every ack survives — from a consumer-reported
gate miss), and two release-ceremony cures (tag-first push so the publish
never queues behind the release commit's CI run; raw-curl npmjs shasum
probe with propagation grace instead of a one-shot false divergence).
The quiet-loss cure regressed recovery: the new typed torn-record error
was correct at identity-read time but threw inside init-time recovery
walks, killing opens that previously survived. The boundary, redrawn:
- IDENTITY READS (get-by-id of a specific record, CAS blob point-get):
typed TornRecordError, unchanged — a caller who asked for THAT record
can act on the answer.
- SET-SHAPED READS AND WALKS (enumeration, pagination, batch hydration —
the paths recovery rebuilds and finds page over): HEAL PAST the torn
victim. The adapter's loud floor (error log + counted gauge) fires at
the encounter; the walk serves the remaining rows. One crash casualty
can no longer kill every query on its shard — or the open itself.
- WRITES OVER TORN RECORDS ARE THE CURE: the save path's read-merge, the
commit path's before-image capture, and the operations' rollback
captures all treat a torn prior as the create sentinel, narrated — the
incoming bytes replace the unreadable ones, and history for the id
honestly restarts at that generation. Corruption can never block its
own heal.
- THE NaN SOURCE: torn mapper state (nextId/entries carrying garbage)
discards with narration and re-derives via the existing rebuild path;
the mint gains a source guard healing a non-integer counter from the
live map. The reopen and first-write RangeError shapes are dead at the
source, both authority branches.
Pinned with the exact fault-injection scenarios: a torn entity record
(including the VFS root) no longer kills the open — walks heal past it,
the keeper rows serve, and the identity read of the victim itself is
typed-or-healed; a torn mapper reopens and mints sanely on the first
post-recovery write.
Gates: tsc 0 · unit 2065/2065 · integration 828 · conformance 31/31.
THE DEFAULT FLIP (ruled on proven evidence — at-ack survived 301/301
acked-writes-through-power-cut in block-layer fault injection; deferred
tree authority demonstrably loses flush-covered acks): a brain with NO
stored authority artifact now ADOPTS LOG AUTHORITY AT OPEN. The oracle
gates the flip exactly as the guarded adoption path always did — curable
divergences baseline-backfilled, the flip lands ONLY on a green verdict —
and a brain that cannot verify STAYS tree-authoritative loudly, with the
refusal recorded on the switch artifact so subsequent opens are cheap.
config logAuthority: 'defer' is the explicit documented opt-out (no
automatic adoption; declared flush-window loss; adoptLogAuthority() flips
later). A stored artifact always wins. RELEASES.md carries the posture.
Two standing .fails debt pins FLIP TO HOLDING under the default: the
at-ack crash-survival gap and the ack-at-log durability target — both now
permanent asserted truths, not aspirations.
POWER-CUT THROW SITES (fault-injection findings, brainy-alone config):
- A manifest-listed-but-unloadable column segment QUARANTINES at
discovery (loud once, counted always, quarantinedSegments() exposed for
the heal) and the field serves its remaining segments DEGRADED — never
a raw throw killing every query on the field. Real storage faults still
propagate untouched.
- Torn generation artifacts (NaN/garbage in manifest or counter) DISCARD
with narration at the store's open and recovery re-derives — plus a
defensive finite-integer guard at the init consumer. Never a RangeError
killing an open.
THE LOUD TORN-RECORD CONTRACT: an existing-but-unparseable stored record
now surfaces as a typed, counted TornRecordError on every entity-read
surface (including fifteen previously-blind per-item batch catches);
ENOENT stays clean-absent; artifact readers with designed absent-recovery
keep null-tolerance behind the loud floor. Disk corruption can no longer
read as silent data invisibility.
Suite migration: the default's pins inverted deliberately, generation
baselines made relative, quarantine-contract pins rewritten to the ruled
behavior.
Gates: tsc 0 · unit 2065/2065 (159 files) · integration 826 (93 files) ·
conformance 31/31 · kill-matrix 15/15 · torn-open guards 2/2.
An internal cross-engine fault-injection run (frozen-platter power-loss
capture) surfaced three release-gating findings; each cured in its owning
layer, each pinned:
1. WHOLE-LOG REPLAY ON UNCLEAN OPEN (the big one): log-authority replay
only covered facts ABOVE the manifest — but live canonical entity
writes are tmp+rename without per-file fsync, and the group-commit
flush syncs staging + manifest, never the live tree. Power loss could
therefore vaporize acked canonical bytes BELOW the manifest while the
log held every fact scan-clean (measured: 299 of 301 acks lost).
Now: a clean close stamps a clean-shutdown marker (fsynced, written
last); every open consumes it; an UNCLEAN open under log authority
folds the ENTIRE log into canonical — whole-entity after-images make
the re-apply idempotent and byte-safe. Zero cost on the happy path;
crash recovery pays one narrated fold. Recovery is replay: a crash is
just bigger lag.
2. TORN WRITER LOCK: power loss legally leaves the lock file present but
empty; the parse failure read as 'no holder' while the O_EXCL claim
EEXISTed forever — a PERMANENT lockout no staleness check could clear.
An unparseable lock is stale by definition (no live holder has one):
unlink loudly and re-loop; a racer rewriting a valid lock first wins.
3. PAIR GUARD: flush() called metadataIndex.stampWatermark unguarded;
a replacement metadata provider without the method killed the pair at
first flush. All three stamp calls are optional-chained — a missing
stamp is a verdict-side rescan, never a flush crash.
Pins: whole-log fold restores rows vanished below the manifest ·
clean-shutdown marker lifecycle (stamp/consume/re-stamp) · torn-lock
recovery with a fresh write after · stampless-provider flush.
Gates: unit 2055/2055 · integration 824 · kill-matrix 15/15.
The write side of the law, ruled 2026-08-03: data is either in main space
where developers can use anything, or it is in system.*.
- The reserved-name write door DIES: add/update/relate/updateRelation
metadata bags accept EVERY name (confidence, type, id, data, level,
content, ...) as ordinary user fields — indexed, filterable, sortable,
aggregatable, identical to any other field. The remap/enforce/warn
machinery, the reservedFieldPolicy config (now a typed init refusal),
and the compile-time metadata key bans are all removed. The one write
refusal left: keys spelled 'system.*' (namespace forgery), now enforced
on all four write doors.
- STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag
nested verbatim under 'metadata', sealed by a format stamp — by-name
storage discrimination is unsound once colliders are admitted. Legacy
flat records stay readable forever through the shape-aware splitters
(sound for them: the old door refused colliders). Time travel rides the
same split (generation store snapshots whole records).
- Name-based index exclusions DIE: user frame indexes every name; the
excludeFields/indexedFields knobs and their silent-[] holes are gone;
bulk-payload protection is value-shape only, uniform across names.
- Consumer-sweep findings fixed in the same wave: per-type counts read
the frozen 'system.type' column (addToIndex sort, affinity tracking,
cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for
pre-rebuild reads); resolveHiddenIds addresses 'system.visibility'
(bare 'visibility' was a silent no-op under the law — VFS/system
entities leaked into default reads).
- Fidelity fallout fixed in the owning layers: readEntityFieldAddress
reads the bag first (colliders were absent-shadowed by its own guard)
and never serves system addresses from the bag; blob history refs read
the bag shape-aware; migration transforms now receive ONE normalized
view (engine fields + nested bag) regardless of stored era, and stray
flat-habit keys refuse with the fix in the message.
- THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as
gates-green): all ten collider names + plumbing names written as user
fields, verified verbatim + queryable across live reads, flush+reopen,
a forced epoch rebuild, and asOf time travel; relation mirror; forgery
refusals; legacy flat-record compat. 8/8 green.
Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance
27/27 (exit 0) · consumer test sweep migrated (10 files).
Also completes the v8.10.2 write-granularity law for the transact() plan
path: a metadata-only batch update never rewrites the vector-bearing noun
record (planUpdate staged the unconditional save the update() fix removed).
Seven pins in tests/integration/level-field-shadow.test.ts including the
reporting consumer's exact repro rows; orderBy JSDoc documents the ordering
contract and the announced field-addressing law.
Also: idle PathResolver stats tick no longer logs NaN% every minute (logs
only on new traffic, via prodLog); graph-lsm-* key family recognized as
system resources (kills the per-boot unknown-key warning on provider-backed
brains). Four regression pins in tests/integration/update-write-granularity.
First stage of the co-frozen D1+D3+repacking unit: the format core,
self-contained under _generations/segments/.
- seg-<firstGen,20pad>.bgs: append-once packs of consecutive
generations (magic BGS1; frame = u32 len + u32 crc32c + msgpack
[generation, timestamp, delta, records, flags]; flags reserves
compressed-payload evolution without a format break). Sealed
segments are immutable — fold refuses overlap with sealed ranges.
- seg-<firstGen>.idx: DERIVED sidecar (per-generation frame offsets +
per-id generation postings + checksums); lost/corrupt sidecars
rebuild from their segment loudly; a damaged segment (frame CRC
mismatch) fails loudly, never serves wrong bytes.
- manifest.json: the one discovery path — open() reads it and never
lists the packed backlog (the scan-wedge class's cure); refuses a
newer manifest version rather than serving partial history.
- D3 semantics: dropSegmentsBelow reclaims WHOLE segments at
boundaries only and bumps compactedBelow durably; archival-profile
enforcement stays with the caller per the co-freeze.
- D8 rider: digestThroughPacked(g) — deterministic crc32c chain over
sealed-segment checksums (+ frame-level prefix mid-segment),
O(segments), reopen-stable.
Also fixes a cross-adapter contract bug the suite caught: memory
storage's deleteObjectFromPath ignored the raw-bytes store, so
deleteRawObject on a raw-bytes path (fact-log or segment files)
silently no-op'd — deletes now match filesystem unlink semantics.
Six pins. Wiring into GenerationStore (two-tier reads, the repacker,
cold-open manifest path) lands with the rest of the unit before its
release; cortex's fact-record/stamp shapes reconcile the sidecar
keying when they post.
clearInterval() stops future heartbeat ticks but not one already in
flight: a straggler tick past its ownership guards could land its
atomic lock rewrite AFTER releaseWriterLock()'s unlink, re-creating
the lock file as a phantom that blocks the next writer until the
stale TTL expires (~60s) — the pool-eviction reopen case. Found as an
ENOENT heartbeat warning during benchmark teardown; the quiet variant
is the harmful one.
releaseWriterLock() now awaits the in-flight tick (tracked per tick,
self-clearing) before reading/unlinking, so a straggler's write always
lands BEFORE the unlink and gets removed with everything else. The
heartbeat's ENOENT is also now benign-by-contract (lock or directory
removed under us — the next acquire recreates it); other errors stay
loud.
Pin: straggler-past-guards simulation — lock file absent after close,
directory immediately claimable (fails on the undrained code).
- acquireWriterLock now CLAIMS with an atomic create-exclusive write (O_EXCL)
inside a bounded retry loop: two processes racing an absent lock can never
both succeed (the old read-then-tmp-rename flow let the loser keep running
unlocked, silently). An EEXIST loser re-evaluates and either throws loudly
with the winner's details or performs a verified stale-takeover (re-read
before unlink so a lock that changed hands mid-deliberation is never
clobbered). Exhausted contention fails loudly instead of degrading into a
lockless open.
- BRAINY_WRITER_LOCKED passes through init() unwrapped: the error documents a
machine-readable contract (err.code + err.lockInfo with the holder's
pid/host/heartbeat), but init's blanket error wrapping stripped both,
leaving consumers a message to regex against.
- Two contract tests added: stale-foreign takeover installs OUR lock via the
atomic claim; the conflict error carries code + lockInfo at the public
init() surface.
- New public API: walks every canonical relationship record, queries the same
read path applications use (related() with all visibility tiers included),
and classifies every discrepancy into its failure family: missingFromReads
(records the read path omits — stale adjacency), danglingEndpoints
(endpoint entity gone — the partial-delete scar class), readOnlyVerbIds
(read-path edges with no canonical record — ghosts). Design-hidden
internal/system edges are counted separately so intentional hiding is never
misclassified as loss. Counts exact; example lists capped with an explicit
truncatedExamples flag; loud narration on incoherence; mutates nothing.
- Classification core lives in src/graph/graphAudit.ts behind injected seams
(canonical walks + the end-to-end read) so the discrepancy logic is testable
in isolation; brainy wires the seams to storage walks and related().
- getNounIdsWithPagination now refuses an undecodable resume cursor loudly —
the third and final pagination walk brought under the 8.5.2 cursor contract.
- Types exported: GraphAuditReport, GraphAuditDiscrepancy. Docs: the
inspection guide gains the audit -> repair -> audit verification ritual.
- Aggregation backfill walks build into a STAGING map and swap in atomically
on completion. A mid-walk failure drops the staging map — the previous live
state keeps serving, the aggregate stays flagged pending, and the storage
error surfaces to the failing query. Previously the walk wiped live state
before a scan that could throw, never cleared the pending flag on failure,
and re-ran a full walk on every subsequent query: a silent wipe/walk/throw
loop at the caller's retry rate.
- Failed walks are latched: retries within a 30s cooldown rethrow the recorded
error instantly instead of re-walking, so a tight caller-side retry loop
costs one loud error per query, never a full store walk per query.
- Persisted aggregation state is stamped with the store's committed generation
at flush; reopen adoption requires stamp equality. Stale state (unclean
shutdown) or over-counting state (a log truncation on a copied store pulled
the watermark back) triggers exactly one loud rescan, never a silent adopt.
- The backfill/adoption path narrates: adoption decisions, walk start/finish
with counts and duration, and failures all log by default.
- getNouns/getVerbs refuse a supplied-but-undecodable pagination cursor with a
loud error instead of silently restarting the walk at offset 0 (which
re-served page 1 forever to any while(hasMore) caller).
- The graph cold-load verb walk aborts loudly on a missing or non-advancing
cursor with hasMore=true.
- A versioned index provider whose generation is AHEAD of the committed
watermark (torn copy / crash-recovery truncation) is now named loudly at
open, alongside the existing behind-direction message.
- AggregationIndex: defineAggregate before init no longer forces a backfill.
init reconciles instead of clobbering: the app definition wins, persisted
state is adopted on hash match, a write landing pre-adoption forces an exact
rescan, and a successful state load clears the backfill flag. New ready()
settles every adoption decision before query paths consult backfill state.
- brainy: backfills are single-flight and batched. Concurrent queries share one
store walk and every pending aggregate fills from that same walk; the old
behavior let each concurrent query wipe the others' partial state and start
its own full walk, so a store under steady aggregate traffic never converged.
queryAggregate also waits for persisted definitions before deciding an
aggregate does not exist.
- paramValidation: recordQuery is telemetry-only. The duration-based cap
ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored
at 1000 - below the documented 10000 auto floor, reported under a stale
basis label) is removed; the cap comes from its construction-time basis or
explicit overrides alone.
- storage: __aggregation_* and singleton system keys (brainy:entityIdMapper)
are recognized before the unknown-key warning fires; routing is unchanged.
- docs: find-limits cap-immutability note, aggregation reopen/adoption
semantics, RELEASES.md 8.5.1 entry.
- storage.committedGeneration(): the committed watermark exposed on the
fact-scan capability, so a provider compares its stamp's
sourceGeneration against the store's truth without parsing the
private manifest format (the capability source now carries both the
live fact log and the committed closure).
- Pinned contract tests: fsync-before-ack (transact passes today; the
single-op path is pinned via it.fails as the documented target —
group commit must become LATENCY batching, ack waiting on the shared
fsync, never durability skipping; the pin flips red when that lands
so there is no cliff to discover) and scan-stability-under-rotation
(a scan snapshot yields exactly its facts — no gaps, duplicates, or
bleed-in — while segments rotate beneath it; the reclaim-during-scan
variant lands with fact compaction, which does not exist yet).
- FactLog accepts a rotateBytes option so tests exercise real rotation.
Three additive surfaces for native index providers, which hold only
`storage` and must never construct their own fact-log reader (the log's
open path is writer-side — it reconciles by truncating/rewriting):
- Fact-scan capability on the storage adapter (the getBinaryBlobPath
pattern): the host brain wires a closure over its LIVE fact log at
init; providers call storage.scanFacts()/factLogHeadGeneration()/
factSegmentPaths() and fall back to the enumeration walk on null.
Restore/reopen swaps the underlying log transparently.
- The family-stamp trio (readFamilyStamp/writeFamilyStamp/
verifyFamilyStamp + types) exported from @soulcraft/brainy/internals,
so 'one verifier' is literally one function shared with the native
side, never two synchronized copies.
- Rollup invariants widened to number|string: content fingerprints
(e.g. a per-tree SHA-256) are valid invariant values; strict equality
either way, and a type mismatch reads as incoherence, never a pass.
Every committed generation now also appends a FACT — an after-image
commit record (what each touched entity/relationship became, or a
body-less tombstone for a removal) — to an append-only, crc32c-framed
segment log under _generations/facts/. The before-image history and the
canonical tree remain authoritative; the fact log gives consumers ONE
sequential, self-verifying stream (index heals, incremental replays)
in place of a per-entity directory walk.
- Wire format: positional msgpack facts [generation, timestamp, ops,
meta, blobHashes]; op = [kind u8, id bin16, record | nil tombstone];
32-byte segment header (magic, formatVersion, firstGeneration,
zeroed+verified reserved); length+crc32c frame per fact; zero-padded
segment names so lexicographic order == generation order; JSON
manifest with an atomic rename flip, manifest-first rotation.
- Commit protocol: facts append+fsync BEFORE the commit point inside
the existing durability window, so a crash can only leave the log
AHEAD of committed truth — open() truncates back (torn tails detected
by CRC). Absent generation = never committed; a scan can never see an
uncommitted fact. transact() facts are durable-on-return; single-op
facts ride the group-commit flush exactly like buffered history. A
fact-append failure fails the write, loudly — a silent gap would be a
lie a later replay discovers.
- New public surface: brain.scanFacts() (sequential batches with heal
telemetry: head/segments/approx up front, per-batch generation range
+ bytes + segment id, loud abort on gaps, summary cross-check) and
brain.factSegmentPaths() (immutable sealed segments for zero-copy
consumers; the mutable tail excluded). Exported types CommitFact,
FactOp, FactScanBatch, FactScanHandle.
- Storage: optional binary raw-byte primitives (appendRawBytes,
readRawBytes, writeRawBytes, rawByteSize) on StorageAdapter —
feature-detected; filesystem + memory adapters implement them; an
adapter without them hosts no fact log. Fact segments are byte-copied
(never hard-linked) into snapshots. The _generations/facts/ namespace
is registered as a protected family (rebuildable: false): no sweeper
or GC may delete under it.
- New crc32c (Castagnoli) utility with RFC known-answer tests.
Persisted entity totals inflated permanently: a delete's count decrement
was sourced from re-reading the record being removed, so a null read
(replace race, or a ghost left by a pre-8.3.1 partial delete) silently
skipped the decrement while the paired add had counted — and
Math.max(persistedTotal, scanned) meant no disk cleanup could ever
lower the reported total. Reported by a production deployment with a
write→delete→re-create repro minting drift within hours.
- The caller's pre-delete read now rides through the entire delete path
(remove()/removeMany()/plan → DeleteNoun/DeleteVerb operations →
deleteNoun/deleteVerb → deleteNounMetadata/deleteVerbMetadata): a
null internal read falls back to the known prior record instead of
skipping the decrement. StorageAdapter.deleteNoun/deleteVerb gain an
optional priorMetadata parameter (additive).
- rebuildTypeCounts() rebuilt only the type-statistics arrays and
computed the total just to LOG it — the persisted scalar
(counts.json) survived every rebuild. One canonical walk now rebuilds
every counter rollup (scalar totals + per-type maps + type
statistics) and persists them; repairIndex() runs the recount
unconditionally, since counters can be inflated over clean shelves.
Two production-reported spine fixes:
- Canonical delete is a FULL removal: remove()/removeMany() deleted the
metadata leg but never the canonical vectors.json leg or the <id>/
directory, leaving ghost rows that inflated enumerated counts forever,
read as damage scars, and caused duplicate readdir entries for
re-created VFS paths (the stale-unpost path). The delete operation now
routes through storage.deleteNoun — both legs + the entity container —
with a full two-leg before-image rollback; deleteVerb symmetric; the
blind 'no metadata file' catch that masked real faults is gone. The
generation log still holds the delete's before-image, so asOf() history
is unchanged. repairIndex() gains a conservative, loud orphan-prune
sweep (containers with no metadata content leg) + count recompute for
stores damaged by earlier versions.
- Family-scoped migration gate: every read blocked on the whole-brain
migration lock even when the migrating index was irrelevant. The gate
now scopes to the index families a read actually consults — canonical
reads never wait; find() waits only on its query shape's families;
graph traversals wait only on the graph family. Writes keep the
conservative whole-brain wait. A read that needs the migrating family
still blocks (bounded) with the retryable MigrationInProgressError.
The canonical enumeration walk (getNounsWithPagination) hydrated each entity's
vector + metadata ONE-AT-A-TIME inside the shard loop — every enumeration paid
N x per-op-latency serially, and every index heal enumerates canonical, so this
was the dominant multiplier in the measured multi-minute heals (cortex heal-cost
decomposition).
- Hydration is now 16-way bounded-concurrency (matching the wave cor's native
rebuild uses): heal wall-clock becomes ~2xN/16 x per-op instead of N x per-op.
Order, cursor resume (skipped nouns still never read), filters, peek/hasMore
and totalCount are all preserved — pages are byte-identical to the serial walk.
- New getNounIdsWithPagination(): the id-only opt-out for callers that own their
IO schedule (an index heal). Unfiltered = ids straight from the shard paths,
ZERO per-entity reads; filtered hydrates metadata only (16-way). Same
cursor/offset/nextCursor contract, so it is page-compatible with the hydrating
walk.
Regression: pagination-parallel-hydration.test.ts pins paged==big-page identity,
ids==items order, zero-read id-only, and filter parity. 103 storage tests green.
The class-killer for the lost-main.dkann incident (ADR-004 §7). A provider can
declare a derived-index blob FAMILY (a set of members that are load-bearing
together, e.g. vector-base = main.dkann + main.slotmap + main.slotrev). Once
declared:
- deleteBinaryBlob / removeRawPrefix REFUSE to remove a declared member, throwing
the new ProtectedArtifactError — an in-process GC / sweeper is now INCAPABLE of
deleting a load-bearing index file (COLD != DEAD). Intentional retirement is an
explicit unregisterDerivedFamily(name) first.
- The declaration persists to _system/derived-artifacts.json, so protection
survives a reopen; clear() resets it with the rest of the derived footprint.
- checkDerivedFamiliesPresent() names any member missing on open (the catch for an
EXTERNAL deleter that bypasses the in-process refusal) → rebuild from canonical.
- Transients (*.tmp.*, *.rebuild-tmp, *.rotate-tmp) are never protected; a
namespace family protects a growing prefix (seg-*).
New StorageAdapter surface (optional): registerDerivedFamily / unregisterDerivedFamily
/ listDerivedFamilies + DerivedFamilyDeclaration; new exported errors
ProtectedArtifactError / DerivedArtifactMissingError. Enforcement is inert until a
provider declares a family (no regression). Cor declares its 6 families + does the
atomic set-swap in 3.0.15 (M2); brainy builds the contract now. 8 tests.
Closes the "dishonest readiness proxy" anti-pattern (Pattern A): size()>0 /
isInitialized were treated as "this index serves queries", but a cold native
index can load its COUNT before its SERVING structure, so a query returned a
silent [] indistinguishable from "no such data". A shared assessIndexReadiness()
now reads only the provider's honest isReady() signal (never size()), applied at
every site:
- Vector: a one-shot verifyVectorLive() guard on the semantic/proximity search
path (a pure semantic find({query}) has no filter, so nothing guarded it). It
prefers isReady(), else a known-vector self-match probe; self-heals via rebuild
or throws the new VectorIndexNotReadyError instead of a silent [].
- Graph: getVerbsBySource/ByTarget skip the fast path when the provider reports
not-ready (falling to the canonical shard scan), plus a one-shot probe that
self-heals a no-isReady provider whose adjacency did not cold-load.
- getIndexStatus(): folds in per-index honest `ready` (making `populated`
honest) + rebuildFailed/rebuildError/degradedIds, so a readiness probe never
200s a brain that is still warming up or degraded.
Unblocked by the native providers now reporting serving-truth (graph via
SSTable-residency readiness, vector via durableBaseLoadFailed). New export:
VectorIndexNotReadyError. getIndexStatus gains additive fields. No breaking API.
13 new tests; existing readiness guards green.
The Pass-1 hardening that makes loadBinaryBlob distinguish genuine absence
(ENOENT -> null) from a real fault (EIO/EACCES/... -> throw) was held out of
8.2.6 because the native accelerator's two column-store read sites still relied
on null-on-error. That accelerator release has now hardened those sites to
handle the throw (a faulted segment read marks the field unavailable and throws
a named error), so the fault-propagating form is restored and ships lockstep.
A present-but-unreadable index blob no longer masquerades as absent (which drove
needless rebuilds / empty reads). Adds the loadBinaryBlob leg to the blob
durability suite (absent -> null; present -> bytes; real fault -> throws).
The Pass-1 hardening made loadBinaryBlob distinguish genuine absence (ENOENT →
null) from a real fault (EIO/EACCES/… → throw), so a present-but-unreadable blob
stops masquerading as "absent". The native column-store still has two call sites
that rely on the old null-on-error contract; publishing the throw ahead of them
would break a consumer running new brainy + old cortex.
Per the cortex coordination (accept: cor first, then brainy), this temporarily
restores the shipped 8.2.5 swallow-on-fault behavior for loadBinaryBlob ONLY, so
the rest of Pass 1 — saveBinaryBlob write-honesty, clear() native wipe,
pending-flush durability, count symmetry, degraded surfacing, aggregation
loudness — can release now. The method carries an inline restore guide; the
throwing form goes back in and ships lockstep with the native hardening. No other
Pass-1 change depends on this behavior (ColumnStore's own fault test uses a
direct-throwing storage stub), so the split is behaviour-neutral for every
consumer versus 8.2.5.
With the unique per-writer temp suffix, a rename ENOENT can no longer mean "a
concurrent idempotent writer already renamed it" — nobody else holds this
writer's temp. It means our just-written temp vanished before the rename, so the
bytes did NOT land. The old code returned success on that ENOENT, acknowledging
a write that persisted nothing; the native provider mmaps these blobs, so a
phantom-acked blob is a silent-loss.
saveBinaryBlob now retries once with a fresh temp (self-healing a transient
external-sweeper/crash-cleanup race), and if the temp vanishes again it throws
loud rather than acknowledging a store-nothing write. Non-ENOENT rename faults
propagate verbatim as before. The unique-temp suffix already eliminated the
concurrent same-key collision that originally motivated the ENOENT shortcut.
clear() removed entities, indexes, system and _cas but left three top-level
trees on disk: _blobs (raw HNSW/LSM segment bytes and the native dkann index),
_id_mapper (the native shared mmap id-mapper), and _column_index (column-store
manifests). A cleared brain therefore re-read stale native state.
Worse, the column store splits its state across two of those trees — manifests
under _column_index/ and their segment bytes under _blobs/_column_index/ — so
removing one without the other stranded a manifest listing segments that no
longer exist, which the hardened segment-load path now (correctly) refuses with
ColumnSegmentLoadError. The three trees now fall together as a set, exactly as
_cas already does. locks/ is deliberately preserved: it is live coordination
state, not data.
ColumnStore silently skipped a manifest-listed segment it could not load: a
corrupt or missing segment dropped every one of its entities out of
filter/rangeQuery/sortTopK with no error, so an inconsistent index read as a
merely short result. It now throws a typed ColumnSegmentLoadError when a listed
segment yields undecodable or no bytes, and lets a genuine storage IO fault
propagate verbatim. Only a field with no manifest at all stays benign (nothing
was ever written for it).
baseStorage.getNoun_internal / getVerb_internal likewise caught every error and
returned null, reporting a present-but-unreadable entity as "not found". They
now return null only for genuine ENOENT-class absence (via isAbsentError) and
rethrow real faults and deserialize errors.
Pattern-B (blind catch) hardening: absence -> null, fault -> loud.
Write/index-spine hardening, first batch of Pass 1. Each fix restores an
invariant the surrounding code already intended; every one has a
fail-before/pass-after test.
- Pattern C, finding 5 (baseStorage): delete now decrements the user-facing
scalar total symmetrically — deleteNounMetadata was decrementing only the
per-type bucket, deleteVerbMetadata neither the bucket nor the scalar, so
getNounCount()/getVerbCount() inflated permanently (the stale scalar wins
pagination via Math.max and is persisted). Invariant now holds:
scalar total === Σ per-type across add/update/delete and reopen.
- Pattern A, finding 3 (graph/lsm/LSMTree): a partial SSTable-load failure no
longer publishes the manifest's full relationship count as healthy. Any
per-SSTable load failure throws after the batch, which resets to honest-empty
and lets the existing size()===0 self-heal rebuild run — size()/isHealthy()
can no longer lie about a partial load.
- Pattern B, finding 6 (hnsw/hnswIndex): deferred flush() no longer clears
dirty nodes whose connections failed to persist — failed nodes stay in the
retry set, and flush() throws HnswFlushError instead of returning a lying
node count. The immediate-mode first-noun saveHNSWSystem is un-swallowed, so
addItem() rejects rather than returning an id for a rootless index.
- Pattern B, finding 11 (part — storage reads): new shared isAbsentError()
helper (utils/errorClassification, ENOENT-only absence) applied to
loadBinaryBlob and readObjectFromPath — a real IO fault (EIO/EACCES/EMFILE)
now propagates loudly instead of masquerading as "absent", which had driven
needless rebuilds / empty reads (loadBinaryBlob feeds the native provider).
Regression: 78 green across the 3 new suites + db-mvcc, generationStore,
temporal-vfs, rollback-trapdoor, restore-nondestructive. Full gate runs before
the Pass-1 release (David-gated). Remaining Pass 1: finding 11 getNoun/getVerb
legs, finding 4 (ColumnStore), finding 8 (pending-flush), finding 10 (degraded),
finding 7 (clear). Pattern A guards (1,2,9) as a follow-up release.
restore() removed the entire live brain directory and then fs.cp'd the
snapshot in, so any copy failure left the store destroyed with only a partial
copy. The sharpest edge: fs.cp materialized the holes of sparse mmap blob
files, so a snapshot that fits on disk could balloon and ENOSPC mid-copy — the
recovery tool destroying the brain it was asked to recover.
Rewrite restoreFromDirectory as stage → verify → atomic swap:
- Copy the snapshot into a _restore_staging area BEFORE touching live data,
sparse-aware (copyTreeSparse/copyFileSparse skip all-zero 4 MiB chunks, so a
mostly-hole store restores at its true allocated size, not its apparent size).
- On any copy failure (ENOSPC included) remove only the half-written staging
area and throw — the live store is left exactly as it was.
- Only after the copy succeeds, fsync a completion marker naming the staged
entries, then swapStagedRestoreIn(): an idempotent per-entry
rm-old + rename-staged-in (same-filesystem renames that cannot ENOSPC),
removing stale live entries the snapshot lacks.
- completeInterruptedRestore(), wired into init() right after the root dir is
ensured, finishes a crash mid-swap FORWARD (resume a committed swap) or
discards an uncommitted staging area (live still authoritative).
_restore_staging is excluded from snapshots. persist() (hard-link snapshot)
was already safe and is unchanged; no public API change.
Regression (tests/integration/restore-nondestructive.test.ts): a forced copy
failure leaves live data fully intact and cleans staging; a normal restore
round-trips to the snapshot; an interrupted-but-committed restore completes on
reopen; an uncommitted staging area is discarded on open; the sparse copy is
byte-identical with allocated blocks far below apparent size.
A committed transact() reported success while its canonical entity writes were
still only in the OS page cache (tmp+rename, not fsync'd), even though the
generation counter and manifest were fsync'd. A hard kill in that window could
leave the durable counter ahead of the persisted entity bytes — phantom
progress for any generation-based consumer resuming from the counter. Reported
from a downstream migration's crash-lifecycle forensics.
Add an optional transaction durability barrier to GenerationStorage
(beginWriteBarrier / flushWriteBarrier). FileSystemStorage implements it:
writeObjectToPath records each successful canonical write and
deleteObjectFromPath records each delete's parent dir, between begin and flush;
flush fsyncs every recorded write (contents + rename dir entries, via
syncRawObjects) and the parent dir of every delete. commitTransaction opens the
barrier before running the planned operations and flushes it after, BEFORE
persisting the counter and manifest — so the batch's entire canonical footprint
is durable before the generation stamp advances. The barrier is optional: the
generation store calls it through optional chaining, so in-memory and
durable-per-call (cloud object-PUT) adapters treat it as a no-op.
The single-op group-commit path is unchanged (deferred durability is the
Model-B design that avoids a 3-5x per-write fsync regression), but its
durability contract is now documented explicitly on commitSingleOp: transact =
durable on return; single-op = durable at the next flush()/close(), with the
counter buffered alongside the data so a crash loses both together (never a
torn counter-ahead-of-state store).
Regression (tests/integration/transact-durability-barrier.test.ts): the entity
writes fsync in an earlier syncRawObjects batch than the manifest for single-op
and multi-op (add+relate) transactions; a precommit-rejected batch opens no
barrier and advances nothing; MemoryStorage exposes no barrier (optional-chain
no-op).
The temporal model had a hole exactly where files were concerned: every
entity write is an immutable generation with before-images, but VFS content
BYTES lived under an eager refCount GC left over from the pre-8.0 design —
unlink could physically destroy bytes that in-window history still
referenced, and overwrite never released the old hash at all (an unbounded
silent leak whose accidental byproduct was the only thing "preserving"
history). Reading the past could therefore return a stale field, a dangling
hash, or nothing, depending on luck.
Fix: blob reclamation becomes a HISTORY decision instead of a LIVENESS
decision. Each blob's metadata now carries historyRefCount alongside the
live refCount:
- The commit seam counts one history reference per persisted before-image
record carrying a content hash (commitTransaction staging and the
group-commit flush), recorded BEFORE the record-set persists and carried
in the generation delta (blobHashes — always present on new deltas, so
compaction only falls back to reading records for pre-contract
generations). An aborted transaction compensates best-effort.
- unlink/rmdir/overwrite drop ONLY the live reference (BlobStorage.delete →
release; overwrite finally releases the superseded hash — cancelling the
dedup increment on same-content rewrites and closing the leak), and only
AFTER the canonical mutation commits, so a failed delete can never leave a
live file whose bytes compaction might reclaim.
- History compaction is the ONE reclamation point: after deleting a
generation's record-set it releases that set's references and physically
reclaims any hash at zero live AND zero history references. Pins are
exempt automatically. Crash ordering is over-count-only in every path
(record before persist, release after delete), so a crash can leak until
the scrub recounts but can never reclaim bytes a retained generation
needs. scrubBlobHistoryRefCounts() restores exactness; existing stores get
a one-time marker-gated backfill on open, failing into leak-safe mode
(reclamation disabled) rather than guessing.
On top of the protected history, the temporal API the generational model
always implied:
- vfs.readFile(path, { asOf }) — the exact bytes as of a generation or Date,
materialized from the history (pinned view released so compaction is
never blocked by a read).
- vfs.history(path) — FileVersion[] ascending ({ generation, timestamp,
hash, size, mimeType? }), the newest entry being the live state.
- Overwrites now refresh the file entity's data/embedding text — semantic
search and the data field previously served the FIRST version's text
forever (the stale-field defect a consumer's incident recovery depended
on by luck).
Integration suite (temporal-vfs.test.ts): per-version exact reads +
history listing, leak-fix + history protection on overwrite, rm keeps bytes
readable, compaction reclaims past-window bytes and preserves in-window
(including the cross-file dedup case where an old file's history and a
newer file's removal share one hash), data freshness, and scrub exactness.
Sweeping the vestigial hnsw sharding machinery surfaced two real defects on
the counts-recovery path (the code that rebuilds totalNounCount/totalVerbCount
when counts.json is lost or corrupted — container restarts, partial copies):
- initializeCountsFromDisk counted files in entities/*/hnsw/ — directories the
8.0 write path never populates — so an established store recovered to ZERO
counts on real data: wrong getNounCount()/stats, a "New installation"-style
boot log, and a mis-sized rebuild-strategy decision at open. The scan now
walks the canonical entities/<kind>/<shard>/<id>/ tree (one entity per id
directory), with the same sampled type-distribution estimate as before.
- The sampler called getNounMetadata — whose ensureInitialized() re-enters
init() — from INSIDE init(), a latent deadlock reachable the moment the scan
found anything to sample. The sampled metadata files are now read directly
with fs (gz-transparent), no guarded accessors inside init.
The dead machinery itself (verified zero callers by reachability analysis
before deletion): the nine 7.x hnsw-layout entity/edge CRUD methods, the
sharding-depth prober + its depth-migration engine (unreachable — the probe
always returned null on 8.0 stores), an orphaned streaming verb paginator,
their path/scan helpers, and the now-unused type aliases and fields. The init
block keeps the truthful new-vs-established boot log introduced in 8.0.13,
now decided directly from the canonical layout. Net -1,138 lines; no public
API touched.
Adds a regression test: delete counts.json from a populated store, reopen,
counts recover exactly from the canonical tree.
