Two release-blocking findings from the durability kill-matrix, both fixed
in the owning layer:
1. LOG-AUTHORITY REPLAY AT OPEN: durable-at-ack fsynced the fact before
the ack, but open() truncated every fact above the manifest — after a
power loss that takes the un-fsynced tmp+rename canonical bytes, the
acked write's ONLY durable copy was discarded. Now: under 'log'
authority, open() REPLAYS intact facts above the manifest into
canonical (FactLog.peekFactsAbove — CRC-gated, order-sorted) and
advances the manifest to cover them; tree-authority brains keep the
truncate contract they were promised. Pinned end to end: the power-loss
row constructs the exact disk state (fsynced log, vanished canonical
rename) and the acked write lives.
2. NO SILENT COMMIT: commitSingleOp buffered the generation BEFORE the
fact append; an append failure (ENOSPC) rejected the caller but the
next flush durably committed the generation with NO fact — a permanent
silent log gap. Now the failure path un-buffers and returns the counter
reservation: nothing commits, the log stays gap-free, and the canonical
execute-residue orphan is the documented crash-equivalent.
Plus: the kill-matrix itself (11 rows — every commit-path fault point ×
reopen-as-crash recovery contract, at-ack variants, disk-full row; five
new zero-cost faultPoint sites), the log-authority pin suite (oracle
green/red/state-differs, flip refusal, switch survives reopen, 9/9), and
the group-commit covering pins (5/5).
Gates: unit 2002/2002 (152 files) · integration 785 · conformance 27/27.
The provider contract (metadata addToIndex/removeFromIndex, vector
addItem/removeItem, id-mapper getOrAssign/remove) gains an optional
trailing generation — evaluated lazily at operation execute time (the
graph surface's thunk pattern, generalized), threaded from all 17
construction sites: undefined during generation-0 bootstrap, the real
committed generation everywhere else. Optional = additive: no existing
provider or caller breaks; native delta logs that stamped literal zero
start hearing truth. JS twins accept the parameter with parity notes.
Pins: provider doubles capture and assert nonzero monotonic generations
across add/update/remove on both surfaces.
The two-implementation contract surface as one pure module (no I/O):
segment header v2 (formatVersion 2 + sealSize in the reserved bytes),
per-record [type u8, version u8] envelope killing the unknown-kind
misclassification trap, the 12-type registry (after-images with minted
ints, tombstones, batch.meta, embed.pending/landed, blob.manifest,
projection.note, bootstrap.baseline, log.genesis with id-space width and
TYPED width-mismatch refusal), vectorLeg inline|{sameAsGeneration} with
writer-enforced single-hop, sector-sealed groups with pad frames, torn-tail
discipline, and GOLDEN BYTE VECTORS pinned so a second (native) reader
implementation can conform byte-for-byte. 50 format pins + a
fault-injecting storage wrapper (tear/drop-sync/fail-append) with 13
self-tests. v1 segments remain readable; nothing writes v2 yet — the
live-format cutover is its own commit.
The storage-authority adoption path, guarded shape: the canonical tree
stays authoritative by default ('tree'); a brain flips to 'log' only
through the verification oracle, and the flip is stored, per-brain,
checked at open only.
- FactLog.ensureSynced(): classic group commit — concurrent writers
append, then join ONE covering fsync (running + queued slots give the
covering guarantee: the sync a caller awaits always starts after its
append landed). Solo writer = immediate sync.
- GenerationStore.logDurability 'deferred' (default, byte-identical to
today: fact durability rides the group-commit flush, ack latency
unchanged) | 'at-ack' (log-authority mode: every single-op ack awaits a
covering log fsync — an acked write's fact survives power loss, by
contract). transact() was already durable-at-return in both modes.
- src/db/logAuthority.ts: the stored switch artifact
(_system/log-authority.json, absent = tree), readLogAuthority, and the
VERIFICATION ORACLE — replay the fact log, fold latest state per id
(digests, never bodies — memory-bounded), diff against the canonical
tree paged; verdict green iff every canonical row is exactly reproduced
AND the log claims nothing canonical denies. Divergences are NAMED by
class (pre-log-record → needs baseline backfill; state-differs;
log-live-canonical-absent; log-tombstone-canonical-present). The flip
REFUSES on red with the first divergence and the cure in the message.
- Brainy: authority read at open (log → durable-at-ack enabled);
logAuthority() / verifyLogAuthority() / adoptLogAuthority() public API.
Nothing flips by itself; nothing changes for existing brains.
A3 of the service-class pair (BRAINY-PROD-LATENCY-TRIAD): a VFS file write
ran the embedder synchronously while the caller waited — 5.6s p50 / 21.4s
p95 per small file on a production deployment, the dominant stage of every
capture write.
- add()/update() gain deferEmbedding: the write acks at durability (data +
metadata persisted, a DURABLE pending marker under
_system/pending_embeds/<id> written BEFORE the commit — orphan-safe
direction); the single-flight background worker embeds the CURRENT data
and swaps the vector in ATOMICALLY (ReplaceInVectorIndex — the row is
never absent from search; a deferred UPDATE keeps serving the OLD vector,
stale-beats-absent per the flicker law). Typed refusals: defer+vector,
defer-without-data.
- CRASH-SAFE: markers are recovered at open by a BOUNDED prefix listing
(never a store walk) and the worker resumes in the background — a crash
can delay a vector, never lose one. A wedged embedder trips a LOUD 60s
hang guard and the worker moves on (marker retained for retry).
- The honest gauges: getIndexStatus().pendingEmbeds + pendingEmbedCount();
awaitPendingEmbeds() is the eventual-vector-index BARRIER for callers
and tests that need searchability before proceeding.
- VFS adopts it everywhere a write path could wait on the embedder:
writeFile (both branches) and directory creation. Pinned in the
strongest form: writeFile resolves while the embedder HANGS FOREVER.
Pins: deferred-embedding 5/5 (ack law · stale-beats-absent · crash
recovery across sessions · VFS hung-embedder ack · typed refusals).
Gates: unit 1928/1928 · integration 765 · conformance 27/27.
DP6/DP8 of the Path Registry (BRAINY-PROD-LATENCY-TRIAD, the proven flicker
mechanism): update paths staged RemoveFromVectorIndex then AddToVectorIndex
as two separately-awaited transaction ops — between them a live row was in
NEITHER index (dark to semantic recall, fine in metadata list). The native
pair widened that window to seconds in production before their side's
visibility-commit fix; the structural cure lands here:
- hnswIndex.updateItem: absent → add; SAME vector → pure no-op (the
production shape — a type-only update re-indexed an unchanged vector,
remove+add did pure damage); changed vector → the node NEVER leaves the
index: synchronous vector swap first (every query from that instant sees
correct distances), then unlink/relink at the node's existing level via
shared internals (linkNode/unlinkNodeEdges refactored out of add/remove;
entry point and maxLevel provably unchanged).
- ReplaceInVectorIndexOperation: ONE transaction leg; feature-detects
provider updateItem (native seam flagged — their side ships updateItem,
then the adjacent remove+add fallback is dead code). Both update staging
sites swapped; delete sites untouched.
- LAZY-OPEN GATE (fleet adoption find, SELF-ENGINE-PAIR-STANDARD): under
disableAutoRebuild, ensureIndexesLoaded assessed ONLY the vector index —
a not-ready native METADATA provider never blocked the completion latch
and every find() silently returned [] on a populated store. All three
providers now vote; any not-ready report falls through to the rebuild.
- docs/path-registry.md: brainy's twin table for the 32 shared path IDs —
service class, budgets, lifecycle, narration, and the cited pin per row;
owed rows named (LC4 doors-open migration, MT4 yielding heals, LC7
downgrade contract) per the lifecycle-sprint choreography.
Pins: update-item-atomic 9/9 (visibility-atomic swap, reverse-index parity
vs fresh rebuild, entry-point invariants) · lazy-notready-honor 2/2.
Gates: unit 1928/1928 (148 files) · integration 760 · conformance 27/27.
A4 of the service-class pair (SELF-ENGINE-LIFECYCLE-SPRINT, David-directed:
'why do we need manual flushes at all?'). The production disease: 829
caller-scheduled per-write flushes convoying into 45-66s write walls —
cadence hand-rolled a layer above the only layer that can see dirty-node
counts and IO pressure.
- BrainyConfig.persistence: policy 'auto' (DEFAULT) | 'manual', with
flushEveryWrites (512) / flushIntervalMs (30s) / flushOnIdleMs (2s)
triggers. Auto = the engine kicks ONE single-flight BACKGROUND flush at
a threshold or when the store goes quiet; write acks NEVER await it (a
hung flush cannot block a write — pinned); a failed background flush is
LOUD and re-arms the trigger. 'manual' restores caller-owned cadence.
- Triggers wired at both write chokepoints (single-op post-commit +
transact post-commit); idle timer unref'd; close() tears the timer down
and drains the flight before its own final flush.
- RECOVERY SEMANTICS documented on the config: canonical records are
durable per-write regardless of policy — a crash between background
flushes loses derived state only, which converges at next open (epoch
machinery + the new incremental aggregation catch-up), bounded by the
un-flushed window. Never data loss.
Pins: write-count trigger fires one background flush with zero caller
calls · idle trigger · manual never self-flushes · THE ACK LAW (writes
acknowledge under a never-resolving flush). Gates: unit 1917/1917 ·
integration 760 · conformance 27/27 — green WITH auto as the default.
SELF-ENGINE-LIFECYCLE-SPRINT + BRAINY-PROD-LATENCY-TRIAD, the four asks:
(a) brain.flush() persists aggregation state stamped at the committed
generation. The stamp used to advance only at close(), so a long-lived
writer that flushes but never closes — the primary production shape —
left every write window behind the stamp, and ANY unclean exit forced a
whole-store backfill walk (per-entity work, measured >60s and
door-starving on a 9k-row production brain) on the first stats call.
(b) BEHIND-stamp adoption becomes adopt + INCREMENTAL CATCH-UP: the exact
missing window (stamp, committed] resolves its affected-id set from the
fact log and reconciles each entity with time-travel before/after reads
(asOf at both window bounds) through the same delta algebra the live
hooks use — cost bounded by writes since the last flush, never store
size, and exact under interleaving because reconciliation targets the
FIXED window end while later writes chain through hooks. Oversized
windows (>5000 affected) and unreadable windows demote to the announced
rescan — never a silent partial serve.
(c) The native provider's parallel rebuildAggregate — on the contract since
8.x but never invoked anywhere — is now the backfill walk's preferred
door: one call per aggregate with source-matched entities, replacing
the per-entity FFI stream.
(d) A delete whose before-image is unavailable can no longer SKIP the
aggregation hook silently (counts drifted upward forever): both delete
paths (remove() and transact) flag an exact rescan, loudly.
Pins: integration (flush stamp; unclean-exit reopen → exact counts through
an add + group-move + delete window with the walk spy proving ZERO
whole-store walks) + unit (provider rebuild invoked once with filtered
entities; flagAllForRescan; reconcile delta algebra). Gates: unit 1913/1913
· integration 760 · conformance 27/27.
BRAINY-PROD-LATENCY-TRIAD Track A1 (David-approved plan): the sort path's
value resolution goes BATCHED — one chunked metadata-record batch pass
serves any N, replacing the serial per-row getNoun loop (62-98ms x 3,224
rows = the measured 199-317 second silent scan on self prod). The
metadata record carries every sortable value: system scalars EXACT
(bucketed-index precision loss can never force a per-row disk read
again) and the user bag via the shape-aware split, both record eras.
- resolveOrderValuesBatch: the one sanctioned value source for ordered
reads (batch door: getNounMetadataBatch -> getMetadataBatch -> chunked
parallel; never serial).
- Column top-K page re-sort and the no-column fallback both rewired.
- B2 down-payment: the no-column fallback ANNOUNCES itself once per
field past 500 rows - silent degradation is illegal.
- THE CALL-SHAPE PIN (tests/unit/utils/metadataIndex-sort-callshape):
zero vector-record reads, batch calls only, latency-blind so it holds
on any machine - the serial loop cannot quietly return. Ordering
contract re-pinned through the batch path (nulls last both directions,
ties by id, never drop).
(! = perf contract change only; no API change. Gates: unit 1904/1904,
integration 758, conformance 27/27.)
The write side of the law, ruled 2026-08-03: data is either in main space
where developers can use anything, or it is in system.*.
- The reserved-name write door DIES: add/update/relate/updateRelation
metadata bags accept EVERY name (confidence, type, id, data, level,
content, ...) as ordinary user fields — indexed, filterable, sortable,
aggregatable, identical to any other field. The remap/enforce/warn
machinery, the reservedFieldPolicy config (now a typed init refusal),
and the compile-time metadata key bans are all removed. The one write
refusal left: keys spelled 'system.*' (namespace forgery), now enforced
on all four write doors.
- STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag
nested verbatim under 'metadata', sealed by a format stamp — by-name
storage discrimination is unsound once colliders are admitted. Legacy
flat records stay readable forever through the shape-aware splitters
(sound for them: the old door refused colliders). Time travel rides the
same split (generation store snapshots whole records).
- Name-based index exclusions DIE: user frame indexes every name; the
excludeFields/indexedFields knobs and their silent-[] holes are gone;
bulk-payload protection is value-shape only, uniform across names.
- Consumer-sweep findings fixed in the same wave: per-type counts read
the frozen 'system.type' column (addToIndex sort, affinity tracking,
cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for
pre-rebuild reads); resolveHiddenIds addresses 'system.visibility'
(bare 'visibility' was a silent no-op under the law — VFS/system
entities leaked into default reads).
- Fidelity fallout fixed in the owning layers: readEntityFieldAddress
reads the bag first (colliders were absent-shadowed by its own guard)
and never serves system addresses from the bag; blob history refs read
the bag shape-aware; migration transforms now receive ONE normalized
view (engine fields + nested bag) regardless of stored era, and stray
flat-habit keys refuse with the fix in the message.
- THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as
gates-green): all ten collider names + plumbing names written as user
fields, verified verbatim + queryable across live reads, flush+reopen,
a forced epoch rebuild, and asOf time travel; relation mirror; forgery
refusals; legacy flat-record compat. 8/8 green.
Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance
27/27 (exit 0) · consumer test sweep migrated (10 files).
Also completes the v8.10.2 write-granularity law for the transact() plan
path: a metadata-only batch update never rewrites the vector-bearing noun
record (planUpdate staged the unconditional save the update() fix removed).
Seven pins in tests/integration/level-field-shadow.test.ts including the
reporting consumer's exact repro rows; orderBy JSDoc documents the ordering
contract and the announced field-addressing law.
Also: idle PathResolver stats tick no longer logs NaN% every minute (logs
only on new traffic, via prodLog); graph-lsm-* key family recognized as
system resources (kills the per-boot unknown-key warning on provider-backed
brains). Four regression pins in tests/integration/update-write-granularity.
A production deployment's warm report showed metadata: 'unavailable' under a
native metadata provider. brain.warm()'s metadata leg only duck-typed the
built-in JS manager's hydrateAll() method, which a native provider has no
reason to implement.
- MetadataIndexProvider (src/plugin.ts) gains an optional warm?(): Promise<void>
hook, mirroring the existing vector and graph provider hooks. brain.warm()
now checks the active provider's own warm() FIRST, falls back to the JS
manager's hydrateAll() when absent, and reports 'unavailable' only when
neither exists -- never init() as a stand-in, since a native provider's
init() may be a cheap verify rather than a real warm.
- Tests (tests/unit/brainy/warm.test.ts): a live provider instance shaped to
have warm() reports 'warmed' and the hook called with no hydrateAll
fallback; shaped to have neither hook reports 'unavailable' (pins the
honest branch); the unmodified built-in JS manager still reports 'warmed'
via hydrateAll(), unchanged.
Additive scope agreed mid-flight with the native-provider team: a
maintenance-debt observability seam so an operator sees a grind coming
instead of discovering it as a CPU storm.
- New optional maintenanceDebt?(): Promise<ProviderMaintenanceDebt> hook on
all three provider contracts (vector, metadata, graph -- the same three
warm?() lives on). ProviderMaintenanceDebt is fields-all-optional: a
provider reports only what it truly measures (pendingBytes, pendingItems,
lastPassCompletedAt, lastPassOutcome, converging), never an estimate
dressed as fact.
- New public brain.maintenanceDebt(): a pure passthrough -- for each surface
it calls only the active provider's own hook and reports the payload
verbatim, or 'unavailable' when absent. No thresholds, no polling, no
JS-side estimation; the provider owns the numbers, the operator owns the
policy.
- ProviderMaintenanceDebt, MaintenanceDebtReport, and MaintenanceDebtOutcome
are exported from the package root.
- Tests (tests/unit/brainy/maintenance-debt.test.ts): hook present reports
'reported' with the exact payload passed through; hook absent reports
'unavailable' on every surface; mixed surfaces resolve independently of
each other.
RELEASES.md gains the 8.10.1 entry covering both fixes above and this
feature, including the no-hot-retry contract from the prior commit.
A production incident: a native-provider op ground 38-40s inside a transaction,
blew the apply-phase budget, rolled back, and a downstream pipeline hot-retried
the identical operation into a 6-minute CPU storm. Brainy itself never
auto-retried the timeout; the gap was that TransactionTimeoutError only said
"retryable" in prose, with nothing machine-readable for a caller to branch on.
- TransactionTimeoutError gains two typed, always-true fields: retryable
(a later attempt may succeed once the slowness resolves or the budget is
raised) and hotRetryUnsafe (an immediate identical retry re-pays the full
cost that just timed out and can cascade into a CPU storm -- callers must
latch and back off, never loop). context's existing telemetry fields
(timeoutMs, operationIndex, elapsedMs, totalOperations, operationName) are
now documented as the caller's backoff inputs.
- Updated the "retryable" doc-prose sites (transact()'s timeoutMs option,
transactionBudgetFloorMs, Transaction.execute()'s contract) to point at
the new fields instead of bare prose.
- Regression pin (tests/unit/transaction/timeout-never-internally-retried.test.ts):
an execution counter proves the engine never re-drives a timed-out
operation, through both the single-op engine TransactionManager/Transaction
drives for every single-record write, and add()'s upsert-race retry loop
(which must exit on the first TransactionTimeoutError, never treat it like
the lost-insert-race signal it retries on).
- Removed TransactionManager.executeTransactionWithResult -- zero callers
anywhere in the codebase.
Ruled today: the project's one public home is source.soulcraft.com. The
old public repo is archived history and no longer part of any release.
- package.json repository/homepage/bugs now point at the forge (this is
what the npm page links as Repository/Homepage/Issues)
- README CI badge reads the forge pipeline; CONTRIBUTING drops the
mirror paragraph (forge account or email patch were already the ruled
contribution paths)
- release.sh: mirror push + external release step removed; publishes go
forge-first (box-held write token, temp userconfig so the token never
hits argv; a forge-publish failure aborts before the storefront so the
pair can never diverge), then npmjs with the scope-override pin (the
fleet npmrc maps @soulcraft to the forge and scope mappings beat
--registry); release page created via forge API when a token is
present, loud skip otherwise; changelog compare links point home
- dead external CI workflow removed (.forgejo/workflows/ci.yml is the
live pipeline)
Historical CHANGELOG links to the archive stay as written - history is
history and the archive serves them read-only.
Ruled today: the project's one public home is source.soulcraft.com. The
old public repo is archived history and no longer part of any release.
- package.json repository/homepage/bugs now point at the forge (this is
what the npm page links as Repository/Homepage/Issues)
- README CI badge reads the forge pipeline; CONTRIBUTING drops the
mirror paragraph (forge account or email patch were already the ruled
contribution paths)
- release.sh: mirror push + external release step removed; publishes go
forge-first (box-held write token, temp userconfig so the token never
hits argv; a forge-publish failure aborts before the storefront so the
pair can never diverge), then npmjs with the scope-override pin (the
fleet npmrc maps @soulcraft to the forge and scope mappings beat
--registry); release page created via forge API when a token is
present, loud skip otherwise; changelog compare links point home
- dead external CI workflow removed (.forgejo/workflows/ci.yml is the
live pipeline)
Historical CHANGELOG links to the archive stay as written - history is
history and the archive serves them read-only.
