soulcraftlabs/open-brainy
Archived
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 345 Packages Wiki Activity

feat(embedding): deferred-embed markers become log records — the sidecar recovery path is deleted
Some checks failed
CI / Node 22 (push) Has been cancelled
Details
CI / Node 24 (push) Has been cancelled
Details
CI / Bun (latest) (push) Has been cancelled
Details

Browse source 
The private recovery discipline, applied to its own machinery: pending-
embed markers stop being sidecar files and become first-class log records
riding the write's OWN commit fact — embed.pending lands in the same
atomic append as its after-image (a marker can never be orphaned from its
write, or vice versa; in durable-at-ack mode it shares the write's
covering fsync — zero extra syncs), and the worker's landing commit rides
embed.landed with the inline vector. Crash recovery is now a FOLD of the
log (pending without a matching landed = recovered), skipped wholesale on
brains with no v2 history; the one-time legacy bridge folds existing
sidecar files in, migrates them as one fact, and deletes them —
idempotent under a crash mid-bridge. No code path writes the sidecar
again.

Plus the ENTITY-TRUTH digest law, found by this train's own pins:
canonical vector wrappers denormalize HNSW residue (connections + the
randomly-assigned node level) that the log deliberately does not carry —
the verification oracle digested it and would have reported false
state-differs on ~any nonzero-level node (a ~15% flake in the cutover pin
was the symptom). Both sides of every oracle comparison now normalize to
entity truth (nounEntityTruth); index residue has its own rebuild path
and is not entity state.

Pins: embed-markers-in-log 5/5 (same-generation marker, landed+fold-to-
zero, crash recovery via the log with the sidecar prefix EMPTY on disk,
legacy bridge, VFS hung-embedder ack) · deferred-embedding 5/5 unchanged
(the contract outlived its mechanism) · kill-matrix 11/11 · cutover 5/5
×10 runs (flake dead) · unit 2031/2031.
This commit is contained in:
David Snelling 2026-08-10 11:27:07 -07:00
parent c95bea8887
commit b47787bbf7
7 changed files with 637 additions and 76 deletions
Download patch file Download diff file Expand all files Collapse all files

19
src/db/factLog.ts
View file

@ -138,9 +138,11 @@ export interface FactOp {
/**
* V2-native records beyond noun/verb ops that a fact may carry through the
* ENCODER (types 6/7/8/9/10 of the v2 registry: embed markers, blob
* manifests, projection notes, bootstrap baselines). Encoder-ready by
* design; nothing produces them yet — the deferred-embed sidecar and blob
* lifecycle remodel onto these records in a later leg.
* manifests, projection notes, bootstrap baselines). The deferred-embedding
* lifecycle PRODUCES types 6/7 today: `embed.pending` rides the deferred
* write's own commit fact and `embed.landed` rides the background worker's
* landing commit (recovery folds the pair back out of the log at open). The
* blob lifecycle remodels onto type 8 in a later leg.
*/
export type FactMarkerRecord =
| EmbedPendingRecord
@ -741,6 +743,17 @@ export class FactLog {
return this.head
}
/**
* True when this log has EVER had a v2 tail — the manifest's `brainId` is
* minted at every v2 tail creation seam and never removed (the tail-version
* check is a belt-and-braces second signal). Only v2 facts can carry marker
* records, so marker folds (e.g. the deferred-embed recovery scan) skip
* v1-only logs WHOLESALE on this one cheap check — no segment is read.
*/
hasV2History(): boolean {
return this.manifest.brainId !== undefined || this.tailVersion === FACT_LOG_FORMAT_V2
}
/**
* Open the log and reconcile it to committed truth: read the manifest,
* establish the tail's intact content (torn-tail scan), then TRUNCATE any