feat(embedding): MT5 — deferred embedding with durable markers; write acks never wait on a neural net
Some checks failed
CI / Node 22 (push) Has been cancelled
CI / Node 24 (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled

A3 of the service-class pair (BRAINY-PROD-LATENCY-TRIAD): a VFS file write
ran the embedder synchronously while the caller waited — 5.6s p50 / 21.4s
p95 per small file on a production deployment, the dominant stage of every
capture write.

- add()/update() gain deferEmbedding: the write acks at durability (data +
  metadata persisted, a DURABLE pending marker under
  _system/pending_embeds/<id> written BEFORE the commit — orphan-safe
  direction); the single-flight background worker embeds the CURRENT data
  and swaps the vector in ATOMICALLY (ReplaceInVectorIndex — the row is
  never absent from search; a deferred UPDATE keeps serving the OLD vector,
  stale-beats-absent per the flicker law). Typed refusals: defer+vector,
  defer-without-data.
- CRASH-SAFE: markers are recovered at open by a BOUNDED prefix listing
  (never a store walk) and the worker resumes in the background — a crash
  can delay a vector, never lose one. A wedged embedder trips a LOUD 60s
  hang guard and the worker moves on (marker retained for retry).
- The honest gauges: getIndexStatus().pendingEmbeds + pendingEmbedCount();
  awaitPendingEmbeds() is the eventual-vector-index BARRIER for callers
  and tests that need searchability before proceeding.
- VFS adopts it everywhere a write path could wait on the embedder:
  writeFile (both branches) and directory creation. Pinned in the
  strongest form: writeFile resolves while the embedder HANGS FOREVER.

Pins: deferred-embedding 5/5 (ack law · stale-beats-absent · crash
recovery across sessions · VFS hung-embedder ack · typed refusals).
Gates: unit 1928/1928 · integration 765 · conformance 27/27.
This commit is contained in:
David Snelling 2026-08-05 16:26:43 -07:00
parent ebe06cdf33
commit 287384cf1e
5 changed files with 477 additions and 22 deletions

View file

@ -694,6 +694,12 @@ export class VirtualFileSystem implements IVirtualFileSystem {
await this.brain.update({
id: existingId,
data: embeddingData,
// MT5: the caller's write acks at durability; the re-embed (a neural
// net — it dominated the measured 5.6s p50 per file write) runs on
// the background worker and swaps in atomically. Content is readable
// and metadata-findable immediately; semantic search converges when
// the embed lands (eventual vector index, the documented contract).
deferEmbedding: true,
metadata
})
@ -729,6 +735,9 @@ export class VirtualFileSystem implements IVirtualFileSystem {
data: embeddingData, // Always provide string for embeddings
type: this.getFileNounType(mimeType),
subtype: 'vfs-file', // Standard subtype for VFS file entities (7.30+)
// MT5: ack at durability; embedding backgrounds (see the overwrite
// branch note above).
deferEmbedding: true,
metadata
})
@ -1117,6 +1126,9 @@ export class VirtualFileSystem implements IVirtualFileSystem {
data: path, // Directory path as string content
type: NounType.Collection,
subtype: 'vfs-directory', // Standard subtype for VFS directory entities (7.30+)
// MT5: a directory creation on a write path must not wait on the
// embedder either — same ack-at-durability contract as file writes.
deferEmbedding: true,
metadata
})