feat(embedding): MT5 — deferred embedding with durable markers; write acks never wait on a neural net
Some checks failed
CI / Node 22 (push) Has been cancelled
CI / Node 24 (push) Has been cancelled
CI / Bun (latest) (push) Has been cancelled

A3 of the service-class pair (BRAINY-PROD-LATENCY-TRIAD): a VFS file write
ran the embedder synchronously while the caller waited — 5.6s p50 / 21.4s
p95 per small file on a production deployment, the dominant stage of every
capture write.

- add()/update() gain deferEmbedding: the write acks at durability (data +
  metadata persisted, a DURABLE pending marker under
  _system/pending_embeds/<id> written BEFORE the commit — orphan-safe
  direction); the single-flight background worker embeds the CURRENT data
  and swaps the vector in ATOMICALLY (ReplaceInVectorIndex — the row is
  never absent from search; a deferred UPDATE keeps serving the OLD vector,
  stale-beats-absent per the flicker law). Typed refusals: defer+vector,
  defer-without-data.
- CRASH-SAFE: markers are recovered at open by a BOUNDED prefix listing
  (never a store walk) and the worker resumes in the background — a crash
  can delay a vector, never lose one. A wedged embedder trips a LOUD 60s
  hang guard and the worker moves on (marker retained for retry).
- The honest gauges: getIndexStatus().pendingEmbeds + pendingEmbedCount();
  awaitPendingEmbeds() is the eventual-vector-index BARRIER for callers
  and tests that need searchability before proceeding.
- VFS adopts it everywhere a write path could wait on the embedder:
  writeFile (both branches) and directory creation. Pinned in the
  strongest form: writeFile resolves while the embedder HANGS FOREVER.

Pins: deferred-embedding 5/5 (ack law · stale-beats-absent · crash
recovery across sessions · VFS hung-embedder ack · typed refusals).
Gates: unit 1928/1928 · integration 765 · conformance 27/27.
This commit is contained in:
David Snelling 2026-08-05 16:26:43 -07:00
parent ebe06cdf33
commit 287384cf1e
5 changed files with 477 additions and 22 deletions

View file

@ -338,6 +338,20 @@ export interface AddParams<T = any> {
id?: string
/** Pre-computed embedding vector (skips auto-embedding when provided) */
vector?: Vector
/**
* DEFER THE EMBEDDING (MT5, the deferred-embedding worker): the write
* acknowledges at durability data + metadata persisted, a durable
* pending-embed marker written and the embedding + vector-index insert
* run on the engine's single-flight background worker. HONEST SEMANTICS:
* the row is findable by id/metadata/path IMMEDIATELY; vector/semantic
* search sees it when the background embed completes (eventual vector
* index `getIndexStatus().pendingEmbeds` counts the backlog, and
* `awaitPendingEmbeds()` is the barrier). CRASH-SAFE: markers persist
* before the ack and are recovered at the next open a crash can DELAY
* a vector, never lose one. Refused (typed) together with `vector`
* a supplied vector has nothing to defer.
*/
deferEmbedding?: boolean
/** Multi-tenancy service identifier */
service?: string
/** Type classification confidence (0-1) */
@ -379,6 +393,15 @@ export interface AddParams<T = any> {
export interface UpdateParams<T = any> {
id: string // Entity to update
data?: any // New content to re-embed
/**
* Defer the re-embedding of new `data` (see `AddParams.deferEmbedding`).
* The write acks at durability; the OLD vector keeps serving semantic
* search stale-but-present, never absent (the flicker law) until the
* background worker embeds the new content and swaps it in atomically.
* `data` reads return the NEW content immediately. Refused (typed) with
* an explicit `vector`.
*/
deferEmbedding?: boolean
type?: NounType // Change type
subtype?: string // Change subtype (set to '' or null-equivalent via dedicated unset is future work)
/**