feat(embedding): MT5 — deferred embedding with durable markers; write acks never wait on a neural net
A3 of the service-class pair (BRAINY-PROD-LATENCY-TRIAD): a VFS file write ran the embedder synchronously while the caller waited — 5.6s p50 / 21.4s p95 per small file on a production deployment, the dominant stage of every capture write. - add()/update() gain deferEmbedding: the write acks at durability (data + metadata persisted, a DURABLE pending marker under _system/pending_embeds/<id> written BEFORE the commit — orphan-safe direction); the single-flight background worker embeds the CURRENT data and swaps the vector in ATOMICALLY (ReplaceInVectorIndex — the row is never absent from search; a deferred UPDATE keeps serving the OLD vector, stale-beats-absent per the flicker law). Typed refusals: defer+vector, defer-without-data. - CRASH-SAFE: markers are recovered at open by a BOUNDED prefix listing (never a store walk) and the worker resumes in the background — a crash can delay a vector, never lose one. A wedged embedder trips a LOUD 60s hang guard and the worker moves on (marker retained for retry). - The honest gauges: getIndexStatus().pendingEmbeds + pendingEmbedCount(); awaitPendingEmbeds() is the eventual-vector-index BARRIER for callers and tests that need searchability before proceeding. - VFS adopts it everywhere a write path could wait on the embedder: writeFile (both branches) and directory creation. Pinned in the strongest form: writeFile resolves while the embedder HANGS FOREVER. Pins: deferred-embedding 5/5 (ack law · stale-beats-absent · crash recovery across sessions · VFS hung-embedder ack · typed refusals). Gates: unit 1928/1928 · integration 765 · conformance 27/27.
This commit is contained in:
parent
ebe06cdf33
commit
287384cf1e
5 changed files with 477 additions and 22 deletions
|
|
@ -540,6 +540,22 @@ function rejectForgedSystemKeys(metadata: Record<string, unknown> | undefined, s
|
|||
|
||||
export function validateAddParams(params: AddParams): void {
|
||||
rejectForgedSystemKeys(params.metadata as Record<string, unknown> | undefined, 'add()')
|
||||
// MT5 deferred embedding: an explicit vector has nothing to defer, and a
|
||||
// deferral without data has nothing to embed — both are caller bugs that
|
||||
// must refuse with the fix, never be silently reinterpreted.
|
||||
if ((params as AddParams & { deferEmbedding?: boolean }).deferEmbedding === true) {
|
||||
if (params.vector) {
|
||||
throw new Error(
|
||||
`add(): deferEmbedding cannot be combined with an explicit 'vector' — ` +
|
||||
`the vector is already computed; drop one of the two.`
|
||||
)
|
||||
}
|
||||
if (!params.data) {
|
||||
throw new Error(
|
||||
`add(): deferEmbedding requires 'data' (the content the background worker will embed).`
|
||||
)
|
||||
}
|
||||
}
|
||||
// Universal truth: must have data or vector
|
||||
if (!params.data && !params.vector) {
|
||||
throw new Error(
|
||||
|
|
@ -581,6 +597,19 @@ export function validateAddParams(params: AddParams): void {
|
|||
*/
|
||||
export function validateUpdateParams(params: UpdateParams): void {
|
||||
rejectForgedSystemKeys(params.metadata as Record<string, unknown> | undefined, 'update()')
|
||||
if ((params as UpdateParams & { deferEmbedding?: boolean }).deferEmbedding === true) {
|
||||
if (params.vector) {
|
||||
throw new Error(
|
||||
`update(): deferEmbedding cannot be combined with an explicit 'vector' — ` +
|
||||
`the vector is already computed; drop one of the two.`
|
||||
)
|
||||
}
|
||||
if (!params.data) {
|
||||
throw new Error(
|
||||
`update(): deferEmbedding requires new 'data' — without a data change there is nothing to re-embed.`
|
||||
)
|
||||
}
|
||||
}
|
||||
// Universal truth: must have an ID
|
||||
if (!params.id) {
|
||||
throw new Error('id is required for update')
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue