feat(8.0): asOf at-gen vector defer — provider-served historical semantic search (#35)
A filtered semantic read at a historical generation (db.asOf(g).find({ query, where }))
no longer unconditionally rebuilds an ephemeral JS-HNSW over every at-g vector
(O(n@G), the OOM ceiling the 2026-06-24 scaling audit flagged). When the vector
index is a VersionedIndexProvider that advertises isGenerationVisible(g), Brainy:
1. resolves the at-g metadata∩graph universe from the record-overlay path (no
materialization — it's the metadata-only historical find),
2. routes the vector leg to the provider with { allowedIds, generation }, and
3. composes the at-g entities ranked by the provider's at-g vector distance.
Without a versioned provider (the JS index, or a native one that refuses the
generation) it falls through to the existing materialization — unchanged.
- Seam (A): VectorIndexProvider.search gains an optional `generation?: bigint`
("omitted = now"), the vector mirror of the graph provider's trailing-gen + the
#46 allowedIds field. JsHnswVectorIndex accepts and ignores it (no per-gen
segments → refuse/fall-back posture; Brainy never routes a historical read there).
- Gate: Db.find tries the native at-gen vector path before materialize(); host
exposes canServeVectorAtGeneration + vectorSearchAtGeneration.
- generation is Brainy's u64 commit counter — the same value handed to the graph
index on writes (graphWriteGeneration), so it maps 1:1 to the native side.
Decided lockstep with the cor team (handoff #35 thread: (A) + vector-only). The
gen-g candidate-vector supply for cor's exact-rerank (part-3) is a separate,
non-blocking seam still being confirmed; the honesty guard keeps this path inactive
(falls through to materialize) until cor's native at-gen rerank is live.
Tested: provider routing + at-gen universe correctness (excludes future-born,
applies the filter) + page window via a mock versioned provider; seam-ignore on the
JS index; 38 db-mvcc/db-temporal historical-read tests green (materialize fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
450084b6ce
commit
1c363e8c4b
7 changed files with 292 additions and 0 deletions
|
|
@ -6929,6 +6929,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
persistPinned: (targetPath, generation) =>
|
||||
this.persistPinnedGeneration(targetPath, generation),
|
||||
materializeAt: (generation) => this.materializeAtGeneration(generation),
|
||||
canServeVectorAtGeneration: (generation) => this.canServeVectorAtGeneration(generation),
|
||||
vectorSearchAtGeneration: (params, allowedIds, k, generation) =>
|
||||
this.vectorSearchAtGeneration(params, allowedIds, k, generation),
|
||||
pinGeneration: (generation) => this.pinGeneration(generation),
|
||||
releaseGeneration: (generation) => this.releaseGeneration(generation),
|
||||
registerDbForFinalization: (db, generation, closeOnRelease) => {
|
||||
|
|
@ -7251,6 +7254,50 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @description 8.0 #35 — true when the vector index is a
|
||||
* {@link VersionedIndexProvider} that advertises `isGenerationVisible(generation)`:
|
||||
* it can serve the at-`generation` vector leg from retained segments, so a
|
||||
* historical filtered semantic read needs no O(n@G) JS-HNSW rebuild. The built-in
|
||||
* `JsHnswVectorIndex` is not versioned (only ever "now"), and a native provider
|
||||
* that cannot honor `generation` MUST return `false` here (refuse, never fabricate
|
||||
* now-vectors-as-at-gen), so the caller falls back to materialization.
|
||||
*/
|
||||
private canServeVectorAtGeneration(generation: number): boolean {
|
||||
return (
|
||||
isVersionedIndexProvider(this.index) &&
|
||||
this.index.isGenerationVisible(BigInt(generation))
|
||||
)
|
||||
}
|
||||
|
||||
/**
|
||||
* @description 8.0 #35 — run the vector kNN AS OF `generation`, restricted to
|
||||
* `allowedIds` (the at-gen metadata∩graph universe the `Db` resolved from the
|
||||
* record layer). The versioned provider serves the at-gen walk; Brainy composes
|
||||
* the metadata half. Only reached when {@link canServeVectorAtGeneration} is true.
|
||||
* @param params - The semantic (`query`) or explicit-`vector` find params.
|
||||
* @param allowedIds - The at-gen candidate universe (membership-correct at `generation`).
|
||||
* @param k - Over-fetch (page + headroom).
|
||||
* @param generation - The as-of generation (Brainy's u64 commit counter).
|
||||
* @returns `[id, distance]` pairs, ascending distance (descending relevance).
|
||||
*/
|
||||
private async vectorSearchAtGeneration(
|
||||
params: FindParams<T>,
|
||||
allowedIds: ReadonlySet<string>,
|
||||
k: number,
|
||||
generation: number
|
||||
): Promise<Array<[string, number]>> {
|
||||
const vector = params.vector || (await this.embed(params.query!))
|
||||
// The provider scores the at-gen vectors restricted to `allowedIds`. (#35 part-3:
|
||||
// the gen-g candidate VECTORS — `atGenerationVectors` — are supplied here once that
|
||||
// seam is confirmed with cor; a provider needing them refuses the generation until
|
||||
// then, so `canServeVectorAtGeneration` gates this path off in the interim.)
|
||||
return this.index.search(vector, k, undefined, {
|
||||
allowedIds,
|
||||
generation: BigInt(generation)
|
||||
})
|
||||
}
|
||||
|
||||
// --- Transact planner ------------------------------------------------------
|
||||
|
||||
/**
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue