feat(8.0): asOf at-gen vector defer — provider-served historical semantic search (#35)
A filtered semantic read at a historical generation (db.asOf(g).find({ query, where }))
no longer unconditionally rebuilds an ephemeral JS-HNSW over every at-g vector
(O(n@G), the OOM ceiling the 2026-06-24 scaling audit flagged). When the vector
index is a VersionedIndexProvider that advertises isGenerationVisible(g), Brainy:
1. resolves the at-g metadata∩graph universe from the record-overlay path (no
materialization — it's the metadata-only historical find),
2. routes the vector leg to the provider with { allowedIds, generation }, and
3. composes the at-g entities ranked by the provider's at-g vector distance.
Without a versioned provider (the JS index, or a native one that refuses the
generation) it falls through to the existing materialization — unchanged.
- Seam (A): VectorIndexProvider.search gains an optional `generation?: bigint`
("omitted = now"), the vector mirror of the graph provider's trailing-gen + the
#46 allowedIds field. JsHnswVectorIndex accepts and ignores it (no per-gen
segments → refuse/fall-back posture; Brainy never routes a historical read there).
- Gate: Db.find tries the native at-gen vector path before materialize(); host
exposes canServeVectorAtGeneration + vectorSearchAtGeneration.
- generation is Brainy's u64 commit counter — the same value handed to the graph
index on writes (graphWriteGeneration), so it maps 1:1 to the native side.
Decided lockstep with the cor team (handoff #35 thread: (A) + vector-only). The
gen-g candidate-vector supply for cor's exact-rerank (part-3) is a separate,
non-blocking seam still being confirmed; the honesty guard keeps this path inactive
(falls through to materialize) until cor's native at-gen rerank is live.
Tested: provider routing + at-gen universe correctness (excludes future-born,
applies the filter) + page window via a mock versioned provider; seam-ignore on the
JS index; 38 db-mvcc/db-temporal historical-read tests green (materialize fallback).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
450084b6ce
commit
1c363e8c4b
7 changed files with 292 additions and 0 deletions
|
|
@ -73,6 +73,14 @@ no separate migration doc. **`8.0.0-rc.2` is live** — install with
|
|||
> matches that DO pass the filter, rather than coming back empty. No API change. With the native
|
||||
> `@soulcraft/cor` 3.0 stack the matched universe is forwarded as an opaque roaring buffer with
|
||||
> zero id materialization in TypeScript; the pure-JS path restricts the beam walk with a string set.
|
||||
> - **Historical (`asOf`) semantic search can skip the rebuild.** A filtered semantic query at a
|
||||
> past generation — `db.asOf(g).find({ query, where })` — no longer always rebuilds an ephemeral
|
||||
> in-memory vector index over every at-`g` vector (O(n@G)). When a native versioned vector engine
|
||||
> is registered and can serve the pinned generation, Brainy resolves the at-`g` filter universe from
|
||||
> its record layer (no rebuild) and routes the vector leg to the engine with the matched ids + the
|
||||
> generation; without one it falls back to the materialization (unchanged). The `VectorIndexProvider.search`
|
||||
> options gained an optional `generation?: bigint` ("omitted = now") to carry this — additive, the
|
||||
> built-in index ignores it. No public API change.
|
||||
> - **`find({ where, orderBy })` bounds the sort to the page.** A broad filter + `orderBy`
|
||||
> returning one page no longer materializes every matching sorted id (it produced the full sorted
|
||||
> match set, then sliced) — the page bound (`offset + limit`) is threaded into the column store's
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue