feat(8.0): #35 part-3 — supply at-gen candidate vectors for the native exact-rerank

Completes brainy's #35 side. When the at-gen vector gate serves a filtered
semantic read, Brainy now ships the historically-correct candidate vectors so the
native provider reranks against them with zero per-vector FFI crossing — the
provider need not duplicate the per-generation retention Brainy already does.

- New `AtGenerationVectors { ids: BigInt64Array; vectors: Float32Array; dim }` on
  the VectorIndexProvider.search options bag (row-major: vectors[i*dim..] == ids[i];
  invariant vectors.length === ids.length*dim; ids = the candidate set; dim must
  match the index dim). The built-in JS index ignores it.
- buildAtGenerationVectors resolves each universe id's vector AS OF the generation
  (the record before-image, or live getNoun when untouched since the pin — not
  readNounRaw, whose canonical path is empty under lazy-vector eviction), interns
  the id via the shared mapper, and packs row-major. Absent/vectorless/wrong-dim
  ids are dropped (not vector-rankable). Bounded by the filtered universe.

Shape + the four clarifications (row-major, ids-are-the-candidate-set, dim-asserted,
filtered-path-only) confirmed with cor in the handoff #35 thread. Inert until cor's
native at-gen rerank advertises isGenerationVisible (honesty guard).

Tested: the mock versioned provider now asserts it receives atGenerationVectors with
the row-major invariant, correct dim, and ids resolving back to the at-gen universe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
David Snelling 2026-06-23 16:50:41 -07:00
parent 6991bbe3d2
commit c9e2169415
4 changed files with 132 additions and 9 deletions

View file

@ -734,6 +734,34 @@ export function isGraphAccelerationProvider(
return typeof c.traverse === 'function' && typeof c.graphCursorOpen === 'function'
}
/**
* Columnar at-`generation` candidate vectors for the 8.0 #35 FILTERED exact-rerank.
* Brainy resolves the per-generation vector before-images for the at-gen filtered
* universe (from its canonical generation records) and ships them flat, so a native
* provider reranks against the historically-correct vectors with ZERO per-vector FFI
* crossing without the provider duplicating the per-gen retention Brainy already does.
*
* - **Row-major:** row `i` is `vectors[i*dim .. (i+1)*dim]` and belongs to `ids[i]`.
* INVARIANT: `vectors.length === ids.length * dim` (the provider asserts + refuses on mismatch).
* - **`ids` IS the candidate set:** entity ints (interned via the shared mapper) for the
* at-gen metadatagraph filtered universe. The provider reranks EXACTLY these
* `(id, vector)` pairs and returns top-k by distance; any `allowedIds` passed alongside
* is redundant on this path (the provider may AND it as belt-and-suspenders).
* - **`dim`** must equal the provider's configured index dimension (asserted; mismatch refuse).
*
* Supplied only when there is a BOUNDED filtered universe; the unfiltered-deep at-gen
* case stays the provider's own retained-segment ANN ({@link VersionedIndexProvider.isGenerationVisible}
* reports false there, so Brainy falls back to its materialization overlay).
*/
export interface AtGenerationVectors {
/** Entity ints (shared-mapper interned), one per candidate; the candidate set. */
ids: BigInt64Array
/** Flat row-major vectors: `vectors[i*dim .. (i+1)*dim]` is `ids[i]`'s at-gen vector. */
vectors: Float32Array
/** Vector dimension; MUST equal the provider's configured index dim. */
dim: number
}
/**
* The object returned by the `'vector'` provider factory Brainy's vector
* index contract. Implementations include Brainy's own JS HNSW index and any
@ -782,6 +810,15 @@ export interface VectorIndexProvider {
* `generation` is the same u64 counter handed to the graph index on writes.
*/
generation?: bigint
/**
* 8.0 #35 part-3: the at-`generation` candidate vectors for the FILTERED
* exact-rerank Brainy supplies the historically-correct vectors so the
* provider need not retain them. Present only alongside `generation` on the
* filtered at-gen path; the provider reranks `atGenerationVectors.ids` by
* distance and returns top-k. See {@link AtGenerationVectors}. The built-in
* JS index ignores it (it serves "now" only).
*/
atGenerationVectors?: AtGenerationVectors
}
): Promise<Array<[string, number]>>
size(): number