feat(8.0): #35 part-3 — supply at-gen candidate vectors for the native exact-rerank

Completes brainy's #35 side. When the at-gen vector gate serves a filtered
semantic read, Brainy now ships the historically-correct candidate vectors so the
native provider reranks against them with zero per-vector FFI crossing — the
provider need not duplicate the per-generation retention Brainy already does.

- New `AtGenerationVectors { ids: BigInt64Array; vectors: Float32Array; dim }` on
  the VectorIndexProvider.search options bag (row-major: vectors[i*dim..] == ids[i];
  invariant vectors.length === ids.length*dim; ids = the candidate set; dim must
  match the index dim). The built-in JS index ignores it.
- buildAtGenerationVectors resolves each universe id's vector AS OF the generation
  (the record before-image, or live getNoun when untouched since the pin — not
  readNounRaw, whose canonical path is empty under lazy-vector eviction), interns
  the id via the shared mapper, and packs row-major. Absent/vectorless/wrong-dim
  ids are dropped (not vector-rankable). Bounded by the filtered universe.

Shape + the four clarifications (row-major, ids-are-the-candidate-set, dim-asserted,
filtered-path-only) confirmed with cor in the handoff #35 thread. Inert until cor's
native at-gen rerank advertises isGenerationVisible (honesty guard).

Tested: the mock versioned provider now asserts it receives atGenerationVectors with
the row-major invariant, correct dim, and ids resolving back to the at-gen universe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
David Snelling 2026-06-23 16:50:41 -07:00
parent 6991bbe3d2
commit c9e2169415
4 changed files with 132 additions and 9 deletions

View file

@ -51,7 +51,8 @@ import type {
CommunitiesOptions,
PathOptions,
MetadataIndexProvider,
OpaqueIdSet
OpaqueIdSet,
AtGenerationVectors
} from './plugin.js'
import type {
BrainyPlugin,
@ -7288,16 +7289,68 @@ export class Brainy<T = any> implements BrainyInterface<T> {
generation: number
): Promise<Array<[string, number]>> {
const vector = params.vector || (await this.embed(params.query!))
// The provider scores the at-gen vectors restricted to `allowedIds`. (#35 part-3:
// the gen-g candidate VECTORS — `atGenerationVectors` — are supplied here once that
// seam is confirmed with cor; a provider needing them refuses the generation until
// then, so `canServeVectorAtGeneration` gates this path off in the interim.)
// #35 part-3: supply the at-gen candidate VECTORS (resolved from Brainy's
// per-generation record before-images) so the provider reranks the historically
// correct vectors with zero per-vector crossing. `atGenerationVectors.ids` is the
// candidate set; `allowedIds` rides along (the provider may AND it).
const atGenerationVectors = await this.buildAtGenerationVectors(allowedIds, generation)
return this.index.search(vector, k, undefined, {
allowedIds,
generation: BigInt(generation)
generation: BigInt(generation),
...(atGenerationVectors && { atGenerationVectors })
})
}
/**
* @description 8.0 #35 part-3 assemble the {@link AtGenerationVectors} columnar
* payload for a filtered at-gen exact-rerank: resolve each universe id's vector AS
* OF `generation` (the canonical record before-image, or live storage when the id
* was untouched since the pin), intern the id via the shared mapper, and pack the
* vectors flat row-major (`vectors[i*dim .. (i+1)*dim]` `ids[i]`). Ids absent at
* the generation, vectorless, or of the wrong dimension are dropped (they cannot be
* vector-ranked). Bounded by the filtered universe far cheaper than the full-corpus
* rebuild it replaces. Returns `undefined` when no dimension is known or nothing maps.
* @param allowedIds - The at-gen filtered universe (candidate ids).
* @param generation - The as-of generation.
* @returns The columnar payload, or `undefined` when there is nothing to ship.
*/
private async buildAtGenerationVectors(
allowedIds: ReadonlySet<string>,
generation: number
): Promise<AtGenerationVectors | undefined> {
const dim = this.dimensions
if (!dim || allowedIds.size === 0) return undefined
const idMapper = this.metadataIndex.getIdMapper()
const ints: bigint[] = []
const rows: number[][] = []
for (const id of allowedIds) {
const resolved = await this.generationStore.resolveAt('noun', id, generation)
let vec: number[] | null = null
if (resolved.source === 'record') {
// The generation record carries the at-gen vector before-image.
vec = resolved.vector
} else if (resolved.source === 'current') {
// Untouched since the pin → the live vector IS the at-gen vector. Use
// getNoun (full hydration, lazy-load aware) — readNounRaw's canonical path
// is empty under lazy-vector eviction.
const noun = await this.storage.getNoun(id)
vec = noun?.vector ?? null
}
// 'absent' / vectorless / wrong-dim → skip (not vector-rankable at this gen).
if (Array.isArray(vec) && vec.length === dim) {
ints.push(BigInt(idMapper.getInt(id) ?? idMapper.getOrAssign(id)))
rows.push(vec)
}
}
if (ints.length === 0) return undefined
// Row-major pack: INVARIANT vectors.length === ids.length * dim.
const vectors = new Float32Array(ints.length * dim)
for (let i = 0; i < rows.length; i++) vectors.set(rows[i], i * dim)
return { ids: BigInt64Array.from(ints), vectors, dim }
}
// --- Transact planner ------------------------------------------------------
/**