feat(8.0): #35 part-3 — supply at-gen candidate vectors for the native exact-rerank
Completes brainy's #35 side. When the at-gen vector gate serves a filtered semantic read, Brainy now ships the historically-correct candidate vectors so the native provider reranks against them with zero per-vector FFI crossing — the provider need not duplicate the per-generation retention Brainy already does. - New `AtGenerationVectors { ids: BigInt64Array; vectors: Float32Array; dim }` on the VectorIndexProvider.search options bag (row-major: vectors[i*dim..] == ids[i]; invariant vectors.length === ids.length*dim; ids = the candidate set; dim must match the index dim). The built-in JS index ignores it. - buildAtGenerationVectors resolves each universe id's vector AS OF the generation (the record before-image, or live getNoun when untouched since the pin — not readNounRaw, whose canonical path is empty under lazy-vector eviction), interns the id via the shared mapper, and packs row-major. Absent/vectorless/wrong-dim ids are dropped (not vector-rankable). Bounded by the filtered universe. Shape + the four clarifications (row-major, ids-are-the-candidate-set, dim-asserted, filtered-path-only) confirmed with cor in the handoff #35 thread. Inert until cor's native at-gen rerank advertises isGenerationVisible (honesty guard). Tested: the mock versioned provider now asserts it receives atGenerationVectors with the row-major invariant, correct dim, and ids resolving back to the at-gen universe. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
parent
6991bbe3d2
commit
c9e2169415
4 changed files with 132 additions and 9 deletions
|
|
@ -51,7 +51,8 @@ import type {
|
|||
CommunitiesOptions,
|
||||
PathOptions,
|
||||
MetadataIndexProvider,
|
||||
OpaqueIdSet
|
||||
OpaqueIdSet,
|
||||
AtGenerationVectors
|
||||
} from './plugin.js'
|
||||
import type {
|
||||
BrainyPlugin,
|
||||
|
|
@ -7288,16 +7289,68 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
generation: number
|
||||
): Promise<Array<[string, number]>> {
|
||||
const vector = params.vector || (await this.embed(params.query!))
|
||||
// The provider scores the at-gen vectors restricted to `allowedIds`. (#35 part-3:
|
||||
// the gen-g candidate VECTORS — `atGenerationVectors` — are supplied here once that
|
||||
// seam is confirmed with cor; a provider needing them refuses the generation until
|
||||
// then, so `canServeVectorAtGeneration` gates this path off in the interim.)
|
||||
// #35 part-3: supply the at-gen candidate VECTORS (resolved from Brainy's
|
||||
// per-generation record before-images) so the provider reranks the historically
|
||||
// correct vectors with zero per-vector crossing. `atGenerationVectors.ids` is the
|
||||
// candidate set; `allowedIds` rides along (the provider may AND it).
|
||||
const atGenerationVectors = await this.buildAtGenerationVectors(allowedIds, generation)
|
||||
return this.index.search(vector, k, undefined, {
|
||||
allowedIds,
|
||||
generation: BigInt(generation)
|
||||
generation: BigInt(generation),
|
||||
...(atGenerationVectors && { atGenerationVectors })
|
||||
})
|
||||
}
|
||||
|
||||
/**
|
||||
* @description 8.0 #35 part-3 — assemble the {@link AtGenerationVectors} columnar
|
||||
* payload for a filtered at-gen exact-rerank: resolve each universe id's vector AS
|
||||
* OF `generation` (the canonical record before-image, or live storage when the id
|
||||
* was untouched since the pin), intern the id via the shared mapper, and pack the
|
||||
* vectors flat row-major (`vectors[i*dim .. (i+1)*dim]` ↔ `ids[i]`). Ids absent at
|
||||
* the generation, vectorless, or of the wrong dimension are dropped (they cannot be
|
||||
* vector-ranked). Bounded by the filtered universe — far cheaper than the full-corpus
|
||||
* rebuild it replaces. Returns `undefined` when no dimension is known or nothing maps.
|
||||
* @param allowedIds - The at-gen filtered universe (candidate ids).
|
||||
* @param generation - The as-of generation.
|
||||
* @returns The columnar payload, or `undefined` when there is nothing to ship.
|
||||
*/
|
||||
private async buildAtGenerationVectors(
|
||||
allowedIds: ReadonlySet<string>,
|
||||
generation: number
|
||||
): Promise<AtGenerationVectors | undefined> {
|
||||
const dim = this.dimensions
|
||||
if (!dim || allowedIds.size === 0) return undefined
|
||||
|
||||
const idMapper = this.metadataIndex.getIdMapper()
|
||||
const ints: bigint[] = []
|
||||
const rows: number[][] = []
|
||||
for (const id of allowedIds) {
|
||||
const resolved = await this.generationStore.resolveAt('noun', id, generation)
|
||||
let vec: number[] | null = null
|
||||
if (resolved.source === 'record') {
|
||||
// The generation record carries the at-gen vector before-image.
|
||||
vec = resolved.vector
|
||||
} else if (resolved.source === 'current') {
|
||||
// Untouched since the pin → the live vector IS the at-gen vector. Use
|
||||
// getNoun (full hydration, lazy-load aware) — readNounRaw's canonical path
|
||||
// is empty under lazy-vector eviction.
|
||||
const noun = await this.storage.getNoun(id)
|
||||
vec = noun?.vector ?? null
|
||||
}
|
||||
// 'absent' / vectorless / wrong-dim → skip (not vector-rankable at this gen).
|
||||
if (Array.isArray(vec) && vec.length === dim) {
|
||||
ints.push(BigInt(idMapper.getInt(id) ?? idMapper.getOrAssign(id)))
|
||||
rows.push(vec)
|
||||
}
|
||||
}
|
||||
if (ints.length === 0) return undefined
|
||||
|
||||
// Row-major pack: INVARIANT vectors.length === ids.length * dim.
|
||||
const vectors = new Float32Array(ints.length * dim)
|
||||
for (let i = 0; i < rows.length; i++) vectors.set(rows[i], i * dim)
|
||||
return { ids: BigInt64Array.from(ints), vectors, dim }
|
||||
}
|
||||
|
||||
// --- Transact planner ------------------------------------------------------
|
||||
|
||||
/**
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue