feat(8.0): #35 part-3 — supply at-gen candidate vectors for the native exact-rerank

Completes brainy's #35 side. When the at-gen vector gate serves a filtered
semantic read, Brainy now ships the historically-correct candidate vectors so the
native provider reranks against them with zero per-vector FFI crossing — the
provider need not duplicate the per-generation retention Brainy already does.

- New `AtGenerationVectors { ids: BigInt64Array; vectors: Float32Array; dim }` on
  the VectorIndexProvider.search options bag (row-major: vectors[i*dim..] == ids[i];
  invariant vectors.length === ids.length*dim; ids = the candidate set; dim must
  match the index dim). The built-in JS index ignores it.
- buildAtGenerationVectors resolves each universe id's vector AS OF the generation
  (the record before-image, or live getNoun when untouched since the pin — not
  readNounRaw, whose canonical path is empty under lazy-vector eviction), interns
  the id via the shared mapper, and packs row-major. Absent/vectorless/wrong-dim
  ids are dropped (not vector-rankable). Bounded by the filtered universe.

Shape + the four clarifications (row-major, ids-are-the-candidate-set, dim-asserted,
filtered-path-only) confirmed with cor in the handoff #35 thread. Inert until cor's
native at-gen rerank advertises isGenerationVisible (honesty guard).

Tested: the mock versioned provider now asserts it receives atGenerationVectors with
the row-major invariant, correct dim, and ids resolving back to the at-gen universe.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
David Snelling 2026-06-23 16:50:41 -07:00
parent 6991bbe3d2
commit c9e2169415
4 changed files with 132 additions and 9 deletions

View file

@ -14,7 +14,7 @@ import { euclideanDistance, calculateDistancesBatch } from '../utils/index.js'
import type { BaseStorage } from '../storage/baseStorage.js'
import { getGlobalCache, UnifiedCache } from '../utils/unifiedCache.js'
import { prodLog } from '../utils/logger.js'
import type { VectorIndexProvider, OpaqueIdSet } from '../plugin.js'
import type { VectorIndexProvider, OpaqueIdSet, AtGenerationVectors } from '../plugin.js'
import { ConnectionsCodec, compressedConnectionsKey } from './connectionsCodec.js'
// Default HNSW parameters
@ -673,6 +673,9 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
// historical read here (its defer-gate checks `isGenerationVisible` first),
// so an ignored `generation` cannot silently return now-vectors-as-at-gen.
generation?: bigint
// 8.0 #35 part-3: at-gen candidate vectors for the native exact-rerank. The JS
// index serves "now" only and never receives this (gated off upstream); ignored.
atGenerationVectors?: AtGenerationVectors
}
): Promise<Array<[string, number]>> {
if (this.nouns.size === 0) {