feat(vector): the vectored-noun scalar joins the count ledger; the open gate closes the vector leg
All checks were successful
CI / Node 24 (push) Successful in 12m22s
CI / Node 22 (push) Successful in 12m32s
CI / Integration + conformance (Node 22) (push) Successful in 19m42s
CI / Bun (latest) (push) Successful in 12m15s

The coverage denominator the health-by-accounting ratification named for
the vector family — never built until now, and its absence was measured as
the exact outage class it existed to prevent: a migrated store with
canonical vectors and no derived index opened with the vector leg EMPTY,
served [] from vector search with no error, and the report-driven read gate
had nothing to refuse on (the provider's coverage invariant was honestly
unledgered — the denominator was ours to supply).

- getCanonicalCounts() gains vectors: { all } — the count of canonical
  nouns holding a REAL vector. Incremented where a vector lands (the
  isNew-gated metadata seam for explicit vectors — the same discipline that
  keeps HNSW neighbor-link re-saves from inflating counts; a narrow
  noteVectorLanded hook for the deferred-embed landing, gated on the
  worker's own pre-embed read). Decremented on a proven delete of a
  vectored noun; a vector-uncertain delete marks the ledger suspect rather
  than guessing (no new reads on the delete path). Recounted by the
  sanctioned recount; legacy counts.json derives it once (a deferred noun's
  vector file exists with an empty vector, so presence requires one
  content read at derivation — never on the hot path).
- The open gate's vector leg: when a health-reporting provider claims
  serving while the index holds zero nodes and the ledger proves vectored
  canonical rows exist, open BUILDS (narrated) — routed through the
  provider's idempotent fillFromCanonical() when exposed (the joint door;
  a partial shortfall stays repair()'s operator business), the JS rebuild
  otherwise — or fails typed pre-serve. Scoped exactly: bare isReady()
  providers, migrating providers, and white-box size stubs open as before.

Pinned end-to-end from the partner gate's probe shape (store with vectored
canonical rows, no derived index, reopen → search serves N, never []),
red-proved against the pre-fix path; the inverse (zero vectored rows) opens
without building and serves [] honestly.
This commit is contained in:
David Snelling 2026-08-25 15:31:19 -07:00
parent bce2593e24
commit 9730835bdf
10 changed files with 851 additions and 32 deletions

View file

@ -2382,6 +2382,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
[{ type: 'embed.landed', id, vector: newVector }],
'system:embed-landing'
)
// Vectored-noun ledger: the landing commit above carries a vector
// write with NO accompanying metadata operation, so the
// saveNounMetadata(..., hasVector) seam never fires for it — the
// narrow storage hook is the only seam left. `oldVector.length===0`
// (already known for free from the pre-embed read above) proves this
// is a GENUINE first landing, not a re-embed of an already-vectored
// row (e.g. a deferred update() on a row that already had a real
// vector) — the latter must never double-count.
if (oldVector.length === 0) {
await this.storage.noteVectorLanded?.(id)
}
this.clearPendingEmbed(id)
} catch (err) {
prodLog.warn(
@ -3000,8 +3011,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const runInsert: TransactionFunction<void> = async (tx) => {
// Operation 1: Save metadata FIRST (TypeAwareStorage caching)
// isNew=true: skip pre-read for rollback (entity doesn't exist yet)
// hasVector: the vectored-noun ledger counts this insert iff its
// vector is real/non-empty (never true for a deferred embed, whose
// stub `vector` is `[]` — it counts later, at landing).
tx.addOperation(
new SaveNounMetadataOperation(this.storage, id, storageMetadata, true)
new SaveNounMetadataOperation(this.storage, id, storageMetadata, true, vector.length > 0)
)
// Operation 2: Save vector data
@ -10568,7 +10582,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
plan.postCommit.push(() => this.kickEmbedWorker())
}
plan.operations.push(
new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew),
// hasVector: see the single-add() insert path's comment — never true
// for a deferred embed (stub vector `[]`; counted later at landing).
new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew, vector.length > 0),
new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew),
...(deferringEmbed
? []
@ -16900,9 +16916,92 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const shouldRebuildMetadata =
!metadataMigrating &&
(epochStale || legNeedsRebuild(this.metadataIndex, metadataStats.totalEntries === 0))
const shouldRebuildVector =
// VECTOR LEG — the two-engine gate's last red: a migrated 7.x-era store
// can hold canonical vectored nouns with NO derived vector index built.
// `legNeedsRebuild`'s size-heuristic fallback (below) only fires off
// `hnswIndexSize === 0`, and its health-report branch trusts a
// provider's own `serving` verdict verbatim — but a provider's health
// report can legitimately say `serving: true` while vector coverage is
// honestly UNLEDGERED on ITS side too (an unledgered invariant never
// flips serving), so neither signal alone can tell "genuinely empty"
// apart from "never built". The canonical vectored-noun ledger
// (`getCanonicalCounts().vectors.all` — Deliverable 1) is the
// denominator that CAN tell them apart, and is compared here:
// - a CONFIDENT (non-suspect) ledger `> 0` while the reported node
// count is 0 is a proven coverage gap — force the build regardless
// of what a health report claims;
// - a CONFIDENT ledger `=== 0` while the node count is 0 proves there
// is nothing to load (e.g. every noun's embed is still deferred) —
// skip the size-heuristic fallback's blunt "always rebuild when
// empty" trigger, which otherwise wastes a full canonical walk for
// zero benefit on every cold open of such a store;
// - an unavailable/suspect ledger changes nothing — loud errors never
// quiet losses, so a doubtful ledger must never suppress a rebuild
// the old heuristic would have run.
// The bare `isReady()` boolean (no report, no `unledgered` concept) is
// NOT overridden — that signal is what fixed the 48-seconds-per-restart
// regression pinned in tests/unit/cold-open-rebuild-gate.test.ts (a
// disk-native provider legitimately reporting 0 resident while durable
// on disk), and re-deriving it from a denominator the provider itself
// has no way to consult would reopen exactly that regression.
const vectorAssessment = assessProviderHealth(this.index)
const vectorLedger = await this.storage.getCanonicalCounts?.()
const vectorLedgerAll = vectorLedger?.vectors.all
const vectorLedgerConfident = vectorLedger !== undefined && !vectorLedger.suspect
const vectorHasCoverageProof = vectorLedgerConfident && (vectorLedgerAll as number) > 0
const vectorConfirmedEmpty = vectorLedgerConfident && vectorLedgerAll === 0
let vectorNeedsRebuild: boolean
if (vectorAssessment.via === 'is-ready') {
// Bare isReady() stays authoritative and UNMODIFIED — see above.
vectorNeedsRebuild = vectorAssessment.readiness === 'not-ready'
} else if (vectorAssessment.via === 'health-report') {
vectorNeedsRebuild =
vectorAssessment.readiness !== 'ready' ||
(hnswIndexSize === 0 && vectorHasCoverageProof)
} else {
// size-heuristic / no provider (the built-in JS engine's own posture).
vectorNeedsRebuild = hnswIndexSize === 0 && !vectorConfirmedEmpty
}
const shouldRebuildVector = !vectorMigrating && (epochStale || vectorNeedsRebuild)
// Narration (and the FAIL-TYPED backstop below) are scoped EXACTLY to
// the defect this gate closes: a provider whose OWN health report
// claims `serving: true` — an affirmative "I am ready" a caller would
// otherwise trust outright — while the canonical ledger proves vector
// coverage is missing. This is deliberately NARROWER than "any branch
// where the ledger contributed to the decision":
// - the bare isReady() branch is untouched, as above (never in scope);
// - the health-report branch's OWN `readiness !== 'ready'` case is
// already an ordinary, PRE-EXISTING rebuild trigger (the provider
// admits not-ready) — not a ledger override, so not a "gap";
// - the size-heuristic/no-provider branch's rebuild-when-empty is the
// SAME blunt trigger the code always had (`hnswIndexSize === 0`)
// — the ledger only ever SUPPRESSES a rebuild there (the confirmed-
// empty case), it never forces one the old heuristic wouldn't
// already have run. Marking that branch a "gap" too made the
// FAIL-TYPED backstop fire on ordinary white-box tests that stub
// rebuild() as a no-op and pin `size()` at 0 to drive OTHER
// assertions (e.g. migration-deference's isMigrating() coverage) —
// those are not silent-empty defects, so they must open exactly as
// before (tests/unit/brainy/migration-deference.test.ts).
const vectorCoverageGap =
!vectorMigrating &&
(epochStale || legNeedsRebuild(this.index, hnswIndexSize === 0))
vectorAssessment.via === 'health-report' &&
vectorAssessment.readiness === 'ready' &&
hnswIndexSize === 0 &&
vectorHasCoverageProof
if (vectorCoverageGap) {
prodLog.warn(
`[Brainy] open(): vector index reports ${hnswIndexSize} node(s) but the canonical ` +
`ledger holds ${vectorLedgerAll} vectored noun(s) — the derived vector index is ` +
`missing or unbuilt on this store. Forcing the vector rebuild rather than serving ` +
`silent-empty search results.`
)
}
const shouldRebuildGraph =
!graphMigrating &&
(epochStale || legNeedsRebuild(this.graphIndex, false))
@ -16955,9 +17054,27 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// provider running its own background migration is skipped here (it owns
// its index until it verifies-and-swaps).
const rebuildStartTime = Date.now()
// The vector leg's build door, by contract with the native provider: a
// provider exposing fillFromCanonical() gets THAT call — idempotent, the
// provider's own init runs it first so this is the backstop — never a
// full rebuild() for a coverage gap. A PARTIAL shortfall deliberately
// triggers nothing here: that is repair()'s operator door. The JS index
// has no fill door and keeps its rebuild.
const vectorBuild = (): Promise<unknown> => {
const fillDoor = (this.index as unknown as { fillFromCanonical?: () => Promise<unknown> })
.fillFromCanonical
if (vectorCoverageGap && typeof fillDoor === 'function') {
prodLog.warn(
`[Brainy] open(): vector coverage gap routes through the provider's ` +
`fillFromCanonical() (idempotent canonical fill), not a full rebuild.`
)
return fillDoor.call(this.index)
}
return this.index.rebuild()
}
await Promise.all([
shouldRebuildMetadata ? this.metadataIndex.rebuild() : Promise.resolve(),
shouldRebuildVector ? this.index.rebuild() : Promise.resolve(),
shouldRebuildVector ? vectorBuild() : Promise.resolve(),
shouldRebuildGraph ? this.graphIndex.rebuild() : Promise.resolve()
])
@ -16998,6 +17115,24 @@ export class Brainy<T = any> implements BrainyInterface<T> {
console.log(`[Brainy] Second rebuild result: ${secondAttempt} entries`)
}
// Vector coverage verification: the coverage-gap rebuild above (see
// `vectorCoverageGap`) MUST have actually restored the ledger's
// vectored nouns. A provider that STILL reports 0 nodes after its own
// rebuild() ran — no JS (or provider) fallback could build from what's
// on disk — cannot silently complete open(): search would then serve
// empty results with no signal, exactly the defect this gate closes.
// FAIL TYPED, pre-serve, rather than let a broken vector leg pass as a
// successful open.
if (vectorCoverageGap && this.index.size() === 0) {
throw new VectorIndexNotReadyError(
`open(): the canonical ledger holds ${vectorLedgerAll} vectored noun(s) but the vector ` +
`index still reports 0 node(s) after rebuild() — the derived vector index could not ` +
`be restored from canonical. Refusing to serve silent-empty search results; ` +
`investigate the vector provider/storage, or repairIndex({ rebuild: ['vector'] }) ` +
`after restoring the underlying data.`
)
}
// 8.0 ⇄ native-provider handshake (NON-DESTRUCTIVE): the derived indexes
// have now rebuilt and verified, so they match this build's epoch —
// re-stamp the marker LAST, only here. A crash anywhere above leaves the