feat(vector): the vectored-noun scalar joins the count ledger; the open gate closes the vector leg
The coverage denominator the health-by-accounting ratification named for
the vector family — never built until now, and its absence was measured as
the exact outage class it existed to prevent: a migrated store with
canonical vectors and no derived index opened with the vector leg EMPTY,
served [] from vector search with no error, and the report-driven read gate
had nothing to refuse on (the provider's coverage invariant was honestly
unledgered — the denominator was ours to supply).
- getCanonicalCounts() gains vectors: { all } — the count of canonical
nouns holding a REAL vector. Incremented where a vector lands (the
isNew-gated metadata seam for explicit vectors — the same discipline that
keeps HNSW neighbor-link re-saves from inflating counts; a narrow
noteVectorLanded hook for the deferred-embed landing, gated on the
worker's own pre-embed read). Decremented on a proven delete of a
vectored noun; a vector-uncertain delete marks the ledger suspect rather
than guessing (no new reads on the delete path). Recounted by the
sanctioned recount; legacy counts.json derives it once (a deferred noun's
vector file exists with an empty vector, so presence requires one
content read at derivation — never on the hot path).
- The open gate's vector leg: when a health-reporting provider claims
serving while the index holds zero nodes and the ledger proves vectored
canonical rows exist, open BUILDS (narrated) — routed through the
provider's idempotent fillFromCanonical() when exposed (the joint door;
a partial shortfall stays repair()'s operator business), the JS rebuild
otherwise — or fails typed pre-serve. Scoped exactly: bare isReady()
providers, migrating providers, and white-box size stubs open as before.
Pinned end-to-end from the partner gate's probe shape (store with vectored
canonical rows, no derived index, reopen → search serves N, never []),
red-proved against the pre-fix path; the inverse (zero vectored rows) opens
without building and serves [] honestly.
This commit is contained in:
parent
bce2593e24
commit
9730835bdf
10 changed files with 851 additions and 32 deletions
145
src/brainy.ts
145
src/brainy.ts
|
|
@ -2382,6 +2382,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
[{ type: 'embed.landed', id, vector: newVector }],
|
||||
'system:embed-landing'
|
||||
)
|
||||
// Vectored-noun ledger: the landing commit above carries a vector
|
||||
// write with NO accompanying metadata operation, so the
|
||||
// saveNounMetadata(..., hasVector) seam never fires for it — the
|
||||
// narrow storage hook is the only seam left. `oldVector.length===0`
|
||||
// (already known for free from the pre-embed read above) proves this
|
||||
// is a GENUINE first landing, not a re-embed of an already-vectored
|
||||
// row (e.g. a deferred update() on a row that already had a real
|
||||
// vector) — the latter must never double-count.
|
||||
if (oldVector.length === 0) {
|
||||
await this.storage.noteVectorLanded?.(id)
|
||||
}
|
||||
this.clearPendingEmbed(id)
|
||||
} catch (err) {
|
||||
prodLog.warn(
|
||||
|
|
@ -3000,8 +3011,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
const runInsert: TransactionFunction<void> = async (tx) => {
|
||||
// Operation 1: Save metadata FIRST (TypeAwareStorage caching)
|
||||
// isNew=true: skip pre-read for rollback (entity doesn't exist yet)
|
||||
// hasVector: the vectored-noun ledger counts this insert iff its
|
||||
// vector is real/non-empty (never true for a deferred embed, whose
|
||||
// stub `vector` is `[]` — it counts later, at landing).
|
||||
tx.addOperation(
|
||||
new SaveNounMetadataOperation(this.storage, id, storageMetadata, true)
|
||||
new SaveNounMetadataOperation(this.storage, id, storageMetadata, true, vector.length > 0)
|
||||
)
|
||||
|
||||
// Operation 2: Save vector data
|
||||
|
|
@ -10568,7 +10582,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
plan.postCommit.push(() => this.kickEmbedWorker())
|
||||
}
|
||||
plan.operations.push(
|
||||
new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew),
|
||||
// hasVector: see the single-add() insert path's comment — never true
|
||||
// for a deferred embed (stub vector `[]`; counted later at landing).
|
||||
new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew, vector.length > 0),
|
||||
new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew),
|
||||
...(deferringEmbed
|
||||
? []
|
||||
|
|
@ -16900,9 +16916,92 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
const shouldRebuildMetadata =
|
||||
!metadataMigrating &&
|
||||
(epochStale || legNeedsRebuild(this.metadataIndex, metadataStats.totalEntries === 0))
|
||||
const shouldRebuildVector =
|
||||
|
||||
// VECTOR LEG — the two-engine gate's last red: a migrated 7.x-era store
|
||||
// can hold canonical vectored nouns with NO derived vector index built.
|
||||
// `legNeedsRebuild`'s size-heuristic fallback (below) only fires off
|
||||
// `hnswIndexSize === 0`, and its health-report branch trusts a
|
||||
// provider's own `serving` verdict verbatim — but a provider's health
|
||||
// report can legitimately say `serving: true` while vector coverage is
|
||||
// honestly UNLEDGERED on ITS side too (an unledgered invariant never
|
||||
// flips serving), so neither signal alone can tell "genuinely empty"
|
||||
// apart from "never built". The canonical vectored-noun ledger
|
||||
// (`getCanonicalCounts().vectors.all` — Deliverable 1) is the
|
||||
// denominator that CAN tell them apart, and is compared here:
|
||||
// - a CONFIDENT (non-suspect) ledger `> 0` while the reported node
|
||||
// count is 0 is a proven coverage gap — force the build regardless
|
||||
// of what a health report claims;
|
||||
// - a CONFIDENT ledger `=== 0` while the node count is 0 proves there
|
||||
// is nothing to load (e.g. every noun's embed is still deferred) —
|
||||
// skip the size-heuristic fallback's blunt "always rebuild when
|
||||
// empty" trigger, which otherwise wastes a full canonical walk for
|
||||
// zero benefit on every cold open of such a store;
|
||||
// - an unavailable/suspect ledger changes nothing — loud errors never
|
||||
// quiet losses, so a doubtful ledger must never suppress a rebuild
|
||||
// the old heuristic would have run.
|
||||
// The bare `isReady()` boolean (no report, no `unledgered` concept) is
|
||||
// NOT overridden — that signal is what fixed the 48-seconds-per-restart
|
||||
// regression pinned in tests/unit/cold-open-rebuild-gate.test.ts (a
|
||||
// disk-native provider legitimately reporting 0 resident while durable
|
||||
// on disk), and re-deriving it from a denominator the provider itself
|
||||
// has no way to consult would reopen exactly that regression.
|
||||
const vectorAssessment = assessProviderHealth(this.index)
|
||||
const vectorLedger = await this.storage.getCanonicalCounts?.()
|
||||
const vectorLedgerAll = vectorLedger?.vectors.all
|
||||
const vectorLedgerConfident = vectorLedger !== undefined && !vectorLedger.suspect
|
||||
const vectorHasCoverageProof = vectorLedgerConfident && (vectorLedgerAll as number) > 0
|
||||
const vectorConfirmedEmpty = vectorLedgerConfident && vectorLedgerAll === 0
|
||||
|
||||
let vectorNeedsRebuild: boolean
|
||||
if (vectorAssessment.via === 'is-ready') {
|
||||
// Bare isReady() stays authoritative and UNMODIFIED — see above.
|
||||
vectorNeedsRebuild = vectorAssessment.readiness === 'not-ready'
|
||||
} else if (vectorAssessment.via === 'health-report') {
|
||||
vectorNeedsRebuild =
|
||||
vectorAssessment.readiness !== 'ready' ||
|
||||
(hnswIndexSize === 0 && vectorHasCoverageProof)
|
||||
} else {
|
||||
// size-heuristic / no provider (the built-in JS engine's own posture).
|
||||
vectorNeedsRebuild = hnswIndexSize === 0 && !vectorConfirmedEmpty
|
||||
}
|
||||
|
||||
const shouldRebuildVector = !vectorMigrating && (epochStale || vectorNeedsRebuild)
|
||||
|
||||
// Narration (and the FAIL-TYPED backstop below) are scoped EXACTLY to
|
||||
// the defect this gate closes: a provider whose OWN health report
|
||||
// claims `serving: true` — an affirmative "I am ready" a caller would
|
||||
// otherwise trust outright — while the canonical ledger proves vector
|
||||
// coverage is missing. This is deliberately NARROWER than "any branch
|
||||
// where the ledger contributed to the decision":
|
||||
// - the bare isReady() branch is untouched, as above (never in scope);
|
||||
// - the health-report branch's OWN `readiness !== 'ready'` case is
|
||||
// already an ordinary, PRE-EXISTING rebuild trigger (the provider
|
||||
// admits not-ready) — not a ledger override, so not a "gap";
|
||||
// - the size-heuristic/no-provider branch's rebuild-when-empty is the
|
||||
// SAME blunt trigger the code always had (`hnswIndexSize === 0`)
|
||||
// — the ledger only ever SUPPRESSES a rebuild there (the confirmed-
|
||||
// empty case), it never forces one the old heuristic wouldn't
|
||||
// already have run. Marking that branch a "gap" too made the
|
||||
// FAIL-TYPED backstop fire on ordinary white-box tests that stub
|
||||
// rebuild() as a no-op and pin `size()` at 0 to drive OTHER
|
||||
// assertions (e.g. migration-deference's isMigrating() coverage) —
|
||||
// those are not silent-empty defects, so they must open exactly as
|
||||
// before (tests/unit/brainy/migration-deference.test.ts).
|
||||
const vectorCoverageGap =
|
||||
!vectorMigrating &&
|
||||
(epochStale || legNeedsRebuild(this.index, hnswIndexSize === 0))
|
||||
vectorAssessment.via === 'health-report' &&
|
||||
vectorAssessment.readiness === 'ready' &&
|
||||
hnswIndexSize === 0 &&
|
||||
vectorHasCoverageProof
|
||||
if (vectorCoverageGap) {
|
||||
prodLog.warn(
|
||||
`[Brainy] open(): vector index reports ${hnswIndexSize} node(s) but the canonical ` +
|
||||
`ledger holds ${vectorLedgerAll} vectored noun(s) — the derived vector index is ` +
|
||||
`missing or unbuilt on this store. Forcing the vector rebuild rather than serving ` +
|
||||
`silent-empty search results.`
|
||||
)
|
||||
}
|
||||
|
||||
const shouldRebuildGraph =
|
||||
!graphMigrating &&
|
||||
(epochStale || legNeedsRebuild(this.graphIndex, false))
|
||||
|
|
@ -16955,9 +17054,27 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// provider running its own background migration is skipped here (it owns
|
||||
// its index until it verifies-and-swaps).
|
||||
const rebuildStartTime = Date.now()
|
||||
// The vector leg's build door, by contract with the native provider: a
|
||||
// provider exposing fillFromCanonical() gets THAT call — idempotent, the
|
||||
// provider's own init runs it first so this is the backstop — never a
|
||||
// full rebuild() for a coverage gap. A PARTIAL shortfall deliberately
|
||||
// triggers nothing here: that is repair()'s operator door. The JS index
|
||||
// has no fill door and keeps its rebuild.
|
||||
const vectorBuild = (): Promise<unknown> => {
|
||||
const fillDoor = (this.index as unknown as { fillFromCanonical?: () => Promise<unknown> })
|
||||
.fillFromCanonical
|
||||
if (vectorCoverageGap && typeof fillDoor === 'function') {
|
||||
prodLog.warn(
|
||||
`[Brainy] open(): vector coverage gap routes through the provider's ` +
|
||||
`fillFromCanonical() (idempotent canonical fill), not a full rebuild.`
|
||||
)
|
||||
return fillDoor.call(this.index)
|
||||
}
|
||||
return this.index.rebuild()
|
||||
}
|
||||
await Promise.all([
|
||||
shouldRebuildMetadata ? this.metadataIndex.rebuild() : Promise.resolve(),
|
||||
shouldRebuildVector ? this.index.rebuild() : Promise.resolve(),
|
||||
shouldRebuildVector ? vectorBuild() : Promise.resolve(),
|
||||
shouldRebuildGraph ? this.graphIndex.rebuild() : Promise.resolve()
|
||||
])
|
||||
|
||||
|
|
@ -16998,6 +17115,24 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
console.log(`[Brainy] Second rebuild result: ${secondAttempt} entries`)
|
||||
}
|
||||
|
||||
// Vector coverage verification: the coverage-gap rebuild above (see
|
||||
// `vectorCoverageGap`) MUST have actually restored the ledger's
|
||||
// vectored nouns. A provider that STILL reports 0 nodes after its own
|
||||
// rebuild() ran — no JS (or provider) fallback could build from what's
|
||||
// on disk — cannot silently complete open(): search would then serve
|
||||
// empty results with no signal, exactly the defect this gate closes.
|
||||
// FAIL TYPED, pre-serve, rather than let a broken vector leg pass as a
|
||||
// successful open.
|
||||
if (vectorCoverageGap && this.index.size() === 0) {
|
||||
throw new VectorIndexNotReadyError(
|
||||
`open(): the canonical ledger holds ${vectorLedgerAll} vectored noun(s) but the vector ` +
|
||||
`index still reports 0 node(s) after rebuild() — the derived vector index could not ` +
|
||||
`be restored from canonical. Refusing to serve silent-empty search results; ` +
|
||||
`investigate the vector provider/storage, or repairIndex({ rebuild: ['vector'] }) ` +
|
||||
`after restoring the underlying data.`
|
||||
)
|
||||
}
|
||||
|
||||
// 8.0 ⇄ native-provider handshake (NON-DESTRUCTIVE): the derived indexes
|
||||
// have now rebuilt and verified, so they match this build's epoch —
|
||||
// re-stamp the marker LAST, only here. A crash anywhere above leaves the
|
||||
|
|
|
|||
Reference in a new issue