fix: cold-open no longer re-derives durable indexes — complete the readiness contract for all three providers

A production deployment measured ~48 seconds on EVERY reopen of an
11k-entity brain. Root cause: brainy's rebuild gate decided from in-memory
size()/count, which read 0 for a durable-but-not-resident index, so it
re-read every entity file to rebuild from scratch. At GA we gave only the
GRAPH provider a readiness contract (init() eager cold-load + isReady()
honest signal) so it would never eat that spurious rebuild; the vector and
metadata providers never got it, and brainy never even eager-inited the
vector provider.

Complete the contract symmetrically:

- plugin.ts: VectorIndexProvider gains optional init()+isReady();
  MetadataIndexProvider gains isReady() — mirroring GraphIndexProvider.
  Additive and optional; a provider that exposes nothing keeps today's
  behavior.
- brainy.ts: eager-init every provider that exposes init() (after metadata
  init() so the id-mapper is hydrated first), then decide per leg in
  precedence order — migrating (skip) -> epoch drift (rebuild) -> isReady()
  -> a per-leg empty fallback. The old instant fast-path keyed off
  this.index.size()>0, a dishonest proxy that skipped the metadata/graph
  checks whenever the vector was warm and never fired on a real cold process
  anyway; removed.

The per-leg fallbacks differ because "empty" means different things: the JS
vector's rebuild() IS its load, so size()===0 correctly triggers it; the
id-mapper backs metadata, so totalEntries===0 (past the empty-store return)
is a real load failure; but entities do not imply edges, so a graph
size()===0 is a valid empty state, not a load failure.

- The JS graph now COLD-LOADS its durable LSM instead of re-deriving from a
  full canonical verb scan on every boot (baseStorage._initializeGraphIndex
  loads the persisted SSTables via a new GraphAdjacencyIndex.init(); it
  self-heals from canonical only when the durable state is genuinely missing).
  This removes an O(E)-per-open cost every filesystem consumer paid.
- LSMTree.loadManifest loads its SSTables BEFORE publishing the relationship
  count, and resets to an honest-empty state on load failure — a tree can no
  longer claim persisted relationships while holding none (the silent-empty
  cold-load class the query-time guards exist to prevent).

Verified end-to-end against a built brain: a warm reopen (with edges and
edgeless) reloads only the JS vector; the graph and metadata cold-load with
no rebuild, and queries return correct results. New tests in
cold-open-rebuild-gate.test.ts pin the contract (isReady() defers, self-heal
still fires); migration-deference updated to drive size-based deference
through the vector, the leg where empty->rebuild remains correct.

Pairs with the native provider's isReady()/init() implementation — brainy's
gate defers only to a signal the provider exposes.
This commit is contained in:
David Snelling 2026-07-07 10:39:00 -07:00
parent 4fde94bc2f
commit 61c247c923
8 changed files with 425 additions and 57 deletions

View file

@ -148,7 +148,7 @@ describe('rc.8 no-freeze migration deference (isMigrating / stampBrainFormat / b
expect(internals._indexEpochStale).toBe(true)
})
it('a migrating provider is skipped even when its size()===0; a non-migrating size()===0 sibling still rebuilds', async () => {
it('a migrating provider is skipped even though its empty-signal would trigger a rebuild; clearing the flag lets the empty leg rebuild', async () => {
const brain = await makeWarmBrain()
const internals = internalsOf(brain)
@ -156,22 +156,29 @@ describe('rc.8 no-freeze migration deference (isMigrating / stampBrainFormat / b
const idxSpy = vi.spyOn(internals.index, 'rebuild').mockResolvedValue(undefined)
const giSpy = vi.spyOn(internals.graphIndex, 'rebuild').mockResolvedValue(undefined)
// No epoch drift this time: the rebuild trigger is purely "index is empty".
// No epoch drift: the only rebuild trigger is a leg's own empty signal. The
// JS vector's rebuild() IS its load path, so size()===0 is its trigger —
// the leg where "empty → rebuild" is architecturally correct — so we drive
// deference through it. (The JS graph cold-loads before this gate, so its
// size()===0 is a valid empty state, not a rebuild trigger; graph deference
// is covered by the epoch-drift case above.)
internals._indexEpochStale = false
// Vector index reports empty AND is migrating → its background swap owns it.
vi.spyOn(internals.index, 'size').mockReturnValue(0)
// Migrating: the provider's background swap owns the index, so the
// size()===0 load trigger is suppressed.
setMigrating(internals.index, true)
// Graph index reports empty and is NOT migrating → brainy must rebuild it.
vi.spyOn(internals.graphIndex, 'size').mockReturnValue(0)
await internals.rebuildIndexesIfNeeded()
// size()===0 would normally force the vector rebuild — deference suppresses it.
expect(idxSpy).toHaveBeenCalledTimes(0)
// The empty, non-migrating graph sibling still rebuilds.
expect(giSpy).toHaveBeenCalledTimes(1)
// Metadata has entries and no drift → no rebuild needed.
// Metadata has entries; the graph has no edges; neither drifted → no rebuild.
expect(miSpy).toHaveBeenCalledTimes(0)
expect(giSpy).toHaveBeenCalledTimes(0)
// Clearing the flag: the same empty, non-migrating vector now rebuilds — and
// this second call re-evaluating at all confirms the gate is not latched.
setMigrating(internals.index, false)
await internals.rebuildIndexesIfNeeded()
expect(idxSpy).toHaveBeenCalledTimes(1)
})
// --- Hook 1: large-path first-query lazy force-rebuild deference ----------