soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: cold-open no longer re-derives durable indexes — complete the readiness contract for all three providers

Browse source 
A production deployment measured ~48 seconds on EVERY reopen of an
11k-entity brain. Root cause: brainy's rebuild gate decided from in-memory
size()/count, which read 0 for a durable-but-not-resident index, so it
re-read every entity file to rebuild from scratch. At GA we gave only the
GRAPH provider a readiness contract (init() eager cold-load + isReady()
honest signal) so it would never eat that spurious rebuild; the vector and
metadata providers never got it, and brainy never even eager-inited the
vector provider.

Complete the contract symmetrically:

- plugin.ts: VectorIndexProvider gains optional init()+isReady();
  MetadataIndexProvider gains isReady() — mirroring GraphIndexProvider.
  Additive and optional; a provider that exposes nothing keeps today's
  behavior.
- brainy.ts: eager-init every provider that exposes init() (after metadata
  init() so the id-mapper is hydrated first), then decide per leg in
  precedence order — migrating (skip) -> epoch drift (rebuild) -> isReady()
  -> a per-leg empty fallback. The old instant fast-path keyed off
  this.index.size()>0, a dishonest proxy that skipped the metadata/graph
  checks whenever the vector was warm and never fired on a real cold process
  anyway; removed.

The per-leg fallbacks differ because "empty" means different things: the JS
vector's rebuild() IS its load, so size()===0 correctly triggers it; the
id-mapper backs metadata, so totalEntries===0 (past the empty-store return)
is a real load failure; but entities do not imply edges, so a graph
size()===0 is a valid empty state, not a load failure.

- The JS graph now COLD-LOADS its durable LSM instead of re-deriving from a
  full canonical verb scan on every boot (baseStorage._initializeGraphIndex
  loads the persisted SSTables via a new GraphAdjacencyIndex.init(); it
  self-heals from canonical only when the durable state is genuinely missing).
  This removes an O(E)-per-open cost every filesystem consumer paid.
- LSMTree.loadManifest loads its SSTables BEFORE publishing the relationship
  count, and resets to an honest-empty state on load failure — a tree can no
  longer claim persisted relationships while holding none (the silent-empty
  cold-load class the query-time guards exist to prevent).

Verified end-to-end against a built brain: a warm reopen (with edges and
edgeless) reloads only the JS vector; the graph and metadata cold-load with
no rebuild, and queries return correct results. New tests in
cold-open-rebuild-gate.test.ts pin the contract (isReady() defers, self-heal
still fires); migration-deference updated to drive size-based deference
through the vector, the leg where empty->rebuild remains correct.

Pairs with the native provider's isReady()/init() implementation — brainy's
gate defers only to a signal the provider exposes.
This commit is contained in:
David Snelling 2026-07-07 10:39:00 -07:00
parent 4fde94bc2f
commit 61c247c923
8 changed files with 425 additions and 57 deletions
Download patch file Download diff file Expand all files Collapse all files

115
src/brainy.ts
View file

@ -961,14 +961,21 @@ export class Brainy<T = any> implements BrainyInterface<T> {
this.graphIndex = graphIndex
}
// Eager graph cold-load (cor contract). The native graph provider lazily
// defers loading its source→target adjacency, so isReady() would report
// false at the rebuild gate below and force a spurious rebuild (failing the
// §7.1 rebuild()==0 acceptance). Trigger the eager cold-load now — AFTER
// metadataIndex.init() above (the id-mapper is hydrated, so a native int
// adjacency resolves endpoints through it: the CTX-BR-RESTORE-REBUILD order)
// and BEFORE rebuildIndexesIfNeeded. Optional: the JS graph has no init()
// and self-loads its adjacency on demand.
// Eager cold-load (readiness contract). A provider that persists its
// derived state exposes init?(): trigger the load NOW — AFTER
// metadataIndex.init() above (the id-mapper is hydrated first, so a
// native int-keyed index resolves endpoints/slots through it — the
// CTX-BR-RESTORE-REBUILD order) and BEFORE the rebuild gate — so a
// durable index reports its real size()/isReady() at the gate instead
// of eating a spurious rebuild-from-canonical on every open (§7.1
// rebuild()==0). Vector first, then graph. JS engines: the JS vector
// index has no init() (rebuild() IS its load path); the JS graph's
// init() already cold-loaded its LSM inside storage.getGraphIndex()
// (idempotent here).
const vectorWithInit = this.index as { init?: () => Promise<void> }
if (typeof vectorWithInit.init === 'function') {
await vectorWithInit.init()
}
const graphWithInit = this.graphIndex as { init?: () => Promise<void> }
if (typeof graphWithInit.init === 'function') {
await graphWithInit.init()
@ -13398,20 +13405,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
return
}
// OPTIMIZATION: Instant check - if index already has data, skip immediately
// This gives 0s startup for warm restarts (vs 50-100ms of async checks).
// The epoch-drift guard suppresses this fast path: an index that loaded
// eagerly (size()>0) but whose on-disk format predates this build
// (`_indexEpochStale`) must still rebuild — that is the exact gap this
// handshake closes (rebuild on a format change, not only on size()===0).
if (this.index.size() > 0 && !force && !this._indexEpochStale) {
if (!this.config.silent) {
console.log(
`✅ Index already populated (${this.index.size().toLocaleString()} entities) - 0s startup!`
)
}
return
}
// No instant fast-path here: the honest per-leg readiness checks below
// are all O(1) (one bounded storage sample + each provider's size()/
// isReady()), and this method runs exactly once per open (init calls it;
// the lazy path passes force=true). The removed shortcut keyed off
// `this.index.size() > 0`, a dishonest proxy — it skipped the metadata
// and graph checks whenever the vector happened to be warm, and it never
// fired on a real cold process (the JS vector size is 0 until it loads).
// BUG #2 FIX: Don't trust counts - check actual storage instead
// Counts can be lost/corrupted in container restarts
@ -13437,17 +13437,23 @@ export class Brainy<T = any> implements BrainyInterface<T> {
// Check if indexes need rebuilding
const metadataStats = await this.metadataIndex.getStats()
const hnswIndexSize = this.index.size()
const graphIndexSize = await this.graphIndex.size()
// Graph rebuild fires when the adjacency is empty OR when the provider's
// honest readiness signal says the edges did not load on a cold open — the
// native int adjacency can report size()>0 (count/manifest loaded) yet have
// NO source→target edges, the silent-empty cold-load failure. Providers
// without isReady() keep the exact prior behavior (size()===0 only).
const graphIsReady = this.graphIndex as GraphAdjacencyIndex & { isReady?: () => boolean }
const needsGraphRebuild =
graphIndexSize === 0 ||
(typeof graphIsReady.isReady === 'function' && !graphIsReady.isReady())
// Readiness contract: when a provider exposes isReady(), that honest
// signal REPLACES the size/count heuristic below — an mmap/disk-native
// index legitimately reports 0 resident entries while fully durable on
// disk, and rebuilding it from canonical re-reads every entity file on
// every boot (the 48-seconds-per-restart class a production deployment
// hit). The signal is honest in BOTH directions: a provider whose
// durable state failed to load returns false and gets its rebuild even
// when size() > 0 (the silent-empty cold-load failure). Providers
// without isReady() keep the exact prior empty-heuristics.
const providerReady = (leg: unknown): boolean | undefined => {
const candidate = leg as { isReady?: () => boolean }
return typeof candidate.isReady === 'function' ? candidate.isReady() : undefined
}
const metadataReady = providerReady(this.metadataIndex)
const vectorReady = providerReady(this.index)
const graphReady = providerReady(this.graphIndex)
// Epoch-drift trigger: a format-version change makes EVERY derived index
// suspect even when each is non-empty, so it forces a rebuild of all
@ -13466,16 +13472,38 @@ export class Brainy<T = any> implements BrainyInterface<T> {
const graphMigrating = this.providerIsMigrating(this.graphIndex)
const anyMigrating = metadataMigrating || vectorMigrating || graphMigrating
// Per-leg decision, in precedence order: a migrating provider owns its
// index (skip) → epoch drift forces a rebuild → an exposed isReady()
// decides → otherwise a per-leg fallback. The fallbacks differ by leg
// because "empty" means different things:
// - METADATA: past the empty-store early-return, entities exist, so the
// id-mapper SHOULD have loaded entries — totalEntries===0 is a real
// load-failure signal, so rebuild (self-heal from canonical).
// - VECTOR: the JS vector has no passive cold-load — rebuild() IS its
// load path — so size()===0 correctly triggers the load.
// - GRAPH: entities do NOT imply edges, so size()===0 is a VALID empty
// state, not a load failure. The JS graph cold-loads (and self-heals
// against canonical) inside storage.getGraphIndex() BEFORE this gate,
// so it is already authoritative here; re-deriving would be spurious
// (a full O(E) verb scan on every open of an edgeless brain). It
// therefore rebuilds only on epoch drift or a native !isReady().
// (verifyGraphAdjacencyLive is the query-time backstop.)
const shouldRebuildMetadata =
(metadataStats.totalEntries === 0 || epochStale) && !metadataMigrating
const shouldRebuildVector = (hnswIndexSize === 0 || epochStale) && !vectorMigrating
const shouldRebuildGraph = (needsGraphRebuild || epochStale) && !graphMigrating
!metadataMigrating &&
(epochStale ||
(metadataReady !== undefined ? !metadataReady : metadataStats.totalEntries === 0))
const shouldRebuildVector =
!vectorMigrating &&
(epochStale || (vectorReady !== undefined ? !vectorReady : hnswIndexSize === 0))
const shouldRebuildGraph =
!graphMigrating &&
(epochStale || (graphReady !== undefined ? !graphReady : false))
const needsRebuild = shouldRebuildMetadata || shouldRebuildVector || shouldRebuildGraph
if (!needsRebuild && !force) {
// All indexes already populated (or owned by a background migration), no
// rebuild needed.
// All indexes report current — durably loaded (isReady/size), or owned
// by a background migration. No rebuild needed.
return
}
@ -13501,7 +13529,18 @@ export class Brainy<T = any> implements BrainyInterface<T> {
: '🔄 Auto-rebuild explicitly enabled'
if (!this.config.silent) {
console.log(`${rebuildReason} - rebuilding all indexes from persisted data...`)
// Name exactly which legs rebuild — "all indexes" was a lie whenever
// the durable legs were skipped (e.g. only the JS vector index loads
// here on a warm reopen), and it misread as a whole-brain rebuild in
// consumer boot logs.
const rebuildingLegs = [
shouldRebuildMetadata && 'metadata',
shouldRebuildVector && 'vector',
shouldRebuildGraph && 'graph'
]
.filter(Boolean)
.join(' + ')
console.log(`${rebuildReason} - loading/rebuilding ${rebuildingLegs || 'no'} index(es) from persisted data...`)
}
// Before the graph rebuild, hydrate the entity id-mapper from the persisted

19
src/graph/graphAdjacencyIndex.ts
View file

@ -195,6 +195,25 @@ export class GraphAdjacencyIndex implements GraphIndexProvider {
return next
}
/**
* Eager cold-load of the persisted adjacency (readiness contract).
*
* Loads the LSM manifests + SSTables so `size()` reports the durable edge
* count at the rebuild gate — a warm reopen must load the persisted index,
* never re-derive it from a full canonical verb scan (the every-boot O(E)
* cost this method exists to eliminate). Idempotent; the lazy read paths
* call the same `ensureInitialized()` on demand.
*
* NOTE: the JS index deliberately does NOT expose `isReady()`. That signal
* (see `GraphIndexProvider.isReady`) asserts "traversals are trustworthy",
* which this side cannot honestly promise without comparing against the
* canonical store — the query-time known-edge probe
* (`verifyGraphAdjacencyLive` Strategy 2) remains the JS trust check.
*/
async init(): Promise<void> {
await this.ensureInitialized()
}
/**
* Initialize the graph index (lazy initialization)
* Added defensive auto-rebuild check for verbIdSet consistency

19
src/graph/lsm/LSMTree.ts
View file

@ -547,13 +547,26 @@ export class LSMTree {
const data = metadata.data as PersistedManifestData
this.manifest.sstables = new Map(Object.entries(data.sstables || {}))
this.manifest.lastCompaction = data.lastCompaction || Date.now()
this.manifest.totalRelationships = data.totalRelationships || 0
// Load SSTables from storage
// Load SSTables from storage BEFORE publishing the persisted count.
// If the SSTable load throws, `size()` must keep reporting 0 — a tree
// that claims its persisted relationships while holding none serves
// silent-empty traversals as truth (the cold-load swallow class), and
// downstream self-heal keys off the honest 0.
await this.loadSSTables()
this.manifest.totalRelationships = data.totalRelationships || 0
}
} catch (error) {
prodLog.debug('LSMTree: No existing manifest found, starting fresh')
// Reset anything partially loaded — an honest empty tree triggers the
// rebuild/self-heal paths; a half-loaded one masks them. (An absent
// manifest on a fresh store also lands here: empty is correct.)
this.manifest.sstables = new Map()
this.manifest.totalRelationships = 0
this.sstablesByLevel.clear()
prodLog.debug(
`LSMTree(${this.config.storagePrefix}): no loadable manifest/SSTables — starting empty ` +
`(${error instanceof Error ? error.message : String(error)})`
)
}
}

41
src/plugin.ts
View file

@ -125,6 +125,20 @@ export interface MetadataIndexProvider {
flush(): Promise<void>
rebuild(): Promise<void>
/**
* @description OPTIONAL honest durability signal (readiness contract,
* mirrors `isReady?()` on the graph and vector providers). `true` ⇔ the
* persisted field postings are loaded (or cheaply demand-loadable) and
* consistent with what the provider last persisted — a rebuild from the
* canonical records would be redundant. When exposed, the rebuild gate
* defers to this signal INSTEAD of the `getStats().totalEntries === 0`
* heuristic (a durable provider may legitimately report 0 resident entries
* on a cold open while its postings sit loadable on disk). Absent → the
* gate keeps the count heuristic. Never return `true` when the durable
* state failed to load.
*/
isReady?(): boolean
/**
* @description OPTIONAL. A native provider returns true from the moment its
* `init()` detects a large epoch-drift until its background
@ -902,6 +916,33 @@ export interface VectorIndexProvider {
flush(): Promise<number>
getPersistMode(): 'immediate' | 'deferred'
/**
* @description OPTIONAL eager cold-load (readiness contract, mirrors
* {@link GraphIndexProvider.init}). Called once during brain init — AFTER the
* metadata provider's `init()` (the id-mapper is hydrated first, so a
* provider whose vector slots resolve through interned ints reads a complete
* mapping) and BEFORE the rebuild gate — so a durable provider loads (or
* verifies it can demand-load) its persisted index and reports
* `isReady() === true` at the gate instead of eating a spurious
* rebuild-from-canonical on every open. The built-in JS index omits it:
* `rebuild()` IS its load path.
*/
init?(): Promise<void>
/**
* @description OPTIONAL honest durability signal (readiness contract,
* mirrors {@link GraphIndexProvider.isReady}). `true` ⇔ the persisted
* derived index is loaded (or cheaply demand-loadable) and consistent with
* what the provider last persisted — a rebuild from the canonical records
* would be redundant work. When exposed, the rebuild gate defers to this
* signal INSTEAD of the `size() === 0` heuristic (an mmap/disk-native index
* may legitimately report 0 resident entries while fully durable). Absent →
* the gate keeps the size heuristic. Never return `true` when the durable
* state failed to load — that converts a recoverable rebuild into silent
* empty results.
*/
isReady?(): boolean
/**
* @description OPTIONAL. A native provider returns true from the moment its
* `init()` detects a large epoch-drift until its background

25
src/storage/baseStorage.ts
View file

@ -2429,11 +2429,26 @@ export abstract class BaseStorage extends BaseStorageAdapter {
// invalidateGraphIndex); on first init Brainy wires it right after.
this.graphIndex = new GraphAdjacencyIndex(this, {}, this.graphEntityIdResolver)
// Check if we need to rebuild from existing data
const sampleVerbs = await this.getVerbs({ pagination: { limit: 1 } })
if (sampleVerbs.items.length > 0) {
prodLog.info('Found existing verbs, rebuilding graph index...')
await this.graphIndex.rebuild()
// Load the PERSISTED adjacency first (LSM manifests + SSTables). A warm
// reopen must load the durable index it already built — the previous
// "any verb exists → rebuild()" check here re-derived the whole graph
// from a full canonical verb scan on EVERY boot, an O(E) cost that
// dominated real deployments' startup.
await this.graphIndex.init()
// Self-heal only when the durable state is genuinely missing: canonical
// records exist but the loaded index is empty (first open on pre-index
// data, a deleted/corrupt _graph dir, or the LSM load failing loud).
// One O(1) probe replaces the unconditional O(E) re-derive.
if (this.graphIndex.size() === 0) {
const sampleVerbs = await this.getVerbs({ pagination: { limit: 1 } })
if (sampleVerbs.items.length > 0) {
prodLog.warn(
'GraphAdjacencyIndex: canonical verbs exist but the persisted adjacency is empty — ' +
'rebuilding from storage (one-time self-heal).'
)
await this.graphIndex.rebuild()
}
}
return this.graphIndex