open-brainy/docs/concepts/index-health.md

title	slug	public	category	template	order	description	next
Index Health	concepts/index-health	true	concepts	concept	8	How Brainy knows whether a derived index can be trusted — exact accounting instead of sampling, the named health report, degraded-but-serving vs. not-ready, and what repairIndex() checks, heals, and rebuilds.	concepts/generation-fact-log guides/inspection

Index Health

Brainy keeps one canonical copy of every entity and relationship, and three derived indexes built from it — vector, metadata, and graph — so find() can answer semantically, by filter, and by traversal without re-deriving the answer from scratch on every query. A derived index is a cache with a serving structure: it can be present but stale, present but only partially loaded, or fully out of sync with canonical after a crash. This page is about how Brainy decides whether to trust one, what it does when it can't, and how you reconcile the two.

Exact accounting instead of sampling

Older health checks worked by inference: does size() return something greater than zero, does a spot-check on one known item come back correct. Both are proxies. A cold index can report a nonzero count while its actual serving structure never loaded, and a spot-check only proves the one item it happened to ask about.

Every derived-index provider may now expose a named, synchronous, O(1) healthReport() — composed from the provider's own exact ledgers (real counters it already maintains on the write path), never a sample or a walk. This is the one signal Brainy's read gate consults. A provider that doesn't yet expose one falls back to an honest isReady() boolean, and finally to a size heuristic for engines with neither — but wherever a healthReport() exists, it wins.

Underneath, storage itself keeps an analogous canonical count ledger: a counted scalar (the user-facing total — what getNounCount() / getVerbCount() return) and an all scalar (every tier, including internal records a derived index's own coverage math needs to compare against). This is the real denominator a provider's healthReport() measures itself by, rather than a total that can only ever ratchet upward. See What suspect counts mean below for the one case that ledger can't stay exact through on its own.

The named report

A HealthReport carries, per provider ('vector' / 'graph' / 'metadata'):

• healthy — true iff every verified invariant holds. An invariant whose family has no ledger yet is unledgered, never counted either way — unknown, not passing.
• serving — can this provider answer a query right now. A failing invariant graded heal: 'repair' or heal: 'none' still leaves serving: true — this is degraded-but-serving: something is off (say, a stale rollup on an employee record's relationship count) but reads keep working. Only a failure graded heal: 'rebuild' flips serving to false — not-ready — because the provider itself is telling you its serving structure cannot answer correctly.
• invariants — each checked condition, with its provenance (source: 'ledger' — an exact count; 'deep' — a full scan, diagnostic-only; 'unledgered' — not yet tracked) and, for a failing one, an exact missing count plus a capped sample of the affected ids — a verdict, never a dump.
• generation — bumps on every ledger mutation and rebuild, so a caller can cache a verdict per generation instead of re-deriving it.

The distinction that matters day to day: healthy: false can be entirely benign — a maintenance window, a divergence repairIndex() will clean up on its own schedule. serving: false is not benign. It means this provider is refusing to answer, on its own word, right now.

How a failure gets its grade — the serving law. A provider grades heal by one question only: could an answer be wrong? — never how expensive is the fix? A missing-postings shortfall, however large, is heal: 'repair' (re-post exactly what the ledger names, reads serving throughout); it can never withhold serving just because healing it takes work. serving is withheld only by a small, named set of rebuild-graded conditions — the index not initialized, its durable state absent, a manifest naming files that are not resident, a replay that did not complete cleanly — the states in which an answer could genuinely be wrong. And a read is only ever refused by the family it actually consults: a metadata filter is answered by the metadata index alone, vector search by the vector index, traversal by the graph index — one family's refusal never blocks another family's reads.

Reads refuse — they never rebuild

A query that reaches a not-serving provider does not trigger a rebuild from inside the read. Brainy retired that path deliberately: a rebuild kicked off by an ordinary find({ where: { status: 'active' } }) call is a dark, unpredictable cost hiding behind a request that looks like a cheap read. Instead, the read throws a typed, catchable error naming the reason:

Error	Thrown when	Meaning
GraphIndexNotReadyError	find({ connected }), neighbors(), related()	The graph adjacency index isn't serving — traversal would otherwise return [] indistinguishable from "no relationships"
MetadataIndexNotReadyError	find({ where })	The metadata/field index isn't serving — a filtered read would otherwise return [] indistinguishable from "no matches"
VectorIndexNotReadyError	find({ query }), similar()	The vector index isn't serving — a semantic search would otherwise return [] indistinguishable from "nothing similar"

All three are exported from @soulcraftlabs/brainy. Catch them where your application needs to distinguish "this index isn't ready yet" from "there's genuinely nothing here" — a health dashboard, a retry policy, an operator alert. The fix is always the same: reconcile the index, either by reopening the brain (which brings every provider to serving before init() returns — see the next section) or by calling repairIndex() explicitly.

try {
  const active = await brain.find({ where: { status: 'active' } })
} catch (err) {
  if (err instanceof MetadataIndexNotReadyError) {
    // not a "no results" — the index itself refused; alert or retry after repair
  } else {
    throw err
  }
}

Rebuilds happen at open, not on first query

brain.init() runs every needed rebuild to completion before it returns, unconditionally, regardless of dataset size. There is no lazy, first-query rebuild path anymore — a brain either finishes opening healthy, or it fails open loudly. disableAutoRebuild: true no longer defers index construction to the first query: it has no effect on when a needed rebuild runs. Full manual control over rebuilds is repairIndex({ rebuild: [...] }) (below), not this flag.

repairIndex() — checking and healing

Bare repairIndex() is report-driven: it only heals what its own checks say actually needs it, and it always returns a full per-family receipt.

const report = await brain.repairIndex()
report.healedTotal   // total items healed across every family
report.durationMs
report.families       // one row per family checked

Each RepairFamilyReport row names what happened:

• checked — was this family actually examined (false means skipped — see skipped for why).
• healed — items re-posted or corrected in place.
• missing — when the check can name what diverged: an exact count plus a capped sample of ids.
• rebuilt — a full generational rebuild ran (as opposed to an incremental heal).
• detail / reason / skipped — the receipt's narration; a row is always either checked or explains why it wasn't. Nothing is silent.

On every call, bare repairIndex():

1. Prunes orphaned canonical containers left by a partial delete.
2. Recomputes the count rollups from one canonical walk (unconditional — this is also what clears a suspect ledger; see below).
3. Reconciles VFS containment edges, if the VFS is initialized.
4. Runs the metadata index's own corruption detection pass.
5. Consults each of the three derived-index providers' own health check and rebuilds only a family whose failing invariant actually asks for it (heal: 'rebuild') — never a provider that reports healthy or a lesser grade.

The explicit rebuild door

options.rebuild skips the health check and rebuilds one or more families unconditionally — the operator override for when you have independent reason to distrust a family regardless of what it self-reports (a suspicious deploy, a storage-layer incident, a support ticket that doesn't match what the health report says):

// Force the graph adjacency to rebuild from canonical, no invariant consulted
await brain.repairIndex({ rebuild: ['graph'] })

// Force all three derived indexes
await brain.repairIndex({ rebuild: 'all' })

A family named this way is recorded with rebuilt: true and reason: 'explicit rebuild requested', and is skipped by the normal health-driven pass in the same call — it was already rebuilt unconditionally.

Reach for the explicit door when you need certainty regardless of self-report; reach for bare repairIndex() for routine maintenance and after any incident where you're not sure which family (if any) needs it.

What suspect counts mean

Storage's canonical count ledger increments the ALL-visibility total on every new record and decrements it on every proven delete — one where the record was read, or the caller supplied its prior image. A delete that cannot prove what it removed existed doesn't guess: it flags the ledger suspect (an operator-visible console.warn, narrated once per session, not once per delete) rather than risk decrementing a total that was never incremented for that record in the first place. This is intentionally rare — it's a defensive fallback for callers on an unusual removal path, not a per-delete cost.

suspect is not directly exposed on any Brainy method today — it lives on the StorageAdapter's optional getCanonicalCounts(), primarily consulted by repairIndex()'s recount step and by custom storage adapters composing their own healthReport(). What matters for an application: a suspect ledger is not incorrect, just unverified since the last recount — and repairIndex()'s unconditional count-rollup step (step 2, above) recomputes the ALL scalars from a real canonical walk on every call, clearing the flag with proof either way.

Practical guidance

• On a normal restart, do nothing — init() brings every provider to serving before it returns, or fails loudly.
• On a *NotReadyError from a live read, reconcile with repairIndex() (report-driven is almost always sufficient) and retry.
• After an incident where you distrust a specific family regardless of what it reports healthy — a storage-layer fault, a suspicious restore — use the explicit door: repairIndex({ rebuild: ['metadata' | 'graph' | 'vector'] }).
• To audit before trusting a report, brain.auditGraph() walks every stored relationship and proves (or disproves) that reads return canonical truth, independent of what any provider self-reports — see Inspecting a Live Brainy.