docs(release): the 10.4.0 entry, the index-health concept doc, and the API surfaces — written from the tree, not the plan

docs/concepts/index-health.md

@@ -0,0 +1,204 @@
+---
+title: Index Health
+slug: concepts/index-health
+public: true
+category: concepts
+template: concept
+order: 8
+description: How Brainy knows whether a derived index can be trusted — exact accounting instead of sampling, the named health report, degraded-but-serving vs. not-ready, and what repairIndex() checks, heals, and rebuilds.
+next:
+  - concepts/generation-fact-log
+  - guides/inspection
+---
+
+# Index Health
+
+Brainy keeps one **canonical** copy of every entity and relationship, and three
+**derived** indexes built from it — vector, metadata, and graph — so `find()` can
+answer semantically, by filter, and by traversal without re-deriving the answer from
+scratch on every query. A derived index is a cache with a serving structure: it can
+be present but stale, present but only partially loaded, or fully out of sync with
+canonical after a crash. This page is about how Brainy decides whether to trust one,
+what it does when it can't, and how you reconcile the two.
+
+## Exact accounting instead of sampling
+
+Older health checks worked by inference: does `size()` return something greater
+than zero, does a spot-check on one known item come back correct. Both are proxies.
+A cold index can report a nonzero count while its actual serving structure never
+loaded, and a spot-check only proves the one item it happened to ask about.
+
+Every derived-index provider may now expose a named, synchronous, O(1)
+`healthReport()` — composed from the provider's own **exact ledgers** (real counters
+it already maintains on the write path), never a sample or a walk. This is the one
+signal Brainy's read gate consults. A provider that doesn't yet expose one falls
+back to an honest `isReady()` boolean, and finally to a size heuristic for engines
+with neither — but wherever a `healthReport()` exists, it wins.
+
+Underneath, storage itself keeps an analogous **canonical count ledger**: a
+`counted` scalar (the user-facing total — what `getNounCount()` / `getVerbCount()`
+return) and an `all` scalar (every tier, including internal records a derived
+index's own coverage math needs to compare against). This is the real denominator
+a provider's `healthReport()` measures itself by, rather than a total that can only
+ever ratchet upward. See [What `suspect` counts mean](#what-suspect-counts-mean)
+below for the one case that ledger can't stay exact through on its own.
+
+## The named report
+
+A `HealthReport` carries, per provider (`'vector'` / `'graph'` / `'metadata'`):
+
+- **`healthy`** — `true` iff every *verified* invariant holds. An invariant whose
+  family has no ledger yet is `unledgered`, never counted either way — unknown,
+  not passing.
+- **`serving`** — can this provider answer a query right now. A failing invariant
+  graded `heal: 'repair'` or `heal: 'none'` still leaves `serving: true` — this is
+  **degraded-but-serving**: something is off (say, a stale rollup on an
+  `employee` record's relationship count) but reads keep working. Only a failure
+  graded `heal: 'rebuild'` flips `serving` to `false` — **not-ready** — because the
+  provider itself is telling you its serving structure cannot answer correctly.
+- **`invariants`** — each checked condition, with its provenance
+  (`source: 'ledger'` — an exact count; `'deep'` — a full scan, diagnostic-only;
+  `'unledgered'` — not yet tracked) and, for a failing one, an exact `missing`
+  count plus a capped sample of the affected ids — a verdict, never a dump.
+- **`generation`** — bumps on every ledger mutation and rebuild, so a caller can
+  cache a verdict per generation instead of re-deriving it.
+
+The distinction that matters day to day: `healthy: false` can be entirely benign —
+a maintenance window, a divergence `repairIndex()` will clean up on its own
+schedule. `serving: false` is not benign. It means this provider is refusing to
+answer, on its own word, right now.
+
+## Reads refuse — they never rebuild
+
+A query that reaches a not-serving provider does not trigger a rebuild from inside
+the read. Brainy retired that path deliberately: a rebuild kicked off by an ordinary
+`find({ where: { status: 'active' } })` call is a dark, unpredictable cost hiding
+behind a request that looks like a cheap read. Instead, the read throws a typed,
+catchable error naming the reason:
+
+| Error | Thrown when | Meaning |
+|---|---|---|
+| `GraphIndexNotReadyError` | `find({ connected })`, `neighbors()`, `related()` | The graph adjacency index isn't serving — traversal would otherwise return `[]` indistinguishable from "no relationships" |
+| `MetadataIndexNotReadyError` | `find({ where })` | The metadata/field index isn't serving — a filtered read would otherwise return `[]` indistinguishable from "no matches" |
+| `VectorIndexNotReadyError` | `find({ query })`, `similar()` | The vector index isn't serving — a semantic search would otherwise return `[]` indistinguishable from "nothing similar" |
+
+All three are exported from `@soulcraft/brainy`. Catch them where your application
+needs to distinguish "this index isn't ready yet" from "there's genuinely nothing
+here" — a health dashboard, a retry policy, an operator alert. The fix is always
+the same: reconcile the index, either by reopening the brain (which brings every
+provider to serving before `init()` returns — see the next section) or by calling
+`repairIndex()` explicitly.
+
+```typescript
+try {
+  const active = await brain.find({ where: { status: 'active' } })
+} catch (err) {
+  if (err instanceof MetadataIndexNotReadyError) {
+    // not a "no results" — the index itself refused; alert or retry after repair
+  } else {
+    throw err
+  }
+}
+```
+
+### Rebuilds happen at open, not on first query
+
+`brain.init()` runs every needed rebuild to completion **before it returns**,
+unconditionally, regardless of dataset size. There is no lazy, first-query
+rebuild path anymore — a brain either finishes opening healthy, or it fails
+open loudly. `disableAutoRebuild: true` no longer defers index construction to
+the first query: it has no effect on *when* a needed rebuild runs. Full manual
+control over rebuilds is `repairIndex({ rebuild: [...] })` (below), not this flag.
+
+## `repairIndex()` — checking and healing
+
+Bare `repairIndex()` is **report-driven**: it only heals what its own checks say
+actually needs it, and it always returns a full per-family receipt.
+
+```typescript
+const report = await brain.repairIndex()
+report.healedTotal   // total items healed across every family
+report.durationMs
+report.families       // one row per family checked
+```
+
+Each `RepairFamilyReport` row names what happened:
+
+- **`checked`** — was this family actually examined (`false` means skipped —
+  see `skipped` for why).
+- **`healed`** — items re-posted or corrected in place.
+- **`missing`** — when the check can name what diverged: an exact `count` plus a
+  capped `sample` of ids.
+- **`rebuilt`** — a full generational rebuild ran (as opposed to an incremental
+  heal).
+- **`detail`** / **`reason`** / **`skipped`** — the receipt's narration; a row is
+  always either checked or explains why it wasn't. Nothing is silent.
+
+On every call, bare `repairIndex()`:
+
+1. Prunes orphaned canonical containers left by a partial delete.
+2. Recomputes the count rollups from one canonical walk (unconditional — this is
+   also what clears a `suspect` ledger; see below).
+3. Reconciles VFS containment edges, if the VFS is initialized.
+4. Runs the metadata index's own corruption detection pass.
+5. Consults each of the three derived-index providers' own health check and
+   rebuilds only a family whose failing invariant actually asks for it
+   (`heal: 'rebuild'`) — never a provider that reports `healthy` or a lesser
+   grade.
+
+### The explicit rebuild door
+
+`options.rebuild` skips the health check and rebuilds one or more families
+**unconditionally** — the operator override for when you have independent reason
+to distrust a family regardless of what it self-reports (a suspicious deploy, a
+storage-layer incident, a support ticket that doesn't match what the health report
+says):
+
+```typescript
+// Force the graph adjacency to rebuild from canonical, no invariant consulted
+await brain.repairIndex({ rebuild: ['graph'] })
+
+// Force all three derived indexes
+await brain.repairIndex({ rebuild: 'all' })
+```
+
+A family named this way is recorded with `rebuilt: true` and
+`reason: 'explicit rebuild requested'`, and is skipped by the normal
+health-driven pass in the same call — it was already rebuilt unconditionally.
+
+Reach for the explicit door when you need certainty regardless of self-report;
+reach for bare `repairIndex()` for routine maintenance and after any incident
+where you're not sure which family (if any) needs it.
+
+## What `suspect` counts mean
+
+Storage's canonical count ledger increments the ALL-visibility total on every new
+record and decrements it on every *proven* delete — one where the record was read,
+or the caller supplied its prior image. A delete that cannot prove what it removed
+existed doesn't guess: it flags the ledger `suspect` (an operator-visible
+`console.warn`, narrated once per session, not once per delete) rather than risk
+decrementing a total that was never incremented for that record in the first
+place. This is intentionally rare — it's a defensive fallback for callers on an
+unusual removal path, not a per-delete cost.
+
+`suspect` is not directly exposed on any `Brainy` method today — it lives on the
+`StorageAdapter`'s optional `getCanonicalCounts()`, primarily consulted by
+`repairIndex()`'s recount step and by custom storage adapters composing their own
+`healthReport()`. What matters for an application: a `suspect` ledger is not
+incorrect, just *unverified since the last recount* — and `repairIndex()`'s
+unconditional count-rollup step (step 2, above) recomputes the ALL scalars from a
+real canonical walk on every call, clearing the flag with proof either way.
+
+## Practical guidance
+
+- **On a normal restart**, do nothing — `init()` brings every provider to
+  serving before it returns, or fails loudly.
+- **On a `*NotReadyError`** from a live read, reconcile with `repairIndex()`
+  (report-driven is almost always sufficient) and retry.
+- **After an incident** where you distrust a specific family regardless of what
+  it reports healthy — a storage-layer fault, a suspicious restore — use the
+  explicit door: `repairIndex({ rebuild: ['metadata' | 'graph' | 'vector'] })`.
+- **To audit before trusting a report**, `brain.auditGraph()` walks every stored
+  relationship and proves (or disproves) that reads return canonical truth,
+  independent of what any provider self-reports — see
+  [Inspecting a Live Brainy](../guides/inspection.md).