docs(release): the 10.4.0 entry, the index-health concept doc, and the API surfaces — written from the tree, not the plan
This commit is contained in:
parent
b9ba50fbec
commit
8cced871a0
8 changed files with 454 additions and 63 deletions
84
RELEASES.md
84
RELEASES.md
|
|
@ -31,6 +31,90 @@ is sometimes cited as a 7.x removal — those methods never existed on 7.x; the
|
|||
|
||||
---
|
||||
|
||||
## v10.4.0 — 2026-08-25 (the health report has a name)
|
||||
|
||||
Three related cures, one root cause: an index deciding whether it could be trusted
|
||||
by sampling itself instead of by exact accounting. This release replaces every
|
||||
sampled self-probe with ledger-derived truth, and a read against an unhealthy index
|
||||
now refuses loudly instead of guessing.
|
||||
|
||||
- **The canonical count ledger.** Storage now tracks two scalars per family
|
||||
(nouns/verbs) on the write path: the user-facing `counted` total — unchanged,
|
||||
still what `getNounCount()` / `getVerbCount()` return — and a new ALL-visibility
|
||||
`all` total covering every tier, the real denominator a derived index's own
|
||||
coverage math needs. The unfiltered storage-level `totalCount` returned by
|
||||
`getNouns()` / `getVerbs()` is now this unclamped ALL scalar; previously it could
|
||||
only ever move up (`Math.max(scalar, scanned)`), so an inflated counter could
|
||||
never self-correct. A delete that cannot prove the record it removed actually
|
||||
existed (no canonical read, no prior image available) no longer decrements on
|
||||
faith — it marks the ledger `suspect` (narrated once per session) instead of
|
||||
silently drifting, and the next `repairIndex()` clears the flag with a real
|
||||
recount.
|
||||
- **One contract for a throwing health probe.** A provider's `validateInvariants()`
|
||||
is documented to never throw — but if one does anyway (a bug, a transient fault),
|
||||
it is now read the same way everywhere: `heal: 'none'`, the error named in the
|
||||
report, never synthesized into a rebuild trigger and never swallowed into "looks
|
||||
fine." A flaky check can no longer buy itself a rebuild. `repairIndex()`'s
|
||||
per-family receipt also gains `missing` (an exact count plus a capped id sample),
|
||||
`rebuilt` (a full rebuild ran, vs. an incremental heal), and `reason`.
|
||||
- **The named health report; reads refuse instead of rebuilding.** Any index
|
||||
provider may now expose a synchronous, O(1) `healthReport()` — composed from the
|
||||
provider's own exact ledgers, never a sample — and this is the one signal
|
||||
Brainy's read gate trusts. The first-query lazy-build path is gone: `brain.init()`
|
||||
now runs every needed rebuild to completion before it returns, always, regardless
|
||||
of dataset size. A read that lands on a provider whose health report says it
|
||||
isn't serving throws a typed error instead of triggering a rebuild mid-query —
|
||||
`GraphIndexNotReadyError`, `MetadataIndexNotReadyError`, or
|
||||
`VectorIndexNotReadyError` (all exported from `@soulcraft/brainy`), naming the
|
||||
reasons. `repairIndex({ rebuild: ['metadata' | 'graph' | 'vector'] | 'all' })` is
|
||||
the new explicit operator door: it rebuilds the named family unconditionally, no
|
||||
health check consulted — reach for it when you have independent reason to
|
||||
distrust a family regardless of what it self-reports. Bare `repairIndex()` is
|
||||
unchanged in spirit: report-driven, heals only what its own checks say needs it.
|
||||
- New concept doc: [Index Health](docs/concepts/index-health.md) walks the whole
|
||||
story from a consumer's side — degraded-but-serving vs. not-ready, what
|
||||
`repairIndex()` checks and heals per family, what `suspect` counts mean.
|
||||
|
||||
**Nothing to change to adopt this.** No API removed, no signature narrowed —
|
||||
`repairIndex()` gains an optional options bag and its return value gains fields,
|
||||
both additive. The honest notes: if your code ever relied on a `find()` against a
|
||||
cold/not-yet-built index quietly triggering a rebuild and returning results a beat
|
||||
later, that behavior is gone — it now throws one of the three typed
|
||||
`*NotReadyError` classes instead (catch them if you need to distinguish "not ready
|
||||
yet" from "no results"). And `disableAutoRebuild: true` no longer defers index
|
||||
construction to the first query — a needed rebuild always runs at `open()` now;
|
||||
the flag has no effect on timing. Full manual control still lives in
|
||||
`repairIndex({ rebuild: [...] })`.
|
||||
|
||||
- **Crash-reopen catchup.** After an unclean shutdown, the metadata index now
|
||||
folds the exact fact window it missed — `find()` serves every acked write on
|
||||
reopen, closing the gap where canonical reads and counts recovered a
|
||||
crash-window write but the index kept serving its pre-crash state until the
|
||||
next full rebuild. Related root-cause fixed alongside: `close()` never
|
||||
stamped the index watermarks (only `flush()` did), so a close without a
|
||||
prior flush caused a needless full rescan verdict on the next open.
|
||||
- **Relation rows are live in the metadata index.** Previously verb rows
|
||||
entered the metadata index only during a rebuild — so a rebuilt store's
|
||||
relation postings went stale from the first `relate()` after it. Relations
|
||||
are now posted and retracted on the live write path (relate / unrelate /
|
||||
updateRelation / remove's cascade, and their `transact()` forms), in the
|
||||
same commit as the graph leg.
|
||||
- **The metadata rebuild is online.** `rebuild()` for the metadata family no
|
||||
longer clears and rebuilds in place (reads went empty for the duration): it
|
||||
builds a complete replacement beside the serving index, mirrors concurrent
|
||||
writes to both, swaps atomically, and persists once after the swap. Reads
|
||||
never observe a partial index. `repairIndex({ rebuild: ['metadata'] })` uses
|
||||
it automatically.
|
||||
- **A broken accelerator install can never read as "not installed."** The
|
||||
auto-detection free pass now requires the resolution error to name the
|
||||
accelerator package itself, exactly — a missing platform-binary sibling
|
||||
package, an inner file path, or a dependency failure is a broken install and
|
||||
`init()` throws loudly. And a plugin that declines activation is narrated on
|
||||
the always-on log channel, so `silent: true` can no longer hide a fallback
|
||||
to the default engines.
|
||||
|
||||
---
|
||||
|
||||
## v10.3.1 — 2026-08-18 (the fold that behaves)
|
||||
|
||||
Three recovery cures from one production first-boot incident (a brain's first
|
||||
|
|
|
|||
|
|
@ -323,58 +323,24 @@ Only the graph adjacency index carries a committed scale assertion:
|
|||
- ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer
|
||||
- ✅ **Zero Stubs**: Every line of code is production-ready
|
||||
|
||||
## Lazy Loading Performance
|
||||
## Index Build at Open (10.4+)
|
||||
|
||||
Brainy supports two initialization modes for optimal performance across different use cases:
|
||||
As of 10.4, `brain.init()` runs every needed index rebuild to completion before
|
||||
it returns — always, regardless of dataset size. There is no lazy,
|
||||
first-query rebuild path: a brain either finishes opening healthy, or `init()`
|
||||
fails loudly. `disableAutoRebuild` no longer defers index construction to a
|
||||
first query; it has no effect on *when* a rebuild runs. Manual control over
|
||||
rebuilds is `repairIndex({ rebuild: [...] })`. See
|
||||
[Index Health](concepts/index-health.md) for the full read-gate contract
|
||||
(providers self-report readiness via `healthReport()`; a read against a
|
||||
not-serving provider throws a typed `*NotReadyError` rather than rebuilding
|
||||
mid-query).
|
||||
|
||||
### Mode 1: Auto-Rebuild (Default)
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy()
|
||||
await brain.init() // Rebuilds indexes during init (~500ms-3s for 10K entities)
|
||||
```
|
||||
|
||||
**Performance:**
|
||||
- Init time: 500ms-3s (depends on dataset size)
|
||||
- First query: Instant (indexes already loaded)
|
||||
- Use case: Traditional applications, long-running servers
|
||||
|
||||
### Mode 2: Lazy Loading
|
||||
|
||||
```javascript
|
||||
const brain = new Brainy({ disableAutoRebuild: true })
|
||||
await brain.init() // Returns instantly (0-10ms)
|
||||
|
||||
const results = await brain.find({ limit: 10 }) // First query triggers rebuild (~50-200ms)
|
||||
const more = await brain.find({ limit: 100 }) // Subsequent queries instant (0ms check)
|
||||
```
|
||||
|
||||
**Performance:**
|
||||
- Init time: 0-10ms (instant)
|
||||
- First query: 50-200ms (includes index rebuild for 1K-10K entities)
|
||||
- Subsequent queries: 0ms check (instant)
|
||||
- Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
|
||||
|
||||
**Concurrency Safety:**
|
||||
```javascript
|
||||
// 100 concurrent queries immediately after init
|
||||
await brain.init()
|
||||
|
||||
const promises = Array.from({ length: 100 }, () =>
|
||||
brain.find({ limit: 10 })
|
||||
)
|
||||
|
||||
const results = await Promise.all(promises)
|
||||
// ✅ Only 1 rebuild triggered (mutex)
|
||||
// ✅ All 100 queries return correct results
|
||||
// ✅ Total time: ~60ms (not 6000ms!)
|
||||
```
|
||||
|
||||
**Use Cases for Lazy Loading:**
|
||||
- **Serverless/Edge**: Minimize cold start time (0-10ms init)
|
||||
- **Development**: Faster restarts during development
|
||||
- **Large datasets**: Defer index loading until needed
|
||||
- **Read-heavy workloads**: Writes don't wait for index rebuild
|
||||
<!-- The pre-10.4 "Mode 2: Lazy Loading on First Query" section previously
|
||||
documented here (disableAutoRebuild deferring index construction to the
|
||||
first find() call) described a real, now-retired code path. Removed
|
||||
rather than left to mislead; the concept doc above is the current
|
||||
contract. -->
|
||||
|
||||
## Zero Configuration Required
|
||||
|
||||
|
|
@ -384,10 +350,6 @@ Brainy is designed to be **smart enough to tune itself dynamically**. No configu
|
|||
// That's it. Brainy handles everything.
|
||||
const brain = new Brainy()
|
||||
await brain.init()
|
||||
|
||||
// Or with lazy loading for serverless
|
||||
const brain = new Brainy({ disableAutoRebuild: true })
|
||||
await brain.init() // Instant (0-10ms)
|
||||
```
|
||||
|
||||
### Automatic Self-Tuning
|
||||
|
|
@ -395,7 +357,6 @@ await brain.init() // Instant (0-10ms)
|
|||
- **Metadata Index**: Auto-builds sorted indices for range queries on first use
|
||||
- **Graph Index**: Auto-flushes every 30 seconds
|
||||
- **Default Tuning**: Research-based vector index defaults
|
||||
- **Lazy Loading**: Indices built only when needed
|
||||
- **Cache Management**: LRU caches with TTL
|
||||
|
||||
### Intelligent Defaults
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ next:
|
|||
- guides/storage-adapters
|
||||
---
|
||||
|
||||
# Plugin Development Guide
|
||||
# Plugin System
|
||||
|
||||
Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer.
|
||||
|
||||
|
|
@ -200,15 +200,30 @@ members so a warm reopen never pays a redundant rebuild-from-canonical:
|
|||
- **`init?(): Promise<void>`** — eager cold-load. Brainy awaits it once during
|
||||
`brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first)
|
||||
and **before the rebuild gate**.
|
||||
- **`isReady?(): boolean`** — honest durability signal. `true` ⇔ the persisted index is
|
||||
loaded (or cheaply demand-loadable) and consistent with what was last persisted. When
|
||||
exposed, the rebuild gate defers to this signal **instead of** the `size() === 0` /
|
||||
`totalEntries === 0` heuristics — a disk-native index may report 0 resident entries
|
||||
while fully durable. Never return `true` if the durable state failed to load: the
|
||||
signal is honest in both directions, and a not-ready provider gets its rebuild even
|
||||
when `size() > 0`.
|
||||
- **`healthReport?(): HealthReport`** — the PREFERRED signal (10.4+). A named,
|
||||
synchronous, O(1) verdict derived from the provider's own exact ledgers — never a
|
||||
sample, never I/O, must never throw for a well-formed provider. Brainy's read gate
|
||||
(`assessProviderHealth()`) reads this INSTEAD of `isReady()` / size heuristics when
|
||||
present: `serving: false` refuses the read with a typed `*NotReadyError` rather than
|
||||
triggering a rebuild — a read never starts a store walk. `healthy` marks every
|
||||
*verified* invariant holding; a family named in `unledgered` counts as neither
|
||||
healthy nor broken. See `HealthReport` / `LedgerInvariantResult` /
|
||||
`InvariantSource` in `src/plugin.ts`, and
|
||||
[Index Health](concepts/index-health.md) for the consumer-facing story.
|
||||
- **`isReady?(): boolean`** — honest durability signal, the fallback when
|
||||
`healthReport()` is absent. `true` ⇔ the persisted index is loaded (or cheaply
|
||||
demand-loadable) and consistent with what was last persisted. When exposed, the
|
||||
gate defers to this signal **instead of** the `size() === 0` / `totalEntries === 0`
|
||||
heuristics — a disk-native index may report 0 resident entries while fully durable.
|
||||
Never return `true` if the durable state failed to load: the signal is honest in
|
||||
both directions, and a not-ready provider gets its rebuild even when `size() > 0`.
|
||||
- **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background
|
||||
migration); brainy skips its rebuild entirely.
|
||||
- **`validateInvariants?(): Promise<ProviderInvariantReport>`** — the async DEEP
|
||||
diagnostic (full scans allowed), distinct from the bounded, sync `healthReport()`.
|
||||
Must never throw — a failure is `healthy: false` data, not an exception; a provider
|
||||
that throws anyway is read as a loud, unverified failure (never as "healthy") by
|
||||
every caller, never silently retried into a rebuild.
|
||||
|
||||
Providers that implement none of these keep the size/count heuristics — correct for
|
||||
engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index).
|
||||
|
|
|
|||
|
|
@ -1451,6 +1451,34 @@ const count = await brain.getVerbCount()
|
|||
|
||||
---
|
||||
|
||||
### The canonical count ledger (`StorageAdapter.getCanonicalCounts()`)
|
||||
|
||||
An OPTIONAL method on the `StorageAdapter` interface (implemented by both
|
||||
built-in adapters), not a method on `Brainy` itself — relevant if you're
|
||||
writing a custom storage adapter or composing a provider's own
|
||||
`healthReport()`. O(1), no I/O. Per family (`nouns`/`verbs`):
|
||||
|
||||
```typescript
|
||||
interface CanonicalCounts {
|
||||
nouns: { counted: number; all: number }
|
||||
verbs: { counted: number; all: number }
|
||||
suspect: boolean
|
||||
}
|
||||
```
|
||||
|
||||
- `counted` mirrors `getNounCount()` / `getVerbCount()` (public + internal tiers).
|
||||
- `all` is the ALL-visibility scalar — every tier, including system/internal
|
||||
records — the denominator a derived index's own coverage math is measured
|
||||
against.
|
||||
- `suspect` is `true` when an unprovable delete has left `all` unverified since
|
||||
the last recount; `brain.repairIndex()` clears it with a real canonical walk.
|
||||
|
||||
Adapters without the ledger omit the method; treat absence as "no
|
||||
denominator," never as zero. See
|
||||
**[Index Health](../concepts/index-health.md)** for the full story.
|
||||
|
||||
---
|
||||
|
||||
### Subtype & facet APIs
|
||||
|
||||
Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
|
||||
|
|
@ -1852,6 +1880,81 @@ await brain.repairIndex({ rebuild: ['graph'] })
|
|||
await brain.repairIndex({ rebuild: 'all' })
|
||||
```
|
||||
|
||||
**`RepairReport`:**
|
||||
- `families: RepairFamilyReport[]` — one row per family checked
|
||||
- `healedTotal: number` — items healed across every family
|
||||
- `durationMs: number`
|
||||
|
||||
**`RepairFamilyReport`** (one row):
|
||||
- `family: string` — e.g. `'orphaned-containers'`, `'count-rollups'`,
|
||||
`'vfs-containment'`, `'metadata-corruption'`, `'provider:metadata'`,
|
||||
`'provider:graph'`, `'provider:vector'`
|
||||
- `checked: boolean` — was this family actually examined (`false` ⇒ see `skipped`)
|
||||
- `healed: number` — items re-posted/corrected in place (the incremental heal count)
|
||||
- `missing?: { count: number; sample: string[] }` — exact count plus a capped id
|
||||
sample when the check can name what diverged (never the full list)
|
||||
- `rebuilt?: boolean` — a full generational rebuild ran (vs. an incremental heal)
|
||||
- `detail?: string` / `reason?: string` — narration
|
||||
- `skipped?: string` — why the family wasn't checked
|
||||
|
||||
Full walkthrough — what each family checks, degraded-but-serving vs. not-ready,
|
||||
and what `suspect` counts mean — in
|
||||
**[Index Health](../concepts/index-health.md)**.
|
||||
|
||||
---
|
||||
|
||||
### Index readiness: typed errors, `healthReport()`, `disableAutoRebuild`
|
||||
|
||||
Every derived-index provider (vector, graph, metadata) may expose a named,
|
||||
synchronous, O(1) `healthReport()` composed from its own exact ledgers — the
|
||||
signal Brainy's read gate trusts over sampling or size heuristics. `init()`
|
||||
brings every provider to serving before it returns; there is no first-query
|
||||
lazy-rebuild path. A read that reaches a provider whose health report says it
|
||||
isn't serving throws instead of rebuilding mid-query:
|
||||
|
||||
| Error | Thrown by | Meaning |
|
||||
|---|---|---|
|
||||
| `GraphIndexNotReadyError` | `find({ connected })`, `neighbors()`, `related()` | Graph adjacency isn't serving |
|
||||
| `MetadataIndexNotReadyError` | `find({ where })` | Metadata/field index isn't serving |
|
||||
| `VectorIndexNotReadyError` | `find({ query })`, `similar()` | Vector index isn't serving |
|
||||
|
||||
All three are exported from `@soulcraft/brainy`. Catch them to distinguish
|
||||
"index not ready" from a genuine empty result:
|
||||
|
||||
```typescript
|
||||
import { MetadataIndexNotReadyError } from '@soulcraft/brainy'
|
||||
|
||||
try {
|
||||
const rows = await brain.find({ where: { status: 'active' } })
|
||||
} catch (err) {
|
||||
if (err instanceof MetadataIndexNotReadyError) {
|
||||
// reconcile: await brain.repairIndex(), then retry
|
||||
} else {
|
||||
throw err
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**`disableAutoRebuild`** no longer defers index construction to the first
|
||||
query. A needed rebuild always runs at `open()`, regardless of this flag or
|
||||
dataset size; the flag has no effect on *when* a rebuild runs. Full manual
|
||||
control lives in `repairIndex({ rebuild: [...] })`, above.
|
||||
|
||||
### `validateIndexConsistency()` → `Promise<...>`
|
||||
|
||||
The deep, async diagnostic counterpart to `healthReport()` — safe to run on a
|
||||
live brain, but does more work (a provider's `validateInvariants()` may run a
|
||||
full scan, not just read a ledger). Aggregates the JS metadata index's own
|
||||
consistency check with every derived-index provider's invariant report.
|
||||
|
||||
```typescript
|
||||
const validation = await brain.validateIndexConsistency()
|
||||
if (!validation.healthy) {
|
||||
console.log(validation.recommendation) // what to run, e.g. repairIndex()
|
||||
console.log(validation.providers) // each provider's own invariant report, when exposed
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Lifecycle
|
||||
|
|
|
|||
|
|
@ -723,6 +723,14 @@ async stats(): Promise<Statistics> {
|
|||
|
||||
### 5. Index Rebuilding (Lazy Loading Support)
|
||||
|
||||
> **Stale as of 10.4 — "Mode 2: Lazy Loading on First Query" below is
|
||||
> RETIRED.** `disableAutoRebuild` no longer defers index construction to a
|
||||
> first query; `brain.init()` now runs every needed rebuild to completion
|
||||
> before it returns, unconditionally, and a read against a not-serving
|
||||
> provider throws a typed `*NotReadyError` instead of rebuilding mid-query.
|
||||
> See `docs/concepts/index-health.md` for the current contract. Left below
|
||||
> as historical background on the rebuild mechanics.
|
||||
|
||||
**Two modes of index loading:**
|
||||
|
||||
#### Mode 1: Auto-Rebuild on init() (default)
|
||||
|
|
|
|||
|
|
@ -1,5 +1,15 @@
|
|||
# Initialization and Rebuild Processes
|
||||
|
||||
> **Stale as of 10.4 — "Mode 2: Lazy Loading on First Query" below is RETIRED.**
|
||||
> `disableAutoRebuild` no longer defers index construction to a first query;
|
||||
> `brain.init()` now runs every needed rebuild to completion before it
|
||||
> returns, unconditionally. A read against a not-serving provider throws a
|
||||
> typed `*NotReadyError` instead of rebuilding mid-query. See
|
||||
> `docs/concepts/index-health.md` for the current contract; this document's
|
||||
> line-number references to `src/brainy.ts` also predate the file's current
|
||||
> size and are unreliable. Left as historical background on the rebuild
|
||||
> mechanics, not as a current API description.
|
||||
|
||||
This document explains how Brainy's four indexes (MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
|
||||
|
||||
## Core Principle: All Indexes Are Disk-Based
|
||||
|
|
|
|||
204
docs/concepts/index-health.md
Normal file
204
docs/concepts/index-health.md
Normal file
|
|
@ -0,0 +1,204 @@
|
|||
---
|
||||
title: Index Health
|
||||
slug: concepts/index-health
|
||||
public: true
|
||||
category: concepts
|
||||
template: concept
|
||||
order: 8
|
||||
description: How Brainy knows whether a derived index can be trusted — exact accounting instead of sampling, the named health report, degraded-but-serving vs. not-ready, and what repairIndex() checks, heals, and rebuilds.
|
||||
next:
|
||||
- concepts/generation-fact-log
|
||||
- guides/inspection
|
||||
---
|
||||
|
||||
# Index Health
|
||||
|
||||
Brainy keeps one **canonical** copy of every entity and relationship, and three
|
||||
**derived** indexes built from it — vector, metadata, and graph — so `find()` can
|
||||
answer semantically, by filter, and by traversal without re-deriving the answer from
|
||||
scratch on every query. A derived index is a cache with a serving structure: it can
|
||||
be present but stale, present but only partially loaded, or fully out of sync with
|
||||
canonical after a crash. This page is about how Brainy decides whether to trust one,
|
||||
what it does when it can't, and how you reconcile the two.
|
||||
|
||||
## Exact accounting instead of sampling
|
||||
|
||||
Older health checks worked by inference: does `size()` return something greater
|
||||
than zero, does a spot-check on one known item come back correct. Both are proxies.
|
||||
A cold index can report a nonzero count while its actual serving structure never
|
||||
loaded, and a spot-check only proves the one item it happened to ask about.
|
||||
|
||||
Every derived-index provider may now expose a named, synchronous, O(1)
|
||||
`healthReport()` — composed from the provider's own **exact ledgers** (real counters
|
||||
it already maintains on the write path), never a sample or a walk. This is the one
|
||||
signal Brainy's read gate consults. A provider that doesn't yet expose one falls
|
||||
back to an honest `isReady()` boolean, and finally to a size heuristic for engines
|
||||
with neither — but wherever a `healthReport()` exists, it wins.
|
||||
|
||||
Underneath, storage itself keeps an analogous **canonical count ledger**: a
|
||||
`counted` scalar (the user-facing total — what `getNounCount()` / `getVerbCount()`
|
||||
return) and an `all` scalar (every tier, including internal records a derived
|
||||
index's own coverage math needs to compare against). This is the real denominator
|
||||
a provider's `healthReport()` measures itself by, rather than a total that can only
|
||||
ever ratchet upward. See [What `suspect` counts mean](#what-suspect-counts-mean)
|
||||
below for the one case that ledger can't stay exact through on its own.
|
||||
|
||||
## The named report
|
||||
|
||||
A `HealthReport` carries, per provider (`'vector'` / `'graph'` / `'metadata'`):
|
||||
|
||||
- **`healthy`** — `true` iff every *verified* invariant holds. An invariant whose
|
||||
family has no ledger yet is `unledgered`, never counted either way — unknown,
|
||||
not passing.
|
||||
- **`serving`** — can this provider answer a query right now. A failing invariant
|
||||
graded `heal: 'repair'` or `heal: 'none'` still leaves `serving: true` — this is
|
||||
**degraded-but-serving**: something is off (say, a stale rollup on an
|
||||
`employee` record's relationship count) but reads keep working. Only a failure
|
||||
graded `heal: 'rebuild'` flips `serving` to `false` — **not-ready** — because the
|
||||
provider itself is telling you its serving structure cannot answer correctly.
|
||||
- **`invariants`** — each checked condition, with its provenance
|
||||
(`source: 'ledger'` — an exact count; `'deep'` — a full scan, diagnostic-only;
|
||||
`'unledgered'` — not yet tracked) and, for a failing one, an exact `missing`
|
||||
count plus a capped sample of the affected ids — a verdict, never a dump.
|
||||
- **`generation`** — bumps on every ledger mutation and rebuild, so a caller can
|
||||
cache a verdict per generation instead of re-deriving it.
|
||||
|
||||
The distinction that matters day to day: `healthy: false` can be entirely benign —
|
||||
a maintenance window, a divergence `repairIndex()` will clean up on its own
|
||||
schedule. `serving: false` is not benign. It means this provider is refusing to
|
||||
answer, on its own word, right now.
|
||||
|
||||
## Reads refuse — they never rebuild
|
||||
|
||||
A query that reaches a not-serving provider does not trigger a rebuild from inside
|
||||
the read. Brainy retired that path deliberately: a rebuild kicked off by an ordinary
|
||||
`find({ where: { status: 'active' } })` call is a dark, unpredictable cost hiding
|
||||
behind a request that looks like a cheap read. Instead, the read throws a typed,
|
||||
catchable error naming the reason:
|
||||
|
||||
| Error | Thrown when | Meaning |
|
||||
|---|---|---|
|
||||
| `GraphIndexNotReadyError` | `find({ connected })`, `neighbors()`, `related()` | The graph adjacency index isn't serving — traversal would otherwise return `[]` indistinguishable from "no relationships" |
|
||||
| `MetadataIndexNotReadyError` | `find({ where })` | The metadata/field index isn't serving — a filtered read would otherwise return `[]` indistinguishable from "no matches" |
|
||||
| `VectorIndexNotReadyError` | `find({ query })`, `similar()` | The vector index isn't serving — a semantic search would otherwise return `[]` indistinguishable from "nothing similar" |
|
||||
|
||||
All three are exported from `@soulcraft/brainy`. Catch them where your application
|
||||
needs to distinguish "this index isn't ready yet" from "there's genuinely nothing
|
||||
here" — a health dashboard, a retry policy, an operator alert. The fix is always
|
||||
the same: reconcile the index, either by reopening the brain (which brings every
|
||||
provider to serving before `init()` returns — see the next section) or by calling
|
||||
`repairIndex()` explicitly.
|
||||
|
||||
```typescript
|
||||
try {
|
||||
const active = await brain.find({ where: { status: 'active' } })
|
||||
} catch (err) {
|
||||
if (err instanceof MetadataIndexNotReadyError) {
|
||||
// not a "no results" — the index itself refused; alert or retry after repair
|
||||
} else {
|
||||
throw err
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Rebuilds happen at open, not on first query
|
||||
|
||||
`brain.init()` runs every needed rebuild to completion **before it returns**,
|
||||
unconditionally, regardless of dataset size. There is no lazy, first-query
|
||||
rebuild path anymore — a brain either finishes opening healthy, or it fails
|
||||
open loudly. `disableAutoRebuild: true` no longer defers index construction to
|
||||
the first query: it has no effect on *when* a needed rebuild runs. Full manual
|
||||
control over rebuilds is `repairIndex({ rebuild: [...] })` (below), not this flag.
|
||||
|
||||
## `repairIndex()` — checking and healing
|
||||
|
||||
Bare `repairIndex()` is **report-driven**: it only heals what its own checks say
|
||||
actually needs it, and it always returns a full per-family receipt.
|
||||
|
||||
```typescript
|
||||
const report = await brain.repairIndex()
|
||||
report.healedTotal // total items healed across every family
|
||||
report.durationMs
|
||||
report.families // one row per family checked
|
||||
```
|
||||
|
||||
Each `RepairFamilyReport` row names what happened:
|
||||
|
||||
- **`checked`** — was this family actually examined (`false` means skipped —
|
||||
see `skipped` for why).
|
||||
- **`healed`** — items re-posted or corrected in place.
|
||||
- **`missing`** — when the check can name what diverged: an exact `count` plus a
|
||||
capped `sample` of ids.
|
||||
- **`rebuilt`** — a full generational rebuild ran (as opposed to an incremental
|
||||
heal).
|
||||
- **`detail`** / **`reason`** / **`skipped`** — the receipt's narration; a row is
|
||||
always either checked or explains why it wasn't. Nothing is silent.
|
||||
|
||||
On every call, bare `repairIndex()`:
|
||||
|
||||
1. Prunes orphaned canonical containers left by a partial delete.
|
||||
2. Recomputes the count rollups from one canonical walk (unconditional — this is
|
||||
also what clears a `suspect` ledger; see below).
|
||||
3. Reconciles VFS containment edges, if the VFS is initialized.
|
||||
4. Runs the metadata index's own corruption detection pass.
|
||||
5. Consults each of the three derived-index providers' own health check and
|
||||
rebuilds only a family whose failing invariant actually asks for it
|
||||
(`heal: 'rebuild'`) — never a provider that reports `healthy` or a lesser
|
||||
grade.
|
||||
|
||||
### The explicit rebuild door
|
||||
|
||||
`options.rebuild` skips the health check and rebuilds one or more families
|
||||
**unconditionally** — the operator override for when you have independent reason
|
||||
to distrust a family regardless of what it self-reports (a suspicious deploy, a
|
||||
storage-layer incident, a support ticket that doesn't match what the health report
|
||||
says):
|
||||
|
||||
```typescript
|
||||
// Force the graph adjacency to rebuild from canonical, no invariant consulted
|
||||
await brain.repairIndex({ rebuild: ['graph'] })
|
||||
|
||||
// Force all three derived indexes
|
||||
await brain.repairIndex({ rebuild: 'all' })
|
||||
```
|
||||
|
||||
A family named this way is recorded with `rebuilt: true` and
|
||||
`reason: 'explicit rebuild requested'`, and is skipped by the normal
|
||||
health-driven pass in the same call — it was already rebuilt unconditionally.
|
||||
|
||||
Reach for the explicit door when you need certainty regardless of self-report;
|
||||
reach for bare `repairIndex()` for routine maintenance and after any incident
|
||||
where you're not sure which family (if any) needs it.
|
||||
|
||||
## What `suspect` counts mean
|
||||
|
||||
Storage's canonical count ledger increments the ALL-visibility total on every new
|
||||
record and decrements it on every *proven* delete — one where the record was read,
|
||||
or the caller supplied its prior image. A delete that cannot prove what it removed
|
||||
existed doesn't guess: it flags the ledger `suspect` (an operator-visible
|
||||
`console.warn`, narrated once per session, not once per delete) rather than risk
|
||||
decrementing a total that was never incremented for that record in the first
|
||||
place. This is intentionally rare — it's a defensive fallback for callers on an
|
||||
unusual removal path, not a per-delete cost.
|
||||
|
||||
`suspect` is not directly exposed on any `Brainy` method today — it lives on the
|
||||
`StorageAdapter`'s optional `getCanonicalCounts()`, primarily consulted by
|
||||
`repairIndex()`'s recount step and by custom storage adapters composing their own
|
||||
`healthReport()`. What matters for an application: a `suspect` ledger is not
|
||||
incorrect, just *unverified since the last recount* — and `repairIndex()`'s
|
||||
unconditional count-rollup step (step 2, above) recomputes the ALL scalars from a
|
||||
real canonical walk on every call, clearing the flag with proof either way.
|
||||
|
||||
## Practical guidance
|
||||
|
||||
- **On a normal restart**, do nothing — `init()` brings every provider to
|
||||
serving before it returns, or fails loudly.
|
||||
- **On a `*NotReadyError`** from a live read, reconcile with `repairIndex()`
|
||||
(report-driven is almost always sufficient) and retry.
|
||||
- **After an incident** where you distrust a specific family regardless of what
|
||||
it reports healthy — a storage-layer fault, a suspicious restore — use the
|
||||
explicit door: `repairIndex({ rebuild: ['metadata' | 'graph' | 'vector'] })`.
|
||||
- **To audit before trusting a report**, `brain.auditGraph()` walks every stored
|
||||
relationship and proves (or disproves) that reads return canonical truth,
|
||||
independent of what any provider self-reports — see
|
||||
[Inspecting a Live Brainy](../guides/inspection.md).
|
||||
|
|
@ -14,3 +14,9 @@ from the message alone. `biography.test.ts` is split into two `it` blocks
|
|||
(Ch1-3, then Ch4-6) purely for reporting; it is still ONE fixed-order story.
|
||||
Chapters must never be reordered, skipped, or made conditional, and a
|
||||
failing chapter's assertion must never be weakened to force green.
|
||||
|
||||
Lab notes (hard-won, keep):
|
||||
- `git reset --hard` does NOT remove untracked files — a "clean" tree can still
|
||||
carry stray test stores; use `git clean -fd tests/lifecycle-tmp` equivalents.
|
||||
- `silent: true` patches `console` process-wide — never assert narration through
|
||||
`console` spies in this lane; the engine's always-on channel is `prodLog`.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue