Compare commits

...

8 commits

Author SHA1 Message Date
0e1286e321 chore(release): 10.4.0-rc.2
All checks were successful
Publish (The Source) / Publish to The Source registry (push) Successful in 12m30s
CI / Node 22 (push) Successful in 12m19s
CI / Node 24 (push) Successful in 12m8s
CI / Bun (latest) (push) Successful in 12m25s
CI / Integration + conformance (Node 22) (push) Successful in 19m30s
2026-08-25 11:38:06 -07:00
39b916a3c0 test(readiness): the report helper's clock freezes — two independently-built reports compared across a millisecond tick made the plant lane red
All checks were successful
CI / Node 22 (push) Successful in 12m14s
CI / Node 24 (push) Successful in 12m12s
CI / Bun (latest) (push) Successful in 12m23s
CI / Integration + conformance (Node 22) (push) Successful in 19m22s
2026-08-25 11:04:30 -07:00
553e0d97ae feat(repair): a heal:'repair' verdict routes to the provider's own incremental repair()
Some checks failed
CI / Node 24 (push) Failing after 7m47s
CI / Node 22 (push) Successful in 12m19s
CI / Bun (latest) (push) Successful in 12m20s
CI / Integration + conformance (Node 22) (push) Successful in 19m22s
repairIndex() acted only on heal:'rebuild' — an invariant asking for the
INCREMENTAL heal (re-post exactly what the ledger names, O(missing), never
a store-sized rebuild) did nothing on brainy's side. A failing 'repair'
verdict now routes to the provider's feature-detected repair(); the
post-heal RE-READ of the report decides success (the acceptance meta-pin's
law — run the named heal once, re-read, nothing may still fail the same
way), and a repair that does not converge is recorded with the escalation
named: repairIndex({ rebuild: [family] }).
2026-08-25 10:47:51 -07:00
ddd5e71928 fix(storage): an unknown nested storage config can never silently land on the shared default root
The factory loudly rejects every REMOVED pre-8.0 path key, but an unknown
nested `config` object (e.g. `storage: { config: { baseDir } }` — a shape
that was never supported) fell through SILENTLY to the zero-config default
directory. Every instance constructed with such a shape wrote to ONE shared
on-disk root while its caller believed each had its own — found live when
two integration tests' brains shared a store across an entire
single-process CI run and a health probe refused on the foreign edges it
sampled. A nested `config` carrying any path-shaped key now throws the same
loud migration error, naming the canonical `path` rename. The two tests are
repaired to the supported shape (and now actually test isolated stores, for
the first time since 8.0).
2026-08-25 10:47:51 -07:00
8cced871a0 docs(release): the 10.4.0 entry, the index-health concept doc, and the API surfaces — written from the tree, not the plan
Some checks failed
CI / Node 22 (push) Successful in 12m19s
CI / Node 24 (push) Successful in 12m16s
CI / Integration + conformance (Node 22) (push) Failing after 15m12s
CI / Bun (latest) (push) Successful in 12m24s
2026-08-25 10:02:25 -07:00
b9ba50fbec fix(plugins): the silent-degrade doors close — a broken accelerator install can never read as absent
The auto-detection "not installed" heuristic accepted any resolution failure
whose message merely CONTAINED the package name, unterminated — so a missing
platform-binary sibling package (what a deploy replacing node_modules
mid-restart leaves behind) read as "the accelerator is not installed", and
brainy silently served the default WASM engines with zero journal lines. A
production restart storm paid 90 seconds of throttled WASM compile behind
exactly that hole. The name must now terminate where it ends (quote,
whitespace, punctuation, end) — a sibling package, an inner file path, or a
dependency failure is a broken install and init() throws, as the guard's own
law always stated.

Second door: activate() returning false (the documented graceful decline)
warned on console.warn, which `silent: true` patches away — an invisible
degrade. The decline now narrates via the always-on channel.

Also exports CanonicalCounts from the public surface (the coverage-ledger
denominator type consumers read through getCanonicalCounts()).

Pinned in tests/unit/plugin-activation-loudness.test.ts (five error shapes;
the decline warn under silent: true).
2026-08-25 10:01:56 -07:00
18f172e098 feat(recovery): the catchup verdict is consumed; verb rows go live; the metadata rebuild goes online
Three cures on the JS metadata index, one seam:

- THE CATCHUP WIRING. The index computed its three-way watermark verdict at
  open and nothing consumed it — after a crash + adopt reopen, find() served
  the pre-crash index while canonical reads and counts recovered (caught by
  the lifecycle lane's first run). The open path now consumes the verdict:
  'adopt' is a no-op, 'catchup' folds the fact window (stamped, committed]
  through the index legs — nouns and verbs, remove-then-add, one mechanism
  for add and update — and 'rescan' runs the explicit rebuild, each narrated.
  The lane's Ch4–6 release-blocking marker comes off: the contract holds.
  Bonus root-cause: close() never stamped the projection watermarks (only
  flush() did), so any close without a prior flush verdicted a needless
  'rescan' on reopen — both doors now stamp.

- THE LIVE VERB PATH. Verb rows entered the metadata index only via rebuild
  walks, so every rebuilt store minted phantom/stale verb postings from its
  first live relate(). relate()/unrelate()/updateRelation() and remove()'s
  cascade now post/retract the verb's row in the same commit as the graph
  leg — transact() planners mirror identically — using the exact record
  shape the rebuild walk uses, so live and rebuilt populations agree.

- THE ONLINE REBUILD. rebuild() was clear-then-walk — every metadata read
  empty for the duration. rebuildMetadataIndexOnline builds a fresh manager
  beside the serving one (shared identity, in-memory build, dual-write via
  a shadow seam with zero call-site changes), atomically swaps the
  reference, and persists exactly once post-swap. A find() polled ~200x
  during a 2k-noun rebuild never dropped below its baseline.
  repairIndex({ rebuild: ['metadata'] }) uses it automatically.
2026-08-25 10:01:56 -07:00
f8f64780b1 feat(health): the gate reads the named report — reads refuse loudly, never rebuild; open serves before it returns; the ceremony door
All checks were successful
CI / Node 22 (push) Successful in 12m19s
CI / Node 24 (push) Successful in 12m16s
CI / Integration + conformance (Node 22) (push) Successful in 18m41s
CI / Bun (latest) (push) Successful in 12m20s
The read gate stops consulting the unnamed isReady() boolean: every provider
may expose healthReport() (sync, O(1), composed from exact ledgers —
HealthReport with a monotonic generation, per-invariant source
ledger|deep|unledgered, missing {count, sample}), and one readiness
authority (assessProviderHealth) derives the verdict. Unledgered families
are UNKNOWN — never healthy, never broken; a report that throws is a loud
not-ready, never a shrug. Reads at the four index choke points refuse with
the typed NotReady errors, narrated once per (provider, generation) — a
read NEVER starts a store walk:

- the first-read lazy build retires (open builds instead, regardless of
  size — the ≥10k deferral and the "lazy loading on first query" branch go;
  disableAutoRebuild is re-meant honestly in its docs);
- the verify*Live read-path rebuild triggers retire (refuse-or-serve);
- the read-time consistency probe that could launch a dark rebuild from an
  ordinary find() retires;
- repairIndex({ rebuild: ['metadata'|'graph'|'vector'] | 'all' }) is the
  one explicit door: rebuilds the named leg unconditionally and reports
  rebuilt per family; bare repairIndex() stays report-driven.

test(lifecycle): the biography lane — a store's whole life, refereed

tests/lifecycle/: an independent shadow model referees every read after
every chapter (founding, a working day, clean restart, crash, repair,
second life). Chapters 1-3 green. Chapters 4-6 assert the true contract and
are marked .fails as a release-blocking finding (the kill-matrix
convention): after a crash + adopt reopen the metadata index computes its
'catchup' watermark verdict and nothing consumes it — find() serves the
pre-crash index while canonical and counts recover. The catchup wiring is
the cure; a passing .fails will force the marker's removal. The lane runs
in the integration gate (config + coverage guard).
2026-08-24 12:45:51 -07:00
37 changed files with 4072 additions and 829 deletions

View file

@ -2,6 +2,17 @@
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines. All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
### [10.4.0-rc.2](https://source.soulcraft.com/soulcraft/brainy/compare/v10.4.0-rc.1...v10.4.0-rc.2) (2026-08-25)
- test(readiness): the report helper's clock freezes — two independently-built reports compared across a millisecond tick made the plant lane red (39b916a3)
- feat(repair): a heal:'repair' verdict routes to the provider's own incremental repair() (553e0d97)
- fix(storage): an unknown nested storage config can never silently land on the shared default root (ddd5e719)
- docs(release): the 10.4.0 entry, the index-health concept doc, and the API surfaces — written from the tree, not the plan (8cced871)
- fix(plugins): the silent-degrade doors close — a broken accelerator install can never read as absent (b9ba50fb)
- feat(recovery): the catchup verdict is consumed; verb rows go live; the metadata rebuild goes online (18f172e0)
- feat(health): the gate reads the named report — reads refuse loudly, never rebuild; open serves before it returns; the ceremony door (f8f64780)
### [10.4.0-rc.1](https://source.soulcraft.com/soulcraft/brainy/compare/v10.3.1...v10.4.0-rc.1) (2026-08-24) ### [10.4.0-rc.1](https://source.soulcraft.com/soulcraft/brainy/compare/v10.3.1...v10.4.0-rc.1) (2026-08-24)
- ci(publish): the home dist-tag follows the version — a prerelease publishes under 'rc' and never moves 'latest' (a1376e4a) - ci(publish): the home dist-tag follows the version — a prerelease publishes under 'rc' and never moves 'latest' (a1376e4a)

View file

@ -31,6 +31,90 @@ is sometimes cited as a 7.x removal — those methods never existed on 7.x; the
--- ---
## v10.4.0 — 2026-08-25 (the health report has a name)
Three related cures, one root cause: an index deciding whether it could be trusted
by sampling itself instead of by exact accounting. This release replaces every
sampled self-probe with ledger-derived truth, and a read against an unhealthy index
now refuses loudly instead of guessing.
- **The canonical count ledger.** Storage now tracks two scalars per family
(nouns/verbs) on the write path: the user-facing `counted` total — unchanged,
still what `getNounCount()` / `getVerbCount()` return — and a new ALL-visibility
`all` total covering every tier, the real denominator a derived index's own
coverage math needs. The unfiltered storage-level `totalCount` returned by
`getNouns()` / `getVerbs()` is now this unclamped ALL scalar; previously it could
only ever move up (`Math.max(scalar, scanned)`), so an inflated counter could
never self-correct. A delete that cannot prove the record it removed actually
existed (no canonical read, no prior image available) no longer decrements on
faith — it marks the ledger `suspect` (narrated once per session) instead of
silently drifting, and the next `repairIndex()` clears the flag with a real
recount.
- **One contract for a throwing health probe.** A provider's `validateInvariants()`
is documented to never throw — but if one does anyway (a bug, a transient fault),
it is now read the same way everywhere: `heal: 'none'`, the error named in the
report, never synthesized into a rebuild trigger and never swallowed into "looks
fine." A flaky check can no longer buy itself a rebuild. `repairIndex()`'s
per-family receipt also gains `missing` (an exact count plus a capped id sample),
`rebuilt` (a full rebuild ran, vs. an incremental heal), and `reason`.
- **The named health report; reads refuse instead of rebuilding.** Any index
provider may now expose a synchronous, O(1) `healthReport()` — composed from the
provider's own exact ledgers, never a sample — and this is the one signal
Brainy's read gate trusts. The first-query lazy-build path is gone: `brain.init()`
now runs every needed rebuild to completion before it returns, always, regardless
of dataset size. A read that lands on a provider whose health report says it
isn't serving throws a typed error instead of triggering a rebuild mid-query —
`GraphIndexNotReadyError`, `MetadataIndexNotReadyError`, or
`VectorIndexNotReadyError` (all exported from `@soulcraft/brainy`), naming the
reasons. `repairIndex({ rebuild: ['metadata' | 'graph' | 'vector'] | 'all' })` is
the new explicit operator door: it rebuilds the named family unconditionally, no
health check consulted — reach for it when you have independent reason to
distrust a family regardless of what it self-reports. Bare `repairIndex()` is
unchanged in spirit: report-driven, heals only what its own checks say needs it.
- New concept doc: [Index Health](docs/concepts/index-health.md) walks the whole
story from a consumer's side — degraded-but-serving vs. not-ready, what
`repairIndex()` checks and heals per family, what `suspect` counts mean.
**Nothing to change to adopt this.** No API removed, no signature narrowed —
`repairIndex()` gains an optional options bag and its return value gains fields,
both additive. The honest notes: if your code ever relied on a `find()` against a
cold/not-yet-built index quietly triggering a rebuild and returning results a beat
later, that behavior is gone — it now throws one of the three typed
`*NotReadyError` classes instead (catch them if you need to distinguish "not ready
yet" from "no results"). And `disableAutoRebuild: true` no longer defers index
construction to the first query — a needed rebuild always runs at `open()` now;
the flag has no effect on timing. Full manual control still lives in
`repairIndex({ rebuild: [...] })`.
- **Crash-reopen catchup.** After an unclean shutdown, the metadata index now
folds the exact fact window it missed — `find()` serves every acked write on
reopen, closing the gap where canonical reads and counts recovered a
crash-window write but the index kept serving its pre-crash state until the
next full rebuild. Related root-cause fixed alongside: `close()` never
stamped the index watermarks (only `flush()` did), so a close without a
prior flush caused a needless full rescan verdict on the next open.
- **Relation rows are live in the metadata index.** Previously verb rows
entered the metadata index only during a rebuild — so a rebuilt store's
relation postings went stale from the first `relate()` after it. Relations
are now posted and retracted on the live write path (relate / unrelate /
updateRelation / remove's cascade, and their `transact()` forms), in the
same commit as the graph leg.
- **The metadata rebuild is online.** `rebuild()` for the metadata family no
longer clears and rebuilds in place (reads went empty for the duration): it
builds a complete replacement beside the serving index, mirrors concurrent
writes to both, swaps atomically, and persists once after the swap. Reads
never observe a partial index. `repairIndex({ rebuild: ['metadata'] })` uses
it automatically.
- **A broken accelerator install can never read as "not installed."** The
auto-detection free pass now requires the resolution error to name the
accelerator package itself, exactly — a missing platform-binary sibling
package, an inner file path, or a dependency failure is a broken install and
`init()` throws loudly. And a plugin that declines activation is narrated on
the always-on log channel, so `silent: true` can no longer hide a fallback
to the default engines.
---
## v10.3.1 — 2026-08-18 (the fold that behaves) ## v10.3.1 — 2026-08-18 (the fold that behaves)
Three recovery cures from one production first-boot incident (a brain's first Three recovery cures from one production first-boot incident (a brain's first

View file

@ -323,58 +323,24 @@ Only the graph adjacency index carries a committed scale assertion:
- ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer - ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer
- ✅ **Zero Stubs**: Every line of code is production-ready - ✅ **Zero Stubs**: Every line of code is production-ready
## Lazy Loading Performance ## Index Build at Open (10.4+)
Brainy supports two initialization modes for optimal performance across different use cases: As of 10.4, `brain.init()` runs every needed index rebuild to completion before
it returns — always, regardless of dataset size. There is no lazy,
first-query rebuild path: a brain either finishes opening healthy, or `init()`
fails loudly. `disableAutoRebuild` no longer defers index construction to a
first query; it has no effect on *when* a rebuild runs. Manual control over
rebuilds is `repairIndex({ rebuild: [...] })`. See
[Index Health](concepts/index-health.md) for the full read-gate contract
(providers self-report readiness via `healthReport()`; a read against a
not-serving provider throws a typed `*NotReadyError` rather than rebuilding
mid-query).
### Mode 1: Auto-Rebuild (Default) <!-- The pre-10.4 "Mode 2: Lazy Loading on First Query" section previously
documented here (disableAutoRebuild deferring index construction to the
```javascript first find() call) described a real, now-retired code path. Removed
const brain = new Brainy() rather than left to mislead; the concept doc above is the current
await brain.init() // Rebuilds indexes during init (~500ms-3s for 10K entities) contract. -->
```
**Performance:**
- Init time: 500ms-3s (depends on dataset size)
- First query: Instant (indexes already loaded)
- Use case: Traditional applications, long-running servers
### Mode 2: Lazy Loading
```javascript
const brain = new Brainy({ disableAutoRebuild: true })
await brain.init() // Returns instantly (0-10ms)
const results = await brain.find({ limit: 10 }) // First query triggers rebuild (~50-200ms)
const more = await brain.find({ limit: 100 }) // Subsequent queries instant (0ms check)
```
**Performance:**
- Init time: 0-10ms (instant)
- First query: 50-200ms (includes index rebuild for 1K-10K entities)
- Subsequent queries: 0ms check (instant)
- Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
**Concurrency Safety:**
```javascript
// 100 concurrent queries immediately after init
await brain.init()
const promises = Array.from({ length: 100 }, () =>
brain.find({ limit: 10 })
)
const results = await Promise.all(promises)
// ✅ Only 1 rebuild triggered (mutex)
// ✅ All 100 queries return correct results
// ✅ Total time: ~60ms (not 6000ms!)
```
**Use Cases for Lazy Loading:**
- **Serverless/Edge**: Minimize cold start time (0-10ms init)
- **Development**: Faster restarts during development
- **Large datasets**: Defer index loading until needed
- **Read-heavy workloads**: Writes don't wait for index rebuild
## Zero Configuration Required ## Zero Configuration Required
@ -384,10 +350,6 @@ Brainy is designed to be **smart enough to tune itself dynamically**. No configu
// That's it. Brainy handles everything. // That's it. Brainy handles everything.
const brain = new Brainy() const brain = new Brainy()
await brain.init() await brain.init()
// Or with lazy loading for serverless
const brain = new Brainy({ disableAutoRebuild: true })
await brain.init() // Instant (0-10ms)
``` ```
### Automatic Self-Tuning ### Automatic Self-Tuning
@ -395,7 +357,6 @@ await brain.init() // Instant (0-10ms)
- **Metadata Index**: Auto-builds sorted indices for range queries on first use - **Metadata Index**: Auto-builds sorted indices for range queries on first use
- **Graph Index**: Auto-flushes every 30 seconds - **Graph Index**: Auto-flushes every 30 seconds
- **Default Tuning**: Research-based vector index defaults - **Default Tuning**: Research-based vector index defaults
- **Lazy Loading**: Indices built only when needed
- **Cache Management**: LRU caches with TTL - **Cache Management**: LRU caches with TTL
### Intelligent Defaults ### Intelligent Defaults

View file

@ -10,7 +10,7 @@ next:
- guides/storage-adapters - guides/storage-adapters
--- ---
# Plugin Development Guide # Plugin System
Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer. Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer.
@ -200,15 +200,30 @@ members so a warm reopen never pays a redundant rebuild-from-canonical:
- **`init?(): Promise<void>`** — eager cold-load. Brainy awaits it once during - **`init?(): Promise<void>`** — eager cold-load. Brainy awaits it once during
`brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first) `brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first)
and **before the rebuild gate**. and **before the rebuild gate**.
- **`isReady?(): boolean`** — honest durability signal. `true` ⇔ the persisted index is - **`healthReport?(): HealthReport`** — the PREFERRED signal (10.4+). A named,
loaded (or cheaply demand-loadable) and consistent with what was last persisted. When synchronous, O(1) verdict derived from the provider's own exact ledgers — never a
exposed, the rebuild gate defers to this signal **instead of** the `size() === 0` / sample, never I/O, must never throw for a well-formed provider. Brainy's read gate
`totalEntries === 0` heuristics — a disk-native index may report 0 resident entries (`assessProviderHealth()`) reads this INSTEAD of `isReady()` / size heuristics when
while fully durable. Never return `true` if the durable state failed to load: the present: `serving: false` refuses the read with a typed `*NotReadyError` rather than
signal is honest in both directions, and a not-ready provider gets its rebuild even triggering a rebuild — a read never starts a store walk. `healthy` marks every
when `size() > 0`. *verified* invariant holding; a family named in `unledgered` counts as neither
healthy nor broken. See `HealthReport` / `LedgerInvariantResult` /
`InvariantSource` in `src/plugin.ts`, and
[Index Health](concepts/index-health.md) for the consumer-facing story.
- **`isReady?(): boolean`** — honest durability signal, the fallback when
`healthReport()` is absent. `true` ⇔ the persisted index is loaded (or cheaply
demand-loadable) and consistent with what was last persisted. When exposed, the
gate defers to this signal **instead of** the `size() === 0` / `totalEntries === 0`
heuristics — a disk-native index may report 0 resident entries while fully durable.
Never return `true` if the durable state failed to load: the signal is honest in
both directions, and a not-ready provider gets its rebuild even when `size() > 0`.
- **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background - **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background
migration); brainy skips its rebuild entirely. migration); brainy skips its rebuild entirely.
- **`validateInvariants?(): Promise<ProviderInvariantReport>`** — the async DEEP
diagnostic (full scans allowed), distinct from the bounded, sync `healthReport()`.
Must never throw — a failure is `healthy: false` data, not an exception; a provider
that throws anyway is read as a loud, unverified failure (never as "healthy") by
every caller, never silently retried into a rebuild.
Providers that implement none of these keep the size/count heuristics — correct for Providers that implement none of these keep the size/count heuristics — correct for
engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index). engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index).

View file

@ -1451,6 +1451,34 @@ const count = await brain.getVerbCount()
--- ---
### The canonical count ledger (`StorageAdapter.getCanonicalCounts()`)
An OPTIONAL method on the `StorageAdapter` interface (implemented by both
built-in adapters), not a method on `Brainy` itself — relevant if you're
writing a custom storage adapter or composing a provider's own
`healthReport()`. O(1), no I/O. Per family (`nouns`/`verbs`):
```typescript
interface CanonicalCounts {
nouns: { counted: number; all: number }
verbs: { counted: number; all: number }
suspect: boolean
}
```
- `counted` mirrors `getNounCount()` / `getVerbCount()` (public + internal tiers).
- `all` is the ALL-visibility scalar — every tier, including system/internal
records — the denominator a derived index's own coverage math is measured
against.
- `suspect` is `true` when an unprovable delete has left `all` unverified since
the last recount; `brain.repairIndex()` clears it with a real canonical walk.
Adapters without the ledger omit the method; treat absence as "no
denominator," never as zero. See
**[Index Health](../concepts/index-health.md)** for the full story.
---
### Subtype & facet APIs ### Subtype & facet APIs
Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**. Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
@ -1831,6 +1859,104 @@ const semanticOnly = await brain.getStats({ excludeVFS: true })
--- ---
### `repairIndex(options?)``Promise<RepairReport>`
The ceremony door for index repair. Bare `repairIndex()` is report-driven: it
prunes orphaned containers, recomputes count rollups, reconciles VFS
containment, and rebuilds only a derived-index family whose own health check
asks for it. Pass `options.rebuild` to force one or more families to rebuild
UNCONDITIONALLY — no health check is consulted — when an operator has
independent reason to reconcile a family regardless of what it self-reports.
```typescript
// Report-driven: only heals what actually needs it
const report = await brain.repairIndex()
console.log(report.healedTotal, report.families)
// Explicit: force the graph adjacency to rebuild from canonical, unconditionally
await brain.repairIndex({ rebuild: ['graph'] })
// Explicit: force all three derived indexes to rebuild
await brain.repairIndex({ rebuild: 'all' })
```
**`RepairReport`:**
- `families: RepairFamilyReport[]` — one row per family checked
- `healedTotal: number` — items healed across every family
- `durationMs: number`
**`RepairFamilyReport`** (one row):
- `family: string` — e.g. `'orphaned-containers'`, `'count-rollups'`,
`'vfs-containment'`, `'metadata-corruption'`, `'provider:metadata'`,
`'provider:graph'`, `'provider:vector'`
- `checked: boolean` — was this family actually examined (`false` ⇒ see `skipped`)
- `healed: number` — items re-posted/corrected in place (the incremental heal count)
- `missing?: { count: number; sample: string[] }` — exact count plus a capped id
sample when the check can name what diverged (never the full list)
- `rebuilt?: boolean` — a full generational rebuild ran (vs. an incremental heal)
- `detail?: string` / `reason?: string` — narration
- `skipped?: string` — why the family wasn't checked
Full walkthrough — what each family checks, degraded-but-serving vs. not-ready,
and what `suspect` counts mean — in
**[Index Health](../concepts/index-health.md)**.
---
### Index readiness: typed errors, `healthReport()`, `disableAutoRebuild`
Every derived-index provider (vector, graph, metadata) may expose a named,
synchronous, O(1) `healthReport()` composed from its own exact ledgers — the
signal Brainy's read gate trusts over sampling or size heuristics. `init()`
brings every provider to serving before it returns; there is no first-query
lazy-rebuild path. A read that reaches a provider whose health report says it
isn't serving throws instead of rebuilding mid-query:
| Error | Thrown by | Meaning |
|---|---|---|
| `GraphIndexNotReadyError` | `find({ connected })`, `neighbors()`, `related()` | Graph adjacency isn't serving |
| `MetadataIndexNotReadyError` | `find({ where })` | Metadata/field index isn't serving |
| `VectorIndexNotReadyError` | `find({ query })`, `similar()` | Vector index isn't serving |
All three are exported from `@soulcraft/brainy`. Catch them to distinguish
"index not ready" from a genuine empty result:
```typescript
import { MetadataIndexNotReadyError } from '@soulcraft/brainy'
try {
const rows = await brain.find({ where: { status: 'active' } })
} catch (err) {
if (err instanceof MetadataIndexNotReadyError) {
// reconcile: await brain.repairIndex(), then retry
} else {
throw err
}
}
```
**`disableAutoRebuild`** no longer defers index construction to the first
query. A needed rebuild always runs at `open()`, regardless of this flag or
dataset size; the flag has no effect on *when* a rebuild runs. Full manual
control lives in `repairIndex({ rebuild: [...] })`, above.
### `validateIndexConsistency()``Promise<...>`
The deep, async diagnostic counterpart to `healthReport()` — safe to run on a
live brain, but does more work (a provider's `validateInvariants()` may run a
full scan, not just read a ledger). Aggregates the JS metadata index's own
consistency check with every derived-index provider's invariant report.
```typescript
const validation = await brain.validateIndexConsistency()
if (!validation.healthy) {
console.log(validation.recommendation) // what to run, e.g. repairIndex()
console.log(validation.providers) // each provider's own invariant report, when exposed
}
```
---
## Lifecycle ## Lifecycle
### Initialization ### Initialization

View file

@ -723,6 +723,14 @@ async stats(): Promise<Statistics> {
### 5. Index Rebuilding (Lazy Loading Support) ### 5. Index Rebuilding (Lazy Loading Support)
> **Stale as of 10.4 — "Mode 2: Lazy Loading on First Query" below is
> RETIRED.** `disableAutoRebuild` no longer defers index construction to a
> first query; `brain.init()` now runs every needed rebuild to completion
> before it returns, unconditionally, and a read against a not-serving
> provider throws a typed `*NotReadyError` instead of rebuilding mid-query.
> See `docs/concepts/index-health.md` for the current contract. Left below
> as historical background on the rebuild mechanics.
**Two modes of index loading:** **Two modes of index loading:**
#### Mode 1: Auto-Rebuild on init() (default) #### Mode 1: Auto-Rebuild on init() (default)

View file

@ -1,5 +1,15 @@
# Initialization and Rebuild Processes # Initialization and Rebuild Processes
> **Stale as of 10.4 — "Mode 2: Lazy Loading on First Query" below is RETIRED.**
> `disableAutoRebuild` no longer defers index construction to a first query;
> `brain.init()` now runs every needed rebuild to completion before it
> returns, unconditionally. A read against a not-serving provider throws a
> typed `*NotReadyError` instead of rebuilding mid-query. See
> `docs/concepts/index-health.md` for the current contract; this document's
> line-number references to `src/brainy.ts` also predate the file's current
> size and are unreliable. Left as historical background on the rebuild
> mechanics, not as a current API description.
This document explains how Brainy's four indexes (MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage. This document explains how Brainy's four indexes (MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
## Core Principle: All Indexes Are Disk-Based ## Core Principle: All Indexes Are Disk-Based

View file

@ -0,0 +1,204 @@
---
title: Index Health
slug: concepts/index-health
public: true
category: concepts
template: concept
order: 8
description: How Brainy knows whether a derived index can be trusted — exact accounting instead of sampling, the named health report, degraded-but-serving vs. not-ready, and what repairIndex() checks, heals, and rebuilds.
next:
- concepts/generation-fact-log
- guides/inspection
---
# Index Health
Brainy keeps one **canonical** copy of every entity and relationship, and three
**derived** indexes built from it — vector, metadata, and graph — so `find()` can
answer semantically, by filter, and by traversal without re-deriving the answer from
scratch on every query. A derived index is a cache with a serving structure: it can
be present but stale, present but only partially loaded, or fully out of sync with
canonical after a crash. This page is about how Brainy decides whether to trust one,
what it does when it can't, and how you reconcile the two.
## Exact accounting instead of sampling
Older health checks worked by inference: does `size()` return something greater
than zero, does a spot-check on one known item come back correct. Both are proxies.
A cold index can report a nonzero count while its actual serving structure never
loaded, and a spot-check only proves the one item it happened to ask about.
Every derived-index provider may now expose a named, synchronous, O(1)
`healthReport()` — composed from the provider's own **exact ledgers** (real counters
it already maintains on the write path), never a sample or a walk. This is the one
signal Brainy's read gate consults. A provider that doesn't yet expose one falls
back to an honest `isReady()` boolean, and finally to a size heuristic for engines
with neither — but wherever a `healthReport()` exists, it wins.
Underneath, storage itself keeps an analogous **canonical count ledger**: a
`counted` scalar (the user-facing total — what `getNounCount()` / `getVerbCount()`
return) and an `all` scalar (every tier, including internal records a derived
index's own coverage math needs to compare against). This is the real denominator
a provider's `healthReport()` measures itself by, rather than a total that can only
ever ratchet upward. See [What `suspect` counts mean](#what-suspect-counts-mean)
below for the one case that ledger can't stay exact through on its own.
## The named report
A `HealthReport` carries, per provider (`'vector'` / `'graph'` / `'metadata'`):
- **`healthy`** — `true` iff every *verified* invariant holds. An invariant whose
family has no ledger yet is `unledgered`, never counted either way — unknown,
not passing.
- **`serving`** — can this provider answer a query right now. A failing invariant
graded `heal: 'repair'` or `heal: 'none'` still leaves `serving: true` — this is
**degraded-but-serving**: something is off (say, a stale rollup on an
`employee` record's relationship count) but reads keep working. Only a failure
graded `heal: 'rebuild'` flips `serving` to `false`**not-ready** — because the
provider itself is telling you its serving structure cannot answer correctly.
- **`invariants`** — each checked condition, with its provenance
(`source: 'ledger'` — an exact count; `'deep'` — a full scan, diagnostic-only;
`'unledgered'` — not yet tracked) and, for a failing one, an exact `missing`
count plus a capped sample of the affected ids — a verdict, never a dump.
- **`generation`** — bumps on every ledger mutation and rebuild, so a caller can
cache a verdict per generation instead of re-deriving it.
The distinction that matters day to day: `healthy: false` can be entirely benign —
a maintenance window, a divergence `repairIndex()` will clean up on its own
schedule. `serving: false` is not benign. It means this provider is refusing to
answer, on its own word, right now.
## Reads refuse — they never rebuild
A query that reaches a not-serving provider does not trigger a rebuild from inside
the read. Brainy retired that path deliberately: a rebuild kicked off by an ordinary
`find({ where: { status: 'active' } })` call is a dark, unpredictable cost hiding
behind a request that looks like a cheap read. Instead, the read throws a typed,
catchable error naming the reason:
| Error | Thrown when | Meaning |
|---|---|---|
| `GraphIndexNotReadyError` | `find({ connected })`, `neighbors()`, `related()` | The graph adjacency index isn't serving — traversal would otherwise return `[]` indistinguishable from "no relationships" |
| `MetadataIndexNotReadyError` | `find({ where })` | The metadata/field index isn't serving — a filtered read would otherwise return `[]` indistinguishable from "no matches" |
| `VectorIndexNotReadyError` | `find({ query })`, `similar()` | The vector index isn't serving — a semantic search would otherwise return `[]` indistinguishable from "nothing similar" |
All three are exported from `@soulcraft/brainy`. Catch them where your application
needs to distinguish "this index isn't ready yet" from "there's genuinely nothing
here" — a health dashboard, a retry policy, an operator alert. The fix is always
the same: reconcile the index, either by reopening the brain (which brings every
provider to serving before `init()` returns — see the next section) or by calling
`repairIndex()` explicitly.
```typescript
try {
const active = await brain.find({ where: { status: 'active' } })
} catch (err) {
if (err instanceof MetadataIndexNotReadyError) {
// not a "no results" — the index itself refused; alert or retry after repair
} else {
throw err
}
}
```
### Rebuilds happen at open, not on first query
`brain.init()` runs every needed rebuild to completion **before it returns**,
unconditionally, regardless of dataset size. There is no lazy, first-query
rebuild path anymore — a brain either finishes opening healthy, or it fails
open loudly. `disableAutoRebuild: true` no longer defers index construction to
the first query: it has no effect on *when* a needed rebuild runs. Full manual
control over rebuilds is `repairIndex({ rebuild: [...] })` (below), not this flag.
## `repairIndex()` — checking and healing
Bare `repairIndex()` is **report-driven**: it only heals what its own checks say
actually needs it, and it always returns a full per-family receipt.
```typescript
const report = await brain.repairIndex()
report.healedTotal // total items healed across every family
report.durationMs
report.families // one row per family checked
```
Each `RepairFamilyReport` row names what happened:
- **`checked`** — was this family actually examined (`false` means skipped —
see `skipped` for why).
- **`healed`** — items re-posted or corrected in place.
- **`missing`** — when the check can name what diverged: an exact `count` plus a
capped `sample` of ids.
- **`rebuilt`** — a full generational rebuild ran (as opposed to an incremental
heal).
- **`detail`** / **`reason`** / **`skipped`** — the receipt's narration; a row is
always either checked or explains why it wasn't. Nothing is silent.
On every call, bare `repairIndex()`:
1. Prunes orphaned canonical containers left by a partial delete.
2. Recomputes the count rollups from one canonical walk (unconditional — this is
also what clears a `suspect` ledger; see below).
3. Reconciles VFS containment edges, if the VFS is initialized.
4. Runs the metadata index's own corruption detection pass.
5. Consults each of the three derived-index providers' own health check and
rebuilds only a family whose failing invariant actually asks for it
(`heal: 'rebuild'`) — never a provider that reports `healthy` or a lesser
grade.
### The explicit rebuild door
`options.rebuild` skips the health check and rebuilds one or more families
**unconditionally** — the operator override for when you have independent reason
to distrust a family regardless of what it self-reports (a suspicious deploy, a
storage-layer incident, a support ticket that doesn't match what the health report
says):
```typescript
// Force the graph adjacency to rebuild from canonical, no invariant consulted
await brain.repairIndex({ rebuild: ['graph'] })
// Force all three derived indexes
await brain.repairIndex({ rebuild: 'all' })
```
A family named this way is recorded with `rebuilt: true` and
`reason: 'explicit rebuild requested'`, and is skipped by the normal
health-driven pass in the same call — it was already rebuilt unconditionally.
Reach for the explicit door when you need certainty regardless of self-report;
reach for bare `repairIndex()` for routine maintenance and after any incident
where you're not sure which family (if any) needs it.
## What `suspect` counts mean
Storage's canonical count ledger increments the ALL-visibility total on every new
record and decrements it on every *proven* delete — one where the record was read,
or the caller supplied its prior image. A delete that cannot prove what it removed
existed doesn't guess: it flags the ledger `suspect` (an operator-visible
`console.warn`, narrated once per session, not once per delete) rather than risk
decrementing a total that was never incremented for that record in the first
place. This is intentionally rare — it's a defensive fallback for callers on an
unusual removal path, not a per-delete cost.
`suspect` is not directly exposed on any `Brainy` method today — it lives on the
`StorageAdapter`'s optional `getCanonicalCounts()`, primarily consulted by
`repairIndex()`'s recount step and by custom storage adapters composing their own
`healthReport()`. What matters for an application: a `suspect` ledger is not
incorrect, just *unverified since the last recount* — and `repairIndex()`'s
unconditional count-rollup step (step 2, above) recomputes the ALL scalars from a
real canonical walk on every call, clearing the flag with proof either way.
## Practical guidance
- **On a normal restart**, do nothing — `init()` brings every provider to
serving before it returns, or fails loudly.
- **On a `*NotReadyError`** from a live read, reconcile with `repairIndex()`
(report-driven is almost always sufficient) and retry.
- **After an incident** where you distrust a specific family regardless of what
it reports healthy — a storage-layer fault, a suspicious restore — use the
explicit door: `repairIndex({ rebuild: ['metadata' | 'graph' | 'vector'] })`.
- **To audit before trusting a report**, `brain.auditGraph()` walks every stored
relationship and proves (or disproves) that reads return canonical truth,
independent of what any provider self-reports — see
[Inspecting a Live Brainy](../guides/inspection.md).

4
package-lock.json generated
View file

@ -1,12 +1,12 @@
{ {
"name": "@soulcraft/brainy", "name": "@soulcraft/brainy",
"version": "10.4.0-rc.1", "version": "10.4.0-rc.2",
"lockfileVersion": 3, "lockfileVersion": 3,
"requires": true, "requires": true,
"packages": { "packages": {
"": { "": {
"name": "@soulcraft/brainy", "name": "@soulcraft/brainy",
"version": "10.4.0-rc.1", "version": "10.4.0-rc.2",
"license": "MIT", "license": "MIT",
"dependencies": { "dependencies": {
"@msgpack/msgpack": "^3.1.2", "@msgpack/msgpack": "^3.1.2",

View file

@ -1,6 +1,6 @@
{ {
"name": "@soulcraft/brainy", "name": "@soulcraft/brainy",
"version": "10.4.0-rc.1", "version": "10.4.0-rc.2",
"description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.", "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.",
"main": "dist/index.js", "main": "dist/index.js",
"module": "dist/index.js", "module": "dist/index.js",

File diff suppressed because it is too large Load diff

View file

@ -269,6 +269,10 @@ export type { FamilyStamp, StampMembers, StampVerdict } from './db/familyStamp.j
export { isVersionedIndexProvider } from './plugin.js' export { isVersionedIndexProvider } from './plugin.js'
export type { VersionedIndexProvider } from './plugin.js' export type { VersionedIndexProvider } from './plugin.js'
export type { ProviderInvariantReport, InvariantResult, InvariantHeal } from './plugin.js' export type { ProviderInvariantReport, InvariantResult, InvariantHeal } from './plugin.js'
// The named, synchronous, O(1) health-report contract (the read gate's ONLY
// source of truth for "can I serve right now") — see HealthReport's
// derivation laws in plugin.ts.
export type { HealthReport, LedgerInvariantResult, InvariantSource } from './plugin.js'
// Optional provider self-report of outstanding background maintenance work // Optional provider self-report of outstanding background maintenance work
// (compaction, deferred writes, etc.) — the payload type for // (compaction, deferred writes, etc.) — the payload type for
// brain.maintenanceDebt(). See the measure-only-what-you-track contract on // brain.maintenanceDebt(). See the measure-only-what-you-track contract on
@ -385,7 +389,10 @@ import type {
HNSWVerb, HNSWVerb,
HNSWConfig, HNSWConfig,
StorageAdapter, StorageAdapter,
DerivedFamilyDeclaration DerivedFamilyDeclaration,
// The canonical count ledger a storage adapter maintains (counted + ALL-visibility
// scalars per family, the coverage-ledger denominators) — see StorageAdapter.getCanonicalCounts.
CanonicalCounts
} from './coreTypes.js' } from './coreTypes.js'
// Export vector index implementation (the JS HNSW path) // Export vector index implementation (the JS HNSW path)

View file

@ -9,6 +9,7 @@
* registered manually via `brain.use()` there is no implicit detection. * registered manually via `brain.use()` there is no implicit detection.
*/ */
import { prodLog } from './utils/logger.js'
import type { import type {
StorageAdapter, StorageAdapter,
Vector, Vector,
@ -171,6 +172,66 @@ export interface ProviderInvariantReport {
durationMs: number durationMs: number
} }
/**
* @description Where a {@link LedgerInvariantResult} verdict came from:
* - `'ledger'` decided from an exact, durable ledger (a real count, not a sample).
* - `'deep'` decided by a full/expensive scan (the `validateInvariants()` diagnostic path only).
* - `'unledgered'` this family has no ledger yet; the verdict is UNKNOWN, never healthy and never broken.
*/
export type InvariantSource = 'ledger' | 'deep' | 'unledgered'
/**
* @description One invariant verdict inside a {@link HealthReport}. Extends
* {@link InvariantResult} with the provenance of the verdict ({@link InvariantSource})
* and, for a failing set-membership invariant, an exact count plus a capped sample
* of the diverging ids a VERDICT, never a dump. `sample` MUST be capped at 16 ids;
* `count` is the exact number even when `sample` is truncated.
*/
export interface LedgerInvariantResult extends InvariantResult {
/** Provenance of this verdict — see {@link InvariantSource}. */
source: InvariantSource
/** Exact count of diverging/missing items plus a capped (≤16 ids) sample. Present only on a failing set-membership invariant. */
missing?: { count: number; sample: string[] }
}
/**
* @description The NAMED, SYNCHRONOUS, O(1) health report a provider exposes via
* {@link MetadataIndexProvider.healthReport} / {@link GraphIndexProvider.healthReport} /
* {@link VectorIndexProvider.healthReport}. This is the read gate's ONLY source of
* truth for "can I serve right now" it replaces sampled self-probes and the
* unnamed `isReady()` latch with an exact, ledger-derived verdict.
*
* Derivation laws (a provider MUST honor these; brainy's read gate assumes them):
* - `healthy` = every VERIFIED invariant in {@link invariants} holds. An invariant
* whose family is named in {@link unledgered} is NEVER counted toward `healthy`
* either way it is unknown, not passing.
* - `serving` = no verified invariant in {@link invariants} FAILS with `heal: 'rebuild'`.
* A failure with `heal: 'repair'` or `heal: 'none'` is degraded-but-serving
* `serving` stays `true`. Only a `'rebuild'`-grade failure makes `serving` `false`.
* - `validateInvariants()` remains the async DEEP diagnostic (full scans allowed,
* `source: 'deep'` results); `healthReport()` MUST be synchronous, O(1) from
* exact ledgers/counters, and MUST NOT throw for a well-formed provider a
* provider that cannot produce a safe verdict reports it as a failing invariant,
* it does not throw (a throw is read by the gate as a CONTRACT VIOLATION, not as
* "unknown").
*/
export interface HealthReport extends ProviderInvariantReport {
/**
* Monotonic per provider: bumps on every ledger mutation and every rebuild
* boundary. Consumers (the read gate's narration dedup, external callers) may
* cache a verdict per generation.
*/
generation: number
/** Each checked invariant, with provenance — see {@link LedgerInvariantResult}. */
invariants: LedgerInvariantResult[]
/**
* Families with no ledger yet. NAMED here so an operator can see what is not
* yet tracked NEVER counted as healthy (they are not verified) and NEVER
* counted as broken (there is nothing to fail).
*/
unledgered: string[]
}
/** /**
* @description A provider's self-report of its own outstanding background * @description A provider's self-report of its own outstanding background
* maintenance work (compaction, deferred writes, a build-newverifyswap in * maintenance work (compaction, deferred writes, a build-newverifyswap in
@ -266,6 +327,20 @@ export interface MetadataIndexProvider {
*/ */
validateInvariants?(): Promise<ProviderInvariantReport> validateInvariants?(): Promise<ProviderInvariantReport>
/**
* @description OPTIONAL. The named, SYNCHRONOUS, O(1) health verdict this
* provider derives from its own exact ledgers see {@link HealthReport} for
* the full derivation laws. MUST NOT perform I/O and MUST NOT throw for a
* well-formed provider (brainy treats a throw as a CONTRACT VIOLATION, never
* as "unknown"). When present, brainy's read gate (`assessProviderHealth()`)
* reads THIS instead of `isReady()` / size heuristics: `serving` decides
* whether reads may proceed; a `false` refuses the read loudly rather than
* triggering a rebuild. Absent the gate falls back to `isReady?()` / the
* size heuristic (this train's JS built-in providers stay on that interim
* path).
*/
healthReport?(): HealthReport
/** /**
* @description OPTIONAL. A native provider returns true from the moment its * @description OPTIONAL. A native provider returns true from the moment its
* `init()` detects a large epoch-drift until its background * `init()` detects a large epoch-drift until its background
@ -462,6 +537,20 @@ export interface GraphIndexProvider {
*/ */
validateInvariants?(): Promise<ProviderInvariantReport> validateInvariants?(): Promise<ProviderInvariantReport>
/**
* @description OPTIONAL. The named, SYNCHRONOUS, O(1) health verdict this
* provider derives from its own exact ledgers see {@link HealthReport} for
* the full derivation laws. MUST NOT perform I/O and MUST NOT throw for a
* well-formed provider (brainy treats a throw as a CONTRACT VIOLATION, never
* as "unknown"). When present, brainy's read gate (`assessProviderHealth()`)
* reads THIS instead of `isReady()` / size heuristics: `serving` decides
* whether reads may proceed; a `false` refuses the read loudly rather than
* triggering a rebuild. Absent the gate falls back to `isReady?()` / the
* size heuristic (this train's JS built-in providers stay on that interim
* path).
*/
healthReport?(): HealthReport
/** /**
* @description OPTIONAL eager cold-load. Called once during brain init AFTER * @description OPTIONAL eager cold-load. Called once during brain init AFTER
* the metadata provider's `init()` (so the id-mapper is hydrated; a native int * the metadata provider's `init()` (so the id-mapper is hydrated; a native int
@ -1225,6 +1314,20 @@ export interface VectorIndexProvider {
*/ */
validateInvariants?(): Promise<ProviderInvariantReport> validateInvariants?(): Promise<ProviderInvariantReport>
/**
* @description OPTIONAL. The named, SYNCHRONOUS, O(1) health verdict this
* provider derives from its own exact ledgers see {@link HealthReport} for
* the full derivation laws. MUST NOT perform I/O and MUST NOT throw for a
* well-formed provider (brainy treats a throw as a CONTRACT VIOLATION, never
* as "unknown"). When present, brainy's read gate (`assessProviderHealth()`)
* reads THIS instead of `isReady()` / size heuristics: `serving` decides
* whether reads may proceed; a `false` refuses the read loudly rather than
* triggering a rebuild. Absent the gate falls back to `isReady?()` / the
* size heuristic (this train's JS built-in providers stay on that interim
* path).
*/
healthReport?(): HealthReport
/** /**
* @description OPTIONAL. A native provider returns true from the moment its * @description OPTIONAL. A native provider returns true from the moment its
* `init()` detects a large epoch-drift until its background * `init()` detects a large epoch-drift until its background
@ -1472,9 +1575,13 @@ export class PluginRegistry {
this.activated.add(name) this.activated.add(name)
activated.push(name) activated.push(name)
} else { } else {
// Documented graceful decline (activate() → false). Surface it loudly so // Documented graceful decline (activate() → false). Surface it on the
// a silent degrade to the default engine never goes unnoticed. // ALWAYS-ON channel: `silent: true` patches console, and a declined
console.warn( // accelerator warned into a patched console is a silent degrade to the
// default engines — the exact invisible-fallback class this registry
// exists to prevent (a production storm ran the WASM engine for 90s
// behind one suppressed warn).
prodLog.warn(
`[brainy] Plugin "${name}" declined activation (activate() returned false); ` + `[brainy] Plugin "${name}" declined activation (activate() returned false); ` +
`the default engine is in use for its providers.` `the default engine is in use for its providers.`
) )

View file

@ -154,6 +154,21 @@ export function resolveFilesystemRoot(
) { ) {
throwRemovedStorageKey('fileSystemStorage.path') throwRemovedStorageKey('fileSystemStorage.path')
} }
// A nested `config` object carrying a path-shaped key is the same hazard in
// a shape nobody ever supported: it used to fall through SILENTLY to the
// shared default root — every instance writing one directory while its
// caller believed each had its own. (Found live: an integration test's
// brains shared one store across a whole single-process run and a health
// probe refused on the foreign edges it sampled.) Loud, with the rename.
const nested = (config as Record<string, unknown>).config
if (nested && typeof nested === 'object') {
const pathish = ['path', 'baseDir', 'rootDirectory', 'rootDir', 'dir', 'directory']
const hit = pathish.find(
(k) => typeof (nested as Record<string, unknown>)[k] === 'string' &&
((nested as Record<string, unknown>)[k] as string).length > 0
)
if (hit) throwRemovedStorageKey(`config.${hit}`)
}
// 3. Zero-config default. A `type: 'filesystem'` with no path lands here // 3. Zero-config default. A `type: 'filesystem'` with no path lands here
// intentionally ("persist, default location"). // intentionally ("persist, default location").

View file

@ -1816,10 +1816,16 @@ export interface BrainyConfig {
| StorageAdapter | StorageAdapter
/** /**
* Disable the automatic index rebuild check during `init()`. By default * RE-MEANT (the health-gate contract): `init()` (open) always verifies the
* Brainy auto-decides from dataset size: small datasets rebuild missing * durable generation of every derived index, and a needed rebuild ALWAYS
* indexes inline, large datasets rebuild lazily on first query. Set `true` * runs at open it is never deferred to the first read, regardless of
* only when an operator wants full manual control via `repairIndex()`. * dataset size or this flag. There is no first-query lazy-build path
* anymore: a read that finds a provider not serving throws a typed
* `*NotReadyError` rather than building anything (see
* `assessProviderHealth` / the read gate in `brainy.ts`). Setting this
* `true` no longer defers index construction to the first query it has
* no effect on WHEN a needed rebuild runs. Full manual control over
* rebuilds remains available via `repairIndex({ rebuild: [...] })`.
*/ */
disableAutoRebuild?: boolean disableAutoRebuild?: boolean

View file

@ -13,13 +13,28 @@
* `size()` or `isInitialized`. When `isReady()` is absent, callers must fall back * `size()` or `isInitialized`. When `isReady()` is absent, callers must fall back
* to a KNOWN-ITEM PROBE (a real search/lookup that must return a known-present * to a KNOWN-ITEM PROBE (a real search/lookup that must return a known-present
* datum) before trusting an empty result never a `size()` proxy. * datum) before trusting an empty result never a `size()` proxy.
*
* {@link assessProviderHealth} is the NEWER, PREFERRED authority: it reads a
* provider's NAMED, synchronous, O(1) {@link import('../plugin.js').HealthReport}
* when one is exposed, and falls back to this file's `isReady()` classifier only
* when the provider does not (yet) expose a health report. Read paths in
* `brainy.ts` call `assessProviderHealth` exclusively `assessIndexReadiness`
* stays exported for the other call sites (`storage/baseStorage.ts`) and for the
* fallback branch inside `assessProviderHealth` itself.
*/ */
import type { HealthReport } from '../plugin.js'
/** A provider that MAY expose the honest cold-load readiness signal. */ /** A provider that MAY expose the honest cold-load readiness signal. */
export interface MaybeReadyProvider { export interface MaybeReadyProvider {
isReady?: () => boolean isReady?: () => boolean
} }
/** A provider that MAY expose the named, synchronous, O(1) health report. */
export interface MaybeHealthReportingProvider {
healthReport?: () => HealthReport
}
/** Three-valued honest-readiness verdict. */ /** Three-valued honest-readiness verdict. */
export type IndexReadiness = 'ready' | 'not-ready' | 'unknown' export type IndexReadiness = 'ready' | 'not-ready' | 'unknown'
@ -36,3 +51,105 @@ export function assessIndexReadiness(provider: unknown): IndexReadiness {
if (p == null || typeof p.isReady !== 'function') return 'unknown' if (p == null || typeof p.isReady !== 'function') return 'unknown'
return p.isReady() ? 'ready' : 'not-ready' return p.isReady() ? 'ready' : 'not-ready'
} }
/**
* @description Which signal {@link assessProviderHealth} actually consulted to
* produce its verdict surfaced so callers can narrate (and tests can pin) how
* a provider was judged, not just what the judgment was.
* - `'health-report'` the provider's `healthReport()` was called (the authority).
* - `'is-ready'` no `healthReport()`; fell back to the provider's `isReady()`.
* - `'size-heuristic'` no `healthReport()` and no `isReady()`; caller must keep its own size-based heuristic.
* - `'none'` there was no provider to assess (`null`/`undefined`).
*/
export type ProviderHealthVia = 'health-report' | 'is-ready' | 'size-heuristic' | 'none'
/** The result of {@link assessProviderHealth}. */
export interface ProviderHealthAssessment {
/** The honest readiness verdict — see {@link IndexReadiness}. */
readiness: IndexReadiness
/** The provider's raw {@link HealthReport}, when one was obtained; `null` otherwise. */
report: HealthReport | null
/** Which signal produced the verdict — see {@link ProviderHealthVia}. */
via: ProviderHealthVia
/** Human-readable reasons: named failing invariants (with `heal`), unledgered families, or the fallback-path explanation. Empty when the provider is healthy and ready. */
reasons: string[]
}
/**
* @description THE read-gate authority. Prefers a provider's NAMED,
* synchronous, O(1) {@link HealthReport} over the older `isReady()` / size
* heuristics; falls back to {@link assessIndexReadiness}'s semantics only when
* a provider does not (yet) expose `healthReport()`.
*
* Derivation:
* - `healthReport()` present call it (wrapped in try/catch). A THROW is a
* CONTRACT VIOLATION, not "unknown": returns `readiness: 'not-ready'`,
* `via: 'health-report'`, and a reason naming the throw never swallowed
* into `'unknown'`.
* - Otherwise `readiness = report.serving ? 'ready' : 'not-ready'`; `reasons`
* names every invariant with `holds: false` (with its `heal`), plus an
* `unledgered: [...]` line when {@link HealthReport.unledgered} is non-empty.
* UNLEDGERED IS UNKNOWN: an unledgered family never flips a serving provider
* to not-ready, and never flips a not-serving provider to ready `serving`
* is always the provider's own verdict, verbatim.
* - No `healthReport()` fall back to {@link assessIndexReadiness}'s semantics:
* `via: 'is-ready'` when `isReady()` exists, `via: 'size-heuristic'` when
* neither hook exists (caller must keep its own size-based heuristic),
* `via: 'none'` when there is no provider at all.
* @param provider - Any index provider (vector / graph / metadata) or `null`/`undefined`.
*/
export function assessProviderHealth(provider: unknown): ProviderHealthAssessment {
const p = provider as (MaybeHealthReportingProvider & MaybeReadyProvider) | null | undefined
if (p == null) {
return { readiness: 'unknown', report: null, via: 'none', reasons: ['no provider to assess'] }
}
if (typeof p.healthReport === 'function') {
let report: HealthReport
try {
report = p.healthReport()
} catch (err) {
const message = err instanceof Error ? err.message : String(err)
return {
readiness: 'not-ready',
report: null,
via: 'health-report',
reasons: [`healthReport() threw: ${message} — a health-report throw is a contract violation, never read as healthy`]
}
}
const reasons: string[] = []
for (const invariant of report.invariants) {
if (!invariant.holds) {
reasons.push(`${invariant.name} (heal:${invariant.heal}): ${invariant.detail}`)
}
}
if (report.unledgered.length > 0) {
reasons.push(`unledgered: ${report.unledgered.join(', ')}`)
}
return {
readiness: report.serving ? 'ready' : 'not-ready',
report,
via: 'health-report',
reasons
}
}
const readiness = assessIndexReadiness(p)
if (readiness === 'unknown') {
return {
readiness,
report: null,
via: 'size-heuristic',
reasons: ['provider exposes neither healthReport() nor isReady() — falling back to the size heuristic']
}
}
return {
readiness,
report: null,
via: 'is-ready',
reasons: readiness === 'not-ready' ? ['isReady() returned false'] : []
}
}

View file

@ -20,6 +20,7 @@ import {
type WatermarkVerdict, type WatermarkVerdict,
type WatermarkVerdictResult type WatermarkVerdictResult
} from './projectionWatermark.js' } from './projectionWatermark.js'
import type { FactScanHandle } from '../db/factLog.js'
import { import {
NounType, NounType,
VerbType, VerbType,
@ -77,6 +78,31 @@ export interface MetadataIndexStats {
indexSize: number // in bytes indexSize: number // in bytes
} }
/**
* @description What {@link MetadataIndexManager.applyWatermarkCatchup} did,
* for the caller's narration.
* - `'noop'` the verdict was `null`/`'adopt'`: the artifact already
* reflects committed truth. Zero index writes.
* - `'rescan'` the verdict was `'rescan'`, OR a `'catchup'` verdict was
* demoted (no window, or no fact log to scan) either way a full
* {@link MetadataIndexManager.rebuild} already ran; `reason` names why.
* - `'caught-up'` the `(from, to]` window folded successfully; the
* artifact is stamped and flushed at `to`.
*/
export interface CatchupApplyResult {
action: 'noop' | 'rescan' | 'caught-up'
/** Present on `'rescan'` — why the fold could not proceed as a catchup. */
reason?: string
/** Present on `'caught-up'` — the fact-log window that was folded. */
window?: { from: number; to: number }
/** Present on `'caught-up'` — noun ops applied (add/update/delete). */
nounsApplied?: number
/** Present on `'caught-up'` — verb ops applied (add/update/delete). */
verbsApplied?: number
/** Present on `'caught-up'` — distinct committed generations folded. */
factsApplied?: number
}
export interface MetadataIndexConfig { export interface MetadataIndexConfig {
maxIndexSize?: number // Max number of entries per field value (default: 10000) maxIndexSize?: number // Max number of entries per field value (default: 10000)
rebuildThreshold?: number // Rebuild if index is this % stale (default: 0.1) rebuildThreshold?: number // Rebuild if index is this % stale (default: 0.1)
@ -147,6 +173,52 @@ export class MetadataIndexManager implements MetadataIndexProvider {
private stampedWatermark: number | null = null private stampedWatermark: number | null = null
/** The three-way verdict computed at init; null until init runs. */ /** The three-way verdict computed at init; null until init runs. */
private loadVerdict: WatermarkVerdictResult | null = null private loadVerdict: WatermarkVerdictResult | null = null
/**
* Set only when {@link loadVerdict}.verdict is `'rescan'`: whether a
* persisted artifact existed at load (even an unstamped/unverifiable
* one) distinguishes genuine first boot (nothing here yet, routine)
* from an artifact whose watermark is unverifiable (the loud case). The
* verdict value alone doesn't carry this distinction; see {@link
* watermarkArtifactPresent}.
*/
private rescanArtifactPresent = false
/**
* @description THE BUILD-BESIDE SEAM (B3 Deliverable 3): when set (via
* {@link beginShadow}), every live `addToIndex`/`removeFromIndex` call on
* THIS instance also applies to the shadow instance so a caller building
* a fresh replacement manager beside this one (walking canonical into it)
* never misses a write that lands during the build. This is the ONE seam
* that makes build-beside possible without touching every call site: every
* existing `AddToMetadataIndexOperation`/`RemoveFromMetadataIndexOperation`
* (and the JS manager's own `rebuild()`/catchup fold) keep calling the SAME
* serving instance exactly as before; only THIS instance knows it is also
* mirroring to a shadow. Null = no build in flight (the overwhelmingly
* common case; the check costs one property read per write).
*/
private shadow: MetadataIndexManager | null = null
/**
* @description Start mirroring every `addToIndex`/`removeFromIndex` call on
* this instance to `shadow` too see {@link shadow}'s JSDoc. The caller
* owns sequencing: writes mirrored WHILE a canonical walk is populating
* `shadow` may be clobbered by the walk's own (possibly stale) reads for
* the same id; the caller closes that window with a bounded fact-log fold
* AFTER the walk (the same mechanism {@link applyWatermarkCatchup} uses)
* before treating `shadow` as authoritative.
* @param shadow - The manager to mirror writes to.
*/
beginShadow(shadow: MetadataIndexManager): void {
this.shadow = shadow
}
/**
* @description Stop mirroring writes to a shadow (see {@link beginShadow}).
* Idempotent; a no-op when no shadow is attached.
*/
endShadow(): void {
this.shadow = null
}
// Cardinality and field statistics tracking // Cardinality and field statistics tracking
private fieldStats = new Map<string, FieldStats>() private fieldStats = new Map<string, FieldStats>()
@ -1604,6 +1676,15 @@ export class MetadataIndexManager implements MetadataIndexProvider {
for (const { field } of fields) { for (const { field } of fields) {
this.metadataCache.invalidatePattern(`field_values_${field}`) this.metadataCache.invalidatePattern(`field_values_${field}`)
} }
// THE BUILD-BESIDE SEAM — see `shadow`'s JSDoc. Mirrors this write to a
// shadow manager under construction, if one is attached. `skipFlush:
// true` always: the shadow's own persistence is the build orchestrator's
// job (it flushes once, after the swap — never mid-build, to avoid
// colliding with this instance's own persisted keys).
if (this.shadow) {
await this.shadow.addToIndex(id, entityOrMetadata, true, false, generation)
}
} }
/** /**
@ -1676,6 +1757,11 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// the real commit watermark (the JS mapper ignores it). // the real commit watermark (the JS mapper ignores it).
this.idMapper.remove(id, generation) this.idMapper.remove(id, generation)
await this.idMapper.flush() await this.idMapper.flush()
// THE BUILD-BESIDE SEAM — see `shadow`'s JSDoc.
if (this.shadow) {
await this.shadow.removeFromIndex(id, metadata, generation)
}
} }
/** /**
@ -2759,8 +2845,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
* `'adopt'` (stamped == committed, zero work), `'catchup'` (stamped < * `'adopt'` (stamped == committed, zero work), `'catchup'` (stamped <
* committed; the gap from {@link watermarkGap} awaits an incremental * committed; the gap from {@link watermarkGap} awaits an incremental
* fold), `'rescan'` (unstamped or stamped above committed never * fold), `'rescan'` (unstamped or stamped above committed never
* trusted). Null until init() has run. Computed and exposed only; no * trusted). Null until init() has run. The coordinator (`Brainy.open()`)
* load behavior changes ride on it yet. * consumes this via {@link applyWatermarkCatchup} right after init.
*/ */
watermarkVerdict(): WatermarkVerdict | null { watermarkVerdict(): WatermarkVerdict | null {
return this.loadVerdict?.verdict ?? null return this.loadVerdict?.verdict ?? null
@ -2774,6 +2860,223 @@ export class MetadataIndexManager implements MetadataIndexProvider {
return this.loadVerdict?.gap ?? null return this.loadVerdict?.gap ?? null
} }
/**
* @description Meaningful only when {@link watermarkVerdict} is
* `'rescan'`: `true` when a persisted artifact existed at load (even an
* unstamped/unverifiable one real prior state, worth narrating loudly);
* `false` for a genuine first boot (nothing persisted yet a caller
* should narrate this at a routine log level, not as an alarm, even
* though the verdict value is the same `'rescan'` either way).
*/
watermarkArtifactPresent(): boolean {
return this.rescanArtifactPresent
}
/**
* @description Consume the three-way watermark verdict {@link
* watermarkVerdict} computed at init the cure for a crash-recovered
* store whose canonical reads/counts recover every acked write but whose
* metadata projection (flushed only periodically, not per-commit) keeps
* serving the pre-crash state. Call once, right after `init()`, before
* anything reads from this projection.
*
* - `null`/`'adopt'` the artifact already reflects the store's
* committed generation. Zero index writes.
* - `'catchup'` the caller-supplied `scan` (expected already opened
* over `(watermarkGap().from, watermarkGap().to]`) is folded in, ONE
* op at a time, through the SAME two legs {@link rebuild} uses (ADR-007
* A4 one mechanism, never a second hand-rolled add/update shape): a
* tombstone (`op.record === null`) retracts id-keyed (this projection
* keeps no per-record delta log, so the pre-crash metadata for that id
* if any is what a value-precise removal would need, and it isn't
* available; the same tradeoff `remove()`'s null-metadata closure
* already accepts elsewhere); an after-image retracts-then-reposts, so
* an update never leaves stale postings under the old field values. A
* fact outside the window is skipped defensively (belt: the scan is
* already opened to the window; suspenders: this loop never trusts an
* over-run). On success the artifact is stamped at `to` and flushed
* the same STAMP-AFTER-DATA door {@link flush} always writes through.
* - `'rescan'` (or a `'catchup'` verdict with no window, or no `scan` to
* fold the store hosts no fact log) the persisted artifact is
* unverifiable; this method runs the existing {@link rebuild} itself
* rather than leave the caller to notice and trigger it separately.
*
* @param scan - An open fact scan covering the catchup window (see
* {@link Brainy.scanFacts}), or `null` when none is available/needed.
* Ignored when the verdict is not `'catchup'`.
* @returns What happened see {@link CatchupApplyResult}.
*/
async applyWatermarkCatchup(scan: FactScanHandle | null): Promise<CatchupApplyResult> {
const verdict = this.watermarkVerdict()
if (verdict === null || verdict === 'adopt') return { action: 'noop' }
if (verdict === 'rescan') {
await this.rebuild()
return {
action: 'rescan',
reason: 'persisted artifact is unverifiable (unstamped, or stamped ABOVE the ' +
"store's committed generation) — never adopting unverifiable state"
}
}
// verdict === 'catchup'
const window = this.watermarkGap()
if (window === null) {
await this.rebuild()
return { action: 'rescan', reason: "'catchup' verdict exposed no window — cannot bound a fold" }
}
if (scan === null) {
await this.rebuild()
return {
action: 'rescan',
reason: `no fact log available to fold the (${window.from}, ${window.to}] catchup window`
}
}
const { nounsApplied, verbsApplied, factsApplied } = await this.foldFactWindow(scan, window.from, window.to)
this.stampWatermark(window.to)
await this.flush()
return { action: 'caught-up', window, nounsApplied, verbsApplied, factsApplied }
}
/**
* @description Fold an open fact scan's `(fromGeneration, toGeneration]`
* window into this projection, ONE op at a time, through the SAME two legs
* {@link rebuild} uses (ADR-007 A4 one mechanism, never a second
* hand-rolled add/update shape): a tombstone retracts id-keyed; an
* after-image retracts-then-reposts. THE CORE LOOP shared by {@link
* applyWatermarkCatchup} (which stamps + flushes after) and {@link
* buildBeside} (which does neither persistence is the caller's job,
* exactly once, after a swap). Never stamps, never flushes, never touches
* storage beyond what `addToIndex`/`removeFromIndex` do internally
* (skipFlush is always forced true).
* @param scan - An open fact scan.
* @param fromGeneration - Window lower bound (exclusive).
* @param toGeneration - Window upper bound (inclusive).
* @returns Counts for the caller's narration.
*/
private async foldFactWindow(
scan: FactScanHandle,
fromGeneration: number,
toGeneration: number
): Promise<{ nounsApplied: number; verbsApplied: number; factsApplied: number }> {
let nounsApplied = 0
let verbsApplied = 0
let factsApplied = 0
for await (const batch of scan.batches()) {
for (const fact of batch.facts) {
// Defensive containment: the scan is already opened to the window,
// but a fact outside it is never applied regardless.
if (fact.generation <= fromGeneration || fact.generation > toGeneration) continue
const generation = BigInt(fact.generation)
for (const op of fact.ops) {
if (op.record === null) {
// TOMBSTONE — the id-keyed removal path (no per-record delta
// log to recover the old field values from).
await this.removeFromIndex(op.id, undefined, generation)
} else {
// AFTER-IMAGE — retract any stale posting for this id, then
// repost the new shape. Covers both a fresh add (nothing to
// retract; a no-op-ish remove) and an update, through the same
// two calls.
await this.removeFromIndex(op.id, undefined, generation)
await this.indexStoredRecord(op.id, op.record.metadata, {
skipFlush: true,
deferWrites: false,
generation
})
}
if (op.kind === 'noun') nounsApplied++
else verbsApplied++
}
factsApplied++
}
}
return { nounsApplied, verbsApplied, factsApplied }
}
/**
* @description B3 Deliverable 3 the shadow-build lifecycle's init: the
* MINIMUM setup {@link buildBeside} needs, deliberately NOT the general
* {@link init} sequence. Two reasons general `init()` is unsafe for a
* build-beside shadow:
* 1. `init()` unconditionally re-initializes the id mapper from storage
* (`idMapper.init()`) safe for a FRESH mapper, but this instance is
* constructed with the CURRENTLY-SERVING manager's SHARED, already-live
* mapper (identity is shared, never a second mapper this train's own
* law). Re-running its init() would DISCARD every not-yet-flushed
* UUIDint assignment sitting in memory, breaking the live manager's
* own serving mid-build.
* 2. `init()` loads the field registry and, on a registry that's
* missing/empty while canonical has entities (exactly the shape a
* rebuild is often invoked to FIX), triggers `rebuild()` itself
* WITHOUT `inMemoryOnly`, which would touch the shared storage keys
* the live manager depends on.
* What this DOES run: the WASM roaring-bitmap library init (idempotent;
* needed before any column-store write) and the column store's OWN
* segment-manifest discovery (read-only against shared storage; needed so
* THIS instance's eventual post-swap flush continues segment numbering
* correctly instead of colliding with the retiring manager's segments).
*/
private async initForShadowBuild(): Promise<void> {
await roaringLibraryInitialize()
try {
await this.columnStore.init(this.storage, this.idMapper)
} catch (err) {
prodLog.warn('[MetadataIndex] shadow build: column store storage discovery failed:', err)
}
}
/**
* @description B3 Deliverable 3 THE ONLINE REBUILD's manager-side half:
* populate THIS instance (expected fresh/empty, constructed with the SAME
* storage + idMapper as the manager it will replace see {@link
* initForShadowBuild}) from canonical storage without ever touching the
* shared storage keys the currently-serving manager depends on no chunk
* deletion, no flush, anywhere in this call. The caller (the brain's
* rebuild-beside orchestrator) is responsible for:
* 1. Attaching this instance as a {@link beginShadow} target on the OLD
* manager BEFORE calling this, so live writes during the walk mirror
* here too (best-effort the walk below may still clobber a mirrored
* write with a stale read for the same id; the fold after the walk is
* what makes the final state authoritative, not the mirror).
* 2. Swapping its own reference to this instance once this resolves.
* 3. Calling {@link stampWatermark} + {@link flush} EXACTLY ONCE, after
* the swap this instance never persists itself.
* @param committedGenerationAtStart - The store's committed generation
* captured by the caller BEFORE this call the fold's lower bound.
* @returns The generation this instance's canonical data reflects once the
* walk + fold settle the fold's upper bound (writes committed after
* this point but before the swap only reach this instance via the live
* {@link beginShadow} mirror, so the caller re-reads the store's
* committed generation right before stamping, rather than trusting this
* return value as final).
* @throws If canonical advanced during the walk but no fact log is
* available to fold the gap never a silently incomplete shadow.
*/
async buildBeside(committedGenerationAtStart: number): Promise<number> {
await this.initForShadowBuild()
await this.rebuild({ inMemoryOnly: true })
const committedAfterWalk = this.storage.committedGeneration?.() ?? committedGenerationAtStart
if (committedAfterWalk > committedGenerationAtStart) {
const scan = this.storage.scanFacts?.({
fromGeneration: committedGenerationAtStart + 1,
toGeneration: committedAfterWalk
}) ?? null
if (scan === null) {
throw new Error(
`MetadataIndexManager.buildBeside: canonical advanced from generation ` +
`${committedGenerationAtStart} to ${committedAfterWalk} during the walk, but this ` +
`store hosts no fact log to fold the gap — refusing a silently incomplete shadow`
)
}
await this.foldFactWindow(scan, committedGenerationAtStart, committedAfterWalk)
}
return committedAfterWalk
}
/** /**
* @description Write the pending watermark stamp as a sidecar record * @description Write the pending watermark stamp as a sidecar record
* always called AFTER the data it certifies is durable. A stamp-write * always called AFTER the data it certifies is durable. A stamp-write
@ -2826,6 +3129,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
if (result.verdict === 'rescan') { if (result.verdict === 'rescan') {
const artifactPresent = this.fieldIndexes.size > 0 || stamped !== null const artifactPresent = this.fieldIndexes.size > 0 || stamped !== null
this.rescanArtifactPresent = artifactPresent
if (artifactPresent) { if (artifactPresent) {
prodLog.warn( prodLog.warn(
`[MetadataIndex] watermark verdict: RESCAN — persisted index is ` + `[MetadataIndex] watermark verdict: RESCAN — persisted index is ` +
@ -3484,13 +3788,48 @@ export class MetadataIndexManager implements MetadataIndexProvider {
} }
} }
/**
* @description Index one raw stored noun/verb record THE ONE add leg
* shared by {@link rebuild}'s canonical walk and {@link
* applyWatermarkCatchup}'s fact-log fold (ADR-007 A4: one mechanism,
* never a second hand-rolled shape). No conversion step is needed here:
* a raw stored record (`storage.getNounMetadata`/`getVerbMetadata`, or a
* fact's after-image `record.metadata`) is byte-identical both read the
* exact same canonical path and already the v2 nested-bag
* ("entity-record") shape {@link extractIndexableFields} expects.
* @param id - Entity/relationship id.
* @param storedMetadata - The raw stored metadata record.
* @param opts.skipFlush - Forwarded to {@link addToIndex}.
* @param opts.deferWrites - Forwarded to {@link addToIndex}.
* @param opts.generation - Forwarded to {@link addToIndex}.
*/
private async indexStoredRecord(
id: string,
storedMetadata: unknown,
opts: { skipFlush: boolean; deferWrites: boolean; generation?: bigint }
): Promise<void> {
await this.addToIndex(id, storedMetadata, opts.skipFlush, opts.deferWrites, opts.generation)
}
/** /**
* Rebuild entire index from scratch using pagination * Rebuild entire index from scratch using pagination
* Non-blocking version that yields control back to event loop * Non-blocking version that yields control back to event loop
* Sparse indices now lazy-loaded via UnifiedCache (no need to clear Map) * Sparse indices now lazy-loaded via UnifiedCache (no need to clear Map)
*
* @param options.inMemoryOnly - B3 Deliverable 3 (build-beside): when
* `true`, this call never touches the shared storage keys another,
* currently-serving `MetadataIndexManager` over the SAME storage may
* depend on it skips deleting persisted legacy chunk files AND skips
* the final `flush()` (which would otherwise write field indexes AND
* flush the column store's tail buffers to shared segment keys,
* colliding with a live manager's own writes). The caller ({@link
* buildBeside}) owns persistence entirely exactly once, after this
* instance becomes the sole owner via an atomic swap. Default `false`
* (every other caller keeps today's clear-then-persist behavior).
*/ */
async rebuild(): Promise<void> { async rebuild(options?: { inMemoryOnly?: boolean }): Promise<void> {
if (this.isRebuilding) return if (this.isRebuilding) return
const inMemoryOnly = options?.inMemoryOnly ?? false
this.isRebuilding = true this.isRebuilding = true
try { try {
@ -3519,15 +3858,22 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// here — it's always saved at the end of rebuild via flush(). This ensures // here — it's always saved at the end of rebuild via flush(). This ensures
// that if rebuild fails partway, the next init() can still discover fields // that if rebuild fails partway, the next init() can still discover fields
// and trigger another rebuild attempt. // and trigger another rebuild attempt.
prodLog.info('Clearing existing metadata index chunks from storage...') //
const existingFields = await this.getPersistedFieldList() // SKIPPED for inMemoryOnly: these are the SHARED storage keys a live
// manager over the same storage may still be reading (see this
// method's JSDoc) — deleting them before the swap is a live-read
// hazard, not a cleanup.
if (!inMemoryOnly) {
prodLog.info('Clearing existing metadata index chunks from storage...')
const existingFields = await this.getPersistedFieldList()
if (existingFields.length > 0) { if (existingFields.length > 0) {
for (const field of existingFields) { for (const field of existingFields) {
await this.deleteFieldChunks(field) await this.deleteFieldChunks(field)
}
prodLog.info(`Cleared ${existingFields.length} field indexes from storage`)
} }
prodLog.info(`Cleared ${existingFields.length} field indexes from storage`)
} }
// EntityIdMapper is intentionally NOT cleared here. Rebuild re-iterates // EntityIdMapper is intentionally NOT cleared here. Rebuild re-iterates
@ -3582,7 +3928,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
for (const noun of result.items) { for (const noun of result.items) {
const metadata = metadataBatch.get(noun.id) const metadata = metadataBatch.get(noun.id)
if (metadata) { if (metadata) {
await this.addToIndex(noun.id, metadata, true, true) await this.indexStoredRecord(noun.id, metadata, { skipFlush: true, deferWrites: true })
} }
} }
@ -3627,7 +3973,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
for (const verb of result.items) { for (const verb of result.items) {
const metadata = verbMetadataBatch.get(verb.id) const metadata = verbMetadataBatch.get(verb.id)
if (metadata) { if (metadata) {
await this.addToIndex(verb.id, metadata, true, true) await this.indexStoredRecord(verb.id, metadata, { skipFlush: true, deferWrites: true })
} }
} }
@ -3637,8 +3983,16 @@ export class MetadataIndexManager implements MetadataIndexProvider {
// Flush to storage. The column store's flush() handles tail-buffer-to- // Flush to storage. The column store's flush() handles tail-buffer-to-
// segment promotion + manifest persistence. // segment promotion + manifest persistence.
prodLog.debug('💾 Flushing metadata index to storage...') //
await this.flush() // SKIPPED for inMemoryOnly — see this method's JSDoc: flush() writes
// the shared field-index keys AND flushes the column store's tail
// buffers to shared segment keys, which would race a live manager's
// own flushes over the SAME storage. The caller flushes exactly once,
// after the swap.
if (!inMemoryOnly) {
prodLog.debug('💾 Flushing metadata index to storage...')
await this.flush()
}
prodLog.info(`✅ Metadata index rebuild completed! Processed ${totalNounsProcessed} nouns and ${totalVerbsProcessed} verbs`) prodLog.info(`✅ Metadata index rebuild completed! Processed ${totalNounsProcessed} nouns and ${totalVerbsProcessed} verbs`)

View file

@ -20,6 +20,9 @@ export default defineConfig({
// Include only integration tests // Include only integration tests
include: [ include: [
'tests/integration/**/*.test.ts', 'tests/integration/**/*.test.ts',
// The lifecycle biography lane (day-in-the-life scenarios; see
// tests/lifecycle/README.md) runs in the integration gate.
'tests/lifecycle/**/*.test.ts',
'tests/**/*.integration.test.ts' 'tests/**/*.integration.test.ts'
], ],

View file

@ -15,13 +15,7 @@ describe('Batch Import with Immediate Relations (v5.7.3 Fix)', () => {
// Initialize brain // Initialize brain
brain = new Brainy({ requireSubtype: false, brain = new Brainy({ requireSubtype: false,
storage: { storage: { type: 'filesystem', path: testDir },
type: 'filesystem',
config: {
baseDir: testDir,
enableCompression: false // Faster tests
}
},
dimensions: 384 dimensions: 384
}) })

View file

@ -1,29 +1,29 @@
/** /**
* @module tests/integration/cold-graph-connected-8.0 * @module tests/integration/cold-graph-connected-8.0
* @description BRAINY-COLD-GRAPH-CONNECTED (8.0) regression coverage for the silent-empty * @description BRAINY-COLD-GRAPH-CONNECTED (8.0) regression coverage for the silent-empty
* graph-traversal bug, gated on the converged 8.0 contract: a sync `graphIndex.isReady()` that * graph-traversal bug, gated on the honest readiness signal: a sync `graphIndex.isReady()`
* is true ONLY when the sourcetarget EDGES are loaded (NOT the membership/manifest count). * that is true ONLY when the sourcetarget EDGES are loaded (NOT the membership/manifest count).
* *
* On the FIRST `find({ connected })` after a cold process start of a LARGE brain (10k nouns, * On the FIRST `find({ connected })` after a cold process start, a native graph adjacency can
* which skips the eager index rebuild), a native graph adjacency can reload its relationship * reload its relationship COUNT (so `size() > 0`) but NOT its edges so `getNeighbors()` returns
* COUNT (so `size() > 0`) but NOT its edges so `getNeighbors()` returns `[]` for EVERY source * `[]` for EVERY source and brainy would serve that `[]` as if the anchor were genuinely edgeless.
* and brainy would serve that `[]` as if the anchor were genuinely edgeless.
* *
* The 8.0 guard (`verifyGraphAdjacencyLive`) prefers the honest `isReady()` signal: * RE-POINTED to the health-gate law: `verifyGraphAdjacencyLive` NEVER rebuilds and NEVER walks the
* - `isReady() === false` hydrate the id-mapper, rebuild from storage, re-check; a still-false * store from a read a read-path rebuild is exactly the dark-rebuild failure mode the law retires
* `isReady()` throws {@link GraphIndexNotReadyError} instead of returning `[]` ('rebuilt' when * (open() alone owns building). The guard now:
* the rebuild heals it); * - `isReady() === false` THROWS {@link GraphIndexNotReadyError} immediately no rebuild attempt;
* - a genuinely edgeless anchor with `isReady() === true` verifies 'live' and the empty result * - a genuinely edgeless anchor with `isReady() === true` verifies 'live' and the empty result
* stands no spurious rebuild, no throw; * stands no spurious throw;
* - a provider WITHOUT `isReady()` falls back to the shipped 7.x known-edge-sample probe. * - a provider WITHOUT `isReady()` falls back to the shipped known-edge-sample probe, which is
* now READ-ONLY: it refuses loudly (throws) rather than self-healing via rebuild.
* *
* These exercise REAL `find({ connected })` against an in-memory brain whose graph index is * These exercise REAL `find({ connected })` against an in-memory brain whose graph index is
* instrumented with a test-double `isReady()` (and, for the fallback case, an empty-then-healed * instrumented with a test-double `isReady()` (and, for the fallback case, an always-empty
* `getNeighbors`). Only the readiness/edge surface is wrapped; the underlying real adjacency * `getNeighbors`). Only the readiness/edge surface is wrapped; the underlying real adjacency
* (built by `relate()`) is unmasked once a rebuild "heals" it. * (built by `relate()`) is what a healthy provider actually serves.
*/ */
import { describe, it, expect, afterEach } from 'vitest' import { describe, it, expect, afterEach, vi } from 'vitest'
import { Brainy } from '../../src/index.js' import { Brainy } from '../../src/index.js'
import { NounType, VerbType } from '../../src/types/graphTypes.js' import { NounType, VerbType } from '../../src/types/graphTypes.js'
import { GraphIndexNotReadyError } from '../../src/errors/brainyError.js' import { GraphIndexNotReadyError } from '../../src/errors/brainyError.js'
@ -63,17 +63,17 @@ async function buildBrain(
} }
/** /**
* Instrument the brain's real graph index with a test-double `isReady()` (the 8.0 contract) plus * Instrument the brain's real graph index with a test-double `isReady()` (the honest-readiness
* an edge surface that goes empty while NOT ready. `getNeighbors` returns `[]` while `!ready` * contract) plus an edge surface that goes empty while NOT ready. `getNeighbors` returns `[]`
* (modelling the cold-unloaded adjacency) and delegates to the REAL index once a rebuild flips * while `!ready` (modelling the cold-unloaded adjacency) and delegates to the REAL index once
* `ready` on. `rebuild` is counted; it heals (`ready = true`) only when `healsOnRebuild` is set. * `ready` flips true (used only by the "healthy" control cases the guard itself never flips
* Pass `failFirstRebuild` to make the FIRST rebuild throw a transient error (without healing) so * this anymore, since it never rebuilds). `rebuild` is counted so tests can assert it is NEVER
* the empty-result re-collect path in executeGraphSearch is exercised. * called by a read.
*/ */
function instrumentIsReady( function instrumentIsReady(
brain: any, brain: any,
opts: { ready: boolean; healsOnRebuild: boolean; failFirstRebuild?: boolean } opts: { ready: boolean }
): { rebuildCalls: number } { ): { rebuildCalls: number; ready: boolean } {
const gi = brain.graphIndex const gi = brain.graphIndex
const origGetNeighbors = gi.getNeighbors.bind(gi) const origGetNeighbors = gi.getNeighbors.bind(gi)
const state = { ready: opts.ready, rebuildCalls: 0 } const state = { ready: opts.ready, rebuildCalls: 0 }
@ -85,10 +85,6 @@ function instrumentIsReady(
gi.rebuild = async (): Promise<void> => { gi.rebuild = async (): Promise<void> => {
state.rebuildCalls++ state.rebuildCalls++
if (opts.failFirstRebuild && state.rebuildCalls === 1) {
throw new Error('transient rebuild hiccup')
}
if (opts.healsOnRebuild) state.ready = true // unmask the real (already-populated) adjacency
} }
return state return state
@ -96,12 +92,12 @@ function instrumentIsReady(
/** /**
* Fallback instrumentation a provider WITHOUT `isReady()` (older cortex / JS baseline). Wraps * Fallback instrumentation a provider WITHOUT `isReady()` (older cortex / JS baseline). Wraps
* `getNeighbors` to return `[]` while `broken` and delegates to the REAL index once a rebuild * `getNeighbors` to always return `[]` while `broken`. This is the shipped known-edge-sample
* heals it. This is the shipped 7.x known-edge-sample probe path on 8.0. * probe path now READ-ONLY: it refuses loudly rather than self-healing.
*/ */
function instrumentNoIsReady( function instrumentNoIsReady(
brain: any, brain: any,
opts: { broken: boolean; healsOnRebuild: boolean } opts: { broken: boolean }
): { rebuildCalls: number } { ): { rebuildCalls: number } {
const gi = brain.graphIndex const gi = brain.graphIndex
// Ensure the provider does NOT expose isReady() — the default JS provider doesn't. // Ensure the provider does NOT expose isReady() — the default JS provider doesn't.
@ -114,13 +110,12 @@ function instrumentNoIsReady(
gi.rebuild = async (): Promise<void> => { gi.rebuild = async (): Promise<void> => {
state.rebuildCalls++ state.rebuildCalls++
if (opts.healsOnRebuild) state.broken = false
} }
return state return state
} }
describe('BRAINY-COLD-GRAPH-CONNECTED 8.0 — isReady()-gated, never serves a silent []', () => { describe('BRAINY-COLD-GRAPH-CONNECTED 8.0 — isReady()-gated, never serves a silent [], never rebuilds from a read', () => {
let brains: any[] = [] let brains: any[] = []
afterEach(async () => { afterEach(async () => {
for (const b of brains) { for (const b of brains) {
@ -131,35 +126,37 @@ describe('BRAINY-COLD-GRAPH-CONNECTED 8.0 — isReady()-gated, never serves a si
} }
} }
brains = [] brains = []
vi.restoreAllMocks()
}) })
it('(a) isReady() false → rebuild heals it true → find({ connected }) returns correct N (rebuilt)', async () => { it('(a) isReady() false → THROWS GraphIndexNotReadyError immediately, no rebuild attempt', async () => {
const { brain, anchorId, targetIds } = await buildBrain({ anchorEdges: true })
brains.push(brain)
const state = instrumentIsReady(brain, { ready: false, healsOnRebuild: true })
const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
expect(state.rebuildCalls).toBeGreaterThanOrEqual(1) // detected not-ready + healed it
const ids = results.map((r: any) => r.id).sort()
expect(ids).toEqual(targetIds.sort()) // B, C, D — the real edges, served after the heal
})
it('(b) isReady() stays false after rebuild → throws GraphIndexNotReadyError (NOT a silent [])', async () => {
const { brain, anchorId } = await buildBrain({ anchorEdges: true }) const { brain, anchorId } = await buildBrain({ anchorEdges: true })
brains.push(brain) brains.push(brain)
instrumentIsReady(brain, { ready: false, healsOnRebuild: false }) // rebuild never makes it ready const state = instrumentIsReady(brain, { ready: false })
await expect( await expect(
brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 }) brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
).rejects.toBeInstanceOf(GraphIndexNotReadyError) ).rejects.toBeInstanceOf(GraphIndexNotReadyError)
expect(state.rebuildCalls).toBe(0) // a read never rebuilds — it refuses loudly instead
})
it('(b) isReady() stays false → throws GraphIndexNotReadyError (NOT a silent [])', async () => {
const { brain, anchorId } = await buildBrain({ anchorEdges: true })
brains.push(brain)
const state = instrumentIsReady(brain, { ready: false })
await expect(
brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
).rejects.toBeInstanceOf(GraphIndexNotReadyError)
expect(state.rebuildCalls).toBe(0)
}) })
it('(c) edgeless anchor + isReady() true → returns [] with NO rebuild and NO throw', async () => { it('(c) edgeless anchor + isReady() true → returns [] with NO rebuild and NO throw', async () => {
// The anchor has no edges, but E -> F does — the adjacency is genuinely loaded (ready). // The anchor has no edges, but E -> F does — the adjacency is genuinely loaded (ready).
const { brain, anchorId } = await buildBrain({ anchorEdges: false }) const { brain, anchorId } = await buildBrain({ anchorEdges: false })
brains.push(brain) brains.push(brain)
const state = instrumentIsReady(brain, { ready: true, healsOnRebuild: false }) const state = instrumentIsReady(brain, { ready: true })
const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 }) const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
@ -170,7 +167,7 @@ describe('BRAINY-COLD-GRAPH-CONNECTED 8.0 — isReady()-gated, never serves a si
it('(d) healthy isReady() true → correct results, NO rebuild', async () => { it('(d) healthy isReady() true → correct results, NO rebuild', async () => {
const { brain, anchorId, targetIds } = await buildBrain({ anchorEdges: true }) const { brain, anchorId, targetIds } = await buildBrain({ anchorEdges: true })
brains.push(brain) brains.push(brain)
const state = instrumentIsReady(brain, { ready: true, healsOnRebuild: false }) const state = instrumentIsReady(brain, { ready: true })
const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 }) const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
@ -179,30 +176,30 @@ describe('BRAINY-COLD-GRAPH-CONNECTED 8.0 — isReady()-gated, never serves a si
expect(ids).toEqual(targetIds.sort()) expect(ids).toEqual(targetIds.sort())
}) })
it('(e) provider WITHOUT isReady() → falls back to the known-edge-sample probe (self-heals)', async () => { it('(e) provider WITHOUT isReady() → the known-edge-sample probe REFUSES LOUDLY (never self-heals)', async () => {
const { brain, anchorId, targetIds } = await buildBrain({ anchorEdges: true }) const { brain, anchorId } = await buildBrain({ anchorEdges: true })
brains.push(brain) brains.push(brain)
const state = instrumentNoIsReady(brain, { broken: true, healsOnRebuild: true }) const state = instrumentNoIsReady(brain, { broken: true })
const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 }) await expect(
brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
).rejects.toBeInstanceOf(GraphIndexNotReadyError)
expect(state.rebuildCalls).toBeGreaterThanOrEqual(1) // detected the empty adjacency + healed it expect(state.rebuildCalls).toBe(0) // the fallback probe is READ-ONLY — it never calls rebuild()
const ids = results.map((r: any) => r.id).sort()
expect(ids).toEqual(targetIds.sort()) // B, C, D — served after the heal
}) })
it('(f) executeGraphSearch re-collect: a transient first rebuild leaves connectedIds empty; the empty-result guard then heals + re-collects', async () => { it('(f) an empty connectedIds set re-verifies against a not-serving adjacency and throws, rather than serving [] as truth', async () => {
// First verify (inside neighbors()) hits a transient rebuild failure → returns 'live' without // executeGraphSearch's cold-load guard (connectedIds.size === 0 → re-verify) used to
// healing, so getNeighbors stays empty and connectedIds is empty. The empty connectedIds set // interpret a healed rebuild as "re-collect and serve." That rebuild-and-heal path is
// then drives executeGraphSearch's own verify, whose rebuild now heals → 'rebuilt' → re-collect. // retired: the re-verify now either confirms a genuinely edgeless anchor ('live', case (c))
const { brain, anchorId, targetIds } = await buildBrain({ anchorEdges: true }) // or — as here — discovers the adjacency itself is not serving, and throws.
const { brain, anchorId } = await buildBrain({ anchorEdges: true })
brains.push(brain) brains.push(brain)
const state = instrumentIsReady(brain, { ready: false, healsOnRebuild: true, failFirstRebuild: true }) const state = instrumentIsReady(brain, { ready: false })
const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 }) await expect(
brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
expect(state.rebuildCalls).toBeGreaterThanOrEqual(2) // first transient, second heals ).rejects.toBeInstanceOf(GraphIndexNotReadyError)
const ids = results.map((r: any) => r.id).sort() expect(state.rebuildCalls).toBe(0)
expect(ids).toEqual(targetIds.sort()) // re-collected after the heal
}) })
}) })

View file

@ -0,0 +1,352 @@
/**
* @module tests/integration/health-gate
* @description Pins for the health-by-accounting read gate: the read gate stops
* consulting an unnamed `isReady()` boolean and reads a NAMED, sync, O(1)
* {@link HealthReport}; no read path may ever start a store walk; the open path
* brings every provider to serving before it returns; an explicit operator door
* (`repairIndex({ rebuild: [...] })`) rebuilds a named leg unconditionally.
*
* Providers here are white-box test doubles: a `healthReport()` (or, for the
* interim-path pins, an `isReady()`) function assigned directly onto the LIVE
* JS provider object, the same pattern `tests/unit/validate-invariants-delegation.test.ts`
* uses for `validateInvariants`. This exercises brainy's real gate/verify code
* against a controlled provider self-report no engine mocks.
*/
import { describe, it, expect, afterEach, vi } from 'vitest'
import { mkdtempSync, rmSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import {
Brainy,
NounType,
VerbType,
GraphIndexNotReadyError,
MetadataIndexNotReadyError,
VectorIndexNotReadyError
} from '../../src/index.js'
import type { HealthReport, LedgerInvariantResult } from '../../src/plugin.js'
import { prodLog } from '../../src/utils/logger.js'
import { createTestConfig } from '../helpers/test-factory.js'
/** The white-box surface these pins drive on a live brain instance. */
interface BrainInternals {
storage: {
getNoun(id: string): Promise<unknown>
getNounMetadata(id: string): Promise<unknown>
getNouns(options?: unknown): Promise<unknown>
getVerbs(options?: unknown): Promise<unknown>
}
index: { healthReport?: () => HealthReport; isReady?: () => boolean; rebuild(): Promise<void> }
metadataIndex: {
healthReport?: () => HealthReport
isReady?: () => boolean
rebuild(): Promise<void>
validateInvariants?: () => Promise<unknown>
}
graphIndex: {
healthReport?: () => HealthReport
isReady?: () => boolean
rebuild(): Promise<void>
validateInvariants?: () => Promise<unknown>
}
rebuildIndexesIfNeeded(force?: boolean): Promise<void>
}
function internalsOf(brain: Brainy): BrainInternals {
return brain as unknown as BrainInternals
}
function invariant(overrides: Partial<LedgerInvariantResult> = {}): LedgerInvariantResult {
return {
name: 'manifest-residency',
holds: true,
detail: 'ok',
heal: 'none',
source: 'ledger',
...overrides
}
}
function healthReport(overrides: Partial<HealthReport> = {}): HealthReport {
return {
provider: 'vector',
healthy: true,
serving: true,
invariants: [],
checkedAt: Date.now(),
durationMs: 1,
generation: 1,
unledgered: [],
...overrides
}
}
const brains: Brainy[] = []
const dirs: string[] = []
afterEach(async () => {
for (const b of brains.splice(0)) await b.close().catch(() => {})
for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true })
vi.restoreAllMocks()
})
describe('health gate (a) — not-serving refuses loudly, ZERO canonical reads during the refusal', () => {
it('metadata not-serving: find() throws MetadataIndexNotReadyError naming the failing invariant', async () => {
const brain = new Brainy(createTestConfig({ silent: true }))
await brain.init()
brains.push(brain)
await brain.add({ data: 'row', type: NounType.Document, metadata: { team: 'atlas' } })
await brain.flush()
const internals = internalsOf(brain)
internals.metadataIndex.healthReport = () =>
healthReport({
provider: 'metadata',
serving: false,
healthy: false,
invariants: [invariant({ name: 'posted-count-floor', holds: false, heal: 'rebuild', detail: 'posted 2 < canonical 5' })]
})
const getNounSpy = vi.spyOn(internals.storage, 'getNoun')
const getNounMetadataSpy = vi.spyOn(internals.storage, 'getNounMetadata')
const getNounsSpy = vi.spyOn(internals.storage, 'getNouns')
await expect(brain.find({ where: { team: 'atlas' } })).rejects.toBeInstanceOf(MetadataIndexNotReadyError)
await expect(brain.find({ where: { team: 'atlas' } })).rejects.toThrow(/posted-count-floor/)
expect(getNounSpy).not.toHaveBeenCalled()
expect(getNounMetadataSpy).not.toHaveBeenCalled()
expect(getNounsSpy).not.toHaveBeenCalled()
delete internals.metadataIndex.healthReport
})
it('graph not-serving: related() throws GraphIndexNotReadyError naming the failing invariant, no canonical reads', async () => {
const brain = new Brainy(createTestConfig({ silent: true }))
await brain.init()
brains.push(brain)
const a = await brain.add({ data: 'a', type: NounType.Person })
const b = await brain.add({ data: 'b', type: NounType.Person })
await brain.relate({ from: a, to: b, type: VerbType.Knows })
await brain.flush()
const internals = internalsOf(brain)
internals.graphIndex.healthReport = () =>
healthReport({
provider: 'graph',
serving: false,
healthy: false,
invariants: [invariant({ name: 'adjacency-residency', holds: false, heal: 'rebuild', detail: 'edges not loaded' })]
})
const getNounSpy = vi.spyOn(internals.storage, 'getNoun')
const getVerbsSpy = vi.spyOn(internals.storage, 'getVerbs')
await expect(brain.related({ from: a })).rejects.toBeInstanceOf(GraphIndexNotReadyError)
await expect(brain.related({ from: a })).rejects.toThrow(/adjacency-residency/)
expect(getNounSpy).not.toHaveBeenCalled()
expect(getVerbsSpy).not.toHaveBeenCalled()
delete internals.graphIndex.healthReport
})
})
describe('health gate (b) — unledgered is unknown: never blocks a serving provider', () => {
it('serving:true with an unledgered family and no failing invariant serves normally; at most one narration', async () => {
const brain = new Brainy(createTestConfig({ silent: true }))
await brain.init()
brains.push(brain)
await brain.add({ data: 'row', type: NounType.Document, metadata: { team: 'atlas' } })
await brain.flush()
const internals = internalsOf(brain)
internals.metadataIndex.healthReport = () =>
healthReport({
provider: 'metadata',
serving: true,
healthy: true,
invariants: [],
unledgered: ['canonical-verb-coverage']
})
const warnSpy = vi.spyOn(prodLog, 'warn')
const r1 = await brain.find({ where: { team: 'atlas' } })
const r2 = await brain.find({ where: { team: 'atlas' } })
expect(r1.length).toBe(1)
expect(r2.length).toBe(1)
const narrations = warnSpy.mock.calls.filter(
([msg]) => typeof msg === 'string' && msg.includes('canonical-verb-coverage')
)
expect(narrations.length).toBe(1) // one narration at most across both reads (same generation)
delete internals.metadataIndex.healthReport
})
})
describe('health gate (c) — degraded-but-serving narrates once per generation', () => {
it('a heal:"repair" failure serves; narrates once per generation, twice across a generation bump', async () => {
const brain = new Brainy(createTestConfig({ silent: true }))
await brain.init()
brains.push(brain)
await brain.add({ data: 'row', type: NounType.Document, metadata: { team: 'atlas' } })
await brain.flush()
const internals = internalsOf(brain)
let generation = 1
internals.index.healthReport = () =>
healthReport({
provider: 'vector',
serving: true,
healthy: false,
invariants: [invariant({ name: 'stale-vector-counter', holds: false, heal: 'repair', detail: 'counter drift' })],
generation
})
const warnSpy = vi.spyOn(prodLog, 'warn')
const countNarrations = () =>
warnSpy.mock.calls.filter(([msg]) => typeof msg === 'string' && msg.includes('stale-vector-counter')).length
await expect(brain.find({ where: { team: 'atlas' } })).resolves.toHaveLength(1)
await expect(brain.find({ where: { team: 'atlas' } })).resolves.toHaveLength(1)
expect(countNarrations()).toBe(1) // same generation both times — one narration
generation = 2
await expect(brain.find({ where: { team: 'atlas' } })).resolves.toHaveLength(1)
expect(countNarrations()).toBe(2) // generation bumped — a second narration
delete internals.index.healthReport
})
})
describe('health gate (d) — interim isReady()-only path (no healthReport) is unchanged', () => {
it('isReady() === true serves; isReady() === false refuses via the typed NotReady error', async () => {
const brain = new Brainy(createTestConfig({ silent: true }))
await brain.init()
brains.push(brain)
await brain.add({ data: 'row', type: NounType.Document, metadata: { team: 'atlas' } })
await brain.flush()
const internals = internalsOf(brain)
internals.metadataIndex.isReady = () => true
await expect(brain.find({ where: { team: 'atlas' } })).resolves.toHaveLength(1)
internals.metadataIndex.isReady = () => false
await expect(brain.find({ where: { team: 'atlas' } })).rejects.toBeInstanceOf(MetadataIndexNotReadyError)
delete internals.metadataIndex.isReady
})
})
describe('health gate (e) — open builds; the first read never does', () => {
it('disableAutoRebuild:true on a populated store: open narrates + builds; the first find() triggers zero rebuilds', async () => {
const dir = mkdtempSync(join(tmpdir(), 'brainy-healthgate-open-'))
dirs.push(dir)
const writer = new Brainy({
storage: { type: 'filesystem', path: dir },
requireSubtype: false,
silent: true,
disableAutoRebuild: true
})
await writer.init()
brains.push(writer)
await writer.add({ data: 'row one', type: NounType.Document, metadata: { team: 'atlas' } })
await writer.flush()
await brains.pop()!.close()
const warnSpy = vi.spyOn(prodLog, 'warn')
const reader = new Brainy({
storage: { type: 'filesystem', path: dir },
requireSubtype: false,
silent: true,
disableAutoRebuild: true
})
const internals = internalsOf(reader)
const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded')
await reader.init()
brains.push(reader)
expect(rebuildSpy).toHaveBeenCalledTimes(1) // open() built it, exactly once
expect(
warnSpy.mock.calls.some(
([msg]) => typeof msg === 'string' && msg.includes('open() is building')
)
).toBe(true)
rebuildSpy.mockClear()
const rows = await reader.find({ where: { team: 'atlas' } })
expect(rebuildSpy).toHaveBeenCalledTimes(0) // the read never builds
expect(rows.length).toBe(1)
}, 30000)
})
describe('health gate (f) — the ceremony door: explicit rebuild bypasses invariant consultation', () => {
it("repairIndex({ rebuild: ['graph'] }) rebuilds unconditionally without consulting validateInvariants", async () => {
const brain = new Brainy(createTestConfig({ silent: true }))
await brain.init()
brains.push(brain)
await brain.add({ data: 'x', type: NounType.Concept })
await brain.flush()
const internals = internalsOf(brain)
let validateCalls = 0
internals.graphIndex.validateInvariants = async () => {
validateCalls++
return healthReport({ provider: 'graph' })
}
const rebuildSpy = vi.spyOn(internals.graphIndex, 'rebuild')
const report = await brain.repairIndex({ rebuild: ['graph'] })
expect(rebuildSpy).toHaveBeenCalledTimes(1)
expect(validateCalls).toBe(0) // the door never consults validateInvariants to decide
const graphFamily = report.families.find((f) => f.family === 'provider:graph')
expect(graphFamily?.rebuilt).toBe(true)
expect(graphFamily?.checked).toBe(true)
expect(graphFamily?.reason).toBe('explicit rebuild requested')
delete internals.graphIndex.validateInvariants
})
it('bare repairIndex() on a healthy provider calls no rebuild()', async () => {
const brain = new Brainy(createTestConfig({ silent: true }))
await brain.init()
brains.push(brain)
await brain.add({ data: 'x', type: NounType.Concept })
await brain.flush()
const internals = internalsOf(brain)
internals.graphIndex.validateInvariants = async () => healthReport({ provider: 'graph', healthy: true, serving: true })
const rebuildSpy = vi.spyOn(internals.graphIndex, 'rebuild')
await brain.repairIndex()
expect(rebuildSpy).not.toHaveBeenCalled()
delete internals.graphIndex.validateInvariants
})
})
describe('health gate (g) — a throwing healthReport() is a contract violation, never read as healthy', () => {
it('healthReport() that throws refuses loudly with the typed NotReady error naming the throw', async () => {
const brain = new Brainy(createTestConfig({ silent: true }))
await brain.init()
brains.push(brain)
await brain.add({ data: 'row', type: NounType.Document, metadata: { team: 'atlas' } })
await brain.flush()
const internals = internalsOf(brain)
internals.index.healthReport = () => {
throw new Error('accelerator: mmap window busy')
}
await expect(brain.find({ where: { team: 'atlas' } })).rejects.toBeInstanceOf(VectorIndexNotReadyError)
await expect(brain.find({ where: { team: 'atlas' } })).rejects.toThrow(/mmap window busy/)
delete internals.index.healthReport
})
})

View file

@ -0,0 +1,167 @@
/**
* @module tests/integration/metadata-online-rebuild
* @description THE ONLINE JS METADATA REBUILD (B3 Deliverable 3) pins.
* `MetadataIndexManager.rebuild()` used to be clear-then-walk reads went
* dark for the duration. `repairIndex({ rebuild: ['metadata'] })` now builds
* a fresh replacement index BESIDE the live one (walk canonical + mirror
* every live write via `beginShadow`/`endShadow` + a bounded fact-log fold),
* then atomically swaps the brain's reference `find()` never observes a
* half-built index, and a write landing DURING the build is never lost.
*/
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
import { describe, it, expect, afterEach } from 'vitest'
import { mkdtempSync, rmSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { Brainy } from '../../src/brainy.js'
import { NounType, VerbType } from '../../src/types/graphTypes.js'
import type { MetadataIndexManager } from '../../src/utils/metadataIndex.js'
const dirs: string[] = []
const brains: Brainy<any>[] = []
afterEach(async () => {
for (const b of brains.splice(0)) await b.close().catch(() => {})
for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true })
})
function metadataIndexOf(brain: Brainy<any>): MetadataIndexManager {
return (brain as unknown as { metadataIndex: MetadataIndexManager }).metadataIndex
}
async function openBrain(): Promise<{ brain: Brainy<any>; dir: string }> {
const dir = mkdtempSync(join(tmpdir(), 'brainy-online-rebuild-'))
dirs.push(dir)
const brain = new Brainy<any>({
requireSubtype: false,
storage: { type: 'filesystem', path: dir },
silent: true,
persistence: { policy: 'manual' },
logAuthority: 'adopt'
})
await brain.init()
brains.push(brain)
return { brain, dir }
}
describe('repairIndex({ rebuild: ["metadata"] }) — the online build-beside rebuild', () => {
it(
'a find() polled throughout the rebuild of a 2k-noun store never returns fewer rows than ' +
'before the build started, and a write landing DURING the build is never lost',
async () => {
const { brain, dir } = await openBrain()
void dir
const N = 2000
const ids: string[] = []
for (let i = 0; i < N; i++) {
ids.push(
await brain.add({
data: `entity ${i}`,
type: NounType.Person,
metadata: { status: i % 2 === 0 ? 'active' : 'inactive' }
})
)
}
for (let i = 0; i < 20; i++) {
await brain.relate({
from: ids[i], to: ids[i + 1], type: VerbType.WorksWith, metadata: { tag: 'orig' }
})
}
await brain.flush()
const baseline = await brain.find({ where: { status: 'active' }, limit: 10000 })
expect(baseline.length).toBe(N / 2)
// Kick off the online rebuild WITHOUT awaiting — poll reads and
// perform a live write concurrently with it.
const repairPromise = brain.repairIndex({ rebuild: ['metadata'] })
let minObserved = Infinity
let polls = 0
const pollPromise = (async () => {
// Poll until the rebuild settles — bounded so a slow CI box can't
// spin forever, generous enough to actually overlap the walk.
while (polls < 200) {
const rows = await brain.find({ where: { status: 'active' }, limit: 10000 })
minObserved = Math.min(minObserved, rows.length)
polls++
await new Promise((resolve) => setTimeout(resolve, 1))
}
})()
const newId = await brain.add({
data: 'added during the rebuild',
type: NounType.Person,
metadata: { status: 'active' }
})
const newRelId = await brain.relate({
from: newId, to: ids[0], type: VerbType.WorksWith, metadata: { tag: 'during-build' }
})
const [report] = await Promise.all([repairPromise, pollPromise])
// THE PIN: never fewer rows than the pre-build baseline, at any polled
// instant — reads served the OLD (fully-populated) manager throughout.
expect(polls).toBeGreaterThan(0)
expect(minObserved).toBeGreaterThanOrEqual(baseline.length)
// The repair report still accounts for the family (same receipt shape
// regardless of which rebuild mechanism actually ran underneath).
const metadataFamily = report.families.find((f) => f.family === 'provider:metadata')
expect(metadataFamily?.checked).toBe(true)
expect(metadataFamily?.rebuilt).toBe(true)
// Post-swap correctness: the live write during the build was never
// lost (the beginShadow mirror + post-walk fold caught it).
const afterActive = await brain.find({ where: { status: 'active' }, limit: 10000 })
expect(afterActive.length).toBe(baseline.length + 1)
expect(afterActive.some((r) => r.id === newId)).toBe(true)
const index = metadataIndexOf(brain)
expect(await index.getIds('tag', 'during-build')).toEqual([newRelId])
expect((await index.getIds('tag', 'orig')).length).toBe(20)
// The swap stamped the watermark — a reopen adopts, zero rebuild.
await brain.close()
brains.length = 0 // already closed above; afterEach must not double-close
const reopened = new Brainy<any>({
requireSubtype: false,
storage: { type: 'filesystem', path: dir },
silent: true,
persistence: { policy: 'manual' },
logAuthority: 'adopt'
})
await reopened.init()
brains.push(reopened)
const reopenedIndex = metadataIndexOf(reopened)
expect(reopenedIndex.watermarkVerdict()).toBe('adopt')
const reopenedActive = await reopened.find({ where: { status: 'active' }, limit: 10000 })
expect(reopenedActive.length).toBe(afterActive.length)
},
60000
)
it('repairIndex({ rebuild: ["metadata"] }) on an empty store is a trivial no-op walk', async () => {
const { brain } = await openBrain()
const report = await brain.repairIndex({ rebuild: ['metadata'] })
const metadataFamily = report.families.find((f) => f.family === 'provider:metadata')
expect(metadataFamily?.checked).toBe(true)
expect(await brain.getNounCount()).toBe(0)
})
it('two consecutive online rebuilds both leave the index correct (idempotent)', async () => {
const { brain } = await openBrain()
const a = await brain.add({ data: 'a', type: NounType.Person, metadata: { status: 'active' } })
await brain.add({ data: 'b', type: NounType.Person, metadata: { status: 'inactive' } })
await brain.flush()
await brain.repairIndex({ rebuild: ['metadata'] })
const first = await brain.find({ where: { status: 'active' } })
expect(first.map((r) => r.id)).toEqual([a])
await brain.repairIndex({ rebuild: ['metadata'] })
const second = await brain.find({ where: { status: 'active' } })
expect(second.map((r) => r.id)).toEqual([a])
})
})

View file

@ -34,13 +34,7 @@ describe('Read-After-Write Consistency (v5.7.2 Bug Fix)', () => {
testDir = join(tmpdir(), `brainy-consistency-${Date.now()}-${Math.random().toString(36).substring(7)}`) testDir = join(tmpdir(), `brainy-consistency-${Date.now()}-${Math.random().toString(36).substring(7)}`)
brain = new Brainy({ requireSubtype: false, brain = new Brainy({ requireSubtype: false,
storage: { storage: { type: 'filesystem', path: testDir },
type: 'filesystem',
config: {
baseDir: testDir,
enableCompression: false // Faster tests
}
},
dimensions: 384 dimensions: 384
}) })

View file

@ -68,4 +68,46 @@ describe('repairIndex per-family receipt', () => {
expect(orphans!.healed, 'the ghost was pruned and receipted').toBeGreaterThan(0) expect(orphans!.healed, 'the ghost was pruned and receipted').toBeGreaterThan(0)
expect(report.healedTotal).toBeGreaterThan(0) expect(report.healedTotal).toBeGreaterThan(0)
}, 120000) }, 120000)
it("a heal:'repair' verdict routes to the provider's own repair(), and the re-read decides", async () => {
// A fake provider report: one failing invariant asking for the INCREMENTAL
// heal. repairIndex must call repair() (never rebuild()) and count the heal
// only when the post-repair re-read clears the same verdict.
const dir = mkdtempSync(join(tmpdir(), 'brainy-repair-route-'))
const brain: any = new Brainy({ storage: { type: 'filesystem', path: dir }, requireSubtype: false, silent: true })
await brain.init()
brains.push(brain)
let repairCalls = 0
let rebuildCalls = 0
let healed = false
const failing = {
provider: 'vector', healthy: false, serving: true,
invariants: [{ name: 'node-coverage', holds: false, detail: 'short 3', heal: 'repair' as const }],
checkedAt: 1, durationMs: 1
}
const clean = {
provider: 'vector', healthy: true, serving: true,
invariants: [{ name: 'node-coverage', holds: true, detail: 'ok', heal: 'none' as const }],
checkedAt: 2, durationMs: 1
}
;(brain.index as any).validateInvariants = async () => (healed ? clean : failing)
;(brain.index as any).repair = async () => { repairCalls++; healed = true; return { repaired: 3 } }
const origRebuild = (brain.index as any).rebuild
;(brain.index as any).rebuild = async () => { rebuildCalls++ }
try {
const report = await brain.repairIndex()
const row = report.families.find((f: any) => f.family === 'provider:vector')
expect(row, 'the provider family is in the receipt').toBeDefined()
expect(repairCalls, 'repair() ran exactly once').toBe(1)
expect(rebuildCalls, "a heal:'repair' verdict never runs rebuild()").toBe(0)
expect(row!.healed, 'the cleared verdict counts as healed').toBe(1)
expect(String(row!.detail)).toMatch(/incremental repair cleared: node-coverage/)
} finally {
delete (brain.index as any).validateInvariants
delete (brain.index as any).repair
;(brain.index as any).rebuild = origRebuild
}
})
}) })

View file

@ -0,0 +1,190 @@
/**
* @module tests/integration/verb-metadata-rows
* @description THE LIVE VERB PATH pins. Before this train, verb rows entered
* the metadata index ONLY via `MetadataIndexManager.rebuild()`'s canonical
* walk every relate()/unrelate()/updateRelation() call, and every
* remove()-cascaded relationship, left the metadata index blind to verb
* writes until the next rebuild. This file pins that `relate()`,
* `unrelate()`, `updateRelation()`, `remove()`'s cascade, and their
* `transact()` mirrors now post/retract the SAME verb rows a rebuild would
* derive from canonical (ADR-007 A4: one mechanism for add/update, live and
* rebuilt).
*/
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
import { Brainy } from '../../src/brainy.js'
import { NounType, VerbType } from '../../src/types/graphTypes.js'
import type { MetadataIndexManager } from '../../src/utils/metadataIndex.js'
/** The JS metadata-index manager backing a memory-storage brain in these
* tests (feature-detected in production code via `instanceof
* MetadataIndexManager`; a narrow test-only reach-in here, matching the
* existing idiom in tests/integration/find-where-zero.test.ts and
* tests/integration/level-field-shadow.test.ts). */
function metadataIndexOf(brain: Brainy<any>): MetadataIndexManager {
return (brain as unknown as { metadataIndex: MetadataIndexManager }).metadataIndex
}
describe('verb metadata rows — the live path matches the rebuild walk', () => {
let brain: Brainy<any>
beforeEach(async () => {
brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, silent: true })
await brain.init()
})
afterEach(async () => {
await brain.close()
})
async function addPerson(label: string): Promise<string> {
return brain.add({
data: `person ${label}`,
type: NounType.Person,
metadata: { label }
})
}
it('(a) relate() posts a metadata-index-backed verb row a query can find', async () => {
const a = await addPerson('a')
const b = await addPerson('b')
const relId = await brain.relate({
from: a, to: b, type: VerbType.WorksWith, metadata: { role: 'lead' }
})
// Read it back the SAME way a rebuild-sourced row is queried — the
// manager's own posting lookup, keyed on the custom field the caller wrote.
const index = metadataIndexOf(brain)
expect(await index.getIds('role', 'lead')).toEqual([relId])
})
it('(b) unrelate() retracts the row', async () => {
const a = await addPerson('a')
const b = await addPerson('b')
const relId = await brain.relate({
from: a, to: b, type: VerbType.WorksWith, metadata: { role: 'lead' }
})
const index = metadataIndexOf(brain)
expect(await index.getIds('role', 'lead')).toEqual([relId])
// Flush BEFORE retracting the field's only occurrence: this durably
// persists the 'role' column (a segment on disk/in the store), so the
// post-retraction query below reads "this field exists, zero live
// postings" (→ []) rather than "this field has never been written"
// (→ FIELD_NOT_INDEXED) — an orthogonal column-store characteristic
// (an unflushed field with its last live posting removed reverts to
// unknown), not a D2 behavior.
await brain.flush()
await brain.unrelate(relId)
expect(await index.getIds('role', 'lead')).toEqual([])
})
it('(c) updateRelation({ metadata }) leaves exactly the new values', async () => {
const a = await addPerson('a')
const b = await addPerson('b')
const relId = await brain.relate({
from: a, to: b, type: VerbType.WorksWith, metadata: { role: 'lead', team: 'core' }
})
const index = metadataIndexOf(brain)
expect(await index.getIds('role', 'lead')).toEqual([relId])
// Flush first — see (b)'s note: 'role'/'team' must be durably known
// fields before their only value is retracted, or the post-update
// "gone" checks below throw FIELD_NOT_INDEXED instead of returning [].
await brain.flush()
await brain.updateRelation({ id: relId, metadata: { role: 'reviewer' }, merge: false })
// Stale values gone (the old shape AND the merge:false-dropped field)…
expect(await index.getIds('role', 'lead')).toEqual([])
expect(await index.getIds('team', 'core')).toEqual([])
// …only the new value serves.
expect(await index.getIds('role', 'reviewer')).toEqual([relId])
})
it("(d) remove(entity) cascade retracts every incident relation's metadata row", async () => {
const a = await addPerson('a')
const b = await addPerson('b')
const c = await addPerson('c')
const rel1 = await brain.relate({
from: a, to: b, type: VerbType.WorksWith, metadata: { tag: 'cascade-test' }
})
const rel2 = await brain.relate({
from: c, to: a, type: VerbType.WorksWith, metadata: { tag: 'cascade-test' }
})
const index = metadataIndexOf(brain)
expect((await index.getIds('tag', 'cascade-test')).sort()).toEqual([rel1, rel2].sort())
// Flush first — see (b)'s note.
await brain.flush()
await brain.remove(a) // a is source of rel1, target of rel2 — both cascade
expect(await index.getIds('tag', 'cascade-test')).toEqual([])
})
it('(e) a rebuild() reproduces exactly the verb-row population the live path built', async () => {
const a = await addPerson('a')
const b = await addPerson('b')
const c = await addPerson('c')
await brain.relate({ from: a, to: b, type: VerbType.WorksWith, metadata: { tag: 'parity', label: 'ab' } })
await brain.relate({ from: b, to: c, type: VerbType.RelatedTo, metadata: { tag: 'parity', label: 'bc' } })
const relId3 = await brain.relate({
from: c, to: a, type: VerbType.WorksWith, metadata: { tag: 'parity', label: 'ca' }
})
await brain.unrelate(relId3) // exercise retraction too — the rebuild must NOT resurrect it
const index = metadataIndexOf(brain)
const beforeIds = (await index.getIds('tag', 'parity')).slice().sort()
expect(beforeIds.length).toBe(2)
const beforeAb = await index.getIds('label', 'ab')
const beforeBc = await index.getIds('label', 'bc')
await index.rebuild()
const afterIds = (await index.getIds('tag', 'parity')).slice().sort()
expect(afterIds).toEqual(beforeIds)
expect(await index.getIds('label', 'ab')).toEqual(beforeAb)
expect(await index.getIds('label', 'bc')).toEqual(beforeBc)
expect(await index.getIds('label', 'ca')).toEqual([]) // the unrelated edge stays gone
})
it('(f) transact() relate/unrelate posts/retracts the same metadata-index rows as single-op', async () => {
const a = await addPerson('a')
const b = await addPerson('b')
const c = await addPerson('c')
const d = await addPerson('d')
// Single-op baseline.
const singleOpId = await brain.relate({
from: a, to: b, type: VerbType.WorksWith, metadata: { tag: 'parity-f' }
})
// transact() mirror.
const relateDb = await brain.transact([
{ op: 'relate', from: c, to: d, type: VerbType.WorksWith, metadata: { tag: 'parity-f' } }
])
const transactId = relateDb.receipt!.ids[0]
await relateDb.release()
const index = metadataIndexOf(brain)
expect((await index.getIds('tag', 'parity-f')).sort()).toEqual([singleOpId, transactId].sort())
// Flush first — see (b)'s note: 'tag' must be durably known before its
// last live posting is retracted below.
await brain.flush()
// Retract both ways — single-op unrelate() and transact() unrelate.
await brain.unrelate(singleOpId)
const unrelateDb = await brain.transact([{ op: 'unrelate', id: transactId }])
await unrelateDb.release()
expect(await index.getIds('tag', 'parity-f')).toEqual([])
})
})

22
tests/lifecycle/README.md Normal file
View file

@ -0,0 +1,22 @@
# The Lifecycle Lane
One brain, driven through founding, a working day, a clean restart, a
crash, a repair, and a second life, checked chapter by chapter against an
independent shadow-model referee (`biographyHarness.ts`). It catches
COMPOSITION regressions unit tests miss — a store fine in one process but
broken across a restart/crash/repair. Runs on the plain JS engine, so it
gates every commit.
Run it: `npx vitest run tests/lifecycle --pool=forks`
A red names the chapter label, the id, and expected-vs-actual — diagnosable
from the message alone. `biography.test.ts` is split into two `it` blocks
(Ch1-3, then Ch4-6) purely for reporting; it is still ONE fixed-order story.
Chapters must never be reordered, skipped, or made conditional, and a
failing chapter's assertion must never be weakened to force green.
Lab notes (hard-won, keep):
- `git reset --hard` does NOT remove untracked files — a "clean" tree can still
carry stray test stores; use `git clean -fd tests/lifecycle-tmp` equivalents.
- `silent: true` patches `console` process-wide — never assert narration through
`console` spies in this lane; the engine's always-on channel is `prodLog`.

View file

@ -0,0 +1,404 @@
/**
* @module tests/lifecycle/biography
* @description THE LIFECYCLE LANE see `tests/lifecycle/README.md` for what
* this proves and how to run it. One scenario, "the working store": a single
* brain driven through founding, a working day, a clean restart, a crash, a
* repair, and a second life, verified chapter by chapter against an
* independent shadow-model referee (`biographyHarness.ts`).
*
* Split into two `it` blocks so a currently-failing later chapter (see the
* second block's header comment a live engine finding, not a defect in
* this lane) never hides the earlier chapters' passing coverage. The two
* blocks share one brain's directory and one shadow model, run in the SAME
* fixed order the single scenario always has (`describe.sequential` below
* exists to say so explicitly, though vitest's own default is sequential
* within a file) this is a split for REPORTING clarity, not a reordering
* or conditional skip of any chapter.
*/
import { describe, it, expect } from 'vitest'
import * as fs from 'node:fs'
import { NounType, VerbType } from '../../src/types/graphTypes.js'
import type { Brainy } from '../../src/brainy.js'
import type { AddParams, RelateParams, UpdateParams, UpdateRelationParams } from '../../src/index.js'
import { abandonAsCrashed, makeTempDir, openBrain, uid } from '../helpers/durabilityKillMatrix.js'
import {
createModel,
getCanonicalCountsFor,
modelAdd,
modelDelete,
modelRelate,
modelUpdate,
modelUpdateRelation,
recordVfsFileWrite,
snapshotVfsBaseline,
verifyChapter,
type HubCheck,
type ShadowModel
} from './biographyHarness.js'
const STATUSES = ['active', 'pending', 'closed', 'archived'] as const
/** Cycle a status value to the next one in the fixed rotation used so
* Ch2's 40 updates provably MOVE entities across find() buckets rather than
* risking a no-op reassignment of the same value. */
function nextStatus(current: unknown): (typeof STATUSES)[number] {
const currentStr = typeof current === 'string' ? current : STATUSES[0]
const idx = STATUSES.indexOf(currentStr as (typeof STATUSES)[number])
return STATUSES[(idx < 0 ? 0 : idx + 1) % STATUSES.length]
}
// ---------------------------------------------------------------------------
// Shared biography state — set up by the first `it`, consumed by the second.
// The two blocks are one continuous story told in two named pieces; nothing
// here resets or diverges between them.
// ---------------------------------------------------------------------------
let dir: string
let model: ShadowModel
let brain: Brainy
let hubs: HubCheck[]
let employees: string[]
let customers: string[]
let invoices: string[]
let tasks: string[]
let projects: string[]
let nonHub: string[]
// ---- Wrappers: every call to the real brain updates the shadow model in
// the same statement, so the two can never drift apart by construction.
// Defined once, closing over the `let` bindings above so both `it` blocks
// (and any future reopen inside them) operate on the current brain/model.
async function doAdd(label: string, params: Omit<AddParams, 'id'>): Promise<string> {
const id = uid(label)
await brain.add({ ...params, id })
modelAdd(model, id, {
type: params.type,
subtype: params.subtype,
metadata: params.metadata ?? {},
visibility: params.visibility
})
return id
}
async function doUpdate(id: string, patch: Omit<UpdateParams, 'id'>): Promise<void> {
await brain.update({ ...patch, id })
modelUpdate(model, id, { metadata: patch.metadata, merge: patch.merge, visibility: patch.visibility })
}
async function doRemove(id: string): Promise<void> {
await brain.remove(id)
modelDelete(model, id)
}
async function doRelate(params: RelateParams): Promise<string> {
const id = await brain.relate(params)
modelRelate(model, id, {
from: params.from,
to: params.to,
type: params.type,
subtype: params.subtype,
metadata: params.metadata
})
return id
}
async function doUpdateRelation(id: string, patch: Omit<UpdateRelationParams, 'id'>): Promise<void> {
await brain.updateRelation({ ...patch, id })
modelUpdateRelation(model, id, { metadata: patch.metadata, merge: patch.merge })
}
async function doVfsWrite(path: string, content: string): Promise<void> {
await brain.vfs.writeFile(path, content)
recordVfsFileWrite(model)
}
describe.sequential('lifecycle — the working store', () => {
it(
'Ch1 FOUNDING -> Ch2 A WORKING DAY -> Ch3 CLEAN RESTART: every read serves truth',
async () => {
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
dir = makeTempDir()
model = createModel()
// logAuthority: 'adopt' from the first open, mirrored across every
// reopen — see write-flow-production-shape.test.ts, which the later
// crash chapter's at-ack law is pinned against.
brain = await openBrain(dir, { logAuthority: 'adopt' })
// =================================================================
// CHAPTER 1 — FOUNDING
// =================================================================
// Baseline MUST be snapshotted before any biography act — it is the
// VFS root's own system-tier footprint, measured, never hardcoded.
await snapshotVfsBaseline(brain, model)
employees = []
for (let i = 0; i < 20; i++) {
employees.push(
await doAdd(`emp-${i}`, {
data: `employee record ${i}`,
type: NounType.Person,
subtype: 'employee',
metadata: { status: STATUSES[i % STATUSES.length], department: ['engineering', 'sales', 'support'][i % 3] }
})
)
}
customers = []
for (let i = 0; i < 20; i++) {
customers.push(
await doAdd(`cust-${i}`, {
data: `customer record ${i}`,
type: NounType.Person,
subtype: 'customer',
metadata: { status: STATUSES[i % STATUSES.length], tier: i % 2 === 0 ? 'gold' : 'standard' }
})
)
}
invoices = []
for (let i = 0; i < 30; i++) {
invoices.push(
await doAdd(`inv-${i}`, {
data: `invoice record ${i}`,
type: NounType.Document,
subtype: 'invoice',
metadata: { status: STATUSES[i % STATUSES.length], amount: 100 + i * 17 }
})
)
}
tasks = []
for (let i = 0; i < 25; i++) {
tasks.push(
await doAdd(`task-${i}`, {
data: `task record ${i}`,
type: NounType.Task,
subtype: 'milestone',
metadata: { status: STATUSES[i % STATUSES.length], priority: (i % 5) + 1 }
})
)
}
projects = []
for (let i = 0; i < 25; i++) {
projects.push(
await doAdd(`proj-${i}`, {
data: `project record ${i}`,
type: NounType.Project,
metadata: { status: STATUSES[i % STATUSES.length], budget: 1000 * (i + 1) }
})
)
}
expect(employees.length + customers.length + invoices.length + tasks.length + projects.length).toBe(120)
// Five hubs (proj-0..proj-4) fan out to tasks (Contains) and employees
// (WorksWith); a residual band of invoice->customer RelatedTo edges is
// unrelated to any hub. Hubs are never touched again for the rest of
// the biography, so they stay valid adjacency samples in every chapter.
const hubIds = projects.slice(0, 5)
for (let h = 0; h < 5; h++) {
for (let k = 0; k < 15; k++) {
const taskIdx = (h * 5 + k) % tasks.length
await doRelate({ from: hubIds[h], to: tasks[taskIdx], type: VerbType.Contains, subtype: 'delivers' })
}
for (let k = 0; k < 10; k++) {
const empIdx = (h * 4 + k) % employees.length
await doRelate({ from: hubIds[h], to: employees[empIdx], type: VerbType.WorksWith })
}
}
for (let j = 0; j < 25; j++) {
await doRelate({ from: invoices[j], to: customers[j % customers.length], type: VerbType.RelatedTo, subtype: 'billed-to' })
}
expect(model.relations.size).toBe(150)
// A handful of VFS files.
for (let i = 0; i < 5; i++) {
await doVfsWrite(`/report-${i}.txt`, `founding report ${i}`)
}
await brain.flush()
hubs = hubIds.map((id) => ({ id, typeFilters: [VerbType.Contains, VerbType.WorksWith] }))
await verifyChapter(brain, model, 'Ch1 FOUNDING', { hubs, bucketField: 'status' })
// =================================================================
// CHAPTER 2 — A WORKING DAY
// =================================================================
// Non-hub pool for every mutation below.
nonHub = [...employees, ...customers, ...invoices, ...tasks, ...projects.slice(5)]
// 40 updates that provably MOVE entities across find() status buckets.
const updateTargets = nonHub.slice(0, 40)
for (const id of updateTargets) {
const current = model.entities.get(id)!.metadata.status
await doUpdate(id, { metadata: { status: nextStatus(current) } })
}
// 10 visibility flips (public -> internal).
const visibilityTargets = nonHub.slice(40, 50)
for (const id of visibilityTargets) {
await doUpdate(id, { visibility: 'internal' })
}
// 15 deletes — some hub members (their edges cascade away), 3 of them
// earmarked for Ch6's resurrection.
const resurrectIds = [tasks[0], tasks[1], employees[0]]
const otherDeletes = [
tasks[2], tasks[3], tasks[4], tasks[5], tasks[6],
employees[1], employees[2], employees[3],
customers[0], customers[1], customers[2], customers[3]
]
const ch2DeleteTargets = [...resurrectIds, ...otherDeletes]
expect(ch2DeleteTargets.length).toBe(15)
for (const id of ch2DeleteTargets) {
await doRemove(id)
}
// 20 new adds.
const ch2NewTypes = [NounType.Person, NounType.Document, NounType.Task]
for (let i = 0; i < 20; i++) {
await doAdd(`ch2-new-${i}`, {
data: `working-day addition ${i}`,
type: ch2NewTypes[i % ch2NewTypes.length],
subtype: 'ad-hoc',
metadata: { status: STATUSES[i % STATUSES.length] }
})
}
// 10 updateRelation metadata patches — read AFTER the deletes above,
// so only relations the cascade left alive are ever targeted.
const survivingRelationIds = [...model.relations.keys()].slice(0, 10)
expect(survivingRelationIds.length).toBe(10)
for (const relId of survivingRelationIds) {
await doUpdateRelation(relId, { metadata: { reviewed: true } })
}
await brain.flush()
await verifyChapter(brain, model, 'Ch2 A WORKING DAY', { hubs, bucketField: 'status' })
// =================================================================
// CHAPTER 3 — CLEAN RESTART
// =================================================================
await brain.close()
brain = await openBrain(dir, { logAuthority: 'adopt' })
await verifyChapter(brain, model, 'Ch3 CLEAN RESTART', { hubs, bucketField: 'status' })
// Leave the brain closed and the directory intact for the next `it`
// (the biography continues there) — do NOT remove `dir` here.
await brain.close()
},
300000
)
it(
'Ch4 CRASH -> Ch5 REPAIR -> Ch6 SECOND LIFE: continues the Ch3 store',
async () => {
try {
brain = await openBrain(dir, { logAuthority: 'adopt' })
// ===============================================================
// CHAPTER 4 — CRASH
// ===============================================================
const ch4Types = [NounType.Person, NounType.Document, NounType.Task, NounType.Project]
for (let i = 0; i < 10; i++) {
await doAdd(`ch4-new-${i}`, {
data: `crash-window addition ${i}`,
type: ch4Types[i % ch4Types.length],
metadata: { status: STATUSES[i % STATUSES.length] }
})
}
const ch4UpdateTargets = nonHub.slice(50, 55) // invoices[10..14] — untouched so far
for (const id of ch4UpdateTargets) {
await doUpdate(id, { metadata: { status: 'active' } })
}
// NO flush — abandon exactly the way process death would (the
// at-ack law: every write already awaited above must survive).
await abandonAsCrashed(brain)
brain = await openBrain(dir, { logAuthority: 'adopt' })
await verifyChapter(brain, model, 'Ch4 CRASH', { hubs, bucketField: 'status' })
// ===============================================================
// CHAPTER 5 — REPAIR
// ===============================================================
const report = await brain.repairIndex()
for (const family of report.families) {
const accounted =
family.checked === true || (family.checked === false && typeof family.skipped === 'string' && family.skipped.length > 0)
expect(
accounted,
`[Ch5 REPAIR] family '${family.family}' must be checked or explicitly skipped with a reason; got ${JSON.stringify(family)}`
).toBe(true)
}
// A healthy store: repair must change nothing the model doesn't
// already expect — verifyChapter against the UNCHANGED model proves it.
await verifyChapter(brain, model, 'Ch5 REPAIR', { hubs, bucketField: 'status' })
// ===============================================================
// CHAPTER 6 — SECOND LIFE
// ===============================================================
const ch6Types = [NounType.Person, NounType.Document, NounType.Task, NounType.Project]
for (let i = 0; i < 10; i++) {
await doAdd(`ch6-new-${i}`, {
data: `second-life addition ${i}`,
type: ch6Types[i % ch6Types.length],
metadata: { status: STATUSES[i % STATUSES.length] }
})
}
const ch6UpdateTargets = nonHub.slice(55, 65) // invoices[15..24] — untouched so far
expect(ch6UpdateTargets.every((id) => model.entities.get(id)!.alive)).toBe(true)
for (const id of ch6UpdateTargets) {
await doUpdate(id, { metadata: { status: 'closed' } })
}
const ch6DeleteTargets = nonHub
.slice(65, 90) // invoices[25..29] + tasks[0..19] (some already dead — filtered below)
.filter((id) => model.entities.get(id)!.alive)
.slice(0, 7)
expect(ch6DeleteTargets.length).toBe(7)
for (const id of ch6DeleteTargets) {
await doRemove(id)
}
// Resurrection: the SAME three ids Ch2 deleted, reinserted with
// BRAND-NEW metadata — the model expects the new metadata only.
await doAdd('task-0', { data: 'resurrected task 0', type: NounType.Task, subtype: 'milestone', metadata: { status: 'active', resurrected: true } })
await doAdd('task-1', { data: 'resurrected task 1', type: NounType.Task, subtype: 'milestone', metadata: { status: 'pending', resurrected: true } })
await doAdd('emp-0', { data: 'resurrected employee 0', type: NounType.Person, subtype: 'employee', metadata: { status: 'active', resurrected: true } })
expect(tasks[0]).toBe(uid('task-0')) // same id as Ch1/Ch2 — the resurrection-adjacent shape
await brain.close()
brain = await openBrain(dir, { logAuthority: 'adopt' })
await verifyChapter(brain, model, 'Ch6 SECOND LIFE', { hubs, bucketField: 'status' })
// Final, standalone getCanonicalCounts() exactness check (beyond
// verifyChapter's own (f) leg) — the whole ledger, in one shot.
const finalCounts = await getCanonicalCountsFor(brain)
const aliveEntities = [...model.entities.values()].filter((e) => e.alive)
const alivePublicEntities = aliveEntities.filter((e) => (e.visibility ?? 'public') === 'public')
const aliveVerbs = model.relations.size
expect(finalCounts, 'final getCanonicalCounts() exactness — Ch6 SECOND LIFE').toEqual({
nouns: {
counted: alivePublicEntities.length + model.vfsFileNouns,
all: aliveEntities.length + model.vfsFileNouns + model.vfsBaselineNouns
},
verbs: {
counted: aliveVerbs + model.vfsContainsVerbs,
all: aliveVerbs + model.vfsContainsVerbs + model.vfsBaselineVerbs
},
suspect: false
})
} finally {
await brain.close().catch(() => {})
// Best-effort, retried: a still-draining background persistence
// write (e.g. count/index write-through) can race a single rmSync
// and leave a partial directory behind — retry a couple of times
// rather than let this temp dir leak.
for (let attempt = 0; attempt < 3; attempt++) {
try {
fs.rmSync(dir, { recursive: true, force: true })
if (!fs.existsSync(dir)) break
} catch {
// ignore and retry
}
await new Promise((resolve) => setTimeout(resolve, 100))
}
}
},
300000
)
})

View file

@ -0,0 +1,389 @@
/**
* @module tests/lifecycle/biographyHarness
* @description The referee for the LIFECYCLE LANE (see `biography.test.ts`):
* a plain in-memory SHADOW MODEL of a brain's contents, updated by every act
* the biography performs (add/update/remove/relate/updateRelation/vfs writes),
* plus `verifyChapter()`, which asserts the live brain agrees with the model
* after every chapter. No engine code runs inside the model it is an
* independent ledger, not a mirror of the implementation under test.
*
* COUNT SEMANTICS this harness encodes (verified against the live engine,
* not assumed see the module-level comments below for how each was
* confirmed):
*
* - `getNounCount()` / `getVerbCount()` count PUBLIC-tier alive records only
* (visibility absent or `'public'`) `'internal'` and `'system'` are both
* excluded. `storage.getCanonicalCounts()` mirrors that same PUBLIC-only
* scalar as `counted`, and additionally reports `all` every tier,
* unfiltered as the coverage-ledger denominator (see
* tests/integration/canonical-count-ledger.test.ts).
* - `brain.vfs.writeFile()` for a brand-new file at a path directly under the
* VFS root creates exactly ONE new File noun plus ONE new `Contains` verb
* (root -> file), and BOTH are ordinary PUBLIC records (no visibility
* field is set) so they count toward `getNounCount()`/`getVerbCount()`
* as well as the canonical `all` scalars. Only the VFS ROOT entity itself
* is `'system'`-tier (created once, at `init()`, before any biography
* chapter runs) that lone record is the only hidden-tier footprint the
* model does not construct explicitly, so it is captured empirically via
* `snapshotVfsBaseline()` immediately after `init()` rather than hardcoded.
* - `related()` filters edges by the RELATION's own visibility tier, not by
* the visibility of the entities the edge connects flipping an entity to
* `'internal'` does not hide its edges from `related()`. This lane never
* sets relation visibility, so every relation the model tracks is exactly
* as reachable as its presence in `model.relations` implies.
* - `remove()` cascades: every relation touching the removed entity (as
* `from` or `to`) is hard-deleted along with it. The model mirrors this by
* deleting the relation entirely from `model.relations` (no relation
* "alive" flag presence in the map IS aliveness).
*/
import { expect } from 'vitest'
import type { Brainy } from '../../src/brainy.js'
import type { NounType, VerbType } from '../../src/types/graphTypes.js'
import type { EntityVisibility, StorageAdapter } from '../../src/coreTypes.js'
/**
* One entity's complete lifecycle-relevant state, as the biography's acts
* leave it. `alive: false` means the model believes the id has been removed
* the entry is KEPT (never deleted from the map) so `verifyChapter` can
* assert the negative half of the contract: a dead id must read as `null`.
*/
export interface ShadowEntity {
type: NounType
subtype?: string
metadata: Record<string, unknown>
visibility?: EntityVisibility
alive: boolean
}
/**
* One relation's complete lifecycle-relevant state. There is no `alive`
* flag here presence in {@link ShadowModel.relations} IS aliveness,
* mirroring the engine's hard delete of the canonical verb record on
* cascade (see the module header).
*/
export interface ShadowRelation {
from: string
to: string
type: VerbType
subtype?: string
metadata: Record<string, unknown>
}
/**
* The independent truth ledger the biography updates on every act it
* performs. `verifyChapter` checks the live brain against this never the
* other way around.
*/
export interface ShadowModel {
entities: Map<string, ShadowEntity>
relations: Map<string, ShadowRelation>
/**
* `getCanonicalCounts()` nouns.all / verbs.all captured right after
* `init()`, before chapter 1 the VFS root's own system-tier footprint.
* Set once via {@link snapshotVfsBaseline}; never hardcoded.
*/
vfsBaselineNouns: number
vfsBaselineVerbs: number
/**
* Public nouns/verbs created by `vfs.writeFile()` for a brand-new file at
* a flat top-level path: exactly one File noun + one Contains verb per
* call (see the module header). Bumped by {@link recordVfsFileWrite}.
*/
vfsFileNouns: number
vfsContainsVerbs: number
}
/** A fresh, empty shadow model — call once before chapter 1. */
export function createModel(): ShadowModel {
return {
entities: new Map(),
relations: new Map(),
vfsBaselineNouns: 0,
vfsBaselineVerbs: 0,
vfsFileNouns: 0,
vfsContainsVerbs: 0
}
}
/** Narrow, documented private-storage access (the same style already used by
* `tests/helpers/durabilityKillMatrix.ts`'s `storeOf()`), needed because
* `getCanonicalCounts()` lives on the storage adapter, not on `Brainy`. */
function storageOf(brain: Brainy): StorageAdapter {
return (brain as unknown as { storage: StorageAdapter }).storage
}
/** Public wrapper around the private-storage `getCanonicalCounts()` read, so
* callers never need their own private-access cast used internally by
* {@link snapshotVfsBaseline} and {@link verifyChapter}, and by
* `biography.test.ts` for its final standalone exactness check. */
export async function getCanonicalCountsFor(brain: Brainy): ReturnType<NonNullable<StorageAdapter['getCanonicalCounts']>> {
const storage = storageOf(brain)
if (!storage.getCanonicalCounts) {
throw new Error(
'lifecycle lane: the storage adapter under test has no getCanonicalCounts() — the canonical-count-exactness leg of this lane is unrepresentable without it.'
)
}
return storage.getCanonicalCounts()
}
/**
* Snapshot the VFS root's own hidden-tier footprint. Call exactly once,
* immediately after `init()` and before chapter 1 does anything this is
* the ONE baseline offset the model does not construct by hand (see the
* module header for why: the root is `'system'`-tier plumbing the biography
* never explicitly creates).
*/
export async function snapshotVfsBaseline(brain: Brainy, model: ShadowModel): Promise<void> {
const counts = await getCanonicalCountsFor(brain)
model.vfsBaselineNouns = counts.nouns.all
model.vfsBaselineVerbs = counts.verbs.all
}
/**
* Record one `brain.vfs.writeFile()` call for a brand-new file at a flat
* top-level path (no intermediate directories). Bumps both the noun and verb
* VFS counters by one, matching the engine's actual write path exactly (see
* the module header) never call this for an overwrite of an existing path,
* a nested path (which would also vivify intermediate directory nouns/edges,
* a different, unmodeled shape), or the biography loses its exactness.
*/
export function recordVfsFileWrite(model: ShadowModel): void {
model.vfsFileNouns += 1
model.vfsContainsVerbs += 1
}
/** Record a fresh `add()` (or a Ch6 resurrection `Map.set` fully replaces
* whatever a prior dead entry held, which is exactly the "new metadata only"
* contract a resurrection must honor). */
export function modelAdd(
model: ShadowModel,
id: string,
entity: { type: NounType; subtype?: string; metadata: Record<string, unknown>; visibility?: EntityVisibility }
): void {
model.entities.set(id, {
type: entity.type,
subtype: entity.subtype,
metadata: { ...entity.metadata },
visibility: entity.visibility,
alive: true
})
}
/** Record an `update()` merges metadata by default, matching the engine's
* `merge: true` default; pass `merge: false` to mirror a full replace. */
export function modelUpdate(
model: ShadowModel,
id: string,
patch: { metadata?: Record<string, unknown>; merge?: boolean; visibility?: EntityVisibility }
): void {
const existing = model.entities.get(id)
if (!existing || !existing.alive) {
throw new Error(`shadow model: update() targeted ${id}, which the model does not have alive — biography sequencing bug`)
}
if (patch.metadata) {
existing.metadata = patch.merge === false ? { ...patch.metadata } : { ...existing.metadata, ...patch.metadata }
}
if (patch.visibility !== undefined) {
existing.visibility = patch.visibility
}
}
/** Record a `remove()` marks the entity dead (entry retained, per
* {@link ShadowEntity}) and cascades: every relation touching it, in either
* direction, is hard-deleted from the model too (matching the engine). */
export function modelDelete(model: ShadowModel, id: string): void {
const existing = model.entities.get(id)
if (!existing || !existing.alive) {
throw new Error(`shadow model: remove() targeted ${id}, which the model does not have alive — biography sequencing bug`)
}
existing.alive = false
for (const [relId, rel] of model.relations) {
if (rel.from === id || rel.to === id) model.relations.delete(relId)
}
}
/** Record a `relate()` — `id` is the relation id the real call returned. */
export function modelRelate(
model: ShadowModel,
id: string,
relation: { from: string; to: string; type: VerbType; subtype?: string; metadata?: Record<string, unknown> }
): void {
model.relations.set(id, {
from: relation.from,
to: relation.to,
type: relation.type,
subtype: relation.subtype,
metadata: { ...(relation.metadata ?? {}) }
})
}
/** Record an `updateRelation()` metadata patch — merges by default. */
export function modelUpdateRelation(
model: ShadowModel,
id: string,
patch: { metadata?: Record<string, unknown>; merge?: boolean }
): void {
const existing = model.relations.get(id)
if (!existing) {
throw new Error(`shadow model: updateRelation() targeted ${id}, which the model does not have — biography sequencing bug`)
}
if (patch.metadata) {
existing.metadata = patch.merge === false ? { ...patch.metadata } : { ...existing.metadata, ...patch.metadata }
}
}
/** Order-independent structural equality for plain JSON-shaped metadata. */
function deepEqual(a: unknown, b: unknown): boolean {
if (a === b) return true
if (typeof a !== typeof b) return false
if (a === null || b === null) return a === b
if (typeof a !== 'object') return false
const aKeys = Object.keys(a as Record<string, unknown>)
const bKeys = Object.keys(b as Record<string, unknown>)
if (aKeys.length !== bKeys.length) return false
for (const k of aKeys) {
if (!deepEqual((a as Record<string, unknown>)[k], (b as Record<string, unknown>)[k])) return false
}
return true
}
/** One hub entity to sample for the `related()` adjacency check, plus the
* verb type(s) it is known (by biography construction) to have OUT-edges
* of, so the type-filtered variant is exercised too. */
export interface HubCheck {
id: string
typeFilters: VerbType[]
}
/** Options steering one `verifyChapter()` call. */
export interface VerifyOptions {
/** Hub entities to sample for the `related()` adjacency check. */
hubs: HubCheck[]
/** The metadata field `find()` bucket-checks against (a bare string field
* every alive entity may or may not carry distinct values present among
* ALIVE model entities are discovered automatically each call, so a
* chapter that moves entities across buckets is re-checked exactly). */
bucketField: string
}
/**
* Assert the live brain agrees with the model, in full, after one chapter.
* Every failure message names the chapter `label`, the id (where
* applicable), and expected-vs-actual a red here must be diagnosable from
* the assertion message alone, with no need to re-read this file.
*/
export async function verifyChapter(brain: Brainy, model: ShadowModel, label: string, opts: VerifyOptions): Promise<void> {
// (a) + (b): every alive entity reads back exactly as modeled; every dead
// entity reads as null.
for (const [id, entity] of model.entities) {
const live = await brain.get(id)
if (entity.alive) {
expect(live, `[${label}] alive entity ${id} (type=${entity.type}) must be readable via get(), got null`).not.toBeNull()
const e = live!
expect(e.type, `[${label}] entity ${id} .type mismatch: expected ${entity.type}, got ${e.type}`).toBe(entity.type)
expect(e.subtype, `[${label}] entity ${id} .subtype mismatch: expected ${JSON.stringify(entity.subtype)}, got ${JSON.stringify(e.subtype)}`).toBe(entity.subtype)
expect(
e.visibility,
`[${label}] entity ${id} .visibility mismatch: expected ${JSON.stringify(entity.visibility)}, got ${JSON.stringify(e.visibility)}`
).toBe(entity.visibility)
const metaMatches = deepEqual(e.metadata ?? {}, entity.metadata)
expect(
metaMatches,
`[${label}] entity ${id} .metadata mismatch: expected ${JSON.stringify(entity.metadata)}, got ${JSON.stringify(e.metadata)}`
).toBe(true)
} else {
expect(live, `[${label}] dead entity ${id} (type=${entity.type}) must read as null, got ${JSON.stringify(live)}`).toBeNull()
}
}
// (c) find({ where: { <bucketField>: value } }) returns exactly the
// model's matching alive set, per distinct value currently present.
const bucketValues = new Set<string>()
for (const entity of model.entities.values()) {
if (!entity.alive) continue
const v = entity.metadata[opts.bucketField]
if (typeof v === 'string') bucketValues.add(v)
}
for (const value of bucketValues) {
const expectedIds = [...model.entities.entries()]
.filter(([, e]) => e.alive && e.metadata[opts.bucketField] === value)
.map(([id]) => id)
.sort()
const results = await brain.find({
where: { [opts.bucketField]: value } as Record<string, unknown>,
includeInternal: true,
limit: 100000
})
const actualIds = results.map((r) => r.id).sort()
expect(
actualIds,
`[${label}] find({ where: { ${opts.bucketField}: ${JSON.stringify(value)} } }) mismatch: expected ${expectedIds.length} ids ${JSON.stringify(expectedIds)}, got ${actualIds.length} ids ${JSON.stringify(actualIds)}`
).toEqual(expectedIds)
}
// (d) related(id) / related(id, { type }) for the hub sample matches the
// model's adjacency exactly (out-edges — related(id) is shorthand for
// { from: id }).
for (const hub of opts.hubs) {
const expectedAll = [...model.relations.entries()]
.filter(([, r]) => r.from === hub.id)
.map(([id]) => id)
.sort()
const liveAll = await brain.related({ from: hub.id, limit: 100000 })
const actualAllIds = liveAll.map((r) => r.id).sort()
expect(
actualAllIds,
`[${label}] related(${hub.id}) mismatch: expected ${expectedAll.length} ids ${JSON.stringify(expectedAll)}, got ${actualAllIds.length} ids ${JSON.stringify(actualAllIds)}`
).toEqual(expectedAll)
for (const typeFilter of hub.typeFilters) {
const expectedTyped = [...model.relations.entries()]
.filter(([, r]) => r.from === hub.id && r.type === typeFilter)
.map(([id]) => id)
.sort()
const liveTyped = await brain.related({ from: hub.id, type: typeFilter, limit: 100000 })
const actualTypedIds = liveTyped.map((r) => r.id).sort()
expect(
actualTypedIds,
`[${label}] related(${hub.id}, { type: '${typeFilter}' }) mismatch: expected ${expectedTyped.length} ids ${JSON.stringify(expectedTyped)}, got ${actualTypedIds.length} ids ${JSON.stringify(actualTypedIds)}`
).toEqual(expectedTyped)
}
}
// (e) getNounCount() / getVerbCount(): PUBLIC-tier alive records
// (visibility absent/'public'; 'internal' and 'system' both excluded — see
// the module header) plus the VFS's own public contributions.
const alivePublicNouns = [...model.entities.values()].filter((e) => e.alive && (e.visibility ?? 'public') === 'public').length
const aliveVerbs = model.relations.size
const expectedNounCount = alivePublicNouns + model.vfsFileNouns
const expectedVerbCount = aliveVerbs + model.vfsContainsVerbs
expect(
await brain.getNounCount(),
`[${label}] getNounCount() mismatch: expected ${expectedNounCount} (alive public entities ${alivePublicNouns} + vfs file nouns ${model.vfsFileNouns})`
).toBe(expectedNounCount)
expect(
await brain.getVerbCount(),
`[${label}] getVerbCount() mismatch: expected ${expectedVerbCount} (alive relations ${aliveVerbs} + vfs contains verbs ${model.vfsContainsVerbs})`
).toBe(expectedVerbCount)
// (f) getCanonicalCounts(): ALL-visibility scalars (every tier) equal the
// model's alive totals including hidden tiers, plus the VFS's own
// contributions (both file nouns/verbs AND the once-measured root
// baseline). suspect must be false — every delete in this biography goes
// through brain.remove(), which always proves the record it decrements.
const ledger = await getCanonicalCountsFor(brain)
const aliveAllNouns = [...model.entities.values()].filter((e) => e.alive).length
const expectedNounsAll = aliveAllNouns + model.vfsFileNouns + model.vfsBaselineNouns
const expectedVerbsAll = aliveVerbs + model.vfsContainsVerbs + model.vfsBaselineVerbs
expect(
ledger.nouns.all,
`[${label}] getCanonicalCounts().nouns.all mismatch: expected ${expectedNounsAll} (alive incl. internal ${aliveAllNouns} + vfs file nouns ${model.vfsFileNouns} + vfs root baseline ${model.vfsBaselineNouns})`
).toBe(expectedNounsAll)
expect(
ledger.verbs.all,
`[${label}] getCanonicalCounts().verbs.all mismatch: expected ${expectedVerbsAll} (alive relations ${aliveVerbs} + vfs contains verbs ${model.vfsContainsVerbs} + vfs root baseline ${model.vfsBaselineVerbs})`
).toBe(expectedVerbsAll)
expect(ledger.nouns.counted, `[${label}] getCanonicalCounts().nouns.counted mismatch (should mirror getNounCount())`).toBe(expectedNounCount)
expect(ledger.verbs.counted, `[${label}] getCanonicalCounts().verbs.counted mismatch (should mirror getVerbCount())`).toBe(expectedVerbCount)
expect(ledger.suspect, `[${label}] getCanonicalCounts().suspect must be false — every delete in this biography proves its record`).toBe(false)
}

View file

@ -1,38 +1,49 @@
/** /**
* @module tests/unit/brainy/lazy-notready-honor * @module tests/unit/brainy/lazy-notready-honor
* @description THE SILENT-EMPTY TRAP pin (found during a fleet adoption, * @description THE SILENT-EMPTY TRAP pin (found during a fleet adoption,
* SELF-ENGINE-PAIR-STANDARD): under `disableAutoRebuild: true`, the lazy * SELF-ENGINE-PAIR-STANDARD): under `disableAutoRebuild: true`, the OLD lazy
* first-query path (`ensureIndexesLoaded`) assessed ONLY the vector index's * first-query path (`ensureIndexesLoaded`) assessed ONLY the vector index's
* readiness a native METADATA provider reporting not-ready (its strand * readiness a native METADATA provider reporting not-ready (its strand
* report) never blocked the completion latch, so the promised lazy rebuild * report) never blocked the completion latch, so the promised lazy rebuild
* never fired and every `find()` silently returned `[]` on a populated * never fired and every `find()` silently returned `[]` on a populated store
* store (measured: 52 entities durable-but-unqueryable, first query * (measured: 52 entities durable-but-unqueryable, first query 0ms/0 rows).
* 0ms/0 rows). The law: a not-ready report from ANY provider falls through *
* to the rebuild never a silent empty. * RE-POINTED to the health-gate law (a read never builds; a rebuild runs
* entirely at open): `ensureIndexesLoaded()` is now a pure CHECK. A not-ready
* report from ANY provider metadata, vector, or graph makes it THROW the
* matching typed `*NotReadyError` rather than silently letting the read
* proceed, and it NEVER calls `rebuildIndexesIfNeeded` (that is entirely
* open()'s job now see the second describe block below). The spirit is
* unchanged: a not-ready report from any single provider can never be
* shadowed into a silent empty result.
* *
* White-box provider-double pattern per tests/unit/brainy/migration-deference. * White-box provider-double pattern per tests/unit/brainy/migration-deference.
*/ */
import { describe, it, expect, afterEach, vi } from 'vitest' import { describe, it, expect, afterEach, vi } from 'vitest'
import { Brainy } from '../../../src/index.js' import { mkdtempSync, rmSync } from 'node:fs'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { Brainy, MetadataIndexNotReadyError } from '../../../src/index.js'
import { NounType } from '../../../src/types/graphTypes.js' import { NounType } from '../../../src/types/graphTypes.js'
import { createTestConfig } from '../../helpers/test-factory.js' import { createTestConfig } from '../../helpers/test-factory.js'
interface BrainInternals { interface BrainInternals {
index: { size(): number } index: { size(): number }
metadataIndex: { isReady?: () => boolean } metadataIndex: { isReady?: () => boolean }
lazyRebuildCompleted: boolean ensureIndexesLoaded(): void
ensureIndexesLoaded(): Promise<void>
rebuildIndexesIfNeeded(force?: boolean): Promise<void> rebuildIndexesIfNeeded(force?: boolean): Promise<void>
} }
const brains: Brainy[] = [] const brains: Brainy[] = []
const dirs: string[] = []
afterEach(async () => { afterEach(async () => {
for (const b of brains.splice(0)) await b.close().catch(() => {}) for (const b of brains.splice(0)) await b.close().catch(() => {})
for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true })
vi.restoreAllMocks() vi.restoreAllMocks()
}) })
async function warmLazyBrain(): Promise<{ brain: Brainy; internals: BrainInternals }> { async function warmBrain(): Promise<{ brain: Brainy; internals: BrainInternals }> {
const brain = new Brainy(createTestConfig({ disableAutoRebuild: true })) const brain = new Brainy(createTestConfig({ disableAutoRebuild: true }))
await brain.init() await brain.init()
brains.push(brain) brains.push(brain)
@ -40,36 +51,59 @@ async function warmLazyBrain(): Promise<{ brain: Brainy; internals: BrainInterna
await brain.add({ data: `row ${i}`, type: NounType.Document, metadata: { i } }) await brain.add({ data: `row ${i}`, type: NounType.Document, metadata: { i } })
} }
const internals = brain as unknown as BrainInternals const internals = brain as unknown as BrainInternals
internals.lazyRebuildCompleted = false // simulate the cold first query
return { brain, internals } return { brain, internals }
} }
describe('lazy path honors EVERY providers not-ready report', () => { describe('the read gate honors EVERY providers not-ready report', () => {
it('a not-ready METADATA provider blocks the completion latch and fires the rebuild', async () => { it('a not-ready METADATA provider refuses loudly — it never lets a read proceed, and it never rebuilds', async () => {
const { internals } = await warmLazyBrain() const { internals } = await warmBrain()
// The trap's shape: vector side looks fine (populated), metadata // The trap's shape: vector side looks fine (populated), metadata
// provider says NOT ready — the old gate latched complete here. // provider says NOT ready — the OLD gate silently latched complete here.
;(internals.metadataIndex as { isReady?: () => boolean }).isReady = () => false // The new gate refuses loudly instead; a read never triggers a rebuild.
const rebuildSpy = vi internals.metadataIndex.isReady = () => false
.spyOn(internals, 'rebuildIndexesIfNeeded') const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded').mockResolvedValue(undefined)
.mockResolvedValue(undefined)
await internals.ensureIndexesLoaded() expect(() => internals.ensureIndexesLoaded()).toThrow(MetadataIndexNotReadyError)
expect(rebuildSpy, 'a read NEVER triggers a rebuild — building is entirely open()\'s job now').not.toHaveBeenCalled()
expect(rebuildSpy, 'not-ready metadata provider must fire the lazy rebuild').toHaveBeenCalledWith(true)
}) })
it('control: all providers ready/unknown+populated → latch completes, no rebuild', async () => { it('control: all providers ready/unknown+populated → the gate lets the read through, no rebuild', async () => {
const { internals } = await warmLazyBrain() const { internals } = await warmBrain()
;(internals.metadataIndex as { isReady?: () => boolean }).isReady = () => true internals.metadataIndex.isReady = () => true
const rebuildSpy = vi const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded').mockResolvedValue(undefined)
.spyOn(internals, 'rebuildIndexesIfNeeded')
.mockResolvedValue(undefined)
await internals.ensureIndexesLoaded()
expect(() => internals.ensureIndexesLoaded()).not.toThrow()
expect(rebuildSpy).not.toHaveBeenCalled() expect(rebuildSpy).not.toHaveBeenCalled()
expect(internals.lazyRebuildCompleted).toBe(true)
}) })
}) })
describe('the open-time build honors the same law: a needed rebuild runs at open, never deferred to a read', () => {
it('disableAutoRebuild:true does not defer a needed rebuild past open() on a reopened, populated store', async () => {
const dir = mkdtempSync(join(tmpdir(), 'brainy-lazy-notready-honor-'))
dirs.push(dir)
const writer = new Brainy(createTestConfig({ disableAutoRebuild: true, storage: { type: 'filesystem', path: dir } }))
await writer.init()
for (let i = 0; i < 3; i++) {
await writer.add({ data: `row ${i}`, type: NounType.Document, metadata: { i } })
}
await writer.flush()
await writer.close()
// Fresh instance over the same store: its derived indexes start empty in
// memory, so open()'s rebuildIndexesIfNeeded MUST fire (and complete)
// before init() returns — even though disableAutoRebuild is true, there
// is no first-query lazy path left to defer to.
const reader = new Brainy(createTestConfig({ disableAutoRebuild: true, storage: { type: 'filesystem', path: dir } }))
const internals = reader as unknown as { rebuildIndexesIfNeeded(force?: boolean): Promise<void> }
const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded')
await reader.init()
brains.push(reader)
expect(rebuildSpy).toHaveBeenCalledTimes(1)
const rows = await reader.find({ where: { i: 1 } })
expect(rows.length).toBe(1)
}, 30000)
})

View file

@ -1,15 +1,18 @@
/** /**
* @module tests/unit/brainy/metadata-provider-contract * @module tests/unit/brainy/metadata-provider-contract
* @description Brainy-side wiring of the two metadata-provider contract additions * @description Brainy-side wiring of the metadata-provider contract.
* confirmed with cor for the lockstep:
* *
* 1. `probeConsistency()` an OPTIONAL O(1) cold-open consistency sampler. On the * `getIdsForFilter(filter, opts?)` brainy passes a page bound on the UNSORTED
* first read, brainy calls it once; on `false` it self-heals via * `find({ type, where, limit })` path so a native provider can early-stop. The JS
* `detectAndRepairCorruption()` (the metadata counterpart of the graph cold-load * index ignores `opts`.
* guard). The native provider implements it; the JS index omits it (no-op). *
* 2. `getIdsForFilter(filter, opts?)` brainy passes a page bound on the UNSORTED * RETIRED (health-gate law): `probeConsistency()` / `ensureMetadataConsistencyProbed()`
* `find({ type, where, limit })` path so a native provider can early-stop. The JS * a read-time consistency probe that launches `detectAndRepairCorruption()` on
* index ignores `opts`. * `false` was exactly the read-triggered dark rebuild the law forbids (a read must
* never start a store walk or a rebuild). The probe's diagnostic value lives on in
* `validateIndexConsistency()` / `repairIndex()`, which remain explicit, operator-invoked
* calls. The pin below confirms the retirement: `probeConsistency()` is never called by
* a read, even when a provider exposes it.
* *
* These are unit tests of brainy's CALL behaviour (the real end-to-end honoring is * These are unit tests of brainy's CALL behaviour (the real end-to-end honoring is
* exercised by cor's combined matrix); they inject probe/spy hooks onto the live JS * exercised by cor's combined matrix); they inject probe/spy hooks onto the live JS
@ -19,7 +22,7 @@ import { describe, it, expect, beforeEach } from 'vitest'
import { Brainy } from '../../../src/brainy' import { Brainy } from '../../../src/brainy'
import { NounType } from '../../../src/types/graphTypes' import { NounType } from '../../../src/types/graphTypes'
describe('metadata-provider contract wiring (probeConsistency + getIdsForFilter opts)', () => { describe('metadata-provider contract wiring (getIdsForFilter opts)', () => {
let brain: Brainy<any> let brain: Brainy<any>
let mi: any let mi: any
@ -29,47 +32,23 @@ describe('metadata-provider contract wiring (probeConsistency + getIdsForFilter
await brain.add({ data: 'a', type: NounType.Thing, metadata: { kind: 'x' } }) await brain.add({ data: 'a', type: NounType.Thing, metadata: { kind: 'x' } })
await brain.add({ data: 'b', type: NounType.Thing, metadata: { kind: 'y' } }) await brain.add({ data: 'b', type: NounType.Thing, metadata: { kind: 'y' } })
mi = (brain as any).metadataIndex mi = (brain as any).metadataIndex
;(brain as any)._metadataConsistencyProbed = false // reset the one-shot guard
}) })
it('calls probeConsistency once on cold open and self-heals via detectAndRepairCorruption on false', async () => { it('RETIRED: a read never calls probeConsistency() / self-heals via detectAndRepairCorruption — that is the read-triggered dark rebuild the health-gate law forbids', async () => {
let probes = 0 let probes = 0
let repairs = 0 let repairs = 0
mi.probeConsistency = async () => { probes++; return false } // corrupt → must repair mi.probeConsistency = async () => { probes++; return false } // would-be corrupt signal
const origRepair = mi.detectAndRepairCorruption.bind(mi) const origRepair = mi.detectAndRepairCorruption.bind(mi)
mi.detectAndRepairCorruption = async () => { repairs++; return origRepair() } mi.detectAndRepairCorruption = async () => { repairs++; return origRepair() }
await brain.find({ where: { kind: 'x' } }) await brain.find({ where: { kind: 'x' } })
expect(probes).toBe(1)
expect(repairs).toBe(1)
// Second read must NOT re-probe (once per brain).
await brain.find({ where: { kind: 'y' } }) await brain.find({ where: { kind: 'y' } })
expect(probes).toBe(1)
expect(repairs).toBe(1)
})
it('does NOT repair when the probe reports healthy', async () => { expect(probes).toBe(0) // no read-time probe exists anymore
let repairs = 0 expect(repairs).toBe(0) // and therefore no read-triggered self-heal either
mi.probeConsistency = async () => true // clean
const origRepair = mi.detectAndRepairCorruption.bind(mi)
mi.detectAndRepairCorruption = async () => { repairs++; return origRepair() }
await brain.find({ where: { kind: 'x' } }) delete mi.probeConsistency
expect(repairs).toBe(0) mi.detectAndRepairCorruption = origRepair
})
it('a probe failure never breaks the read (best-effort, retried next time)', async () => {
let probes = 0
mi.probeConsistency = async () => { probes++; throw new Error('probe boom') }
// The read still succeeds despite the throwing probe.
const rows = await brain.find({ where: { kind: 'x' } })
expect(rows.length).toBe(1)
expect(probes).toBe(1)
// Guard reset on failure → the next read retries the probe.
await brain.find({ where: { kind: 'y' } })
expect(probes).toBe(2)
}) })
it('passes a page bound to getIdsForFilter on the unsorted find path (offset 0, brainy re-windows)', async () => { it('passes a page bound to getIdsForFilter on the unsorted find path (offset 0, brainy re-windows)', async () => {

View file

@ -25,7 +25,7 @@
*/ */
import { describe, it, expect, afterEach, vi } from 'vitest' import { describe, it, expect, afterEach, vi } from 'vitest'
import { Brainy } from '../../../src/index.js' import { Brainy, VectorIndexNotReadyError } from '../../../src/index.js'
import { NounType } from '../../../src/types/graphTypes.js' import { NounType } from '../../../src/types/graphTypes.js'
import { createTestConfig } from '../../helpers/test-factory.js' import { createTestConfig } from '../../helpers/test-factory.js'
import { BaseStorage } from '../../../src/storage/baseStorage.js' import { BaseStorage } from '../../../src/storage/baseStorage.js'
@ -43,9 +43,8 @@ interface BrainInternals {
metadataIndex: { rebuild(...a: unknown[]): Promise<unknown> } metadataIndex: { rebuild(...a: unknown[]): Promise<unknown> }
graphIndex: { size(): number; rebuild(...a: unknown[]): Promise<unknown> } graphIndex: { size(): number; rebuild(...a: unknown[]): Promise<unknown> }
_indexEpochStale: boolean _indexEpochStale: boolean
lazyRebuildCompleted: boolean
rebuildIndexesIfNeeded(force?: boolean): Promise<void> rebuildIndexesIfNeeded(force?: boolean): Promise<void>
ensureIndexesLoaded(): Promise<void> ensureIndexesLoaded(): void
storage: { readRawObject(p: string): Promise<unknown> } storage: { readRawObject(p: string): Promise<unknown> }
} }
@ -181,40 +180,40 @@ describe('rc.8 no-freeze migration deference (isMigrating / stampBrainFormat / b
expect(idxSpy).toHaveBeenCalledTimes(1) expect(idxSpy).toHaveBeenCalledTimes(1)
}) })
// --- Hook 1: large-path first-query lazy force-rebuild deference ---------- // --- Hook 1: read-gate deference (RE-POINTED — the health-gate law retired
// the first-query lazy force-rebuild entirely: ensureIndexesLoaded() is now
// a pure CHECK that never calls rebuildIndexesIfNeeded, migrating or not.
// What survives from the original law is the DEFERENCE itself: a migrating
// provider's report is never judged by the gate — it neither throws nor
// rebuilds — while the exact same not-ready report on a NON-migrating
// provider throws the typed error instead of ever rebuilding.) ------------
it('lazy first-query force-rebuild is SKIPPED when the vector provider isMigrating()', async () => { it('the read gate defers to a migrating vector provider — a not-ready report neither throws nor rebuilds', async () => {
// disableAutoRebuild routes first queries through ensureIndexesLoaded() (the
// large-brain lazy path that would otherwise force a blocking rebuild).
const brain = await makeWarmBrain(2, { disableAutoRebuild: true }) const brain = await makeWarmBrain(2, { disableAutoRebuild: true })
const internals = internalsOf(brain) const internals = internalsOf(brain)
const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded').mockResolvedValue(undefined) const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded').mockResolvedValue(undefined)
// Simulate a cold/empty live vector index (cor is mid-swap, serving canonical). // Simulate a not-ready live vector index (cor is mid-swap, serving canonical).
vi.spyOn(internals.index, 'size').mockReturnValue(0) ;(internals.index as unknown as { isReady?: () => boolean }).isReady = () => false
internals.lazyRebuildCompleted = false
setMigrating(internals.index, true) setMigrating(internals.index, true)
await internals.ensureIndexesLoaded() expect(() => internals.ensureIndexesLoaded()).not.toThrow()
// A query during cor's background swap must not trigger brainy's own
// A query during cor's background swap must not trigger brainy's blocking rebuild. // rebuild — reads never rebuild in any case, migrating or not.
expect(rebuildSpy).toHaveBeenCalledTimes(0) expect(rebuildSpy).toHaveBeenCalledTimes(0)
}) })
it('lazy first-query force-rebuild STILL fires when the vector provider is not migrating (control)', async () => { it('the read gate THROWS for the same not-ready vector provider once migration clears (control)', async () => {
const brain = await makeWarmBrain(2, { disableAutoRebuild: true }) const brain = await makeWarmBrain(2, { disableAutoRebuild: true })
const internals = internalsOf(brain) const internals = internalsOf(brain)
const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded').mockResolvedValue(undefined) const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded').mockResolvedValue(undefined)
vi.spyOn(internals.index, 'size').mockReturnValue(0) ;(internals.index as unknown as { isReady?: () => boolean }).isReady = () => false
internals.lazyRebuildCompleted = false
// No isMigrating → not deferring. // No isMigrating → not deferring.
await internals.ensureIndexesLoaded() expect(() => internals.ensureIndexesLoaded()).toThrow(VectorIndexNotReadyError)
// Still never rebuilds — the gate refuses loudly instead.
// Without deference, the cold empty index drives the lazy force-rebuild. expect(rebuildSpy).toHaveBeenCalledTimes(0)
expect(rebuildSpy).toHaveBeenCalledTimes(1)
expect(rebuildSpy).toHaveBeenCalledWith(true)
}) })
// --- Hook 2: public stampBrainFormat() ----------------------------------- // --- Hook 2: public stampBrainFormat() -----------------------------------

View file

@ -3,10 +3,14 @@
* reported cold `find({ where })` returning a silent `[]` on a freshly-opened * reported cold `find({ where })` returning a silent `[]` on a freshly-opened
* brain (a native metadata index that reports data but has not loaded its field * brain (a native metadata index that reports data but has not loaded its field
* postings). This guard, the field-index counterpart of verifyGraphAdjacencyLive, * postings). This guard, the field-index counterpart of verifyGraphAdjacencyLive,
* probes a known persisted value on the first filtered find(): if the index does * probes a known persisted value on the first filtered find().
* not serve it, brainy rebuilds and re-probes, and raises a loud *
* MetadataIndexNotReadyError only if the rebuild still can't serve never a * RE-POINTED to the health-gate law: the guard NEVER rebuilds and NEVER walks
* silent empty result that misrepresents existing data. * the store from a read a read-path rebuild is exactly the dark-rebuild
* failure mode the law retires (open() alone owns building). When the probe
* cannot serve the known value it raises a loud MetadataIndexNotReadyError
* IMMEDIATELY, with no rebuild attempt in between never a silent empty
* result that misrepresents existing data.
* *
* The 8.0 JS index cold-loads correctly, so we simulate the cold native failure * The 8.0 JS index cold-loads correctly, so we simulate the cold native failure
* mode by intercepting the provider's getIdsForFilter/rebuild. * mode by intercepting the provider's getIdsForFilter/rebuild.
@ -42,37 +46,19 @@ describe('Metadata cold-read guard (#venue silent-[])', () => {
mi.rebuild = origRebuild mi.rebuild = origRebuild
}) })
it('cold index: verifyMetadataLive self-heals via rebuild — find({where}) is correct, NOT silent []', async () => { it('cold index: verifyMetadataLive REFUSES immediately — find({where}) throws MetadataIndexNotReadyError, NEVER a silent [], and NEVER a rebuild attempt', async () => {
const mi = brain.metadataIndex const mi = brain.metadataIndex
const origGetIds = mi.getIdsForFilter.bind(mi) const origGetIds = mi.getIdsForFilter.bind(mi)
let rebuilds = 0
const origRebuild = mi.rebuild.bind(mi) const origRebuild = mi.rebuild.bind(mi)
let cold = true
brain._metadataVerified = false // re-arm the one-shot for this scenario brain._metadataVerified = false // re-arm the one-shot for this scenario
mi.getIdsForFilter = async (...a: any[]) => (cold ? [] : origGetIds(...a)) mi.getIdsForFilter = async () => [] // cold: the known value never resolves
mi.rebuild = async () => { mi.rebuild = async () => { rebuilds++; return origRebuild() }
await origRebuild()
cold = false // the rebuild warms the postings
}
try {
const res = await brain.find({ where: { status: 'active' }, limit: 100 })
expect(res.length).toBe(1) // self-healed — the known entity is returned
} finally {
mi.getIdsForFilter = origGetIds
mi.rebuild = origRebuild
}
})
it('unrecoverably cold index: find({where}) throws MetadataIndexNotReadyError — never a silent []', async () => {
const mi = brain.metadataIndex
const origGetIds = mi.getIdsForFilter.bind(mi)
const origRebuild = mi.rebuild.bind(mi)
brain._metadataVerified = false
mi.getIdsForFilter = async () => [] // always cold; rebuild can't fix it
mi.rebuild = async () => {}
try { try {
await expect(brain.find({ where: { status: 'active' }, limit: 100 })).rejects.toBeInstanceOf( await expect(brain.find({ where: { status: 'active' }, limit: 100 })).rejects.toBeInstanceOf(
MetadataIndexNotReadyError MetadataIndexNotReadyError
) )
expect(rebuilds).toBe(0) // the guard never rebuilds from a read — it refuses loudly instead
} finally { } finally {
mi.getIdsForFilter = origGetIds mi.getIdsForFilter = origGetIds
mi.rebuild = origRebuild mi.rebuild = origRebuild

View file

@ -0,0 +1,71 @@
/**
* @module tests/unit/plugin-activation-loudness
* @description The plugin-activation swallow closes. Two laws:
* (1) THE NOT-INSTALLED FREE PASS IS EXACT a resolution failure earns the
* silent skip ONLY when it names the probed package itself, terminated
* where the name ends. A missing platform-binary SIBLING package
* ("<pkg>-linux-x64-gnu" what a deploy replacing node_modules
* mid-restart leaves), an inner file path, or a dependency failure is a
* BROKEN install and must fail loud. A production storm ran 90s of
* throttled WASM behind this exact prefix-match hole.
* (2) A GRACEFUL DECLINE IS NARRATED ON THE ALWAYS-ON CHANNEL activate()
* returning false warns via prodLog, which `silent: true` cannot patch
* away; a declined accelerator is never an invisible degrade.
*/
import { describe, it, expect, vi, afterEach } from 'vitest'
import { Brainy } from '../../src/brainy.js'
import { prodLog } from '../../src/utils/logger.js'
const isNotInstalled = (error: unknown, pkg: string): boolean =>
(Brainy as unknown as {
isPackageNotInstalledError(e: unknown, p: string): boolean
}).isPackageNotInstalledError(error, pkg)
const resolutionError = (message: string): Error => {
const e = new Error(message) as Error & { code?: string }
e.code = 'ERR_MODULE_NOT_FOUND'
return e
}
describe('the not-installed free pass is exact', () => {
const PKG = '@soulcraft/cor'
it('the package itself, quoted or bare → not-installed (the one free path)', () => {
expect(isNotInstalled(resolutionError(`Cannot find package '${PKG}' imported from /app/x.js`), PKG)).toBe(true)
expect(isNotInstalled(resolutionError(`Cannot find module ${PKG}`), PKG)).toBe(true)
})
it('a missing platform-binary SIBLING package is a broken install, never not-installed', () => {
expect(isNotInstalled(resolutionError(`Cannot find package '${PKG}-linux-x64-gnu' imported from /app`), PKG)).toBe(false)
expect(isNotInstalled(resolutionError(`Failed to resolve ${PKG}-darwin-arm64`), PKG)).toBe(false)
})
it('an inner file path or a non-resolution error is never not-installed', () => {
expect(isNotInstalled(resolutionError(`Cannot find module '/app/node_modules/${PKG}/native/b.node'`), PKG)).toBe(false)
expect(isNotInstalled(new Error(`dlopen failed: wrong ELF class in ${PKG}`), PKG)).toBe(false)
})
})
describe('a graceful decline is narrated on the always-on channel', () => {
afterEach(() => vi.restoreAllMocks())
it('activate() → false warns via prodLog even under silent: true', async () => {
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
const warn = vi.spyOn(prodLog, 'warn')
const brain: any = new Brainy({
requireSubtype: false,
storage: { type: 'memory' },
silent: true,
dimensions: 384
})
brain.use({ name: 'declining-accelerator', activate: async () => false })
await brain.init()
try {
expect(
warn.mock.calls.some((c) => String(c[0]).includes('"declining-accelerator" declined activation'))
).toBe(true)
} finally {
await brain.close().catch(() => {})
}
})
})

View file

@ -63,6 +63,9 @@ function inGate(rel: string): boolean {
return ( return (
rel.startsWith('tests/unit/') || rel.startsWith('tests/unit/') ||
rel.startsWith('tests/integration/') || rel.startsWith('tests/integration/') ||
// The lifecycle biography lane — included by the integration config
// ('tests/lifecycle/**/*.test.ts'; see tests/lifecycle/README.md).
rel.startsWith('tests/lifecycle/') ||
rel.endsWith('.unit.test.ts') || rel.endsWith('.unit.test.ts') ||
rel.endsWith('.integration.test.ts') rel.endsWith('.integration.test.ts')
) )

View file

@ -0,0 +1,157 @@
/**
* @module tests/unit/utils/indexReadiness
* @description Pins for the read-gate authority, {@link assessProviderHealth}, and
* its older sibling {@link assessIndexReadiness}. The health-gate law: a provider's
* NAMED, synchronous, O(1) health report when exposed REPLACES the `isReady()`/
* size-heuristic fallback as the read gate's source of truth. A throw from
* `healthReport()` is a CONTRACT VIOLATION (never read as healthy, never swallowed
* into "unknown"); an `unledgered` family is UNKNOWN (never healthy, never broken
* `serving` is always the provider's own verdict, verbatim).
*/
import { describe, it, expect } from 'vitest'
import { assessIndexReadiness, assessProviderHealth } from '../../../src/utils/indexReadiness.js'
import type { HealthReport, LedgerInvariantResult } from '../../../src/plugin.js'
function invariant(overrides: Partial<LedgerInvariantResult> = {}): LedgerInvariantResult {
return {
name: 'manifest-residency',
holds: true,
detail: 'ok',
heal: 'none',
source: 'ledger',
...overrides
}
}
function report(overrides: Partial<HealthReport> = {}): HealthReport {
return {
provider: 'vector',
healthy: true,
serving: true,
invariants: [],
// A FIXED stamp, never Date.now(): the pin at :98 compares two
// independently-built reports, and a live clock made them differ by 1ms
// whenever the millisecond ticked between the two calls — a plant-lane
// red that had nothing to do with the code under test.
checkedAt: 1_700_000_000_000,
durationMs: 1,
generation: 1,
unledgered: [],
...overrides
}
}
describe('assessIndexReadiness (legacy isReady() classifier)', () => {
it('unknown when the provider is null/undefined', () => {
expect(assessIndexReadiness(null)).toBe('unknown')
expect(assessIndexReadiness(undefined)).toBe('unknown')
})
it('unknown when isReady() is absent', () => {
expect(assessIndexReadiness({})).toBe('unknown')
})
it('ready / not-ready mirror isReady()', () => {
expect(assessIndexReadiness({ isReady: () => true })).toBe('ready')
expect(assessIndexReadiness({ isReady: () => false })).toBe('not-ready')
})
})
describe('assessProviderHealth — the read-gate authority', () => {
it('via "none": no provider at all', () => {
const a = assessProviderHealth(null)
expect(a.via).toBe('none')
expect(a.readiness).toBe('unknown')
expect(a.report).toBeNull()
expect(a.reasons.length).toBeGreaterThan(0)
})
it('via "size-heuristic": provider exposes neither healthReport() nor isReady()', () => {
const a = assessProviderHealth({})
expect(a.via).toBe('size-heuristic')
expect(a.readiness).toBe('unknown')
expect(a.report).toBeNull()
})
it('via "is-ready": provider exposes isReady() but no healthReport() — ready', () => {
const a = assessProviderHealth({ isReady: () => true })
expect(a.via).toBe('is-ready')
expect(a.readiness).toBe('ready')
expect(a.reasons).toEqual([])
})
it('via "is-ready": isReady() === false — not-ready with a reason', () => {
const a = assessProviderHealth({ isReady: () => false })
expect(a.via).toBe('is-ready')
expect(a.readiness).toBe('not-ready')
expect(a.reasons.length).toBeGreaterThan(0)
})
it('healthReport() present REPLACES isReady() — serving:true wins even if isReady() lies false', () => {
const p = { isReady: () => false, healthReport: () => report({ serving: true }) }
const a = assessProviderHealth(p)
expect(a.via).toBe('health-report')
expect(a.readiness).toBe('ready')
})
it('serving:true, healthy:true, no invariants failing → ready, no reasons', () => {
const p = { healthReport: () => report({ serving: true, healthy: true }) }
const a = assessProviderHealth(p)
expect(a.readiness).toBe('ready')
expect(a.reasons).toEqual([])
expect(a.report).toEqual(report({ serving: true, healthy: true }))
})
it('serving:false with a named heal:"rebuild" failing invariant → not-ready, reason names it', () => {
const failing = invariant({ name: 'posted-count-floor', holds: false, heal: 'rebuild', detail: 'posted 10 < canonical 20' })
const p = { healthReport: () => report({ serving: false, healthy: false, invariants: [failing] }) }
const a = assessProviderHealth(p)
expect(a.readiness).toBe('not-ready')
expect(a.reasons.some((r) => r.includes('posted-count-floor') && r.includes('heal:rebuild') && r.includes('posted 10 < canonical 20'))).toBe(true)
})
it('unledgered-only report (serving:true, no failing invariant) → ready, reason names the unledgered family', () => {
const p = { healthReport: () => report({ serving: true, healthy: true, unledgered: ['canonical-verb-coverage'] }) }
const a = assessProviderHealth(p)
expect(a.readiness).toBe('ready')
expect(a.reasons.some((r) => r.includes('unledgered') && r.includes('canonical-verb-coverage'))).toBe(true)
})
it('UNLEDGERED IS UNKNOWN: an unledgered family never flips a NOT-serving provider to ready', () => {
const failing = invariant({ holds: false, heal: 'rebuild', name: 'x' })
const p = { healthReport: () => report({ serving: false, healthy: false, invariants: [failing], unledgered: ['some-family'] }) }
const a = assessProviderHealth(p)
expect(a.readiness).toBe('not-ready')
})
it('serving:true, healthy:false with a heal:"repair" failure → still ready (degraded-but-serving)', () => {
const failing = invariant({ name: 'stale-counter', holds: false, heal: 'repair', detail: 'counter drift' })
const p = { healthReport: () => report({ serving: true, healthy: false, invariants: [failing] }) }
const a = assessProviderHealth(p)
expect(a.readiness).toBe('ready')
expect(a.reasons.some((r) => r.includes('stale-counter') && r.includes('heal:repair'))).toBe(true)
})
it('healthReport() that THROWS is a CONTRACT VIOLATION: not-ready, via health-report, reason names the throw — never "unknown"', () => {
const p = { healthReport: () => { throw new Error('mmap window busy') } }
const a = assessProviderHealth(p)
expect(a.via).toBe('health-report')
expect(a.readiness).toBe('not-ready')
expect(a.report).toBeNull()
expect(a.reasons.some((r) => r.includes('mmap window busy'))).toBe(true)
expect(a.readiness).not.toBe('unknown')
})
it('healthReport() that throws a non-Error value still produces a named reason (String(err))', () => {
const p = { healthReport: () => { throw 'boom' } }
const a = assessProviderHealth(p)
expect(a.readiness).toBe('not-ready')
expect(a.reasons.some((r) => r.includes('boom'))).toBe(true)
})
it('the returned report carries the generation for narration dedup', () => {
const p = { healthReport: () => report({ generation: 42 }) }
const a = assessProviderHealth(p)
expect(a.report?.generation).toBe(42)
})
})

View file

@ -11,8 +11,11 @@
* Same rule, same verdict names as the shipped aggregation machinery * Same rule, same verdict names as the shipped aggregation machinery
* (AggregationIndex.stateAdoptionVerdict). * (AggregationIndex.stateAdoptionVerdict).
* *
* The verdict is COMPUTED AND EXPOSED only these pins assert no rebuild * The verdict is computed at init and consumed via
* trigger changed; acting on 'catchup' lands with the coordinator's wiring. * {@link MetadataIndexManager.applyWatermarkCatchup} the coordinator
* (`Brainy.performInit`) calls it right after `init()`, with an open fact
* scan when the verdict is `'catchup'`. This file pins both halves: the
* verdict computation (above) and the fold/no-op/demotion behavior below.
*/ */
import { describe, it, expect, vi, afterEach } from 'vitest' import { describe, it, expect, vi, afterEach } from 'vitest'
import { v4 as uuidv4 } from 'uuid' import { v4 as uuidv4 } from 'uuid'
@ -22,6 +25,46 @@ import {
} from '../../../src/utils/metadataIndex.js' } from '../../../src/utils/metadataIndex.js'
import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js' import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js'
import { prodLog } from '../../../src/utils/logger.js' import { prodLog } from '../../../src/utils/logger.js'
import type { CommitFact, FactScanBatch, FactScanHandle } from '../../../src/db/factLog.js'
/** A fact scan handle over an in-memory list of facts batches them one
* fact at a time (batch size is irrelevant to the fold, which reads
* `batch.facts` only). */
function fakeScan(facts: CommitFact[]): FactScanHandle {
return {
headGeneration: facts.length > 0 ? facts[facts.length - 1].generation : 0,
segmentCount: 1,
approxFactCount: facts.length,
async *batches(): AsyncGenerator<FactScanBatch> {
for (const fact of facts) {
yield {
facts: [fact],
firstGeneration: fact.generation,
lastGeneration: fact.generation,
factCount: 1,
byteSize: 0,
segmentId: 'fake'
}
}
},
summary: () => ({ factsYielded: facts.length, segmentsRead: 1 })
}
}
/** One noun after-image fact the flat-record shape (no nested `metadata`
* key), matching this file's existing `writeArtifact` convention. */
function nounAdd(generation: number, id: string, metadata: Record<string, unknown>): CommitFact {
return {
generation,
timestamp: Date.now(),
ops: [{ kind: 'noun', id, record: { metadata, vector: null } }]
}
}
/** One noun tombstone fact. */
function nounDelete(generation: number, id: string): CommitFact {
return { generation, timestamp: Date.now(), ops: [{ kind: 'noun', id, record: null }] }
}
/** Fresh storage with a controllable committed generation. */ /** Fresh storage with a controllable committed generation. */
async function makeStorage(committed: number | null): Promise<MemoryStorage> { async function makeStorage(committed: number | null): Promise<MemoryStorage> {
@ -169,3 +212,106 @@ describe('metadata index — watermark stamp + three-way load verdict', () => {
expect(await storage.getMetadata(METADATA_INDEX_STAMP_KEY)).toBeNull() expect(await storage.getMetadata(METADATA_INDEX_STAMP_KEY)).toBeNull()
}) })
}) })
describe('metadata index — applyWatermarkCatchup (the coordinator door)', () => {
it("an 'adopt' verdict performs zero index writes", async () => {
const storage = await makeStorage(5)
await writeArtifact(storage, 5)
const index = await reopen(storage)
expect(index.watermarkVerdict()).toBe('adopt')
const addSpy = vi.spyOn(index, 'addToIndex')
const removeSpy = vi.spyOn(index, 'removeFromIndex')
const result = await index.applyWatermarkCatchup(null)
expect(result).toEqual({ action: 'noop' })
expect(addSpy).not.toHaveBeenCalled()
expect(removeSpy).not.toHaveBeenCalled()
})
it('a catchup window folding an add, an update (same id twice), and a delete → the index serves exactly the final state', async () => {
const storage = await makeStorage(5)
// Session 1: two pre-existing entities, stamped at generation 5.
const survivorId = uuidv4()
const deletedId = uuidv4()
{
const index = new MetadataIndexManager(storage)
await index.init()
await index.addToIndex(survivorId, { status: 'active' })
await index.addToIndex(deletedId, { status: 'active' })
index.stampWatermark(5)
await index.flush()
}
// The store advanced to generation 8 without another metadata flush —
// the exact shape a crash-then-adopt-reopen leaves behind.
setCommitted(storage, 8)
const index = await reopen(storage)
expect(index.watermarkVerdict()).toBe('catchup')
expect(index.watermarkGap()).toEqual({ from: 5, to: 8 })
const addedId = uuidv4()
const scan = fakeScan([
nounAdd(6, addedId, { status: 'new' }), // add
nounAdd(7, addedId, { status: 'updated' }), // update — same id twice
nounDelete(8, deletedId) // delete
])
const result = await index.applyWatermarkCatchup(scan)
expect(result.action).toBe('caught-up')
expect(result.window).toEqual({ from: 5, to: 8 })
expect(result.factsApplied).toBe(3)
expect(result.nounsApplied).toBe(3)
expect(result.verbsApplied).toBe(0)
// Final state: the added/updated id serves ONLY its final value...
expect(await index.getIds('status', 'updated')).toEqual([addedId])
expect(await index.getIds('status', 'new')).toEqual([]) // stale value gone
// ...the deleted id is gone...
expect(await index.getIds('status', 'active')).toEqual([survivorId])
// ...and the untouched survivor is unaffected.
expect(await index.getIds('status', 'active')).toContain(survivorId)
// The window is certified: watermark stamped at `to`, and a fresh
// reopen now verdicts 'adopt'.
expect(index.watermark()).toBe(8)
const reopened = await reopen(storage)
expect(reopened.watermarkVerdict()).toBe('adopt')
})
it("a 'rescan' verdict runs the existing rebuild path instead of folding", async () => {
const storage = await makeStorage(9)
await writeArtifact(storage, 9)
setCommitted(storage, 4) // a truncated log pulled the watermark back — stamp ABOVE committed → rescan
const index = await reopen(storage)
expect(index.watermarkVerdict()).toBe('rescan')
const rebuildSpy = vi.spyOn(index, 'rebuild')
const result = await index.applyWatermarkCatchup(null)
expect(result.action).toBe('rescan')
expect(result.reason).toBeTruthy()
expect(rebuildSpy).toHaveBeenCalledTimes(1)
})
it("a 'catchup' verdict with no fact log available demotes to rebuild, narrated", async () => {
const storage = await makeStorage(5)
await writeArtifact(storage, 5)
setCommitted(storage, 8)
const index = await reopen(storage)
expect(index.watermarkVerdict()).toBe('catchup')
const rebuildSpy = vi.spyOn(index, 'rebuild')
const result = await index.applyWatermarkCatchup(null) // no scan — no fact log
expect(result.action).toBe('rescan')
expect(result.reason).toContain('no fact log')
expect(rebuildSpy).toHaveBeenCalledTimes(1)
})
})

View file

@ -3,9 +3,14 @@
* @description Pattern-A / Finding 1: a pure semantic find({ query }) has no * @description Pattern-A / Finding 1: a pure semantic find({ query }) has no
* filter, so verifyMetadataLive never fires nothing guarded the vector index. * filter, so verifyMetadataLive never fires nothing guarded the vector index.
* A cold native vector index that loaded its COUNT but not its serving structure * A cold native vector index that loaded its COUNT but not its serving structure
* returned a silent []. verifyVectorLive() closes that: honest isReady() first, * returned a silent []. verifyVectorLive() closes that: the health-report/isReady()
* else a known-vector self-match probe; self-heal (rebuild) or throw * authority first, else a known-vector self-match probe.
* VectorIndexNotReadyError never a silent empty result. *
* RE-POINTED to the health-gate law: the guard NEVER rebuilds and NEVER walks
* the store from a read a read-path rebuild is exactly the dark-rebuild
* failure mode the law retires (open() alone owns building). A not-serving
* signal (from either strategy) THROWS VectorIndexNotReadyError immediately,
* with no rebuild attempt in between never a silent empty result.
*/ */
import { describe, it, expect, beforeEach } from 'vitest' import { describe, it, expect, beforeEach } from 'vitest'
import { Brainy, NounType, VectorIndexNotReadyError } from '../../src/index.js' import { Brainy, NounType, VectorIndexNotReadyError } from '../../src/index.js'
@ -34,50 +39,37 @@ describe('Vector cold-read guard (verifyVectorLive) — silent-[] on cold semant
vi.rebuild = origRebuild vi.rebuild = origRebuild
}) })
it('cold index: verifyVectorLive self-heals via rebuild — semantic find is correct, NOT silent []', async () => { it('cold index (no isReady()): verifyVectorLive REFUSES immediately — throws VectorIndexNotReadyError, NEVER rebuilds', async () => {
const vi = brain.index
const origSearch = vi.search.bind(vi)
const origRebuild = vi.rebuild.bind(vi)
let cold = true
brain._vectorVerified = false
// size()>0 (count present) but search returns nothing until a rebuild warms it.
vi.search = async (...a: any[]) => (cold ? [] : origSearch(...a))
vi.rebuild = async (...a: any[]) => { await origRebuild(...a); cold = false }
try {
const res = await brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
expect(res.length).toBeGreaterThan(0) // self-healed
} finally {
vi.search = origSearch; vi.rebuild = origRebuild
}
})
it('unrecoverably cold index: semantic find throws VectorIndexNotReadyError', async () => {
const vi = brain.index const vi = brain.index
const origSearch = vi.search.bind(vi) const origSearch = vi.search.bind(vi)
let rebuilds = 0
const origRebuild = vi.rebuild.bind(vi) const origRebuild = vi.rebuild.bind(vi)
brain._vectorVerified = false brain._vectorVerified = false
vi.search = async () => [] // always cold; rebuild can't fix it // size()>0 (count present) but search never returns a hit for the known vector.
vi.rebuild = async () => {} vi.search = async () => []
vi.rebuild = async (...a: any[]) => { rebuilds++; return origRebuild(...a) }
try { try {
await expect( await expect(
brain.find({ query: 'x', searchMode: 'semantic', limit: 100 }) brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
).rejects.toBeInstanceOf(VectorIndexNotReadyError) ).rejects.toBeInstanceOf(VectorIndexNotReadyError)
expect(rebuilds).toBe(0) // the guard never rebuilds from a read — it refuses loudly instead
} finally { } finally {
vi.search = origSearch; vi.rebuild = origRebuild vi.search = origSearch; vi.rebuild = origRebuild
} }
}) })
it('native provider reporting isReady()===false rebuilds, then serves', async () => { it('native provider reporting isReady()===false THROWS immediately — never rebuilds', async () => {
const vi = brain.index const vi = brain.index
let rebuilds = 0
const origRebuild = vi.rebuild.bind(vi) const origRebuild = vi.rebuild.bind(vi)
let ready = false
brain._vectorVerified = false brain._vectorVerified = false
vi.isReady = () => ready vi.isReady = () => false
vi.rebuild = async (...a: any[]) => { await origRebuild(...a); ready = true } vi.rebuild = async (...a: any[]) => { rebuilds++; return origRebuild(...a) }
try { try {
const res = await brain.find({ query: 'x', searchMode: 'semantic', limit: 100 }) await expect(
expect(ready).toBe(true) // rebuild ran because isReady() was false brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
expect(res).toBeDefined() ).rejects.toBeInstanceOf(VectorIndexNotReadyError)
expect(rebuilds).toBe(0) // a not-ready report throws immediately — it is never a rebuild trigger
} finally { } finally {
delete vi.isReady; vi.rebuild = origRebuild delete vi.isReady; vi.rebuild = origRebuild
} }