Compare commits
No commits in common. "v10.4.0-rc.2" and "v10.4.0-rc.1" have entirely different histories.
v10.4.0-rc
...
v10.4.0-rc
37 changed files with 829 additions and 4072 deletions
11
CHANGELOG.md
11
CHANGELOG.md
|
|
@ -2,17 +2,6 @@
|
||||||
|
|
||||||
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
|
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
|
||||||
|
|
||||||
### [10.4.0-rc.2](https://source.soulcraft.com/soulcraft/brainy/compare/v10.4.0-rc.1...v10.4.0-rc.2) (2026-08-25)
|
|
||||||
|
|
||||||
- test(readiness): the report helper's clock freezes — two independently-built reports compared across a millisecond tick made the plant lane red (39b916a3)
|
|
||||||
- feat(repair): a heal:'repair' verdict routes to the provider's own incremental repair() (553e0d97)
|
|
||||||
- fix(storage): an unknown nested storage config can never silently land on the shared default root (ddd5e719)
|
|
||||||
- docs(release): the 10.4.0 entry, the index-health concept doc, and the API surfaces — written from the tree, not the plan (8cced871)
|
|
||||||
- fix(plugins): the silent-degrade doors close — a broken accelerator install can never read as absent (b9ba50fb)
|
|
||||||
- feat(recovery): the catchup verdict is consumed; verb rows go live; the metadata rebuild goes online (18f172e0)
|
|
||||||
- feat(health): the gate reads the named report — reads refuse loudly, never rebuild; open serves before it returns; the ceremony door (f8f64780)
|
|
||||||
|
|
||||||
|
|
||||||
### [10.4.0-rc.1](https://source.soulcraft.com/soulcraft/brainy/compare/v10.3.1...v10.4.0-rc.1) (2026-08-24)
|
### [10.4.0-rc.1](https://source.soulcraft.com/soulcraft/brainy/compare/v10.3.1...v10.4.0-rc.1) (2026-08-24)
|
||||||
|
|
||||||
- ci(publish): the home dist-tag follows the version — a prerelease publishes under 'rc' and never moves 'latest' (a1376e4a)
|
- ci(publish): the home dist-tag follows the version — a prerelease publishes under 'rc' and never moves 'latest' (a1376e4a)
|
||||||
|
|
|
||||||
84
RELEASES.md
84
RELEASES.md
|
|
@ -31,90 +31,6 @@ is sometimes cited as a 7.x removal — those methods never existed on 7.x; the
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## v10.4.0 — 2026-08-25 (the health report has a name)
|
|
||||||
|
|
||||||
Three related cures, one root cause: an index deciding whether it could be trusted
|
|
||||||
by sampling itself instead of by exact accounting. This release replaces every
|
|
||||||
sampled self-probe with ledger-derived truth, and a read against an unhealthy index
|
|
||||||
now refuses loudly instead of guessing.
|
|
||||||
|
|
||||||
- **The canonical count ledger.** Storage now tracks two scalars per family
|
|
||||||
(nouns/verbs) on the write path: the user-facing `counted` total — unchanged,
|
|
||||||
still what `getNounCount()` / `getVerbCount()` return — and a new ALL-visibility
|
|
||||||
`all` total covering every tier, the real denominator a derived index's own
|
|
||||||
coverage math needs. The unfiltered storage-level `totalCount` returned by
|
|
||||||
`getNouns()` / `getVerbs()` is now this unclamped ALL scalar; previously it could
|
|
||||||
only ever move up (`Math.max(scalar, scanned)`), so an inflated counter could
|
|
||||||
never self-correct. A delete that cannot prove the record it removed actually
|
|
||||||
existed (no canonical read, no prior image available) no longer decrements on
|
|
||||||
faith — it marks the ledger `suspect` (narrated once per session) instead of
|
|
||||||
silently drifting, and the next `repairIndex()` clears the flag with a real
|
|
||||||
recount.
|
|
||||||
- **One contract for a throwing health probe.** A provider's `validateInvariants()`
|
|
||||||
is documented to never throw — but if one does anyway (a bug, a transient fault),
|
|
||||||
it is now read the same way everywhere: `heal: 'none'`, the error named in the
|
|
||||||
report, never synthesized into a rebuild trigger and never swallowed into "looks
|
|
||||||
fine." A flaky check can no longer buy itself a rebuild. `repairIndex()`'s
|
|
||||||
per-family receipt also gains `missing` (an exact count plus a capped id sample),
|
|
||||||
`rebuilt` (a full rebuild ran, vs. an incremental heal), and `reason`.
|
|
||||||
- **The named health report; reads refuse instead of rebuilding.** Any index
|
|
||||||
provider may now expose a synchronous, O(1) `healthReport()` — composed from the
|
|
||||||
provider's own exact ledgers, never a sample — and this is the one signal
|
|
||||||
Brainy's read gate trusts. The first-query lazy-build path is gone: `brain.init()`
|
|
||||||
now runs every needed rebuild to completion before it returns, always, regardless
|
|
||||||
of dataset size. A read that lands on a provider whose health report says it
|
|
||||||
isn't serving throws a typed error instead of triggering a rebuild mid-query —
|
|
||||||
`GraphIndexNotReadyError`, `MetadataIndexNotReadyError`, or
|
|
||||||
`VectorIndexNotReadyError` (all exported from `@soulcraft/brainy`), naming the
|
|
||||||
reasons. `repairIndex({ rebuild: ['metadata' | 'graph' | 'vector'] | 'all' })` is
|
|
||||||
the new explicit operator door: it rebuilds the named family unconditionally, no
|
|
||||||
health check consulted — reach for it when you have independent reason to
|
|
||||||
distrust a family regardless of what it self-reports. Bare `repairIndex()` is
|
|
||||||
unchanged in spirit: report-driven, heals only what its own checks say needs it.
|
|
||||||
- New concept doc: [Index Health](docs/concepts/index-health.md) walks the whole
|
|
||||||
story from a consumer's side — degraded-but-serving vs. not-ready, what
|
|
||||||
`repairIndex()` checks and heals per family, what `suspect` counts mean.
|
|
||||||
|
|
||||||
**Nothing to change to adopt this.** No API removed, no signature narrowed —
|
|
||||||
`repairIndex()` gains an optional options bag and its return value gains fields,
|
|
||||||
both additive. The honest notes: if your code ever relied on a `find()` against a
|
|
||||||
cold/not-yet-built index quietly triggering a rebuild and returning results a beat
|
|
||||||
later, that behavior is gone — it now throws one of the three typed
|
|
||||||
`*NotReadyError` classes instead (catch them if you need to distinguish "not ready
|
|
||||||
yet" from "no results"). And `disableAutoRebuild: true` no longer defers index
|
|
||||||
construction to the first query — a needed rebuild always runs at `open()` now;
|
|
||||||
the flag has no effect on timing. Full manual control still lives in
|
|
||||||
`repairIndex({ rebuild: [...] })`.
|
|
||||||
|
|
||||||
- **Crash-reopen catchup.** After an unclean shutdown, the metadata index now
|
|
||||||
folds the exact fact window it missed — `find()` serves every acked write on
|
|
||||||
reopen, closing the gap where canonical reads and counts recovered a
|
|
||||||
crash-window write but the index kept serving its pre-crash state until the
|
|
||||||
next full rebuild. Related root-cause fixed alongside: `close()` never
|
|
||||||
stamped the index watermarks (only `flush()` did), so a close without a
|
|
||||||
prior flush caused a needless full rescan verdict on the next open.
|
|
||||||
- **Relation rows are live in the metadata index.** Previously verb rows
|
|
||||||
entered the metadata index only during a rebuild — so a rebuilt store's
|
|
||||||
relation postings went stale from the first `relate()` after it. Relations
|
|
||||||
are now posted and retracted on the live write path (relate / unrelate /
|
|
||||||
updateRelation / remove's cascade, and their `transact()` forms), in the
|
|
||||||
same commit as the graph leg.
|
|
||||||
- **The metadata rebuild is online.** `rebuild()` for the metadata family no
|
|
||||||
longer clears and rebuilds in place (reads went empty for the duration): it
|
|
||||||
builds a complete replacement beside the serving index, mirrors concurrent
|
|
||||||
writes to both, swaps atomically, and persists once after the swap. Reads
|
|
||||||
never observe a partial index. `repairIndex({ rebuild: ['metadata'] })` uses
|
|
||||||
it automatically.
|
|
||||||
- **A broken accelerator install can never read as "not installed."** The
|
|
||||||
auto-detection free pass now requires the resolution error to name the
|
|
||||||
accelerator package itself, exactly — a missing platform-binary sibling
|
|
||||||
package, an inner file path, or a dependency failure is a broken install and
|
|
||||||
`init()` throws loudly. And a plugin that declines activation is narrated on
|
|
||||||
the always-on log channel, so `silent: true` can no longer hide a fallback
|
|
||||||
to the default engines.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## v10.3.1 — 2026-08-18 (the fold that behaves)
|
## v10.3.1 — 2026-08-18 (the fold that behaves)
|
||||||
|
|
||||||
Three recovery cures from one production first-boot incident (a brain's first
|
Three recovery cures from one production first-boot incident (a brain's first
|
||||||
|
|
|
||||||
|
|
@ -323,24 +323,58 @@ Only the graph adjacency index carries a committed scale assertion:
|
||||||
- ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer
|
- ✅ **Single-Node by Design**: One process owns one `path`; scale out at the service layer
|
||||||
- ✅ **Zero Stubs**: Every line of code is production-ready
|
- ✅ **Zero Stubs**: Every line of code is production-ready
|
||||||
|
|
||||||
## Index Build at Open (10.4+)
|
## Lazy Loading Performance
|
||||||
|
|
||||||
As of 10.4, `brain.init()` runs every needed index rebuild to completion before
|
Brainy supports two initialization modes for optimal performance across different use cases:
|
||||||
it returns — always, regardless of dataset size. There is no lazy,
|
|
||||||
first-query rebuild path: a brain either finishes opening healthy, or `init()`
|
|
||||||
fails loudly. `disableAutoRebuild` no longer defers index construction to a
|
|
||||||
first query; it has no effect on *when* a rebuild runs. Manual control over
|
|
||||||
rebuilds is `repairIndex({ rebuild: [...] })`. See
|
|
||||||
[Index Health](concepts/index-health.md) for the full read-gate contract
|
|
||||||
(providers self-report readiness via `healthReport()`; a read against a
|
|
||||||
not-serving provider throws a typed `*NotReadyError` rather than rebuilding
|
|
||||||
mid-query).
|
|
||||||
|
|
||||||
<!-- The pre-10.4 "Mode 2: Lazy Loading on First Query" section previously
|
### Mode 1: Auto-Rebuild (Default)
|
||||||
documented here (disableAutoRebuild deferring index construction to the
|
|
||||||
first find() call) described a real, now-retired code path. Removed
|
```javascript
|
||||||
rather than left to mislead; the concept doc above is the current
|
const brain = new Brainy()
|
||||||
contract. -->
|
await brain.init() // Rebuilds indexes during init (~500ms-3s for 10K entities)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Performance:**
|
||||||
|
- Init time: 500ms-3s (depends on dataset size)
|
||||||
|
- First query: Instant (indexes already loaded)
|
||||||
|
- Use case: Traditional applications, long-running servers
|
||||||
|
|
||||||
|
### Mode 2: Lazy Loading
|
||||||
|
|
||||||
|
```javascript
|
||||||
|
const brain = new Brainy({ disableAutoRebuild: true })
|
||||||
|
await brain.init() // Returns instantly (0-10ms)
|
||||||
|
|
||||||
|
const results = await brain.find({ limit: 10 }) // First query triggers rebuild (~50-200ms)
|
||||||
|
const more = await brain.find({ limit: 100 }) // Subsequent queries instant (0ms check)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Performance:**
|
||||||
|
- Init time: 0-10ms (instant)
|
||||||
|
- First query: 50-200ms (includes index rebuild for 1K-10K entities)
|
||||||
|
- Subsequent queries: 0ms check (instant)
|
||||||
|
- Concurrent queries: Wait for same rebuild (mutex prevents duplicates)
|
||||||
|
|
||||||
|
**Concurrency Safety:**
|
||||||
|
```javascript
|
||||||
|
// 100 concurrent queries immediately after init
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
|
const promises = Array.from({ length: 100 }, () =>
|
||||||
|
brain.find({ limit: 10 })
|
||||||
|
)
|
||||||
|
|
||||||
|
const results = await Promise.all(promises)
|
||||||
|
// ✅ Only 1 rebuild triggered (mutex)
|
||||||
|
// ✅ All 100 queries return correct results
|
||||||
|
// ✅ Total time: ~60ms (not 6000ms!)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Use Cases for Lazy Loading:**
|
||||||
|
- **Serverless/Edge**: Minimize cold start time (0-10ms init)
|
||||||
|
- **Development**: Faster restarts during development
|
||||||
|
- **Large datasets**: Defer index loading until needed
|
||||||
|
- **Read-heavy workloads**: Writes don't wait for index rebuild
|
||||||
|
|
||||||
## Zero Configuration Required
|
## Zero Configuration Required
|
||||||
|
|
||||||
|
|
@ -350,6 +384,10 @@ Brainy is designed to be **smart enough to tune itself dynamically**. No configu
|
||||||
// That's it. Brainy handles everything.
|
// That's it. Brainy handles everything.
|
||||||
const brain = new Brainy()
|
const brain = new Brainy()
|
||||||
await brain.init()
|
await brain.init()
|
||||||
|
|
||||||
|
// Or with lazy loading for serverless
|
||||||
|
const brain = new Brainy({ disableAutoRebuild: true })
|
||||||
|
await brain.init() // Instant (0-10ms)
|
||||||
```
|
```
|
||||||
|
|
||||||
### Automatic Self-Tuning
|
### Automatic Self-Tuning
|
||||||
|
|
@ -357,6 +395,7 @@ await brain.init()
|
||||||
- **Metadata Index**: Auto-builds sorted indices for range queries on first use
|
- **Metadata Index**: Auto-builds sorted indices for range queries on first use
|
||||||
- **Graph Index**: Auto-flushes every 30 seconds
|
- **Graph Index**: Auto-flushes every 30 seconds
|
||||||
- **Default Tuning**: Research-based vector index defaults
|
- **Default Tuning**: Research-based vector index defaults
|
||||||
|
- **Lazy Loading**: Indices built only when needed
|
||||||
- **Cache Management**: LRU caches with TTL
|
- **Cache Management**: LRU caches with TTL
|
||||||
|
|
||||||
### Intelligent Defaults
|
### Intelligent Defaults
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,7 @@ next:
|
||||||
- guides/storage-adapters
|
- guides/storage-adapters
|
||||||
---
|
---
|
||||||
|
|
||||||
# Plugin System
|
# Plugin Development Guide
|
||||||
|
|
||||||
Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer.
|
Brainy has a plugin system that allows third-party packages to replace internal subsystems with custom implementations. This is how `@soulcraft/cor` provides optional native acceleration, and it's the same system available to any developer.
|
||||||
|
|
||||||
|
|
@ -200,30 +200,15 @@ members so a warm reopen never pays a redundant rebuild-from-canonical:
|
||||||
- **`init?(): Promise<void>`** — eager cold-load. Brainy awaits it once during
|
- **`init?(): Promise<void>`** — eager cold-load. Brainy awaits it once during
|
||||||
`brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first)
|
`brain.init()`, after the metadata provider's `init()` (the id-mapper hydrates first)
|
||||||
and **before the rebuild gate**.
|
and **before the rebuild gate**.
|
||||||
- **`healthReport?(): HealthReport`** — the PREFERRED signal (10.4+). A named,
|
- **`isReady?(): boolean`** — honest durability signal. `true` ⇔ the persisted index is
|
||||||
synchronous, O(1) verdict derived from the provider's own exact ledgers — never a
|
loaded (or cheaply demand-loadable) and consistent with what was last persisted. When
|
||||||
sample, never I/O, must never throw for a well-formed provider. Brainy's read gate
|
exposed, the rebuild gate defers to this signal **instead of** the `size() === 0` /
|
||||||
(`assessProviderHealth()`) reads this INSTEAD of `isReady()` / size heuristics when
|
`totalEntries === 0` heuristics — a disk-native index may report 0 resident entries
|
||||||
present: `serving: false` refuses the read with a typed `*NotReadyError` rather than
|
while fully durable. Never return `true` if the durable state failed to load: the
|
||||||
triggering a rebuild — a read never starts a store walk. `healthy` marks every
|
signal is honest in both directions, and a not-ready provider gets its rebuild even
|
||||||
*verified* invariant holding; a family named in `unledgered` counts as neither
|
when `size() > 0`.
|
||||||
healthy nor broken. See `HealthReport` / `LedgerInvariantResult` /
|
|
||||||
`InvariantSource` in `src/plugin.ts`, and
|
|
||||||
[Index Health](concepts/index-health.md) for the consumer-facing story.
|
|
||||||
- **`isReady?(): boolean`** — honest durability signal, the fallback when
|
|
||||||
`healthReport()` is absent. `true` ⇔ the persisted index is loaded (or cheaply
|
|
||||||
demand-loadable) and consistent with what was last persisted. When exposed, the
|
|
||||||
gate defers to this signal **instead of** the `size() === 0` / `totalEntries === 0`
|
|
||||||
heuristics — a disk-native index may report 0 resident entries while fully durable.
|
|
||||||
Never return `true` if the durable state failed to load: the signal is honest in
|
|
||||||
both directions, and a not-ready provider gets its rebuild even when `size() > 0`.
|
|
||||||
- **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background
|
- **`isMigrating?(): boolean`** — while `true`, the provider owns its index (background
|
||||||
migration); brainy skips its rebuild entirely.
|
migration); brainy skips its rebuild entirely.
|
||||||
- **`validateInvariants?(): Promise<ProviderInvariantReport>`** — the async DEEP
|
|
||||||
diagnostic (full scans allowed), distinct from the bounded, sync `healthReport()`.
|
|
||||||
Must never throw — a failure is `healthy: false` data, not an exception; a provider
|
|
||||||
that throws anyway is read as a loud, unverified failure (never as "healthy") by
|
|
||||||
every caller, never silently retried into a rebuild.
|
|
||||||
|
|
||||||
Providers that implement none of these keep the size/count heuristics — correct for
|
Providers that implement none of these keep the size/count heuristics — correct for
|
||||||
engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index).
|
engines whose `rebuild()` *is* their load path (like brainy's built-in JS vector index).
|
||||||
|
|
|
||||||
|
|
@ -1451,34 +1451,6 @@ const count = await brain.getVerbCount()
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### The canonical count ledger (`StorageAdapter.getCanonicalCounts()`)
|
|
||||||
|
|
||||||
An OPTIONAL method on the `StorageAdapter` interface (implemented by both
|
|
||||||
built-in adapters), not a method on `Brainy` itself — relevant if you're
|
|
||||||
writing a custom storage adapter or composing a provider's own
|
|
||||||
`healthReport()`. O(1), no I/O. Per family (`nouns`/`verbs`):
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
interface CanonicalCounts {
|
|
||||||
nouns: { counted: number; all: number }
|
|
||||||
verbs: { counted: number; all: number }
|
|
||||||
suspect: boolean
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
- `counted` mirrors `getNounCount()` / `getVerbCount()` (public + internal tiers).
|
|
||||||
- `all` is the ALL-visibility scalar — every tier, including system/internal
|
|
||||||
records — the denominator a derived index's own coverage math is measured
|
|
||||||
against.
|
|
||||||
- `suspect` is `true` when an unprovable delete has left `all` unverified since
|
|
||||||
the last recount; `brain.repairIndex()` clears it with a real canonical walk.
|
|
||||||
|
|
||||||
Adapters without the ledger omit the method; treat absence as "no
|
|
||||||
denominator," never as zero. See
|
|
||||||
**[Index Health](../concepts/index-health.md)** for the full story.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Subtype & facet APIs
|
### Subtype & facet APIs
|
||||||
|
|
||||||
Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
|
Full guide: **[Subtypes & Facets](../guides/subtypes-and-facets.md)**.
|
||||||
|
|
@ -1859,104 +1831,6 @@ const semanticOnly = await brain.getStats({ excludeVFS: true })
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### `repairIndex(options?)` → `Promise<RepairReport>`
|
|
||||||
|
|
||||||
The ceremony door for index repair. Bare `repairIndex()` is report-driven: it
|
|
||||||
prunes orphaned containers, recomputes count rollups, reconciles VFS
|
|
||||||
containment, and rebuilds only a derived-index family whose own health check
|
|
||||||
asks for it. Pass `options.rebuild` to force one or more families to rebuild
|
|
||||||
UNCONDITIONALLY — no health check is consulted — when an operator has
|
|
||||||
independent reason to reconcile a family regardless of what it self-reports.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Report-driven: only heals what actually needs it
|
|
||||||
const report = await brain.repairIndex()
|
|
||||||
console.log(report.healedTotal, report.families)
|
|
||||||
|
|
||||||
// Explicit: force the graph adjacency to rebuild from canonical, unconditionally
|
|
||||||
await brain.repairIndex({ rebuild: ['graph'] })
|
|
||||||
|
|
||||||
// Explicit: force all three derived indexes to rebuild
|
|
||||||
await brain.repairIndex({ rebuild: 'all' })
|
|
||||||
```
|
|
||||||
|
|
||||||
**`RepairReport`:**
|
|
||||||
- `families: RepairFamilyReport[]` — one row per family checked
|
|
||||||
- `healedTotal: number` — items healed across every family
|
|
||||||
- `durationMs: number`
|
|
||||||
|
|
||||||
**`RepairFamilyReport`** (one row):
|
|
||||||
- `family: string` — e.g. `'orphaned-containers'`, `'count-rollups'`,
|
|
||||||
`'vfs-containment'`, `'metadata-corruption'`, `'provider:metadata'`,
|
|
||||||
`'provider:graph'`, `'provider:vector'`
|
|
||||||
- `checked: boolean` — was this family actually examined (`false` ⇒ see `skipped`)
|
|
||||||
- `healed: number` — items re-posted/corrected in place (the incremental heal count)
|
|
||||||
- `missing?: { count: number; sample: string[] }` — exact count plus a capped id
|
|
||||||
sample when the check can name what diverged (never the full list)
|
|
||||||
- `rebuilt?: boolean` — a full generational rebuild ran (vs. an incremental heal)
|
|
||||||
- `detail?: string` / `reason?: string` — narration
|
|
||||||
- `skipped?: string` — why the family wasn't checked
|
|
||||||
|
|
||||||
Full walkthrough — what each family checks, degraded-but-serving vs. not-ready,
|
|
||||||
and what `suspect` counts mean — in
|
|
||||||
**[Index Health](../concepts/index-health.md)**.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Index readiness: typed errors, `healthReport()`, `disableAutoRebuild`
|
|
||||||
|
|
||||||
Every derived-index provider (vector, graph, metadata) may expose a named,
|
|
||||||
synchronous, O(1) `healthReport()` composed from its own exact ledgers — the
|
|
||||||
signal Brainy's read gate trusts over sampling or size heuristics. `init()`
|
|
||||||
brings every provider to serving before it returns; there is no first-query
|
|
||||||
lazy-rebuild path. A read that reaches a provider whose health report says it
|
|
||||||
isn't serving throws instead of rebuilding mid-query:
|
|
||||||
|
|
||||||
| Error | Thrown by | Meaning |
|
|
||||||
|---|---|---|
|
|
||||||
| `GraphIndexNotReadyError` | `find({ connected })`, `neighbors()`, `related()` | Graph adjacency isn't serving |
|
|
||||||
| `MetadataIndexNotReadyError` | `find({ where })` | Metadata/field index isn't serving |
|
|
||||||
| `VectorIndexNotReadyError` | `find({ query })`, `similar()` | Vector index isn't serving |
|
|
||||||
|
|
||||||
All three are exported from `@soulcraft/brainy`. Catch them to distinguish
|
|
||||||
"index not ready" from a genuine empty result:
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
import { MetadataIndexNotReadyError } from '@soulcraft/brainy'
|
|
||||||
|
|
||||||
try {
|
|
||||||
const rows = await brain.find({ where: { status: 'active' } })
|
|
||||||
} catch (err) {
|
|
||||||
if (err instanceof MetadataIndexNotReadyError) {
|
|
||||||
// reconcile: await brain.repairIndex(), then retry
|
|
||||||
} else {
|
|
||||||
throw err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**`disableAutoRebuild`** no longer defers index construction to the first
|
|
||||||
query. A needed rebuild always runs at `open()`, regardless of this flag or
|
|
||||||
dataset size; the flag has no effect on *when* a rebuild runs. Full manual
|
|
||||||
control lives in `repairIndex({ rebuild: [...] })`, above.
|
|
||||||
|
|
||||||
### `validateIndexConsistency()` → `Promise<...>`
|
|
||||||
|
|
||||||
The deep, async diagnostic counterpart to `healthReport()` — safe to run on a
|
|
||||||
live brain, but does more work (a provider's `validateInvariants()` may run a
|
|
||||||
full scan, not just read a ledger). Aggregates the JS metadata index's own
|
|
||||||
consistency check with every derived-index provider's invariant report.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const validation = await brain.validateIndexConsistency()
|
|
||||||
if (!validation.healthy) {
|
|
||||||
console.log(validation.recommendation) // what to run, e.g. repairIndex()
|
|
||||||
console.log(validation.providers) // each provider's own invariant report, when exposed
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Lifecycle
|
## Lifecycle
|
||||||
|
|
||||||
### Initialization
|
### Initialization
|
||||||
|
|
|
||||||
|
|
@ -723,14 +723,6 @@ async stats(): Promise<Statistics> {
|
||||||
|
|
||||||
### 5. Index Rebuilding (Lazy Loading Support)
|
### 5. Index Rebuilding (Lazy Loading Support)
|
||||||
|
|
||||||
> **Stale as of 10.4 — "Mode 2: Lazy Loading on First Query" below is
|
|
||||||
> RETIRED.** `disableAutoRebuild` no longer defers index construction to a
|
|
||||||
> first query; `brain.init()` now runs every needed rebuild to completion
|
|
||||||
> before it returns, unconditionally, and a read against a not-serving
|
|
||||||
> provider throws a typed `*NotReadyError` instead of rebuilding mid-query.
|
|
||||||
> See `docs/concepts/index-health.md` for the current contract. Left below
|
|
||||||
> as historical background on the rebuild mechanics.
|
|
||||||
|
|
||||||
**Two modes of index loading:**
|
**Two modes of index loading:**
|
||||||
|
|
||||||
#### Mode 1: Auto-Rebuild on init() (default)
|
#### Mode 1: Auto-Rebuild on init() (default)
|
||||||
|
|
|
||||||
|
|
@ -1,15 +1,5 @@
|
||||||
# Initialization and Rebuild Processes
|
# Initialization and Rebuild Processes
|
||||||
|
|
||||||
> **Stale as of 10.4 — "Mode 2: Lazy Loading on First Query" below is RETIRED.**
|
|
||||||
> `disableAutoRebuild` no longer defers index construction to a first query;
|
|
||||||
> `brain.init()` now runs every needed rebuild to completion before it
|
|
||||||
> returns, unconditionally. A read against a not-serving provider throws a
|
|
||||||
> typed `*NotReadyError` instead of rebuilding mid-query. See
|
|
||||||
> `docs/concepts/index-health.md` for the current contract; this document's
|
|
||||||
> line-number references to `src/brainy.ts` also predate the file's current
|
|
||||||
> size and are unreliable. Left as historical background on the rebuild
|
|
||||||
> mechanics, not as a current API description.
|
|
||||||
|
|
||||||
This document explains how Brainy's four indexes (MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
|
This document explains how Brainy's four indexes (MetadataIndex, vector index, GraphAdjacencyIndex, DeletedItemsIndex) initialize and rebuild from persisted storage.
|
||||||
|
|
||||||
## Core Principle: All Indexes Are Disk-Based
|
## Core Principle: All Indexes Are Disk-Based
|
||||||
|
|
|
||||||
|
|
@ -1,204 +0,0 @@
|
||||||
---
|
|
||||||
title: Index Health
|
|
||||||
slug: concepts/index-health
|
|
||||||
public: true
|
|
||||||
category: concepts
|
|
||||||
template: concept
|
|
||||||
order: 8
|
|
||||||
description: How Brainy knows whether a derived index can be trusted — exact accounting instead of sampling, the named health report, degraded-but-serving vs. not-ready, and what repairIndex() checks, heals, and rebuilds.
|
|
||||||
next:
|
|
||||||
- concepts/generation-fact-log
|
|
||||||
- guides/inspection
|
|
||||||
---
|
|
||||||
|
|
||||||
# Index Health
|
|
||||||
|
|
||||||
Brainy keeps one **canonical** copy of every entity and relationship, and three
|
|
||||||
**derived** indexes built from it — vector, metadata, and graph — so `find()` can
|
|
||||||
answer semantically, by filter, and by traversal without re-deriving the answer from
|
|
||||||
scratch on every query. A derived index is a cache with a serving structure: it can
|
|
||||||
be present but stale, present but only partially loaded, or fully out of sync with
|
|
||||||
canonical after a crash. This page is about how Brainy decides whether to trust one,
|
|
||||||
what it does when it can't, and how you reconcile the two.
|
|
||||||
|
|
||||||
## Exact accounting instead of sampling
|
|
||||||
|
|
||||||
Older health checks worked by inference: does `size()` return something greater
|
|
||||||
than zero, does a spot-check on one known item come back correct. Both are proxies.
|
|
||||||
A cold index can report a nonzero count while its actual serving structure never
|
|
||||||
loaded, and a spot-check only proves the one item it happened to ask about.
|
|
||||||
|
|
||||||
Every derived-index provider may now expose a named, synchronous, O(1)
|
|
||||||
`healthReport()` — composed from the provider's own **exact ledgers** (real counters
|
|
||||||
it already maintains on the write path), never a sample or a walk. This is the one
|
|
||||||
signal Brainy's read gate consults. A provider that doesn't yet expose one falls
|
|
||||||
back to an honest `isReady()` boolean, and finally to a size heuristic for engines
|
|
||||||
with neither — but wherever a `healthReport()` exists, it wins.
|
|
||||||
|
|
||||||
Underneath, storage itself keeps an analogous **canonical count ledger**: a
|
|
||||||
`counted` scalar (the user-facing total — what `getNounCount()` / `getVerbCount()`
|
|
||||||
return) and an `all` scalar (every tier, including internal records a derived
|
|
||||||
index's own coverage math needs to compare against). This is the real denominator
|
|
||||||
a provider's `healthReport()` measures itself by, rather than a total that can only
|
|
||||||
ever ratchet upward. See [What `suspect` counts mean](#what-suspect-counts-mean)
|
|
||||||
below for the one case that ledger can't stay exact through on its own.
|
|
||||||
|
|
||||||
## The named report
|
|
||||||
|
|
||||||
A `HealthReport` carries, per provider (`'vector'` / `'graph'` / `'metadata'`):
|
|
||||||
|
|
||||||
- **`healthy`** — `true` iff every *verified* invariant holds. An invariant whose
|
|
||||||
family has no ledger yet is `unledgered`, never counted either way — unknown,
|
|
||||||
not passing.
|
|
||||||
- **`serving`** — can this provider answer a query right now. A failing invariant
|
|
||||||
graded `heal: 'repair'` or `heal: 'none'` still leaves `serving: true` — this is
|
|
||||||
**degraded-but-serving**: something is off (say, a stale rollup on an
|
|
||||||
`employee` record's relationship count) but reads keep working. Only a failure
|
|
||||||
graded `heal: 'rebuild'` flips `serving` to `false` — **not-ready** — because the
|
|
||||||
provider itself is telling you its serving structure cannot answer correctly.
|
|
||||||
- **`invariants`** — each checked condition, with its provenance
|
|
||||||
(`source: 'ledger'` — an exact count; `'deep'` — a full scan, diagnostic-only;
|
|
||||||
`'unledgered'` — not yet tracked) and, for a failing one, an exact `missing`
|
|
||||||
count plus a capped sample of the affected ids — a verdict, never a dump.
|
|
||||||
- **`generation`** — bumps on every ledger mutation and rebuild, so a caller can
|
|
||||||
cache a verdict per generation instead of re-deriving it.
|
|
||||||
|
|
||||||
The distinction that matters day to day: `healthy: false` can be entirely benign —
|
|
||||||
a maintenance window, a divergence `repairIndex()` will clean up on its own
|
|
||||||
schedule. `serving: false` is not benign. It means this provider is refusing to
|
|
||||||
answer, on its own word, right now.
|
|
||||||
|
|
||||||
## Reads refuse — they never rebuild
|
|
||||||
|
|
||||||
A query that reaches a not-serving provider does not trigger a rebuild from inside
|
|
||||||
the read. Brainy retired that path deliberately: a rebuild kicked off by an ordinary
|
|
||||||
`find({ where: { status: 'active' } })` call is a dark, unpredictable cost hiding
|
|
||||||
behind a request that looks like a cheap read. Instead, the read throws a typed,
|
|
||||||
catchable error naming the reason:
|
|
||||||
|
|
||||||
| Error | Thrown when | Meaning |
|
|
||||||
|---|---|---|
|
|
||||||
| `GraphIndexNotReadyError` | `find({ connected })`, `neighbors()`, `related()` | The graph adjacency index isn't serving — traversal would otherwise return `[]` indistinguishable from "no relationships" |
|
|
||||||
| `MetadataIndexNotReadyError` | `find({ where })` | The metadata/field index isn't serving — a filtered read would otherwise return `[]` indistinguishable from "no matches" |
|
|
||||||
| `VectorIndexNotReadyError` | `find({ query })`, `similar()` | The vector index isn't serving — a semantic search would otherwise return `[]` indistinguishable from "nothing similar" |
|
|
||||||
|
|
||||||
All three are exported from `@soulcraft/brainy`. Catch them where your application
|
|
||||||
needs to distinguish "this index isn't ready yet" from "there's genuinely nothing
|
|
||||||
here" — a health dashboard, a retry policy, an operator alert. The fix is always
|
|
||||||
the same: reconcile the index, either by reopening the brain (which brings every
|
|
||||||
provider to serving before `init()` returns — see the next section) or by calling
|
|
||||||
`repairIndex()` explicitly.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
try {
|
|
||||||
const active = await brain.find({ where: { status: 'active' } })
|
|
||||||
} catch (err) {
|
|
||||||
if (err instanceof MetadataIndexNotReadyError) {
|
|
||||||
// not a "no results" — the index itself refused; alert or retry after repair
|
|
||||||
} else {
|
|
||||||
throw err
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Rebuilds happen at open, not on first query
|
|
||||||
|
|
||||||
`brain.init()` runs every needed rebuild to completion **before it returns**,
|
|
||||||
unconditionally, regardless of dataset size. There is no lazy, first-query
|
|
||||||
rebuild path anymore — a brain either finishes opening healthy, or it fails
|
|
||||||
open loudly. `disableAutoRebuild: true` no longer defers index construction to
|
|
||||||
the first query: it has no effect on *when* a needed rebuild runs. Full manual
|
|
||||||
control over rebuilds is `repairIndex({ rebuild: [...] })` (below), not this flag.
|
|
||||||
|
|
||||||
## `repairIndex()` — checking and healing
|
|
||||||
|
|
||||||
Bare `repairIndex()` is **report-driven**: it only heals what its own checks say
|
|
||||||
actually needs it, and it always returns a full per-family receipt.
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
const report = await brain.repairIndex()
|
|
||||||
report.healedTotal // total items healed across every family
|
|
||||||
report.durationMs
|
|
||||||
report.families // one row per family checked
|
|
||||||
```
|
|
||||||
|
|
||||||
Each `RepairFamilyReport` row names what happened:
|
|
||||||
|
|
||||||
- **`checked`** — was this family actually examined (`false` means skipped —
|
|
||||||
see `skipped` for why).
|
|
||||||
- **`healed`** — items re-posted or corrected in place.
|
|
||||||
- **`missing`** — when the check can name what diverged: an exact `count` plus a
|
|
||||||
capped `sample` of ids.
|
|
||||||
- **`rebuilt`** — a full generational rebuild ran (as opposed to an incremental
|
|
||||||
heal).
|
|
||||||
- **`detail`** / **`reason`** / **`skipped`** — the receipt's narration; a row is
|
|
||||||
always either checked or explains why it wasn't. Nothing is silent.
|
|
||||||
|
|
||||||
On every call, bare `repairIndex()`:
|
|
||||||
|
|
||||||
1. Prunes orphaned canonical containers left by a partial delete.
|
|
||||||
2. Recomputes the count rollups from one canonical walk (unconditional — this is
|
|
||||||
also what clears a `suspect` ledger; see below).
|
|
||||||
3. Reconciles VFS containment edges, if the VFS is initialized.
|
|
||||||
4. Runs the metadata index's own corruption detection pass.
|
|
||||||
5. Consults each of the three derived-index providers' own health check and
|
|
||||||
rebuilds only a family whose failing invariant actually asks for it
|
|
||||||
(`heal: 'rebuild'`) — never a provider that reports `healthy` or a lesser
|
|
||||||
grade.
|
|
||||||
|
|
||||||
### The explicit rebuild door
|
|
||||||
|
|
||||||
`options.rebuild` skips the health check and rebuilds one or more families
|
|
||||||
**unconditionally** — the operator override for when you have independent reason
|
|
||||||
to distrust a family regardless of what it self-reports (a suspicious deploy, a
|
|
||||||
storage-layer incident, a support ticket that doesn't match what the health report
|
|
||||||
says):
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// Force the graph adjacency to rebuild from canonical, no invariant consulted
|
|
||||||
await brain.repairIndex({ rebuild: ['graph'] })
|
|
||||||
|
|
||||||
// Force all three derived indexes
|
|
||||||
await brain.repairIndex({ rebuild: 'all' })
|
|
||||||
```
|
|
||||||
|
|
||||||
A family named this way is recorded with `rebuilt: true` and
|
|
||||||
`reason: 'explicit rebuild requested'`, and is skipped by the normal
|
|
||||||
health-driven pass in the same call — it was already rebuilt unconditionally.
|
|
||||||
|
|
||||||
Reach for the explicit door when you need certainty regardless of self-report;
|
|
||||||
reach for bare `repairIndex()` for routine maintenance and after any incident
|
|
||||||
where you're not sure which family (if any) needs it.
|
|
||||||
|
|
||||||
## What `suspect` counts mean
|
|
||||||
|
|
||||||
Storage's canonical count ledger increments the ALL-visibility total on every new
|
|
||||||
record and decrements it on every *proven* delete — one where the record was read,
|
|
||||||
or the caller supplied its prior image. A delete that cannot prove what it removed
|
|
||||||
existed doesn't guess: it flags the ledger `suspect` (an operator-visible
|
|
||||||
`console.warn`, narrated once per session, not once per delete) rather than risk
|
|
||||||
decrementing a total that was never incremented for that record in the first
|
|
||||||
place. This is intentionally rare — it's a defensive fallback for callers on an
|
|
||||||
unusual removal path, not a per-delete cost.
|
|
||||||
|
|
||||||
`suspect` is not directly exposed on any `Brainy` method today — it lives on the
|
|
||||||
`StorageAdapter`'s optional `getCanonicalCounts()`, primarily consulted by
|
|
||||||
`repairIndex()`'s recount step and by custom storage adapters composing their own
|
|
||||||
`healthReport()`. What matters for an application: a `suspect` ledger is not
|
|
||||||
incorrect, just *unverified since the last recount* — and `repairIndex()`'s
|
|
||||||
unconditional count-rollup step (step 2, above) recomputes the ALL scalars from a
|
|
||||||
real canonical walk on every call, clearing the flag with proof either way.
|
|
||||||
|
|
||||||
## Practical guidance
|
|
||||||
|
|
||||||
- **On a normal restart**, do nothing — `init()` brings every provider to
|
|
||||||
serving before it returns, or fails loudly.
|
|
||||||
- **On a `*NotReadyError`** from a live read, reconcile with `repairIndex()`
|
|
||||||
(report-driven is almost always sufficient) and retry.
|
|
||||||
- **After an incident** where you distrust a specific family regardless of what
|
|
||||||
it reports healthy — a storage-layer fault, a suspicious restore — use the
|
|
||||||
explicit door: `repairIndex({ rebuild: ['metadata' | 'graph' | 'vector'] })`.
|
|
||||||
- **To audit before trusting a report**, `brain.auditGraph()` walks every stored
|
|
||||||
relationship and proves (or disproves) that reads return canonical truth,
|
|
||||||
independent of what any provider self-reports — see
|
|
||||||
[Inspecting a Live Brainy](../guides/inspection.md).
|
|
||||||
4
package-lock.json
generated
4
package-lock.json
generated
|
|
@ -1,12 +1,12 @@
|
||||||
{
|
{
|
||||||
"name": "@soulcraft/brainy",
|
"name": "@soulcraft/brainy",
|
||||||
"version": "10.4.0-rc.2",
|
"version": "10.4.0-rc.1",
|
||||||
"lockfileVersion": 3,
|
"lockfileVersion": 3,
|
||||||
"requires": true,
|
"requires": true,
|
||||||
"packages": {
|
"packages": {
|
||||||
"": {
|
"": {
|
||||||
"name": "@soulcraft/brainy",
|
"name": "@soulcraft/brainy",
|
||||||
"version": "10.4.0-rc.2",
|
"version": "10.4.0-rc.1",
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
"dependencies": {
|
"dependencies": {
|
||||||
"@msgpack/msgpack": "^3.1.2",
|
"@msgpack/msgpack": "^3.1.2",
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
{
|
{
|
||||||
"name": "@soulcraft/brainy",
|
"name": "@soulcraft/brainy",
|
||||||
"version": "10.4.0-rc.2",
|
"version": "10.4.0-rc.1",
|
||||||
"description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.",
|
"description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.",
|
||||||
"main": "dist/index.js",
|
"main": "dist/index.js",
|
||||||
"module": "dist/index.js",
|
"module": "dist/index.js",
|
||||||
|
|
|
||||||
1273
src/brainy.ts
1273
src/brainy.ts
File diff suppressed because it is too large
Load diff
|
|
@ -269,10 +269,6 @@ export type { FamilyStamp, StampMembers, StampVerdict } from './db/familyStamp.j
|
||||||
export { isVersionedIndexProvider } from './plugin.js'
|
export { isVersionedIndexProvider } from './plugin.js'
|
||||||
export type { VersionedIndexProvider } from './plugin.js'
|
export type { VersionedIndexProvider } from './plugin.js'
|
||||||
export type { ProviderInvariantReport, InvariantResult, InvariantHeal } from './plugin.js'
|
export type { ProviderInvariantReport, InvariantResult, InvariantHeal } from './plugin.js'
|
||||||
// The named, synchronous, O(1) health-report contract (the read gate's ONLY
|
|
||||||
// source of truth for "can I serve right now") — see HealthReport's
|
|
||||||
// derivation laws in plugin.ts.
|
|
||||||
export type { HealthReport, LedgerInvariantResult, InvariantSource } from './plugin.js'
|
|
||||||
// Optional provider self-report of outstanding background maintenance work
|
// Optional provider self-report of outstanding background maintenance work
|
||||||
// (compaction, deferred writes, etc.) — the payload type for
|
// (compaction, deferred writes, etc.) — the payload type for
|
||||||
// brain.maintenanceDebt(). See the measure-only-what-you-track contract on
|
// brain.maintenanceDebt(). See the measure-only-what-you-track contract on
|
||||||
|
|
@ -389,10 +385,7 @@ import type {
|
||||||
HNSWVerb,
|
HNSWVerb,
|
||||||
HNSWConfig,
|
HNSWConfig,
|
||||||
StorageAdapter,
|
StorageAdapter,
|
||||||
DerivedFamilyDeclaration,
|
DerivedFamilyDeclaration
|
||||||
// The canonical count ledger a storage adapter maintains (counted + ALL-visibility
|
|
||||||
// scalars per family, the coverage-ledger denominators) — see StorageAdapter.getCanonicalCounts.
|
|
||||||
CanonicalCounts
|
|
||||||
} from './coreTypes.js'
|
} from './coreTypes.js'
|
||||||
|
|
||||||
// Export vector index implementation (the JS HNSW path)
|
// Export vector index implementation (the JS HNSW path)
|
||||||
|
|
|
||||||
113
src/plugin.ts
113
src/plugin.ts
|
|
@ -9,7 +9,6 @@
|
||||||
* registered manually via `brain.use()` — there is no implicit detection.
|
* registered manually via `brain.use()` — there is no implicit detection.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { prodLog } from './utils/logger.js'
|
|
||||||
import type {
|
import type {
|
||||||
StorageAdapter,
|
StorageAdapter,
|
||||||
Vector,
|
Vector,
|
||||||
|
|
@ -172,66 +171,6 @@ export interface ProviderInvariantReport {
|
||||||
durationMs: number
|
durationMs: number
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Where a {@link LedgerInvariantResult} verdict came from:
|
|
||||||
* - `'ledger'` — decided from an exact, durable ledger (a real count, not a sample).
|
|
||||||
* - `'deep'` — decided by a full/expensive scan (the `validateInvariants()` diagnostic path only).
|
|
||||||
* - `'unledgered'` — this family has no ledger yet; the verdict is UNKNOWN, never healthy and never broken.
|
|
||||||
*/
|
|
||||||
export type InvariantSource = 'ledger' | 'deep' | 'unledgered'
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description One invariant verdict inside a {@link HealthReport}. Extends
|
|
||||||
* {@link InvariantResult} with the provenance of the verdict ({@link InvariantSource})
|
|
||||||
* and, for a failing set-membership invariant, an exact count plus a capped sample
|
|
||||||
* of the diverging ids — a VERDICT, never a dump. `sample` MUST be capped at 16 ids;
|
|
||||||
* `count` is the exact number even when `sample` is truncated.
|
|
||||||
*/
|
|
||||||
export interface LedgerInvariantResult extends InvariantResult {
|
|
||||||
/** Provenance of this verdict — see {@link InvariantSource}. */
|
|
||||||
source: InvariantSource
|
|
||||||
/** Exact count of diverging/missing items plus a capped (≤16 ids) sample. Present only on a failing set-membership invariant. */
|
|
||||||
missing?: { count: number; sample: string[] }
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description The NAMED, SYNCHRONOUS, O(1) health report a provider exposes via
|
|
||||||
* {@link MetadataIndexProvider.healthReport} / {@link GraphIndexProvider.healthReport} /
|
|
||||||
* {@link VectorIndexProvider.healthReport}. This is the read gate's ONLY source of
|
|
||||||
* truth for "can I serve right now" — it replaces sampled self-probes and the
|
|
||||||
* unnamed `isReady()` latch with an exact, ledger-derived verdict.
|
|
||||||
*
|
|
||||||
* Derivation laws (a provider MUST honor these; brainy's read gate assumes them):
|
|
||||||
* - `healthy` = every VERIFIED invariant in {@link invariants} holds. An invariant
|
|
||||||
* whose family is named in {@link unledgered} is NEVER counted toward `healthy`
|
|
||||||
* either way — it is unknown, not passing.
|
|
||||||
* - `serving` = no verified invariant in {@link invariants} FAILS with `heal: 'rebuild'`.
|
|
||||||
* A failure with `heal: 'repair'` or `heal: 'none'` is degraded-but-serving —
|
|
||||||
* `serving` stays `true`. Only a `'rebuild'`-grade failure makes `serving` `false`.
|
|
||||||
* - `validateInvariants()` remains the async DEEP diagnostic (full scans allowed,
|
|
||||||
* `source: 'deep'` results); `healthReport()` MUST be synchronous, O(1) from
|
|
||||||
* exact ledgers/counters, and MUST NOT throw for a well-formed provider — a
|
|
||||||
* provider that cannot produce a safe verdict reports it as a failing invariant,
|
|
||||||
* it does not throw (a throw is read by the gate as a CONTRACT VIOLATION, not as
|
|
||||||
* "unknown").
|
|
||||||
*/
|
|
||||||
export interface HealthReport extends ProviderInvariantReport {
|
|
||||||
/**
|
|
||||||
* Monotonic per provider: bumps on every ledger mutation and every rebuild
|
|
||||||
* boundary. Consumers (the read gate's narration dedup, external callers) may
|
|
||||||
* cache a verdict per generation.
|
|
||||||
*/
|
|
||||||
generation: number
|
|
||||||
/** Each checked invariant, with provenance — see {@link LedgerInvariantResult}. */
|
|
||||||
invariants: LedgerInvariantResult[]
|
|
||||||
/**
|
|
||||||
* Families with no ledger yet. NAMED here so an operator can see what is not
|
|
||||||
* yet tracked — NEVER counted as healthy (they are not verified) and NEVER
|
|
||||||
* counted as broken (there is nothing to fail).
|
|
||||||
*/
|
|
||||||
unledgered: string[]
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description A provider's self-report of its own outstanding background
|
* @description A provider's self-report of its own outstanding background
|
||||||
* maintenance work (compaction, deferred writes, a build-new→verify→swap in
|
* maintenance work (compaction, deferred writes, a build-new→verify→swap in
|
||||||
|
|
@ -327,20 +266,6 @@ export interface MetadataIndexProvider {
|
||||||
*/
|
*/
|
||||||
validateInvariants?(): Promise<ProviderInvariantReport>
|
validateInvariants?(): Promise<ProviderInvariantReport>
|
||||||
|
|
||||||
/**
|
|
||||||
* @description OPTIONAL. The named, SYNCHRONOUS, O(1) health verdict this
|
|
||||||
* provider derives from its own exact ledgers — see {@link HealthReport} for
|
|
||||||
* the full derivation laws. MUST NOT perform I/O and MUST NOT throw for a
|
|
||||||
* well-formed provider (brainy treats a throw as a CONTRACT VIOLATION, never
|
|
||||||
* as "unknown"). When present, brainy's read gate (`assessProviderHealth()`)
|
|
||||||
* reads THIS instead of `isReady()` / size heuristics: `serving` decides
|
|
||||||
* whether reads may proceed; a `false` refuses the read loudly rather than
|
|
||||||
* triggering a rebuild. Absent → the gate falls back to `isReady?()` / the
|
|
||||||
* size heuristic (this train's JS built-in providers stay on that interim
|
|
||||||
* path).
|
|
||||||
*/
|
|
||||||
healthReport?(): HealthReport
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description OPTIONAL. A native provider returns true from the moment its
|
* @description OPTIONAL. A native provider returns true from the moment its
|
||||||
* `init()` detects a large epoch-drift until its background
|
* `init()` detects a large epoch-drift until its background
|
||||||
|
|
@ -537,20 +462,6 @@ export interface GraphIndexProvider {
|
||||||
*/
|
*/
|
||||||
validateInvariants?(): Promise<ProviderInvariantReport>
|
validateInvariants?(): Promise<ProviderInvariantReport>
|
||||||
|
|
||||||
/**
|
|
||||||
* @description OPTIONAL. The named, SYNCHRONOUS, O(1) health verdict this
|
|
||||||
* provider derives from its own exact ledgers — see {@link HealthReport} for
|
|
||||||
* the full derivation laws. MUST NOT perform I/O and MUST NOT throw for a
|
|
||||||
* well-formed provider (brainy treats a throw as a CONTRACT VIOLATION, never
|
|
||||||
* as "unknown"). When present, brainy's read gate (`assessProviderHealth()`)
|
|
||||||
* reads THIS instead of `isReady()` / size heuristics: `serving` decides
|
|
||||||
* whether reads may proceed; a `false` refuses the read loudly rather than
|
|
||||||
* triggering a rebuild. Absent → the gate falls back to `isReady?()` / the
|
|
||||||
* size heuristic (this train's JS built-in providers stay on that interim
|
|
||||||
* path).
|
|
||||||
*/
|
|
||||||
healthReport?(): HealthReport
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description OPTIONAL eager cold-load. Called once during brain init — AFTER
|
* @description OPTIONAL eager cold-load. Called once during brain init — AFTER
|
||||||
* the metadata provider's `init()` (so the id-mapper is hydrated; a native int
|
* the metadata provider's `init()` (so the id-mapper is hydrated; a native int
|
||||||
|
|
@ -1314,20 +1225,6 @@ export interface VectorIndexProvider {
|
||||||
*/
|
*/
|
||||||
validateInvariants?(): Promise<ProviderInvariantReport>
|
validateInvariants?(): Promise<ProviderInvariantReport>
|
||||||
|
|
||||||
/**
|
|
||||||
* @description OPTIONAL. The named, SYNCHRONOUS, O(1) health verdict this
|
|
||||||
* provider derives from its own exact ledgers — see {@link HealthReport} for
|
|
||||||
* the full derivation laws. MUST NOT perform I/O and MUST NOT throw for a
|
|
||||||
* well-formed provider (brainy treats a throw as a CONTRACT VIOLATION, never
|
|
||||||
* as "unknown"). When present, brainy's read gate (`assessProviderHealth()`)
|
|
||||||
* reads THIS instead of `isReady()` / size heuristics: `serving` decides
|
|
||||||
* whether reads may proceed; a `false` refuses the read loudly rather than
|
|
||||||
* triggering a rebuild. Absent → the gate falls back to `isReady?()` / the
|
|
||||||
* size heuristic (this train's JS built-in providers stay on that interim
|
|
||||||
* path).
|
|
||||||
*/
|
|
||||||
healthReport?(): HealthReport
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description OPTIONAL. A native provider returns true from the moment its
|
* @description OPTIONAL. A native provider returns true from the moment its
|
||||||
* `init()` detects a large epoch-drift until its background
|
* `init()` detects a large epoch-drift until its background
|
||||||
|
|
@ -1575,13 +1472,9 @@ export class PluginRegistry {
|
||||||
this.activated.add(name)
|
this.activated.add(name)
|
||||||
activated.push(name)
|
activated.push(name)
|
||||||
} else {
|
} else {
|
||||||
// Documented graceful decline (activate() → false). Surface it on the
|
// Documented graceful decline (activate() → false). Surface it loudly so
|
||||||
// ALWAYS-ON channel: `silent: true` patches console, and a declined
|
// a silent degrade to the default engine never goes unnoticed.
|
||||||
// accelerator warned into a patched console is a silent degrade to the
|
console.warn(
|
||||||
// default engines — the exact invisible-fallback class this registry
|
|
||||||
// exists to prevent (a production storm ran the WASM engine for 90s
|
|
||||||
// behind one suppressed warn).
|
|
||||||
prodLog.warn(
|
|
||||||
`[brainy] Plugin "${name}" declined activation (activate() returned false); ` +
|
`[brainy] Plugin "${name}" declined activation (activate() returned false); ` +
|
||||||
`the default engine is in use for its providers.`
|
`the default engine is in use for its providers.`
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -154,21 +154,6 @@ export function resolveFilesystemRoot(
|
||||||
) {
|
) {
|
||||||
throwRemovedStorageKey('fileSystemStorage.path')
|
throwRemovedStorageKey('fileSystemStorage.path')
|
||||||
}
|
}
|
||||||
// A nested `config` object carrying a path-shaped key is the same hazard in
|
|
||||||
// a shape nobody ever supported: it used to fall through SILENTLY to the
|
|
||||||
// shared default root — every instance writing one directory while its
|
|
||||||
// caller believed each had its own. (Found live: an integration test's
|
|
||||||
// brains shared one store across a whole single-process run and a health
|
|
||||||
// probe refused on the foreign edges it sampled.) Loud, with the rename.
|
|
||||||
const nested = (config as Record<string, unknown>).config
|
|
||||||
if (nested && typeof nested === 'object') {
|
|
||||||
const pathish = ['path', 'baseDir', 'rootDirectory', 'rootDir', 'dir', 'directory']
|
|
||||||
const hit = pathish.find(
|
|
||||||
(k) => typeof (nested as Record<string, unknown>)[k] === 'string' &&
|
|
||||||
((nested as Record<string, unknown>)[k] as string).length > 0
|
|
||||||
)
|
|
||||||
if (hit) throwRemovedStorageKey(`config.${hit}`)
|
|
||||||
}
|
|
||||||
|
|
||||||
// 3. Zero-config default. A `type: 'filesystem'` with no path lands here
|
// 3. Zero-config default. A `type: 'filesystem'` with no path lands here
|
||||||
// intentionally ("persist, default location").
|
// intentionally ("persist, default location").
|
||||||
|
|
|
||||||
|
|
@ -1816,16 +1816,10 @@ export interface BrainyConfig {
|
||||||
| StorageAdapter
|
| StorageAdapter
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* RE-MEANT (the health-gate contract): `init()` (open) always verifies the
|
* Disable the automatic index rebuild check during `init()`. By default
|
||||||
* durable generation of every derived index, and a needed rebuild ALWAYS
|
* Brainy auto-decides from dataset size: small datasets rebuild missing
|
||||||
* runs at open — it is never deferred to the first read, regardless of
|
* indexes inline, large datasets rebuild lazily on first query. Set `true`
|
||||||
* dataset size or this flag. There is no first-query lazy-build path
|
* only when an operator wants full manual control via `repairIndex()`.
|
||||||
* anymore: a read that finds a provider not serving throws a typed
|
|
||||||
* `*NotReadyError` rather than building anything (see
|
|
||||||
* `assessProviderHealth` / the read gate in `brainy.ts`). Setting this
|
|
||||||
* `true` no longer defers index construction to the first query — it has
|
|
||||||
* no effect on WHEN a needed rebuild runs. Full manual control over
|
|
||||||
* rebuilds remains available via `repairIndex({ rebuild: [...] })`.
|
|
||||||
*/
|
*/
|
||||||
disableAutoRebuild?: boolean
|
disableAutoRebuild?: boolean
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -13,28 +13,13 @@
|
||||||
* `size()` or `isInitialized`. When `isReady()` is absent, callers must fall back
|
* `size()` or `isInitialized`. When `isReady()` is absent, callers must fall back
|
||||||
* to a KNOWN-ITEM PROBE (a real search/lookup that must return a known-present
|
* to a KNOWN-ITEM PROBE (a real search/lookup that must return a known-present
|
||||||
* datum) before trusting an empty result — never a `size()` proxy.
|
* datum) before trusting an empty result — never a `size()` proxy.
|
||||||
*
|
|
||||||
* {@link assessProviderHealth} is the NEWER, PREFERRED authority: it reads a
|
|
||||||
* provider's NAMED, synchronous, O(1) {@link import('../plugin.js').HealthReport}
|
|
||||||
* when one is exposed, and falls back to this file's `isReady()` classifier only
|
|
||||||
* when the provider does not (yet) expose a health report. Read paths in
|
|
||||||
* `brainy.ts` call `assessProviderHealth` exclusively — `assessIndexReadiness`
|
|
||||||
* stays exported for the other call sites (`storage/baseStorage.ts`) and for the
|
|
||||||
* fallback branch inside `assessProviderHealth` itself.
|
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { HealthReport } from '../plugin.js'
|
|
||||||
|
|
||||||
/** A provider that MAY expose the honest cold-load readiness signal. */
|
/** A provider that MAY expose the honest cold-load readiness signal. */
|
||||||
export interface MaybeReadyProvider {
|
export interface MaybeReadyProvider {
|
||||||
isReady?: () => boolean
|
isReady?: () => boolean
|
||||||
}
|
}
|
||||||
|
|
||||||
/** A provider that MAY expose the named, synchronous, O(1) health report. */
|
|
||||||
export interface MaybeHealthReportingProvider {
|
|
||||||
healthReport?: () => HealthReport
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Three-valued honest-readiness verdict. */
|
/** Three-valued honest-readiness verdict. */
|
||||||
export type IndexReadiness = 'ready' | 'not-ready' | 'unknown'
|
export type IndexReadiness = 'ready' | 'not-ready' | 'unknown'
|
||||||
|
|
||||||
|
|
@ -51,105 +36,3 @@ export function assessIndexReadiness(provider: unknown): IndexReadiness {
|
||||||
if (p == null || typeof p.isReady !== 'function') return 'unknown'
|
if (p == null || typeof p.isReady !== 'function') return 'unknown'
|
||||||
return p.isReady() ? 'ready' : 'not-ready'
|
return p.isReady() ? 'ready' : 'not-ready'
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Which signal {@link assessProviderHealth} actually consulted to
|
|
||||||
* produce its verdict — surfaced so callers can narrate (and tests can pin) how
|
|
||||||
* a provider was judged, not just what the judgment was.
|
|
||||||
* - `'health-report'` — the provider's `healthReport()` was called (the authority).
|
|
||||||
* - `'is-ready'` — no `healthReport()`; fell back to the provider's `isReady()`.
|
|
||||||
* - `'size-heuristic'` — no `healthReport()` and no `isReady()`; caller must keep its own size-based heuristic.
|
|
||||||
* - `'none'` — there was no provider to assess (`null`/`undefined`).
|
|
||||||
*/
|
|
||||||
export type ProviderHealthVia = 'health-report' | 'is-ready' | 'size-heuristic' | 'none'
|
|
||||||
|
|
||||||
/** The result of {@link assessProviderHealth}. */
|
|
||||||
export interface ProviderHealthAssessment {
|
|
||||||
/** The honest readiness verdict — see {@link IndexReadiness}. */
|
|
||||||
readiness: IndexReadiness
|
|
||||||
/** The provider's raw {@link HealthReport}, when one was obtained; `null` otherwise. */
|
|
||||||
report: HealthReport | null
|
|
||||||
/** Which signal produced the verdict — see {@link ProviderHealthVia}. */
|
|
||||||
via: ProviderHealthVia
|
|
||||||
/** Human-readable reasons: named failing invariants (with `heal`), unledgered families, or the fallback-path explanation. Empty when the provider is healthy and ready. */
|
|
||||||
reasons: string[]
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description THE read-gate authority. Prefers a provider's NAMED,
|
|
||||||
* synchronous, O(1) {@link HealthReport} over the older `isReady()` / size
|
|
||||||
* heuristics; falls back to {@link assessIndexReadiness}'s semantics only when
|
|
||||||
* a provider does not (yet) expose `healthReport()`.
|
|
||||||
*
|
|
||||||
* Derivation:
|
|
||||||
* - `healthReport()` present → call it (wrapped in try/catch). A THROW is a
|
|
||||||
* CONTRACT VIOLATION, not "unknown": returns `readiness: 'not-ready'`,
|
|
||||||
* `via: 'health-report'`, and a reason naming the throw — never swallowed
|
|
||||||
* into `'unknown'`.
|
|
||||||
* - Otherwise → `readiness = report.serving ? 'ready' : 'not-ready'`; `reasons`
|
|
||||||
* names every invariant with `holds: false` (with its `heal`), plus an
|
|
||||||
* `unledgered: [...]` line when {@link HealthReport.unledgered} is non-empty.
|
|
||||||
* UNLEDGERED IS UNKNOWN: an unledgered family never flips a serving provider
|
|
||||||
* to not-ready, and never flips a not-serving provider to ready — `serving`
|
|
||||||
* is always the provider's own verdict, verbatim.
|
|
||||||
* - No `healthReport()` → fall back to {@link assessIndexReadiness}'s semantics:
|
|
||||||
* `via: 'is-ready'` when `isReady()` exists, `via: 'size-heuristic'` when
|
|
||||||
* neither hook exists (caller must keep its own size-based heuristic),
|
|
||||||
* `via: 'none'` when there is no provider at all.
|
|
||||||
* @param provider - Any index provider (vector / graph / metadata) or `null`/`undefined`.
|
|
||||||
*/
|
|
||||||
export function assessProviderHealth(provider: unknown): ProviderHealthAssessment {
|
|
||||||
const p = provider as (MaybeHealthReportingProvider & MaybeReadyProvider) | null | undefined
|
|
||||||
|
|
||||||
if (p == null) {
|
|
||||||
return { readiness: 'unknown', report: null, via: 'none', reasons: ['no provider to assess'] }
|
|
||||||
}
|
|
||||||
|
|
||||||
if (typeof p.healthReport === 'function') {
|
|
||||||
let report: HealthReport
|
|
||||||
try {
|
|
||||||
report = p.healthReport()
|
|
||||||
} catch (err) {
|
|
||||||
const message = err instanceof Error ? err.message : String(err)
|
|
||||||
return {
|
|
||||||
readiness: 'not-ready',
|
|
||||||
report: null,
|
|
||||||
via: 'health-report',
|
|
||||||
reasons: [`healthReport() threw: ${message} — a health-report throw is a contract violation, never read as healthy`]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const reasons: string[] = []
|
|
||||||
for (const invariant of report.invariants) {
|
|
||||||
if (!invariant.holds) {
|
|
||||||
reasons.push(`${invariant.name} (heal:${invariant.heal}): ${invariant.detail}`)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (report.unledgered.length > 0) {
|
|
||||||
reasons.push(`unledgered: ${report.unledgered.join(', ')}`)
|
|
||||||
}
|
|
||||||
|
|
||||||
return {
|
|
||||||
readiness: report.serving ? 'ready' : 'not-ready',
|
|
||||||
report,
|
|
||||||
via: 'health-report',
|
|
||||||
reasons
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const readiness = assessIndexReadiness(p)
|
|
||||||
if (readiness === 'unknown') {
|
|
||||||
return {
|
|
||||||
readiness,
|
|
||||||
report: null,
|
|
||||||
via: 'size-heuristic',
|
|
||||||
reasons: ['provider exposes neither healthReport() nor isReady() — falling back to the size heuristic']
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return {
|
|
||||||
readiness,
|
|
||||||
report: null,
|
|
||||||
via: 'is-ready',
|
|
||||||
reasons: readiness === 'not-ready' ? ['isReady() returned false'] : []
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
|
||||||
|
|
@ -20,7 +20,6 @@ import {
|
||||||
type WatermarkVerdict,
|
type WatermarkVerdict,
|
||||||
type WatermarkVerdictResult
|
type WatermarkVerdictResult
|
||||||
} from './projectionWatermark.js'
|
} from './projectionWatermark.js'
|
||||||
import type { FactScanHandle } from '../db/factLog.js'
|
|
||||||
import {
|
import {
|
||||||
NounType,
|
NounType,
|
||||||
VerbType,
|
VerbType,
|
||||||
|
|
@ -78,31 +77,6 @@ export interface MetadataIndexStats {
|
||||||
indexSize: number // in bytes
|
indexSize: number // in bytes
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* @description What {@link MetadataIndexManager.applyWatermarkCatchup} did,
|
|
||||||
* for the caller's narration.
|
|
||||||
* - `'noop'` — the verdict was `null`/`'adopt'`: the artifact already
|
|
||||||
* reflects committed truth. Zero index writes.
|
|
||||||
* - `'rescan'` — the verdict was `'rescan'`, OR a `'catchup'` verdict was
|
|
||||||
* demoted (no window, or no fact log to scan) — either way a full
|
|
||||||
* {@link MetadataIndexManager.rebuild} already ran; `reason` names why.
|
|
||||||
* - `'caught-up'` — the `(from, to]` window folded successfully; the
|
|
||||||
* artifact is stamped and flushed at `to`.
|
|
||||||
*/
|
|
||||||
export interface CatchupApplyResult {
|
|
||||||
action: 'noop' | 'rescan' | 'caught-up'
|
|
||||||
/** Present on `'rescan'` — why the fold could not proceed as a catchup. */
|
|
||||||
reason?: string
|
|
||||||
/** Present on `'caught-up'` — the fact-log window that was folded. */
|
|
||||||
window?: { from: number; to: number }
|
|
||||||
/** Present on `'caught-up'` — noun ops applied (add/update/delete). */
|
|
||||||
nounsApplied?: number
|
|
||||||
/** Present on `'caught-up'` — verb ops applied (add/update/delete). */
|
|
||||||
verbsApplied?: number
|
|
||||||
/** Present on `'caught-up'` — distinct committed generations folded. */
|
|
||||||
factsApplied?: number
|
|
||||||
}
|
|
||||||
|
|
||||||
export interface MetadataIndexConfig {
|
export interface MetadataIndexConfig {
|
||||||
maxIndexSize?: number // Max number of entries per field value (default: 10000)
|
maxIndexSize?: number // Max number of entries per field value (default: 10000)
|
||||||
rebuildThreshold?: number // Rebuild if index is this % stale (default: 0.1)
|
rebuildThreshold?: number // Rebuild if index is this % stale (default: 0.1)
|
||||||
|
|
@ -173,52 +147,6 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
private stampedWatermark: number | null = null
|
private stampedWatermark: number | null = null
|
||||||
/** The three-way verdict computed at init; null until init runs. */
|
/** The three-way verdict computed at init; null until init runs. */
|
||||||
private loadVerdict: WatermarkVerdictResult | null = null
|
private loadVerdict: WatermarkVerdictResult | null = null
|
||||||
/**
|
|
||||||
* Set only when {@link loadVerdict}.verdict is `'rescan'`: whether a
|
|
||||||
* persisted artifact existed at load (even an unstamped/unverifiable
|
|
||||||
* one) — distinguishes genuine first boot (nothing here yet, routine)
|
|
||||||
* from an artifact whose watermark is unverifiable (the loud case). The
|
|
||||||
* verdict value alone doesn't carry this distinction; see {@link
|
|
||||||
* watermarkArtifactPresent}.
|
|
||||||
*/
|
|
||||||
private rescanArtifactPresent = false
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description THE BUILD-BESIDE SEAM (B3 Deliverable 3): when set (via
|
|
||||||
* {@link beginShadow}), every live `addToIndex`/`removeFromIndex` call on
|
|
||||||
* THIS instance also applies to the shadow instance — so a caller building
|
|
||||||
* a fresh replacement manager beside this one (walking canonical into it)
|
|
||||||
* never misses a write that lands during the build. This is the ONE seam
|
|
||||||
* that makes build-beside possible without touching every call site: every
|
|
||||||
* existing `AddToMetadataIndexOperation`/`RemoveFromMetadataIndexOperation`
|
|
||||||
* (and the JS manager's own `rebuild()`/catchup fold) keep calling the SAME
|
|
||||||
* serving instance exactly as before; only THIS instance knows it is also
|
|
||||||
* mirroring to a shadow. Null = no build in flight (the overwhelmingly
|
|
||||||
* common case; the check costs one property read per write).
|
|
||||||
*/
|
|
||||||
private shadow: MetadataIndexManager | null = null
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Start mirroring every `addToIndex`/`removeFromIndex` call on
|
|
||||||
* this instance to `shadow` too — see {@link shadow}'s JSDoc. The caller
|
|
||||||
* owns sequencing: writes mirrored WHILE a canonical walk is populating
|
|
||||||
* `shadow` may be clobbered by the walk's own (possibly stale) reads for
|
|
||||||
* the same id; the caller closes that window with a bounded fact-log fold
|
|
||||||
* AFTER the walk (the same mechanism {@link applyWatermarkCatchup} uses)
|
|
||||||
* before treating `shadow` as authoritative.
|
|
||||||
* @param shadow - The manager to mirror writes to.
|
|
||||||
*/
|
|
||||||
beginShadow(shadow: MetadataIndexManager): void {
|
|
||||||
this.shadow = shadow
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Stop mirroring writes to a shadow (see {@link beginShadow}).
|
|
||||||
* Idempotent; a no-op when no shadow is attached.
|
|
||||||
*/
|
|
||||||
endShadow(): void {
|
|
||||||
this.shadow = null
|
|
||||||
}
|
|
||||||
|
|
||||||
// Cardinality and field statistics tracking
|
// Cardinality and field statistics tracking
|
||||||
private fieldStats = new Map<string, FieldStats>()
|
private fieldStats = new Map<string, FieldStats>()
|
||||||
|
|
@ -1676,15 +1604,6 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
for (const { field } of fields) {
|
for (const { field } of fields) {
|
||||||
this.metadataCache.invalidatePattern(`field_values_${field}`)
|
this.metadataCache.invalidatePattern(`field_values_${field}`)
|
||||||
}
|
}
|
||||||
|
|
||||||
// THE BUILD-BESIDE SEAM — see `shadow`'s JSDoc. Mirrors this write to a
|
|
||||||
// shadow manager under construction, if one is attached. `skipFlush:
|
|
||||||
// true` always: the shadow's own persistence is the build orchestrator's
|
|
||||||
// job (it flushes once, after the swap — never mid-build, to avoid
|
|
||||||
// colliding with this instance's own persisted keys).
|
|
||||||
if (this.shadow) {
|
|
||||||
await this.shadow.addToIndex(id, entityOrMetadata, true, false, generation)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -1757,11 +1676,6 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
// the real commit watermark (the JS mapper ignores it).
|
// the real commit watermark (the JS mapper ignores it).
|
||||||
this.idMapper.remove(id, generation)
|
this.idMapper.remove(id, generation)
|
||||||
await this.idMapper.flush()
|
await this.idMapper.flush()
|
||||||
|
|
||||||
// THE BUILD-BESIDE SEAM — see `shadow`'s JSDoc.
|
|
||||||
if (this.shadow) {
|
|
||||||
await this.shadow.removeFromIndex(id, metadata, generation)
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -2845,8 +2759,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
* `'adopt'` (stamped == committed, zero work), `'catchup'` (stamped <
|
* `'adopt'` (stamped == committed, zero work), `'catchup'` (stamped <
|
||||||
* committed; the gap from {@link watermarkGap} awaits an incremental
|
* committed; the gap from {@link watermarkGap} awaits an incremental
|
||||||
* fold), `'rescan'` (unstamped or stamped above committed — never
|
* fold), `'rescan'` (unstamped or stamped above committed — never
|
||||||
* trusted). Null until init() has run. The coordinator (`Brainy.open()`)
|
* trusted). Null until init() has run. Computed and exposed only; no
|
||||||
* consumes this via {@link applyWatermarkCatchup} right after init.
|
* load behavior changes ride on it yet.
|
||||||
*/
|
*/
|
||||||
watermarkVerdict(): WatermarkVerdict | null {
|
watermarkVerdict(): WatermarkVerdict | null {
|
||||||
return this.loadVerdict?.verdict ?? null
|
return this.loadVerdict?.verdict ?? null
|
||||||
|
|
@ -2860,223 +2774,6 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
return this.loadVerdict?.gap ?? null
|
return this.loadVerdict?.gap ?? null
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Meaningful only when {@link watermarkVerdict} is
|
|
||||||
* `'rescan'`: `true` when a persisted artifact existed at load (even an
|
|
||||||
* unstamped/unverifiable one — real prior state, worth narrating loudly);
|
|
||||||
* `false` for a genuine first boot (nothing persisted yet — a caller
|
|
||||||
* should narrate this at a routine log level, not as an alarm, even
|
|
||||||
* though the verdict value is the same `'rescan'` either way).
|
|
||||||
*/
|
|
||||||
watermarkArtifactPresent(): boolean {
|
|
||||||
return this.rescanArtifactPresent
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Consume the three-way watermark verdict {@link
|
|
||||||
* watermarkVerdict} computed at init — the cure for a crash-recovered
|
|
||||||
* store whose canonical reads/counts recover every acked write but whose
|
|
||||||
* metadata projection (flushed only periodically, not per-commit) keeps
|
|
||||||
* serving the pre-crash state. Call once, right after `init()`, before
|
|
||||||
* anything reads from this projection.
|
|
||||||
*
|
|
||||||
* - `null`/`'adopt'` → the artifact already reflects the store's
|
|
||||||
* committed generation. Zero index writes.
|
|
||||||
* - `'catchup'` → the caller-supplied `scan` (expected already opened
|
|
||||||
* over `(watermarkGap().from, watermarkGap().to]`) is folded in, ONE
|
|
||||||
* op at a time, through the SAME two legs {@link rebuild} uses (ADR-007
|
|
||||||
* A4 — one mechanism, never a second hand-rolled add/update shape): a
|
|
||||||
* tombstone (`op.record === null`) retracts id-keyed (this projection
|
|
||||||
* keeps no per-record delta log, so the pre-crash metadata for that id
|
|
||||||
* — if any — is what a value-precise removal would need, and it isn't
|
|
||||||
* available; the same tradeoff `remove()`'s null-metadata closure
|
|
||||||
* already accepts elsewhere); an after-image retracts-then-reposts, so
|
|
||||||
* an update never leaves stale postings under the old field values. A
|
|
||||||
* fact outside the window is skipped defensively (belt: the scan is
|
|
||||||
* already opened to the window; suspenders: this loop never trusts an
|
|
||||||
* over-run). On success the artifact is stamped at `to` and flushed —
|
|
||||||
* the same STAMP-AFTER-DATA door {@link flush} always writes through.
|
|
||||||
* - `'rescan'` (or a `'catchup'` verdict with no window, or no `scan` to
|
|
||||||
* fold — the store hosts no fact log) → the persisted artifact is
|
|
||||||
* unverifiable; this method runs the existing {@link rebuild} itself
|
|
||||||
* rather than leave the caller to notice and trigger it separately.
|
|
||||||
*
|
|
||||||
* @param scan - An open fact scan covering the catchup window (see
|
|
||||||
* {@link Brainy.scanFacts}), or `null` when none is available/needed.
|
|
||||||
* Ignored when the verdict is not `'catchup'`.
|
|
||||||
* @returns What happened — see {@link CatchupApplyResult}.
|
|
||||||
*/
|
|
||||||
async applyWatermarkCatchup(scan: FactScanHandle | null): Promise<CatchupApplyResult> {
|
|
||||||
const verdict = this.watermarkVerdict()
|
|
||||||
if (verdict === null || verdict === 'adopt') return { action: 'noop' }
|
|
||||||
|
|
||||||
if (verdict === 'rescan') {
|
|
||||||
await this.rebuild()
|
|
||||||
return {
|
|
||||||
action: 'rescan',
|
|
||||||
reason: 'persisted artifact is unverifiable (unstamped, or stamped ABOVE the ' +
|
|
||||||
"store's committed generation) — never adopting unverifiable state"
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// verdict === 'catchup'
|
|
||||||
const window = this.watermarkGap()
|
|
||||||
if (window === null) {
|
|
||||||
await this.rebuild()
|
|
||||||
return { action: 'rescan', reason: "'catchup' verdict exposed no window — cannot bound a fold" }
|
|
||||||
}
|
|
||||||
if (scan === null) {
|
|
||||||
await this.rebuild()
|
|
||||||
return {
|
|
||||||
action: 'rescan',
|
|
||||||
reason: `no fact log available to fold the (${window.from}, ${window.to}] catchup window`
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const { nounsApplied, verbsApplied, factsApplied } = await this.foldFactWindow(scan, window.from, window.to)
|
|
||||||
|
|
||||||
this.stampWatermark(window.to)
|
|
||||||
await this.flush()
|
|
||||||
return { action: 'caught-up', window, nounsApplied, verbsApplied, factsApplied }
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Fold an open fact scan's `(fromGeneration, toGeneration]`
|
|
||||||
* window into this projection, ONE op at a time, through the SAME two legs
|
|
||||||
* {@link rebuild} uses (ADR-007 A4 — one mechanism, never a second
|
|
||||||
* hand-rolled add/update shape): a tombstone retracts id-keyed; an
|
|
||||||
* after-image retracts-then-reposts. THE CORE LOOP shared by {@link
|
|
||||||
* applyWatermarkCatchup} (which stamps + flushes after) and {@link
|
|
||||||
* buildBeside} (which does neither — persistence is the caller's job,
|
|
||||||
* exactly once, after a swap). Never stamps, never flushes, never touches
|
|
||||||
* storage beyond what `addToIndex`/`removeFromIndex` do internally
|
|
||||||
* (skipFlush is always forced true).
|
|
||||||
* @param scan - An open fact scan.
|
|
||||||
* @param fromGeneration - Window lower bound (exclusive).
|
|
||||||
* @param toGeneration - Window upper bound (inclusive).
|
|
||||||
* @returns Counts for the caller's narration.
|
|
||||||
*/
|
|
||||||
private async foldFactWindow(
|
|
||||||
scan: FactScanHandle,
|
|
||||||
fromGeneration: number,
|
|
||||||
toGeneration: number
|
|
||||||
): Promise<{ nounsApplied: number; verbsApplied: number; factsApplied: number }> {
|
|
||||||
let nounsApplied = 0
|
|
||||||
let verbsApplied = 0
|
|
||||||
let factsApplied = 0
|
|
||||||
for await (const batch of scan.batches()) {
|
|
||||||
for (const fact of batch.facts) {
|
|
||||||
// Defensive containment: the scan is already opened to the window,
|
|
||||||
// but a fact outside it is never applied regardless.
|
|
||||||
if (fact.generation <= fromGeneration || fact.generation > toGeneration) continue
|
|
||||||
const generation = BigInt(fact.generation)
|
|
||||||
for (const op of fact.ops) {
|
|
||||||
if (op.record === null) {
|
|
||||||
// TOMBSTONE — the id-keyed removal path (no per-record delta
|
|
||||||
// log to recover the old field values from).
|
|
||||||
await this.removeFromIndex(op.id, undefined, generation)
|
|
||||||
} else {
|
|
||||||
// AFTER-IMAGE — retract any stale posting for this id, then
|
|
||||||
// repost the new shape. Covers both a fresh add (nothing to
|
|
||||||
// retract; a no-op-ish remove) and an update, through the same
|
|
||||||
// two calls.
|
|
||||||
await this.removeFromIndex(op.id, undefined, generation)
|
|
||||||
await this.indexStoredRecord(op.id, op.record.metadata, {
|
|
||||||
skipFlush: true,
|
|
||||||
deferWrites: false,
|
|
||||||
generation
|
|
||||||
})
|
|
||||||
}
|
|
||||||
if (op.kind === 'noun') nounsApplied++
|
|
||||||
else verbsApplied++
|
|
||||||
}
|
|
||||||
factsApplied++
|
|
||||||
}
|
|
||||||
}
|
|
||||||
return { nounsApplied, verbsApplied, factsApplied }
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description B3 Deliverable 3 — the shadow-build lifecycle's init: the
|
|
||||||
* MINIMUM setup {@link buildBeside} needs, deliberately NOT the general
|
|
||||||
* {@link init} sequence. Two reasons general `init()` is unsafe for a
|
|
||||||
* build-beside shadow:
|
|
||||||
* 1. `init()` unconditionally re-initializes the id mapper from storage
|
|
||||||
* (`idMapper.init()`) — safe for a FRESH mapper, but this instance is
|
|
||||||
* constructed with the CURRENTLY-SERVING manager's SHARED, already-live
|
|
||||||
* mapper (identity is shared, never a second mapper — this train's own
|
|
||||||
* law). Re-running its init() would DISCARD every not-yet-flushed
|
|
||||||
* UUID↔int assignment sitting in memory, breaking the live manager's
|
|
||||||
* own serving mid-build.
|
|
||||||
* 2. `init()` loads the field registry and, on a registry that's
|
|
||||||
* missing/empty while canonical has entities (exactly the shape a
|
|
||||||
* rebuild is often invoked to FIX), triggers `rebuild()` itself —
|
|
||||||
* WITHOUT `inMemoryOnly`, which would touch the shared storage keys
|
|
||||||
* the live manager depends on.
|
|
||||||
* What this DOES run: the WASM roaring-bitmap library init (idempotent;
|
|
||||||
* needed before any column-store write) and the column store's OWN
|
|
||||||
* segment-manifest discovery (read-only against shared storage; needed so
|
|
||||||
* THIS instance's eventual post-swap flush continues segment numbering
|
|
||||||
* correctly instead of colliding with the retiring manager's segments).
|
|
||||||
*/
|
|
||||||
private async initForShadowBuild(): Promise<void> {
|
|
||||||
await roaringLibraryInitialize()
|
|
||||||
try {
|
|
||||||
await this.columnStore.init(this.storage, this.idMapper)
|
|
||||||
} catch (err) {
|
|
||||||
prodLog.warn('[MetadataIndex] shadow build: column store storage discovery failed:', err)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* @description B3 Deliverable 3 — THE ONLINE REBUILD's manager-side half:
|
|
||||||
* populate THIS instance (expected fresh/empty, constructed with the SAME
|
|
||||||
* storage + idMapper as the manager it will replace — see {@link
|
|
||||||
* initForShadowBuild}) from canonical storage without ever touching the
|
|
||||||
* shared storage keys the currently-serving manager depends on — no chunk
|
|
||||||
* deletion, no flush, anywhere in this call. The caller (the brain's
|
|
||||||
* rebuild-beside orchestrator) is responsible for:
|
|
||||||
* 1. Attaching this instance as a {@link beginShadow} target on the OLD
|
|
||||||
* manager BEFORE calling this, so live writes during the walk mirror
|
|
||||||
* here too (best-effort — the walk below may still clobber a mirrored
|
|
||||||
* write with a stale read for the same id; the fold after the walk is
|
|
||||||
* what makes the final state authoritative, not the mirror).
|
|
||||||
* 2. Swapping its own reference to this instance once this resolves.
|
|
||||||
* 3. Calling {@link stampWatermark} + {@link flush} EXACTLY ONCE, after
|
|
||||||
* the swap — this instance never persists itself.
|
|
||||||
* @param committedGenerationAtStart - The store's committed generation
|
|
||||||
* captured by the caller BEFORE this call — the fold's lower bound.
|
|
||||||
* @returns The generation this instance's canonical data reflects once the
|
|
||||||
* walk + fold settle — the fold's upper bound (writes committed after
|
|
||||||
* this point but before the swap only reach this instance via the live
|
|
||||||
* {@link beginShadow} mirror, so the caller re-reads the store's
|
|
||||||
* committed generation right before stamping, rather than trusting this
|
|
||||||
* return value as final).
|
|
||||||
* @throws If canonical advanced during the walk but no fact log is
|
|
||||||
* available to fold the gap — never a silently incomplete shadow.
|
|
||||||
*/
|
|
||||||
async buildBeside(committedGenerationAtStart: number): Promise<number> {
|
|
||||||
await this.initForShadowBuild()
|
|
||||||
await this.rebuild({ inMemoryOnly: true })
|
|
||||||
|
|
||||||
const committedAfterWalk = this.storage.committedGeneration?.() ?? committedGenerationAtStart
|
|
||||||
if (committedAfterWalk > committedGenerationAtStart) {
|
|
||||||
const scan = this.storage.scanFacts?.({
|
|
||||||
fromGeneration: committedGenerationAtStart + 1,
|
|
||||||
toGeneration: committedAfterWalk
|
|
||||||
}) ?? null
|
|
||||||
if (scan === null) {
|
|
||||||
throw new Error(
|
|
||||||
`MetadataIndexManager.buildBeside: canonical advanced from generation ` +
|
|
||||||
`${committedGenerationAtStart} to ${committedAfterWalk} during the walk, but this ` +
|
|
||||||
`store hosts no fact log to fold the gap — refusing a silently incomplete shadow`
|
|
||||||
)
|
|
||||||
}
|
|
||||||
await this.foldFactWindow(scan, committedGenerationAtStart, committedAfterWalk)
|
|
||||||
}
|
|
||||||
return committedAfterWalk
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description Write the pending watermark stamp as a sidecar record —
|
* @description Write the pending watermark stamp as a sidecar record —
|
||||||
* always called AFTER the data it certifies is durable. A stamp-write
|
* always called AFTER the data it certifies is durable. A stamp-write
|
||||||
|
|
@ -3129,7 +2826,6 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
|
|
||||||
if (result.verdict === 'rescan') {
|
if (result.verdict === 'rescan') {
|
||||||
const artifactPresent = this.fieldIndexes.size > 0 || stamped !== null
|
const artifactPresent = this.fieldIndexes.size > 0 || stamped !== null
|
||||||
this.rescanArtifactPresent = artifactPresent
|
|
||||||
if (artifactPresent) {
|
if (artifactPresent) {
|
||||||
prodLog.warn(
|
prodLog.warn(
|
||||||
`[MetadataIndex] watermark verdict: RESCAN — persisted index is ` +
|
`[MetadataIndex] watermark verdict: RESCAN — persisted index is ` +
|
||||||
|
|
@ -3788,48 +3484,13 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
|
||||||
* @description Index one raw stored noun/verb record — THE ONE add leg
|
|
||||||
* shared by {@link rebuild}'s canonical walk and {@link
|
|
||||||
* applyWatermarkCatchup}'s fact-log fold (ADR-007 A4: one mechanism,
|
|
||||||
* never a second hand-rolled shape). No conversion step is needed here:
|
|
||||||
* a raw stored record (`storage.getNounMetadata`/`getVerbMetadata`, or a
|
|
||||||
* fact's after-image `record.metadata`) is byte-identical — both read the
|
|
||||||
* exact same canonical path — and already the v2 nested-bag
|
|
||||||
* ("entity-record") shape {@link extractIndexableFields} expects.
|
|
||||||
* @param id - Entity/relationship id.
|
|
||||||
* @param storedMetadata - The raw stored metadata record.
|
|
||||||
* @param opts.skipFlush - Forwarded to {@link addToIndex}.
|
|
||||||
* @param opts.deferWrites - Forwarded to {@link addToIndex}.
|
|
||||||
* @param opts.generation - Forwarded to {@link addToIndex}.
|
|
||||||
*/
|
|
||||||
private async indexStoredRecord(
|
|
||||||
id: string,
|
|
||||||
storedMetadata: unknown,
|
|
||||||
opts: { skipFlush: boolean; deferWrites: boolean; generation?: bigint }
|
|
||||||
): Promise<void> {
|
|
||||||
await this.addToIndex(id, storedMetadata, opts.skipFlush, opts.deferWrites, opts.generation)
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Rebuild entire index from scratch using pagination
|
* Rebuild entire index from scratch using pagination
|
||||||
* Non-blocking version that yields control back to event loop
|
* Non-blocking version that yields control back to event loop
|
||||||
* Sparse indices now lazy-loaded via UnifiedCache (no need to clear Map)
|
* Sparse indices now lazy-loaded via UnifiedCache (no need to clear Map)
|
||||||
*
|
|
||||||
* @param options.inMemoryOnly - B3 Deliverable 3 (build-beside): when
|
|
||||||
* `true`, this call never touches the shared storage keys another,
|
|
||||||
* currently-serving `MetadataIndexManager` over the SAME storage may
|
|
||||||
* depend on — it skips deleting persisted legacy chunk files AND skips
|
|
||||||
* the final `flush()` (which would otherwise write field indexes AND
|
|
||||||
* flush the column store's tail buffers to shared segment keys,
|
|
||||||
* colliding with a live manager's own writes). The caller ({@link
|
|
||||||
* buildBeside}) owns persistence entirely — exactly once, after this
|
|
||||||
* instance becomes the sole owner via an atomic swap. Default `false`
|
|
||||||
* (every other caller keeps today's clear-then-persist behavior).
|
|
||||||
*/
|
*/
|
||||||
async rebuild(options?: { inMemoryOnly?: boolean }): Promise<void> {
|
async rebuild(): Promise<void> {
|
||||||
if (this.isRebuilding) return
|
if (this.isRebuilding) return
|
||||||
const inMemoryOnly = options?.inMemoryOnly ?? false
|
|
||||||
|
|
||||||
this.isRebuilding = true
|
this.isRebuilding = true
|
||||||
try {
|
try {
|
||||||
|
|
@ -3858,12 +3519,6 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
// here — it's always saved at the end of rebuild via flush(). This ensures
|
// here — it's always saved at the end of rebuild via flush(). This ensures
|
||||||
// that if rebuild fails partway, the next init() can still discover fields
|
// that if rebuild fails partway, the next init() can still discover fields
|
||||||
// and trigger another rebuild attempt.
|
// and trigger another rebuild attempt.
|
||||||
//
|
|
||||||
// SKIPPED for inMemoryOnly: these are the SHARED storage keys a live
|
|
||||||
// manager over the same storage may still be reading (see this
|
|
||||||
// method's JSDoc) — deleting them before the swap is a live-read
|
|
||||||
// hazard, not a cleanup.
|
|
||||||
if (!inMemoryOnly) {
|
|
||||||
prodLog.info('Clearing existing metadata index chunks from storage...')
|
prodLog.info('Clearing existing metadata index chunks from storage...')
|
||||||
const existingFields = await this.getPersistedFieldList()
|
const existingFields = await this.getPersistedFieldList()
|
||||||
|
|
||||||
|
|
@ -3874,7 +3529,6 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
|
|
||||||
prodLog.info(`Cleared ${existingFields.length} field indexes from storage`)
|
prodLog.info(`Cleared ${existingFields.length} field indexes from storage`)
|
||||||
}
|
}
|
||||||
}
|
|
||||||
|
|
||||||
// EntityIdMapper is intentionally NOT cleared here. Rebuild re-iterates
|
// EntityIdMapper is intentionally NOT cleared here. Rebuild re-iterates
|
||||||
// every entity in storage and calls idMapper.getOrAssign(uuid), which
|
// every entity in storage and calls idMapper.getOrAssign(uuid), which
|
||||||
|
|
@ -3928,7 +3582,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
for (const noun of result.items) {
|
for (const noun of result.items) {
|
||||||
const metadata = metadataBatch.get(noun.id)
|
const metadata = metadataBatch.get(noun.id)
|
||||||
if (metadata) {
|
if (metadata) {
|
||||||
await this.indexStoredRecord(noun.id, metadata, { skipFlush: true, deferWrites: true })
|
await this.addToIndex(noun.id, metadata, true, true)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -3973,7 +3627,7 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
for (const verb of result.items) {
|
for (const verb of result.items) {
|
||||||
const metadata = verbMetadataBatch.get(verb.id)
|
const metadata = verbMetadataBatch.get(verb.id)
|
||||||
if (metadata) {
|
if (metadata) {
|
||||||
await this.indexStoredRecord(verb.id, metadata, { skipFlush: true, deferWrites: true })
|
await this.addToIndex(verb.id, metadata, true, true)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -3983,16 +3637,8 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
|
|
||||||
// Flush to storage. The column store's flush() handles tail-buffer-to-
|
// Flush to storage. The column store's flush() handles tail-buffer-to-
|
||||||
// segment promotion + manifest persistence.
|
// segment promotion + manifest persistence.
|
||||||
//
|
|
||||||
// SKIPPED for inMemoryOnly — see this method's JSDoc: flush() writes
|
|
||||||
// the shared field-index keys AND flushes the column store's tail
|
|
||||||
// buffers to shared segment keys, which would race a live manager's
|
|
||||||
// own flushes over the SAME storage. The caller flushes exactly once,
|
|
||||||
// after the swap.
|
|
||||||
if (!inMemoryOnly) {
|
|
||||||
prodLog.debug('💾 Flushing metadata index to storage...')
|
prodLog.debug('💾 Flushing metadata index to storage...')
|
||||||
await this.flush()
|
await this.flush()
|
||||||
}
|
|
||||||
|
|
||||||
prodLog.info(`✅ Metadata index rebuild completed! Processed ${totalNounsProcessed} nouns and ${totalVerbsProcessed} verbs`)
|
prodLog.info(`✅ Metadata index rebuild completed! Processed ${totalNounsProcessed} nouns and ${totalVerbsProcessed} verbs`)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -20,9 +20,6 @@ export default defineConfig({
|
||||||
// Include only integration tests
|
// Include only integration tests
|
||||||
include: [
|
include: [
|
||||||
'tests/integration/**/*.test.ts',
|
'tests/integration/**/*.test.ts',
|
||||||
// The lifecycle biography lane (day-in-the-life scenarios; see
|
|
||||||
// tests/lifecycle/README.md) runs in the integration gate.
|
|
||||||
'tests/lifecycle/**/*.test.ts',
|
|
||||||
'tests/**/*.integration.test.ts'
|
'tests/**/*.integration.test.ts'
|
||||||
],
|
],
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -15,7 +15,13 @@ describe('Batch Import with Immediate Relations (v5.7.3 Fix)', () => {
|
||||||
|
|
||||||
// Initialize brain
|
// Initialize brain
|
||||||
brain = new Brainy({ requireSubtype: false,
|
brain = new Brainy({ requireSubtype: false,
|
||||||
storage: { type: 'filesystem', path: testDir },
|
storage: {
|
||||||
|
type: 'filesystem',
|
||||||
|
config: {
|
||||||
|
baseDir: testDir,
|
||||||
|
enableCompression: false // Faster tests
|
||||||
|
}
|
||||||
|
},
|
||||||
dimensions: 384
|
dimensions: 384
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,29 +1,29 @@
|
||||||
/**
|
/**
|
||||||
* @module tests/integration/cold-graph-connected-8.0
|
* @module tests/integration/cold-graph-connected-8.0
|
||||||
* @description BRAINY-COLD-GRAPH-CONNECTED (8.0) — regression coverage for the silent-empty
|
* @description BRAINY-COLD-GRAPH-CONNECTED (8.0) — regression coverage for the silent-empty
|
||||||
* graph-traversal bug, gated on the honest readiness signal: a sync `graphIndex.isReady()`
|
* graph-traversal bug, gated on the converged 8.0 contract: a sync `graphIndex.isReady()` that
|
||||||
* that is true ONLY when the source→target EDGES are loaded (NOT the membership/manifest count).
|
* is true ONLY when the source→target EDGES are loaded (NOT the membership/manifest count).
|
||||||
*
|
*
|
||||||
* On the FIRST `find({ connected })` after a cold process start, a native graph adjacency can
|
* On the FIRST `find({ connected })` after a cold process start of a LARGE brain (≥10k nouns,
|
||||||
* reload its relationship COUNT (so `size() > 0`) but NOT its edges — so `getNeighbors()` returns
|
* which skips the eager index rebuild), a native graph adjacency can reload its relationship
|
||||||
* `[]` for EVERY source and brainy would serve that `[]` as if the anchor were genuinely edgeless.
|
* COUNT (so `size() > 0`) but NOT its edges — so `getNeighbors()` returns `[]` for EVERY source
|
||||||
|
* and brainy would serve that `[]` as if the anchor were genuinely edgeless.
|
||||||
*
|
*
|
||||||
* RE-POINTED to the health-gate law: `verifyGraphAdjacencyLive` NEVER rebuilds and NEVER walks the
|
* The 8.0 guard (`verifyGraphAdjacencyLive`) prefers the honest `isReady()` signal:
|
||||||
* store from a read — a read-path rebuild is exactly the dark-rebuild failure mode the law retires
|
* - `isReady() === false` → hydrate the id-mapper, rebuild from storage, re-check; a still-false
|
||||||
* (open() alone owns building). The guard now:
|
* `isReady()` throws {@link GraphIndexNotReadyError} instead of returning `[]` ('rebuilt' when
|
||||||
* - `isReady() === false` → THROWS {@link GraphIndexNotReadyError} immediately — no rebuild attempt;
|
* the rebuild heals it);
|
||||||
* - a genuinely edgeless anchor with `isReady() === true` verifies 'live' and the empty result
|
* - a genuinely edgeless anchor with `isReady() === true` verifies 'live' and the empty result
|
||||||
* stands — no spurious throw;
|
* stands — no spurious rebuild, no throw;
|
||||||
* - a provider WITHOUT `isReady()` falls back to the shipped known-edge-sample probe, which is
|
* - a provider WITHOUT `isReady()` falls back to the shipped 7.x known-edge-sample probe.
|
||||||
* now READ-ONLY: it refuses loudly (throws) rather than self-healing via rebuild.
|
|
||||||
*
|
*
|
||||||
* These exercise REAL `find({ connected })` against an in-memory brain whose graph index is
|
* These exercise REAL `find({ connected })` against an in-memory brain whose graph index is
|
||||||
* instrumented with a test-double `isReady()` (and, for the fallback case, an always-empty
|
* instrumented with a test-double `isReady()` (and, for the fallback case, an empty-then-healed
|
||||||
* `getNeighbors`). Only the readiness/edge surface is wrapped; the underlying real adjacency
|
* `getNeighbors`). Only the readiness/edge surface is wrapped; the underlying real adjacency
|
||||||
* (built by `relate()`) is what a healthy provider actually serves.
|
* (built by `relate()`) is unmasked once a rebuild "heals" it.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { describe, it, expect, afterEach, vi } from 'vitest'
|
import { describe, it, expect, afterEach } from 'vitest'
|
||||||
import { Brainy } from '../../src/index.js'
|
import { Brainy } from '../../src/index.js'
|
||||||
import { NounType, VerbType } from '../../src/types/graphTypes.js'
|
import { NounType, VerbType } from '../../src/types/graphTypes.js'
|
||||||
import { GraphIndexNotReadyError } from '../../src/errors/brainyError.js'
|
import { GraphIndexNotReadyError } from '../../src/errors/brainyError.js'
|
||||||
|
|
@ -63,17 +63,17 @@ async function buildBrain(
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Instrument the brain's real graph index with a test-double `isReady()` (the honest-readiness
|
* Instrument the brain's real graph index with a test-double `isReady()` (the 8.0 contract) plus
|
||||||
* contract) plus an edge surface that goes empty while NOT ready. `getNeighbors` returns `[]`
|
* an edge surface that goes empty while NOT ready. `getNeighbors` returns `[]` while `!ready`
|
||||||
* while `!ready` (modelling the cold-unloaded adjacency) and delegates to the REAL index once
|
* (modelling the cold-unloaded adjacency) and delegates to the REAL index once a rebuild flips
|
||||||
* `ready` flips true (used only by the "healthy" control cases — the guard itself never flips
|
* `ready` on. `rebuild` is counted; it heals (`ready = true`) only when `healsOnRebuild` is set.
|
||||||
* this anymore, since it never rebuilds). `rebuild` is counted so tests can assert it is NEVER
|
* Pass `failFirstRebuild` to make the FIRST rebuild throw a transient error (without healing) so
|
||||||
* called by a read.
|
* the empty-result re-collect path in executeGraphSearch is exercised.
|
||||||
*/
|
*/
|
||||||
function instrumentIsReady(
|
function instrumentIsReady(
|
||||||
brain: any,
|
brain: any,
|
||||||
opts: { ready: boolean }
|
opts: { ready: boolean; healsOnRebuild: boolean; failFirstRebuild?: boolean }
|
||||||
): { rebuildCalls: number; ready: boolean } {
|
): { rebuildCalls: number } {
|
||||||
const gi = brain.graphIndex
|
const gi = brain.graphIndex
|
||||||
const origGetNeighbors = gi.getNeighbors.bind(gi)
|
const origGetNeighbors = gi.getNeighbors.bind(gi)
|
||||||
const state = { ready: opts.ready, rebuildCalls: 0 }
|
const state = { ready: opts.ready, rebuildCalls: 0 }
|
||||||
|
|
@ -85,6 +85,10 @@ function instrumentIsReady(
|
||||||
|
|
||||||
gi.rebuild = async (): Promise<void> => {
|
gi.rebuild = async (): Promise<void> => {
|
||||||
state.rebuildCalls++
|
state.rebuildCalls++
|
||||||
|
if (opts.failFirstRebuild && state.rebuildCalls === 1) {
|
||||||
|
throw new Error('transient rebuild hiccup')
|
||||||
|
}
|
||||||
|
if (opts.healsOnRebuild) state.ready = true // unmask the real (already-populated) adjacency
|
||||||
}
|
}
|
||||||
|
|
||||||
return state
|
return state
|
||||||
|
|
@ -92,12 +96,12 @@ function instrumentIsReady(
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Fallback instrumentation — a provider WITHOUT `isReady()` (older cortex / JS baseline). Wraps
|
* Fallback instrumentation — a provider WITHOUT `isReady()` (older cortex / JS baseline). Wraps
|
||||||
* `getNeighbors` to always return `[]` while `broken`. This is the shipped known-edge-sample
|
* `getNeighbors` to return `[]` while `broken` and delegates to the REAL index once a rebuild
|
||||||
* probe path — now READ-ONLY: it refuses loudly rather than self-healing.
|
* heals it. This is the shipped 7.x known-edge-sample probe path on 8.0.
|
||||||
*/
|
*/
|
||||||
function instrumentNoIsReady(
|
function instrumentNoIsReady(
|
||||||
brain: any,
|
brain: any,
|
||||||
opts: { broken: boolean }
|
opts: { broken: boolean; healsOnRebuild: boolean }
|
||||||
): { rebuildCalls: number } {
|
): { rebuildCalls: number } {
|
||||||
const gi = brain.graphIndex
|
const gi = brain.graphIndex
|
||||||
// Ensure the provider does NOT expose isReady() — the default JS provider doesn't.
|
// Ensure the provider does NOT expose isReady() — the default JS provider doesn't.
|
||||||
|
|
@ -110,12 +114,13 @@ function instrumentNoIsReady(
|
||||||
|
|
||||||
gi.rebuild = async (): Promise<void> => {
|
gi.rebuild = async (): Promise<void> => {
|
||||||
state.rebuildCalls++
|
state.rebuildCalls++
|
||||||
|
if (opts.healsOnRebuild) state.broken = false
|
||||||
}
|
}
|
||||||
|
|
||||||
return state
|
return state
|
||||||
}
|
}
|
||||||
|
|
||||||
describe('BRAINY-COLD-GRAPH-CONNECTED 8.0 — isReady()-gated, never serves a silent [], never rebuilds from a read', () => {
|
describe('BRAINY-COLD-GRAPH-CONNECTED 8.0 — isReady()-gated, never serves a silent []', () => {
|
||||||
let brains: any[] = []
|
let brains: any[] = []
|
||||||
afterEach(async () => {
|
afterEach(async () => {
|
||||||
for (const b of brains) {
|
for (const b of brains) {
|
||||||
|
|
@ -126,37 +131,35 @@ describe('BRAINY-COLD-GRAPH-CONNECTED 8.0 — isReady()-gated, never serves a si
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
brains = []
|
brains = []
|
||||||
vi.restoreAllMocks()
|
|
||||||
})
|
})
|
||||||
|
|
||||||
it('(a) isReady() false → THROWS GraphIndexNotReadyError immediately, no rebuild attempt', async () => {
|
it('(a) isReady() false → rebuild heals it true → find({ connected }) returns correct N (rebuilt)', async () => {
|
||||||
|
const { brain, anchorId, targetIds } = await buildBrain({ anchorEdges: true })
|
||||||
|
brains.push(brain)
|
||||||
|
const state = instrumentIsReady(brain, { ready: false, healsOnRebuild: true })
|
||||||
|
|
||||||
|
const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
|
||||||
|
|
||||||
|
expect(state.rebuildCalls).toBeGreaterThanOrEqual(1) // detected not-ready + healed it
|
||||||
|
const ids = results.map((r: any) => r.id).sort()
|
||||||
|
expect(ids).toEqual(targetIds.sort()) // B, C, D — the real edges, served after the heal
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(b) isReady() stays false after rebuild → throws GraphIndexNotReadyError (NOT a silent [])', async () => {
|
||||||
const { brain, anchorId } = await buildBrain({ anchorEdges: true })
|
const { brain, anchorId } = await buildBrain({ anchorEdges: true })
|
||||||
brains.push(brain)
|
brains.push(brain)
|
||||||
const state = instrumentIsReady(brain, { ready: false })
|
instrumentIsReady(brain, { ready: false, healsOnRebuild: false }) // rebuild never makes it ready
|
||||||
|
|
||||||
await expect(
|
await expect(
|
||||||
brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
|
brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
|
||||||
).rejects.toBeInstanceOf(GraphIndexNotReadyError)
|
).rejects.toBeInstanceOf(GraphIndexNotReadyError)
|
||||||
|
|
||||||
expect(state.rebuildCalls).toBe(0) // a read never rebuilds — it refuses loudly instead
|
|
||||||
})
|
|
||||||
|
|
||||||
it('(b) isReady() stays false → throws GraphIndexNotReadyError (NOT a silent [])', async () => {
|
|
||||||
const { brain, anchorId } = await buildBrain({ anchorEdges: true })
|
|
||||||
brains.push(brain)
|
|
||||||
const state = instrumentIsReady(brain, { ready: false })
|
|
||||||
|
|
||||||
await expect(
|
|
||||||
brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
|
|
||||||
).rejects.toBeInstanceOf(GraphIndexNotReadyError)
|
|
||||||
expect(state.rebuildCalls).toBe(0)
|
|
||||||
})
|
})
|
||||||
|
|
||||||
it('(c) edgeless anchor + isReady() true → returns [] with NO rebuild and NO throw', async () => {
|
it('(c) edgeless anchor + isReady() true → returns [] with NO rebuild and NO throw', async () => {
|
||||||
// The anchor has no edges, but E -> F does — the adjacency is genuinely loaded (ready).
|
// The anchor has no edges, but E -> F does — the adjacency is genuinely loaded (ready).
|
||||||
const { brain, anchorId } = await buildBrain({ anchorEdges: false })
|
const { brain, anchorId } = await buildBrain({ anchorEdges: false })
|
||||||
brains.push(brain)
|
brains.push(brain)
|
||||||
const state = instrumentIsReady(brain, { ready: true })
|
const state = instrumentIsReady(brain, { ready: true, healsOnRebuild: false })
|
||||||
|
|
||||||
const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
|
const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
|
||||||
|
|
||||||
|
|
@ -167,7 +170,7 @@ describe('BRAINY-COLD-GRAPH-CONNECTED 8.0 — isReady()-gated, never serves a si
|
||||||
it('(d) healthy isReady() true → correct results, NO rebuild', async () => {
|
it('(d) healthy isReady() true → correct results, NO rebuild', async () => {
|
||||||
const { brain, anchorId, targetIds } = await buildBrain({ anchorEdges: true })
|
const { brain, anchorId, targetIds } = await buildBrain({ anchorEdges: true })
|
||||||
brains.push(brain)
|
brains.push(brain)
|
||||||
const state = instrumentIsReady(brain, { ready: true })
|
const state = instrumentIsReady(brain, { ready: true, healsOnRebuild: false })
|
||||||
|
|
||||||
const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
|
const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
|
||||||
|
|
||||||
|
|
@ -176,30 +179,30 @@ describe('BRAINY-COLD-GRAPH-CONNECTED 8.0 — isReady()-gated, never serves a si
|
||||||
expect(ids).toEqual(targetIds.sort())
|
expect(ids).toEqual(targetIds.sort())
|
||||||
})
|
})
|
||||||
|
|
||||||
it('(e) provider WITHOUT isReady() → the known-edge-sample probe REFUSES LOUDLY (never self-heals)', async () => {
|
it('(e) provider WITHOUT isReady() → falls back to the known-edge-sample probe (self-heals)', async () => {
|
||||||
const { brain, anchorId } = await buildBrain({ anchorEdges: true })
|
const { brain, anchorId, targetIds } = await buildBrain({ anchorEdges: true })
|
||||||
brains.push(brain)
|
brains.push(brain)
|
||||||
const state = instrumentNoIsReady(brain, { broken: true })
|
const state = instrumentNoIsReady(brain, { broken: true, healsOnRebuild: true })
|
||||||
|
|
||||||
await expect(
|
const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
|
||||||
brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
|
|
||||||
).rejects.toBeInstanceOf(GraphIndexNotReadyError)
|
|
||||||
|
|
||||||
expect(state.rebuildCalls).toBe(0) // the fallback probe is READ-ONLY — it never calls rebuild()
|
expect(state.rebuildCalls).toBeGreaterThanOrEqual(1) // detected the empty adjacency + healed it
|
||||||
|
const ids = results.map((r: any) => r.id).sort()
|
||||||
|
expect(ids).toEqual(targetIds.sort()) // B, C, D — served after the heal
|
||||||
})
|
})
|
||||||
|
|
||||||
it('(f) an empty connectedIds set re-verifies against a not-serving adjacency and throws, rather than serving [] as truth', async () => {
|
it('(f) executeGraphSearch re-collect: a transient first rebuild leaves connectedIds empty; the empty-result guard then heals + re-collects', async () => {
|
||||||
// executeGraphSearch's cold-load guard (connectedIds.size === 0 → re-verify) used to
|
// First verify (inside neighbors()) hits a transient rebuild failure → returns 'live' without
|
||||||
// interpret a healed rebuild as "re-collect and serve." That rebuild-and-heal path is
|
// healing, so getNeighbors stays empty and connectedIds is empty. The empty connectedIds set
|
||||||
// retired: the re-verify now either confirms a genuinely edgeless anchor ('live', case (c))
|
// then drives executeGraphSearch's own verify, whose rebuild now heals → 'rebuilt' → re-collect.
|
||||||
// or — as here — discovers the adjacency itself is not serving, and throws.
|
const { brain, anchorId, targetIds } = await buildBrain({ anchorEdges: true })
|
||||||
const { brain, anchorId } = await buildBrain({ anchorEdges: true })
|
|
||||||
brains.push(brain)
|
brains.push(brain)
|
||||||
const state = instrumentIsReady(brain, { ready: false })
|
const state = instrumentIsReady(brain, { ready: false, healsOnRebuild: true, failFirstRebuild: true })
|
||||||
|
|
||||||
await expect(
|
const results = await brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
|
||||||
brain.find({ connected: { from: anchorId, direction: 'out' }, limit: 10 })
|
|
||||||
).rejects.toBeInstanceOf(GraphIndexNotReadyError)
|
expect(state.rebuildCalls).toBeGreaterThanOrEqual(2) // first transient, second heals
|
||||||
expect(state.rebuildCalls).toBe(0)
|
const ids = results.map((r: any) => r.id).sort()
|
||||||
|
expect(ids).toEqual(targetIds.sort()) // re-collected after the heal
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
|
||||||
|
|
@ -1,352 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/integration/health-gate
|
|
||||||
* @description Pins for the health-by-accounting read gate: the read gate stops
|
|
||||||
* consulting an unnamed `isReady()` boolean and reads a NAMED, sync, O(1)
|
|
||||||
* {@link HealthReport}; no read path may ever start a store walk; the open path
|
|
||||||
* brings every provider to serving before it returns; an explicit operator door
|
|
||||||
* (`repairIndex({ rebuild: [...] })`) rebuilds a named leg unconditionally.
|
|
||||||
*
|
|
||||||
* Providers here are white-box test doubles: a `healthReport()` (or, for the
|
|
||||||
* interim-path pins, an `isReady()`) function assigned directly onto the LIVE
|
|
||||||
* JS provider object, the same pattern `tests/unit/validate-invariants-delegation.test.ts`
|
|
||||||
* uses for `validateInvariants`. This exercises brainy's real gate/verify code
|
|
||||||
* against a controlled provider self-report — no engine mocks.
|
|
||||||
*/
|
|
||||||
import { describe, it, expect, afterEach, vi } from 'vitest'
|
|
||||||
import { mkdtempSync, rmSync } from 'node:fs'
|
|
||||||
import { tmpdir } from 'node:os'
|
|
||||||
import { join } from 'node:path'
|
|
||||||
import {
|
|
||||||
Brainy,
|
|
||||||
NounType,
|
|
||||||
VerbType,
|
|
||||||
GraphIndexNotReadyError,
|
|
||||||
MetadataIndexNotReadyError,
|
|
||||||
VectorIndexNotReadyError
|
|
||||||
} from '../../src/index.js'
|
|
||||||
import type { HealthReport, LedgerInvariantResult } from '../../src/plugin.js'
|
|
||||||
import { prodLog } from '../../src/utils/logger.js'
|
|
||||||
import { createTestConfig } from '../helpers/test-factory.js'
|
|
||||||
|
|
||||||
/** The white-box surface these pins drive on a live brain instance. */
|
|
||||||
interface BrainInternals {
|
|
||||||
storage: {
|
|
||||||
getNoun(id: string): Promise<unknown>
|
|
||||||
getNounMetadata(id: string): Promise<unknown>
|
|
||||||
getNouns(options?: unknown): Promise<unknown>
|
|
||||||
getVerbs(options?: unknown): Promise<unknown>
|
|
||||||
}
|
|
||||||
index: { healthReport?: () => HealthReport; isReady?: () => boolean; rebuild(): Promise<void> }
|
|
||||||
metadataIndex: {
|
|
||||||
healthReport?: () => HealthReport
|
|
||||||
isReady?: () => boolean
|
|
||||||
rebuild(): Promise<void>
|
|
||||||
validateInvariants?: () => Promise<unknown>
|
|
||||||
}
|
|
||||||
graphIndex: {
|
|
||||||
healthReport?: () => HealthReport
|
|
||||||
isReady?: () => boolean
|
|
||||||
rebuild(): Promise<void>
|
|
||||||
validateInvariants?: () => Promise<unknown>
|
|
||||||
}
|
|
||||||
rebuildIndexesIfNeeded(force?: boolean): Promise<void>
|
|
||||||
}
|
|
||||||
|
|
||||||
function internalsOf(brain: Brainy): BrainInternals {
|
|
||||||
return brain as unknown as BrainInternals
|
|
||||||
}
|
|
||||||
|
|
||||||
function invariant(overrides: Partial<LedgerInvariantResult> = {}): LedgerInvariantResult {
|
|
||||||
return {
|
|
||||||
name: 'manifest-residency',
|
|
||||||
holds: true,
|
|
||||||
detail: 'ok',
|
|
||||||
heal: 'none',
|
|
||||||
source: 'ledger',
|
|
||||||
...overrides
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function healthReport(overrides: Partial<HealthReport> = {}): HealthReport {
|
|
||||||
return {
|
|
||||||
provider: 'vector',
|
|
||||||
healthy: true,
|
|
||||||
serving: true,
|
|
||||||
invariants: [],
|
|
||||||
checkedAt: Date.now(),
|
|
||||||
durationMs: 1,
|
|
||||||
generation: 1,
|
|
||||||
unledgered: [],
|
|
||||||
...overrides
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const brains: Brainy[] = []
|
|
||||||
const dirs: string[] = []
|
|
||||||
afterEach(async () => {
|
|
||||||
for (const b of brains.splice(0)) await b.close().catch(() => {})
|
|
||||||
for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true })
|
|
||||||
vi.restoreAllMocks()
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('health gate (a) — not-serving refuses loudly, ZERO canonical reads during the refusal', () => {
|
|
||||||
it('metadata not-serving: find() throws MetadataIndexNotReadyError naming the failing invariant', async () => {
|
|
||||||
const brain = new Brainy(createTestConfig({ silent: true }))
|
|
||||||
await brain.init()
|
|
||||||
brains.push(brain)
|
|
||||||
await brain.add({ data: 'row', type: NounType.Document, metadata: { team: 'atlas' } })
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
const internals = internalsOf(brain)
|
|
||||||
internals.metadataIndex.healthReport = () =>
|
|
||||||
healthReport({
|
|
||||||
provider: 'metadata',
|
|
||||||
serving: false,
|
|
||||||
healthy: false,
|
|
||||||
invariants: [invariant({ name: 'posted-count-floor', holds: false, heal: 'rebuild', detail: 'posted 2 < canonical 5' })]
|
|
||||||
})
|
|
||||||
|
|
||||||
const getNounSpy = vi.spyOn(internals.storage, 'getNoun')
|
|
||||||
const getNounMetadataSpy = vi.spyOn(internals.storage, 'getNounMetadata')
|
|
||||||
const getNounsSpy = vi.spyOn(internals.storage, 'getNouns')
|
|
||||||
|
|
||||||
await expect(brain.find({ where: { team: 'atlas' } })).rejects.toBeInstanceOf(MetadataIndexNotReadyError)
|
|
||||||
await expect(brain.find({ where: { team: 'atlas' } })).rejects.toThrow(/posted-count-floor/)
|
|
||||||
|
|
||||||
expect(getNounSpy).not.toHaveBeenCalled()
|
|
||||||
expect(getNounMetadataSpy).not.toHaveBeenCalled()
|
|
||||||
expect(getNounsSpy).not.toHaveBeenCalled()
|
|
||||||
|
|
||||||
delete internals.metadataIndex.healthReport
|
|
||||||
})
|
|
||||||
|
|
||||||
it('graph not-serving: related() throws GraphIndexNotReadyError naming the failing invariant, no canonical reads', async () => {
|
|
||||||
const brain = new Brainy(createTestConfig({ silent: true }))
|
|
||||||
await brain.init()
|
|
||||||
brains.push(brain)
|
|
||||||
const a = await brain.add({ data: 'a', type: NounType.Person })
|
|
||||||
const b = await brain.add({ data: 'b', type: NounType.Person })
|
|
||||||
await brain.relate({ from: a, to: b, type: VerbType.Knows })
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
const internals = internalsOf(brain)
|
|
||||||
internals.graphIndex.healthReport = () =>
|
|
||||||
healthReport({
|
|
||||||
provider: 'graph',
|
|
||||||
serving: false,
|
|
||||||
healthy: false,
|
|
||||||
invariants: [invariant({ name: 'adjacency-residency', holds: false, heal: 'rebuild', detail: 'edges not loaded' })]
|
|
||||||
})
|
|
||||||
|
|
||||||
const getNounSpy = vi.spyOn(internals.storage, 'getNoun')
|
|
||||||
const getVerbsSpy = vi.spyOn(internals.storage, 'getVerbs')
|
|
||||||
|
|
||||||
await expect(brain.related({ from: a })).rejects.toBeInstanceOf(GraphIndexNotReadyError)
|
|
||||||
await expect(brain.related({ from: a })).rejects.toThrow(/adjacency-residency/)
|
|
||||||
|
|
||||||
expect(getNounSpy).not.toHaveBeenCalled()
|
|
||||||
expect(getVerbsSpy).not.toHaveBeenCalled()
|
|
||||||
|
|
||||||
delete internals.graphIndex.healthReport
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('health gate (b) — unledgered is unknown: never blocks a serving provider', () => {
|
|
||||||
it('serving:true with an unledgered family and no failing invariant serves normally; at most one narration', async () => {
|
|
||||||
const brain = new Brainy(createTestConfig({ silent: true }))
|
|
||||||
await brain.init()
|
|
||||||
brains.push(brain)
|
|
||||||
await brain.add({ data: 'row', type: NounType.Document, metadata: { team: 'atlas' } })
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
const internals = internalsOf(brain)
|
|
||||||
internals.metadataIndex.healthReport = () =>
|
|
||||||
healthReport({
|
|
||||||
provider: 'metadata',
|
|
||||||
serving: true,
|
|
||||||
healthy: true,
|
|
||||||
invariants: [],
|
|
||||||
unledgered: ['canonical-verb-coverage']
|
|
||||||
})
|
|
||||||
|
|
||||||
const warnSpy = vi.spyOn(prodLog, 'warn')
|
|
||||||
|
|
||||||
const r1 = await brain.find({ where: { team: 'atlas' } })
|
|
||||||
const r2 = await brain.find({ where: { team: 'atlas' } })
|
|
||||||
expect(r1.length).toBe(1)
|
|
||||||
expect(r2.length).toBe(1)
|
|
||||||
|
|
||||||
const narrations = warnSpy.mock.calls.filter(
|
|
||||||
([msg]) => typeof msg === 'string' && msg.includes('canonical-verb-coverage')
|
|
||||||
)
|
|
||||||
expect(narrations.length).toBe(1) // one narration at most across both reads (same generation)
|
|
||||||
|
|
||||||
delete internals.metadataIndex.healthReport
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('health gate (c) — degraded-but-serving narrates once per generation', () => {
|
|
||||||
it('a heal:"repair" failure serves; narrates once per generation, twice across a generation bump', async () => {
|
|
||||||
const brain = new Brainy(createTestConfig({ silent: true }))
|
|
||||||
await brain.init()
|
|
||||||
brains.push(brain)
|
|
||||||
await brain.add({ data: 'row', type: NounType.Document, metadata: { team: 'atlas' } })
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
const internals = internalsOf(brain)
|
|
||||||
let generation = 1
|
|
||||||
internals.index.healthReport = () =>
|
|
||||||
healthReport({
|
|
||||||
provider: 'vector',
|
|
||||||
serving: true,
|
|
||||||
healthy: false,
|
|
||||||
invariants: [invariant({ name: 'stale-vector-counter', holds: false, heal: 'repair', detail: 'counter drift' })],
|
|
||||||
generation
|
|
||||||
})
|
|
||||||
|
|
||||||
const warnSpy = vi.spyOn(prodLog, 'warn')
|
|
||||||
const countNarrations = () =>
|
|
||||||
warnSpy.mock.calls.filter(([msg]) => typeof msg === 'string' && msg.includes('stale-vector-counter')).length
|
|
||||||
|
|
||||||
await expect(brain.find({ where: { team: 'atlas' } })).resolves.toHaveLength(1)
|
|
||||||
await expect(brain.find({ where: { team: 'atlas' } })).resolves.toHaveLength(1)
|
|
||||||
expect(countNarrations()).toBe(1) // same generation both times — one narration
|
|
||||||
|
|
||||||
generation = 2
|
|
||||||
await expect(brain.find({ where: { team: 'atlas' } })).resolves.toHaveLength(1)
|
|
||||||
expect(countNarrations()).toBe(2) // generation bumped — a second narration
|
|
||||||
|
|
||||||
delete internals.index.healthReport
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('health gate (d) — interim isReady()-only path (no healthReport) is unchanged', () => {
|
|
||||||
it('isReady() === true serves; isReady() === false refuses via the typed NotReady error', async () => {
|
|
||||||
const brain = new Brainy(createTestConfig({ silent: true }))
|
|
||||||
await brain.init()
|
|
||||||
brains.push(brain)
|
|
||||||
await brain.add({ data: 'row', type: NounType.Document, metadata: { team: 'atlas' } })
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
const internals = internalsOf(brain)
|
|
||||||
internals.metadataIndex.isReady = () => true
|
|
||||||
await expect(brain.find({ where: { team: 'atlas' } })).resolves.toHaveLength(1)
|
|
||||||
|
|
||||||
internals.metadataIndex.isReady = () => false
|
|
||||||
await expect(brain.find({ where: { team: 'atlas' } })).rejects.toBeInstanceOf(MetadataIndexNotReadyError)
|
|
||||||
|
|
||||||
delete internals.metadataIndex.isReady
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('health gate (e) — open builds; the first read never does', () => {
|
|
||||||
it('disableAutoRebuild:true on a populated store: open narrates + builds; the first find() triggers zero rebuilds', async () => {
|
|
||||||
const dir = mkdtempSync(join(tmpdir(), 'brainy-healthgate-open-'))
|
|
||||||
dirs.push(dir)
|
|
||||||
|
|
||||||
const writer = new Brainy({
|
|
||||||
storage: { type: 'filesystem', path: dir },
|
|
||||||
requireSubtype: false,
|
|
||||||
silent: true,
|
|
||||||
disableAutoRebuild: true
|
|
||||||
})
|
|
||||||
await writer.init()
|
|
||||||
brains.push(writer)
|
|
||||||
await writer.add({ data: 'row one', type: NounType.Document, metadata: { team: 'atlas' } })
|
|
||||||
await writer.flush()
|
|
||||||
await brains.pop()!.close()
|
|
||||||
|
|
||||||
const warnSpy = vi.spyOn(prodLog, 'warn')
|
|
||||||
const reader = new Brainy({
|
|
||||||
storage: { type: 'filesystem', path: dir },
|
|
||||||
requireSubtype: false,
|
|
||||||
silent: true,
|
|
||||||
disableAutoRebuild: true
|
|
||||||
})
|
|
||||||
const internals = internalsOf(reader)
|
|
||||||
const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded')
|
|
||||||
|
|
||||||
await reader.init()
|
|
||||||
brains.push(reader)
|
|
||||||
|
|
||||||
expect(rebuildSpy).toHaveBeenCalledTimes(1) // open() built it, exactly once
|
|
||||||
expect(
|
|
||||||
warnSpy.mock.calls.some(
|
|
||||||
([msg]) => typeof msg === 'string' && msg.includes('open() is building')
|
|
||||||
)
|
|
||||||
).toBe(true)
|
|
||||||
|
|
||||||
rebuildSpy.mockClear()
|
|
||||||
const rows = await reader.find({ where: { team: 'atlas' } })
|
|
||||||
expect(rebuildSpy).toHaveBeenCalledTimes(0) // the read never builds
|
|
||||||
expect(rows.length).toBe(1)
|
|
||||||
}, 30000)
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('health gate (f) — the ceremony door: explicit rebuild bypasses invariant consultation', () => {
|
|
||||||
it("repairIndex({ rebuild: ['graph'] }) rebuilds unconditionally without consulting validateInvariants", async () => {
|
|
||||||
const brain = new Brainy(createTestConfig({ silent: true }))
|
|
||||||
await brain.init()
|
|
||||||
brains.push(brain)
|
|
||||||
await brain.add({ data: 'x', type: NounType.Concept })
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
const internals = internalsOf(brain)
|
|
||||||
let validateCalls = 0
|
|
||||||
internals.graphIndex.validateInvariants = async () => {
|
|
||||||
validateCalls++
|
|
||||||
return healthReport({ provider: 'graph' })
|
|
||||||
}
|
|
||||||
const rebuildSpy = vi.spyOn(internals.graphIndex, 'rebuild')
|
|
||||||
|
|
||||||
const report = await brain.repairIndex({ rebuild: ['graph'] })
|
|
||||||
|
|
||||||
expect(rebuildSpy).toHaveBeenCalledTimes(1)
|
|
||||||
expect(validateCalls).toBe(0) // the door never consults validateInvariants to decide
|
|
||||||
|
|
||||||
const graphFamily = report.families.find((f) => f.family === 'provider:graph')
|
|
||||||
expect(graphFamily?.rebuilt).toBe(true)
|
|
||||||
expect(graphFamily?.checked).toBe(true)
|
|
||||||
expect(graphFamily?.reason).toBe('explicit rebuild requested')
|
|
||||||
|
|
||||||
delete internals.graphIndex.validateInvariants
|
|
||||||
})
|
|
||||||
|
|
||||||
it('bare repairIndex() on a healthy provider calls no rebuild()', async () => {
|
|
||||||
const brain = new Brainy(createTestConfig({ silent: true }))
|
|
||||||
await brain.init()
|
|
||||||
brains.push(brain)
|
|
||||||
await brain.add({ data: 'x', type: NounType.Concept })
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
const internals = internalsOf(brain)
|
|
||||||
internals.graphIndex.validateInvariants = async () => healthReport({ provider: 'graph', healthy: true, serving: true })
|
|
||||||
const rebuildSpy = vi.spyOn(internals.graphIndex, 'rebuild')
|
|
||||||
|
|
||||||
await brain.repairIndex()
|
|
||||||
|
|
||||||
expect(rebuildSpy).not.toHaveBeenCalled()
|
|
||||||
|
|
||||||
delete internals.graphIndex.validateInvariants
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('health gate (g) — a throwing healthReport() is a contract violation, never read as healthy', () => {
|
|
||||||
it('healthReport() that throws refuses loudly with the typed NotReady error naming the throw', async () => {
|
|
||||||
const brain = new Brainy(createTestConfig({ silent: true }))
|
|
||||||
await brain.init()
|
|
||||||
brains.push(brain)
|
|
||||||
await brain.add({ data: 'row', type: NounType.Document, metadata: { team: 'atlas' } })
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
const internals = internalsOf(brain)
|
|
||||||
internals.index.healthReport = () => {
|
|
||||||
throw new Error('accelerator: mmap window busy')
|
|
||||||
}
|
|
||||||
|
|
||||||
await expect(brain.find({ where: { team: 'atlas' } })).rejects.toBeInstanceOf(VectorIndexNotReadyError)
|
|
||||||
await expect(brain.find({ where: { team: 'atlas' } })).rejects.toThrow(/mmap window busy/)
|
|
||||||
|
|
||||||
delete internals.index.healthReport
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
@ -1,167 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/integration/metadata-online-rebuild
|
|
||||||
* @description THE ONLINE JS METADATA REBUILD (B3 Deliverable 3) pins.
|
|
||||||
* `MetadataIndexManager.rebuild()` used to be clear-then-walk — reads went
|
|
||||||
* dark for the duration. `repairIndex({ rebuild: ['metadata'] })` now builds
|
|
||||||
* a fresh replacement index BESIDE the live one (walk canonical + mirror
|
|
||||||
* every live write via `beginShadow`/`endShadow` + a bounded fact-log fold),
|
|
||||||
* then atomically swaps the brain's reference — `find()` never observes a
|
|
||||||
* half-built index, and a write landing DURING the build is never lost.
|
|
||||||
*/
|
|
||||||
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
|
|
||||||
|
|
||||||
import { describe, it, expect, afterEach } from 'vitest'
|
|
||||||
import { mkdtempSync, rmSync } from 'node:fs'
|
|
||||||
import { tmpdir } from 'node:os'
|
|
||||||
import { join } from 'node:path'
|
|
||||||
import { Brainy } from '../../src/brainy.js'
|
|
||||||
import { NounType, VerbType } from '../../src/types/graphTypes.js'
|
|
||||||
import type { MetadataIndexManager } from '../../src/utils/metadataIndex.js'
|
|
||||||
|
|
||||||
const dirs: string[] = []
|
|
||||||
const brains: Brainy<any>[] = []
|
|
||||||
afterEach(async () => {
|
|
||||||
for (const b of brains.splice(0)) await b.close().catch(() => {})
|
|
||||||
for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true })
|
|
||||||
})
|
|
||||||
|
|
||||||
function metadataIndexOf(brain: Brainy<any>): MetadataIndexManager {
|
|
||||||
return (brain as unknown as { metadataIndex: MetadataIndexManager }).metadataIndex
|
|
||||||
}
|
|
||||||
|
|
||||||
async function openBrain(): Promise<{ brain: Brainy<any>; dir: string }> {
|
|
||||||
const dir = mkdtempSync(join(tmpdir(), 'brainy-online-rebuild-'))
|
|
||||||
dirs.push(dir)
|
|
||||||
const brain = new Brainy<any>({
|
|
||||||
requireSubtype: false,
|
|
||||||
storage: { type: 'filesystem', path: dir },
|
|
||||||
silent: true,
|
|
||||||
persistence: { policy: 'manual' },
|
|
||||||
logAuthority: 'adopt'
|
|
||||||
})
|
|
||||||
await brain.init()
|
|
||||||
brains.push(brain)
|
|
||||||
return { brain, dir }
|
|
||||||
}
|
|
||||||
|
|
||||||
describe('repairIndex({ rebuild: ["metadata"] }) — the online build-beside rebuild', () => {
|
|
||||||
it(
|
|
||||||
'a find() polled throughout the rebuild of a 2k-noun store never returns fewer rows than ' +
|
|
||||||
'before the build started, and a write landing DURING the build is never lost',
|
|
||||||
async () => {
|
|
||||||
const { brain, dir } = await openBrain()
|
|
||||||
void dir
|
|
||||||
|
|
||||||
const N = 2000
|
|
||||||
const ids: string[] = []
|
|
||||||
for (let i = 0; i < N; i++) {
|
|
||||||
ids.push(
|
|
||||||
await brain.add({
|
|
||||||
data: `entity ${i}`,
|
|
||||||
type: NounType.Person,
|
|
||||||
metadata: { status: i % 2 === 0 ? 'active' : 'inactive' }
|
|
||||||
})
|
|
||||||
)
|
|
||||||
}
|
|
||||||
for (let i = 0; i < 20; i++) {
|
|
||||||
await brain.relate({
|
|
||||||
from: ids[i], to: ids[i + 1], type: VerbType.WorksWith, metadata: { tag: 'orig' }
|
|
||||||
})
|
|
||||||
}
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
const baseline = await brain.find({ where: { status: 'active' }, limit: 10000 })
|
|
||||||
expect(baseline.length).toBe(N / 2)
|
|
||||||
|
|
||||||
// Kick off the online rebuild WITHOUT awaiting — poll reads and
|
|
||||||
// perform a live write concurrently with it.
|
|
||||||
const repairPromise = brain.repairIndex({ rebuild: ['metadata'] })
|
|
||||||
|
|
||||||
let minObserved = Infinity
|
|
||||||
let polls = 0
|
|
||||||
const pollPromise = (async () => {
|
|
||||||
// Poll until the rebuild settles — bounded so a slow CI box can't
|
|
||||||
// spin forever, generous enough to actually overlap the walk.
|
|
||||||
while (polls < 200) {
|
|
||||||
const rows = await brain.find({ where: { status: 'active' }, limit: 10000 })
|
|
||||||
minObserved = Math.min(minObserved, rows.length)
|
|
||||||
polls++
|
|
||||||
await new Promise((resolve) => setTimeout(resolve, 1))
|
|
||||||
}
|
|
||||||
})()
|
|
||||||
|
|
||||||
const newId = await brain.add({
|
|
||||||
data: 'added during the rebuild',
|
|
||||||
type: NounType.Person,
|
|
||||||
metadata: { status: 'active' }
|
|
||||||
})
|
|
||||||
const newRelId = await brain.relate({
|
|
||||||
from: newId, to: ids[0], type: VerbType.WorksWith, metadata: { tag: 'during-build' }
|
|
||||||
})
|
|
||||||
|
|
||||||
const [report] = await Promise.all([repairPromise, pollPromise])
|
|
||||||
|
|
||||||
// THE PIN: never fewer rows than the pre-build baseline, at any polled
|
|
||||||
// instant — reads served the OLD (fully-populated) manager throughout.
|
|
||||||
expect(polls).toBeGreaterThan(0)
|
|
||||||
expect(minObserved).toBeGreaterThanOrEqual(baseline.length)
|
|
||||||
|
|
||||||
// The repair report still accounts for the family (same receipt shape
|
|
||||||
// regardless of which rebuild mechanism actually ran underneath).
|
|
||||||
const metadataFamily = report.families.find((f) => f.family === 'provider:metadata')
|
|
||||||
expect(metadataFamily?.checked).toBe(true)
|
|
||||||
expect(metadataFamily?.rebuilt).toBe(true)
|
|
||||||
|
|
||||||
// Post-swap correctness: the live write during the build was never
|
|
||||||
// lost (the beginShadow mirror + post-walk fold caught it).
|
|
||||||
const afterActive = await brain.find({ where: { status: 'active' }, limit: 10000 })
|
|
||||||
expect(afterActive.length).toBe(baseline.length + 1)
|
|
||||||
expect(afterActive.some((r) => r.id === newId)).toBe(true)
|
|
||||||
|
|
||||||
const index = metadataIndexOf(brain)
|
|
||||||
expect(await index.getIds('tag', 'during-build')).toEqual([newRelId])
|
|
||||||
expect((await index.getIds('tag', 'orig')).length).toBe(20)
|
|
||||||
|
|
||||||
// The swap stamped the watermark — a reopen adopts, zero rebuild.
|
|
||||||
await brain.close()
|
|
||||||
brains.length = 0 // already closed above; afterEach must not double-close
|
|
||||||
const reopened = new Brainy<any>({
|
|
||||||
requireSubtype: false,
|
|
||||||
storage: { type: 'filesystem', path: dir },
|
|
||||||
silent: true,
|
|
||||||
persistence: { policy: 'manual' },
|
|
||||||
logAuthority: 'adopt'
|
|
||||||
})
|
|
||||||
await reopened.init()
|
|
||||||
brains.push(reopened)
|
|
||||||
const reopenedIndex = metadataIndexOf(reopened)
|
|
||||||
expect(reopenedIndex.watermarkVerdict()).toBe('adopt')
|
|
||||||
const reopenedActive = await reopened.find({ where: { status: 'active' }, limit: 10000 })
|
|
||||||
expect(reopenedActive.length).toBe(afterActive.length)
|
|
||||||
},
|
|
||||||
60000
|
|
||||||
)
|
|
||||||
|
|
||||||
it('repairIndex({ rebuild: ["metadata"] }) on an empty store is a trivial no-op walk', async () => {
|
|
||||||
const { brain } = await openBrain()
|
|
||||||
const report = await brain.repairIndex({ rebuild: ['metadata'] })
|
|
||||||
const metadataFamily = report.families.find((f) => f.family === 'provider:metadata')
|
|
||||||
expect(metadataFamily?.checked).toBe(true)
|
|
||||||
expect(await brain.getNounCount()).toBe(0)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('two consecutive online rebuilds both leave the index correct (idempotent)', async () => {
|
|
||||||
const { brain } = await openBrain()
|
|
||||||
const a = await brain.add({ data: 'a', type: NounType.Person, metadata: { status: 'active' } })
|
|
||||||
await brain.add({ data: 'b', type: NounType.Person, metadata: { status: 'inactive' } })
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
await brain.repairIndex({ rebuild: ['metadata'] })
|
|
||||||
const first = await brain.find({ where: { status: 'active' } })
|
|
||||||
expect(first.map((r) => r.id)).toEqual([a])
|
|
||||||
|
|
||||||
await brain.repairIndex({ rebuild: ['metadata'] })
|
|
||||||
const second = await brain.find({ where: { status: 'active' } })
|
|
||||||
expect(second.map((r) => r.id)).toEqual([a])
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
@ -34,7 +34,13 @@ describe('Read-After-Write Consistency (v5.7.2 Bug Fix)', () => {
|
||||||
testDir = join(tmpdir(), `brainy-consistency-${Date.now()}-${Math.random().toString(36).substring(7)}`)
|
testDir = join(tmpdir(), `brainy-consistency-${Date.now()}-${Math.random().toString(36).substring(7)}`)
|
||||||
|
|
||||||
brain = new Brainy({ requireSubtype: false,
|
brain = new Brainy({ requireSubtype: false,
|
||||||
storage: { type: 'filesystem', path: testDir },
|
storage: {
|
||||||
|
type: 'filesystem',
|
||||||
|
config: {
|
||||||
|
baseDir: testDir,
|
||||||
|
enableCompression: false // Faster tests
|
||||||
|
}
|
||||||
|
},
|
||||||
dimensions: 384
|
dimensions: 384
|
||||||
})
|
})
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -68,46 +68,4 @@ describe('repairIndex per-family receipt', () => {
|
||||||
expect(orphans!.healed, 'the ghost was pruned and receipted').toBeGreaterThan(0)
|
expect(orphans!.healed, 'the ghost was pruned and receipted').toBeGreaterThan(0)
|
||||||
expect(report.healedTotal).toBeGreaterThan(0)
|
expect(report.healedTotal).toBeGreaterThan(0)
|
||||||
}, 120000)
|
}, 120000)
|
||||||
|
|
||||||
|
|
||||||
it("a heal:'repair' verdict routes to the provider's own repair(), and the re-read decides", async () => {
|
|
||||||
// A fake provider report: one failing invariant asking for the INCREMENTAL
|
|
||||||
// heal. repairIndex must call repair() (never rebuild()) and count the heal
|
|
||||||
// only when the post-repair re-read clears the same verdict.
|
|
||||||
const dir = mkdtempSync(join(tmpdir(), 'brainy-repair-route-'))
|
|
||||||
const brain: any = new Brainy({ storage: { type: 'filesystem', path: dir }, requireSubtype: false, silent: true })
|
|
||||||
await brain.init()
|
|
||||||
brains.push(brain)
|
|
||||||
let repairCalls = 0
|
|
||||||
let rebuildCalls = 0
|
|
||||||
let healed = false
|
|
||||||
const failing = {
|
|
||||||
provider: 'vector', healthy: false, serving: true,
|
|
||||||
invariants: [{ name: 'node-coverage', holds: false, detail: 'short 3', heal: 'repair' as const }],
|
|
||||||
checkedAt: 1, durationMs: 1
|
|
||||||
}
|
|
||||||
const clean = {
|
|
||||||
provider: 'vector', healthy: true, serving: true,
|
|
||||||
invariants: [{ name: 'node-coverage', holds: true, detail: 'ok', heal: 'none' as const }],
|
|
||||||
checkedAt: 2, durationMs: 1
|
|
||||||
}
|
|
||||||
;(brain.index as any).validateInvariants = async () => (healed ? clean : failing)
|
|
||||||
;(brain.index as any).repair = async () => { repairCalls++; healed = true; return { repaired: 3 } }
|
|
||||||
const origRebuild = (brain.index as any).rebuild
|
|
||||||
;(brain.index as any).rebuild = async () => { rebuildCalls++ }
|
|
||||||
try {
|
|
||||||
const report = await brain.repairIndex()
|
|
||||||
const row = report.families.find((f: any) => f.family === 'provider:vector')
|
|
||||||
expect(row, 'the provider family is in the receipt').toBeDefined()
|
|
||||||
expect(repairCalls, 'repair() ran exactly once').toBe(1)
|
|
||||||
expect(rebuildCalls, "a heal:'repair' verdict never runs rebuild()").toBe(0)
|
|
||||||
expect(row!.healed, 'the cleared verdict counts as healed').toBe(1)
|
|
||||||
expect(String(row!.detail)).toMatch(/incremental repair cleared: node-coverage/)
|
|
||||||
} finally {
|
|
||||||
delete (brain.index as any).validateInvariants
|
|
||||||
delete (brain.index as any).repair
|
|
||||||
;(brain.index as any).rebuild = origRebuild
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
})
|
})
|
||||||
|
|
|
||||||
|
|
@ -1,190 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/integration/verb-metadata-rows
|
|
||||||
* @description THE LIVE VERB PATH pins. Before this train, verb rows entered
|
|
||||||
* the metadata index ONLY via `MetadataIndexManager.rebuild()`'s canonical
|
|
||||||
* walk — every relate()/unrelate()/updateRelation() call, and every
|
|
||||||
* remove()-cascaded relationship, left the metadata index blind to verb
|
|
||||||
* writes until the next rebuild. This file pins that `relate()`,
|
|
||||||
* `unrelate()`, `updateRelation()`, `remove()`'s cascade, and their
|
|
||||||
* `transact()` mirrors now post/retract the SAME verb rows a rebuild would
|
|
||||||
* derive from canonical (ADR-007 A4: one mechanism for add/update, live and
|
|
||||||
* rebuilt).
|
|
||||||
*/
|
|
||||||
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
|
|
||||||
|
|
||||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
|
||||||
import { Brainy } from '../../src/brainy.js'
|
|
||||||
import { NounType, VerbType } from '../../src/types/graphTypes.js'
|
|
||||||
import type { MetadataIndexManager } from '../../src/utils/metadataIndex.js'
|
|
||||||
|
|
||||||
/** The JS metadata-index manager backing a memory-storage brain in these
|
|
||||||
* tests (feature-detected in production code via `instanceof
|
|
||||||
* MetadataIndexManager`; a narrow test-only reach-in here, matching the
|
|
||||||
* existing idiom in tests/integration/find-where-zero.test.ts and
|
|
||||||
* tests/integration/level-field-shadow.test.ts). */
|
|
||||||
function metadataIndexOf(brain: Brainy<any>): MetadataIndexManager {
|
|
||||||
return (brain as unknown as { metadataIndex: MetadataIndexManager }).metadataIndex
|
|
||||||
}
|
|
||||||
|
|
||||||
describe('verb metadata rows — the live path matches the rebuild walk', () => {
|
|
||||||
let brain: Brainy<any>
|
|
||||||
|
|
||||||
beforeEach(async () => {
|
|
||||||
brain = new Brainy({ requireSubtype: false, storage: { type: 'memory' }, silent: true })
|
|
||||||
await brain.init()
|
|
||||||
})
|
|
||||||
|
|
||||||
afterEach(async () => {
|
|
||||||
await brain.close()
|
|
||||||
})
|
|
||||||
|
|
||||||
async function addPerson(label: string): Promise<string> {
|
|
||||||
return brain.add({
|
|
||||||
data: `person ${label}`,
|
|
||||||
type: NounType.Person,
|
|
||||||
metadata: { label }
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
it('(a) relate() posts a metadata-index-backed verb row a query can find', async () => {
|
|
||||||
const a = await addPerson('a')
|
|
||||||
const b = await addPerson('b')
|
|
||||||
const relId = await brain.relate({
|
|
||||||
from: a, to: b, type: VerbType.WorksWith, metadata: { role: 'lead' }
|
|
||||||
})
|
|
||||||
|
|
||||||
// Read it back the SAME way a rebuild-sourced row is queried — the
|
|
||||||
// manager's own posting lookup, keyed on the custom field the caller wrote.
|
|
||||||
const index = metadataIndexOf(brain)
|
|
||||||
expect(await index.getIds('role', 'lead')).toEqual([relId])
|
|
||||||
})
|
|
||||||
|
|
||||||
it('(b) unrelate() retracts the row', async () => {
|
|
||||||
const a = await addPerson('a')
|
|
||||||
const b = await addPerson('b')
|
|
||||||
const relId = await brain.relate({
|
|
||||||
from: a, to: b, type: VerbType.WorksWith, metadata: { role: 'lead' }
|
|
||||||
})
|
|
||||||
|
|
||||||
const index = metadataIndexOf(brain)
|
|
||||||
expect(await index.getIds('role', 'lead')).toEqual([relId])
|
|
||||||
|
|
||||||
// Flush BEFORE retracting the field's only occurrence: this durably
|
|
||||||
// persists the 'role' column (a segment on disk/in the store), so the
|
|
||||||
// post-retraction query below reads "this field exists, zero live
|
|
||||||
// postings" (→ []) rather than "this field has never been written"
|
|
||||||
// (→ FIELD_NOT_INDEXED) — an orthogonal column-store characteristic
|
|
||||||
// (an unflushed field with its last live posting removed reverts to
|
|
||||||
// unknown), not a D2 behavior.
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
await brain.unrelate(relId)
|
|
||||||
|
|
||||||
expect(await index.getIds('role', 'lead')).toEqual([])
|
|
||||||
})
|
|
||||||
|
|
||||||
it('(c) updateRelation({ metadata }) leaves exactly the new values', async () => {
|
|
||||||
const a = await addPerson('a')
|
|
||||||
const b = await addPerson('b')
|
|
||||||
const relId = await brain.relate({
|
|
||||||
from: a, to: b, type: VerbType.WorksWith, metadata: { role: 'lead', team: 'core' }
|
|
||||||
})
|
|
||||||
|
|
||||||
const index = metadataIndexOf(brain)
|
|
||||||
expect(await index.getIds('role', 'lead')).toEqual([relId])
|
|
||||||
|
|
||||||
// Flush first — see (b)'s note: 'role'/'team' must be durably known
|
|
||||||
// fields before their only value is retracted, or the post-update
|
|
||||||
// "gone" checks below throw FIELD_NOT_INDEXED instead of returning [].
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
await brain.updateRelation({ id: relId, metadata: { role: 'reviewer' }, merge: false })
|
|
||||||
|
|
||||||
// Stale values gone (the old shape AND the merge:false-dropped field)…
|
|
||||||
expect(await index.getIds('role', 'lead')).toEqual([])
|
|
||||||
expect(await index.getIds('team', 'core')).toEqual([])
|
|
||||||
// …only the new value serves.
|
|
||||||
expect(await index.getIds('role', 'reviewer')).toEqual([relId])
|
|
||||||
})
|
|
||||||
|
|
||||||
it("(d) remove(entity) cascade retracts every incident relation's metadata row", async () => {
|
|
||||||
const a = await addPerson('a')
|
|
||||||
const b = await addPerson('b')
|
|
||||||
const c = await addPerson('c')
|
|
||||||
const rel1 = await brain.relate({
|
|
||||||
from: a, to: b, type: VerbType.WorksWith, metadata: { tag: 'cascade-test' }
|
|
||||||
})
|
|
||||||
const rel2 = await brain.relate({
|
|
||||||
from: c, to: a, type: VerbType.WorksWith, metadata: { tag: 'cascade-test' }
|
|
||||||
})
|
|
||||||
|
|
||||||
const index = metadataIndexOf(brain)
|
|
||||||
expect((await index.getIds('tag', 'cascade-test')).sort()).toEqual([rel1, rel2].sort())
|
|
||||||
|
|
||||||
// Flush first — see (b)'s note.
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
await brain.remove(a) // a is source of rel1, target of rel2 — both cascade
|
|
||||||
|
|
||||||
expect(await index.getIds('tag', 'cascade-test')).toEqual([])
|
|
||||||
})
|
|
||||||
|
|
||||||
it('(e) a rebuild() reproduces exactly the verb-row population the live path built', async () => {
|
|
||||||
const a = await addPerson('a')
|
|
||||||
const b = await addPerson('b')
|
|
||||||
const c = await addPerson('c')
|
|
||||||
await brain.relate({ from: a, to: b, type: VerbType.WorksWith, metadata: { tag: 'parity', label: 'ab' } })
|
|
||||||
await brain.relate({ from: b, to: c, type: VerbType.RelatedTo, metadata: { tag: 'parity', label: 'bc' } })
|
|
||||||
const relId3 = await brain.relate({
|
|
||||||
from: c, to: a, type: VerbType.WorksWith, metadata: { tag: 'parity', label: 'ca' }
|
|
||||||
})
|
|
||||||
await brain.unrelate(relId3) // exercise retraction too — the rebuild must NOT resurrect it
|
|
||||||
|
|
||||||
const index = metadataIndexOf(brain)
|
|
||||||
const beforeIds = (await index.getIds('tag', 'parity')).slice().sort()
|
|
||||||
expect(beforeIds.length).toBe(2)
|
|
||||||
const beforeAb = await index.getIds('label', 'ab')
|
|
||||||
const beforeBc = await index.getIds('label', 'bc')
|
|
||||||
|
|
||||||
await index.rebuild()
|
|
||||||
|
|
||||||
const afterIds = (await index.getIds('tag', 'parity')).slice().sort()
|
|
||||||
expect(afterIds).toEqual(beforeIds)
|
|
||||||
expect(await index.getIds('label', 'ab')).toEqual(beforeAb)
|
|
||||||
expect(await index.getIds('label', 'bc')).toEqual(beforeBc)
|
|
||||||
expect(await index.getIds('label', 'ca')).toEqual([]) // the unrelated edge stays gone
|
|
||||||
})
|
|
||||||
|
|
||||||
it('(f) transact() relate/unrelate posts/retracts the same metadata-index rows as single-op', async () => {
|
|
||||||
const a = await addPerson('a')
|
|
||||||
const b = await addPerson('b')
|
|
||||||
const c = await addPerson('c')
|
|
||||||
const d = await addPerson('d')
|
|
||||||
|
|
||||||
// Single-op baseline.
|
|
||||||
const singleOpId = await brain.relate({
|
|
||||||
from: a, to: b, type: VerbType.WorksWith, metadata: { tag: 'parity-f' }
|
|
||||||
})
|
|
||||||
|
|
||||||
// transact() mirror.
|
|
||||||
const relateDb = await brain.transact([
|
|
||||||
{ op: 'relate', from: c, to: d, type: VerbType.WorksWith, metadata: { tag: 'parity-f' } }
|
|
||||||
])
|
|
||||||
const transactId = relateDb.receipt!.ids[0]
|
|
||||||
await relateDb.release()
|
|
||||||
|
|
||||||
const index = metadataIndexOf(brain)
|
|
||||||
expect((await index.getIds('tag', 'parity-f')).sort()).toEqual([singleOpId, transactId].sort())
|
|
||||||
|
|
||||||
// Flush first — see (b)'s note: 'tag' must be durably known before its
|
|
||||||
// last live posting is retracted below.
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
// Retract both ways — single-op unrelate() and transact() unrelate.
|
|
||||||
await brain.unrelate(singleOpId)
|
|
||||||
const unrelateDb = await brain.transact([{ op: 'unrelate', id: transactId }])
|
|
||||||
await unrelateDb.release()
|
|
||||||
|
|
||||||
expect(await index.getIds('tag', 'parity-f')).toEqual([])
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
@ -1,22 +0,0 @@
|
||||||
# The Lifecycle Lane
|
|
||||||
|
|
||||||
One brain, driven through founding, a working day, a clean restart, a
|
|
||||||
crash, a repair, and a second life, checked chapter by chapter against an
|
|
||||||
independent shadow-model referee (`biographyHarness.ts`). It catches
|
|
||||||
COMPOSITION regressions unit tests miss — a store fine in one process but
|
|
||||||
broken across a restart/crash/repair. Runs on the plain JS engine, so it
|
|
||||||
gates every commit.
|
|
||||||
|
|
||||||
Run it: `npx vitest run tests/lifecycle --pool=forks`
|
|
||||||
|
|
||||||
A red names the chapter label, the id, and expected-vs-actual — diagnosable
|
|
||||||
from the message alone. `biography.test.ts` is split into two `it` blocks
|
|
||||||
(Ch1-3, then Ch4-6) purely for reporting; it is still ONE fixed-order story.
|
|
||||||
Chapters must never be reordered, skipped, or made conditional, and a
|
|
||||||
failing chapter's assertion must never be weakened to force green.
|
|
||||||
|
|
||||||
Lab notes (hard-won, keep):
|
|
||||||
- `git reset --hard` does NOT remove untracked files — a "clean" tree can still
|
|
||||||
carry stray test stores; use `git clean -fd tests/lifecycle-tmp` equivalents.
|
|
||||||
- `silent: true` patches `console` process-wide — never assert narration through
|
|
||||||
`console` spies in this lane; the engine's always-on channel is `prodLog`.
|
|
||||||
|
|
@ -1,404 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/lifecycle/biography
|
|
||||||
* @description THE LIFECYCLE LANE — see `tests/lifecycle/README.md` for what
|
|
||||||
* this proves and how to run it. One scenario, "the working store": a single
|
|
||||||
* brain driven through founding, a working day, a clean restart, a crash, a
|
|
||||||
* repair, and a second life, verified chapter by chapter against an
|
|
||||||
* independent shadow-model referee (`biographyHarness.ts`).
|
|
||||||
*
|
|
||||||
* Split into two `it` blocks so a currently-failing later chapter (see the
|
|
||||||
* second block's header comment — a live engine finding, not a defect in
|
|
||||||
* this lane) never hides the earlier chapters' passing coverage. The two
|
|
||||||
* blocks share one brain's directory and one shadow model, run in the SAME
|
|
||||||
* fixed order the single scenario always has (`describe.sequential` below
|
|
||||||
* exists to say so explicitly, though vitest's own default is sequential
|
|
||||||
* within a file) — this is a split for REPORTING clarity, not a reordering
|
|
||||||
* or conditional skip of any chapter.
|
|
||||||
*/
|
|
||||||
import { describe, it, expect } from 'vitest'
|
|
||||||
import * as fs from 'node:fs'
|
|
||||||
import { NounType, VerbType } from '../../src/types/graphTypes.js'
|
|
||||||
import type { Brainy } from '../../src/brainy.js'
|
|
||||||
import type { AddParams, RelateParams, UpdateParams, UpdateRelationParams } from '../../src/index.js'
|
|
||||||
import { abandonAsCrashed, makeTempDir, openBrain, uid } from '../helpers/durabilityKillMatrix.js'
|
|
||||||
import {
|
|
||||||
createModel,
|
|
||||||
getCanonicalCountsFor,
|
|
||||||
modelAdd,
|
|
||||||
modelDelete,
|
|
||||||
modelRelate,
|
|
||||||
modelUpdate,
|
|
||||||
modelUpdateRelation,
|
|
||||||
recordVfsFileWrite,
|
|
||||||
snapshotVfsBaseline,
|
|
||||||
verifyChapter,
|
|
||||||
type HubCheck,
|
|
||||||
type ShadowModel
|
|
||||||
} from './biographyHarness.js'
|
|
||||||
|
|
||||||
const STATUSES = ['active', 'pending', 'closed', 'archived'] as const
|
|
||||||
|
|
||||||
/** Cycle a status value to the next one in the fixed rotation — used so
|
|
||||||
* Ch2's 40 updates provably MOVE entities across find() buckets rather than
|
|
||||||
* risking a no-op reassignment of the same value. */
|
|
||||||
function nextStatus(current: unknown): (typeof STATUSES)[number] {
|
|
||||||
const currentStr = typeof current === 'string' ? current : STATUSES[0]
|
|
||||||
const idx = STATUSES.indexOf(currentStr as (typeof STATUSES)[number])
|
|
||||||
return STATUSES[(idx < 0 ? 0 : idx + 1) % STATUSES.length]
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
// Shared biography state — set up by the first `it`, consumed by the second.
|
|
||||||
// The two blocks are one continuous story told in two named pieces; nothing
|
|
||||||
// here resets or diverges between them.
|
|
||||||
// ---------------------------------------------------------------------------
|
|
||||||
let dir: string
|
|
||||||
let model: ShadowModel
|
|
||||||
let brain: Brainy
|
|
||||||
let hubs: HubCheck[]
|
|
||||||
let employees: string[]
|
|
||||||
let customers: string[]
|
|
||||||
let invoices: string[]
|
|
||||||
let tasks: string[]
|
|
||||||
let projects: string[]
|
|
||||||
let nonHub: string[]
|
|
||||||
|
|
||||||
// ---- Wrappers: every call to the real brain updates the shadow model in
|
|
||||||
// the same statement, so the two can never drift apart by construction.
|
|
||||||
// Defined once, closing over the `let` bindings above so both `it` blocks
|
|
||||||
// (and any future reopen inside them) operate on the current brain/model.
|
|
||||||
async function doAdd(label: string, params: Omit<AddParams, 'id'>): Promise<string> {
|
|
||||||
const id = uid(label)
|
|
||||||
await brain.add({ ...params, id })
|
|
||||||
modelAdd(model, id, {
|
|
||||||
type: params.type,
|
|
||||||
subtype: params.subtype,
|
|
||||||
metadata: params.metadata ?? {},
|
|
||||||
visibility: params.visibility
|
|
||||||
})
|
|
||||||
return id
|
|
||||||
}
|
|
||||||
|
|
||||||
async function doUpdate(id: string, patch: Omit<UpdateParams, 'id'>): Promise<void> {
|
|
||||||
await brain.update({ ...patch, id })
|
|
||||||
modelUpdate(model, id, { metadata: patch.metadata, merge: patch.merge, visibility: patch.visibility })
|
|
||||||
}
|
|
||||||
|
|
||||||
async function doRemove(id: string): Promise<void> {
|
|
||||||
await brain.remove(id)
|
|
||||||
modelDelete(model, id)
|
|
||||||
}
|
|
||||||
|
|
||||||
async function doRelate(params: RelateParams): Promise<string> {
|
|
||||||
const id = await brain.relate(params)
|
|
||||||
modelRelate(model, id, {
|
|
||||||
from: params.from,
|
|
||||||
to: params.to,
|
|
||||||
type: params.type,
|
|
||||||
subtype: params.subtype,
|
|
||||||
metadata: params.metadata
|
|
||||||
})
|
|
||||||
return id
|
|
||||||
}
|
|
||||||
|
|
||||||
async function doUpdateRelation(id: string, patch: Omit<UpdateRelationParams, 'id'>): Promise<void> {
|
|
||||||
await brain.updateRelation({ ...patch, id })
|
|
||||||
modelUpdateRelation(model, id, { metadata: patch.metadata, merge: patch.merge })
|
|
||||||
}
|
|
||||||
|
|
||||||
async function doVfsWrite(path: string, content: string): Promise<void> {
|
|
||||||
await brain.vfs.writeFile(path, content)
|
|
||||||
recordVfsFileWrite(model)
|
|
||||||
}
|
|
||||||
|
|
||||||
describe.sequential('lifecycle — the working store', () => {
|
|
||||||
it(
|
|
||||||
'Ch1 FOUNDING -> Ch2 A WORKING DAY -> Ch3 CLEAN RESTART: every read serves truth',
|
|
||||||
async () => {
|
|
||||||
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
|
|
||||||
dir = makeTempDir()
|
|
||||||
model = createModel()
|
|
||||||
|
|
||||||
// logAuthority: 'adopt' from the first open, mirrored across every
|
|
||||||
// reopen — see write-flow-production-shape.test.ts, which the later
|
|
||||||
// crash chapter's at-ack law is pinned against.
|
|
||||||
brain = await openBrain(dir, { logAuthority: 'adopt' })
|
|
||||||
|
|
||||||
// =================================================================
|
|
||||||
// CHAPTER 1 — FOUNDING
|
|
||||||
// =================================================================
|
|
||||||
// Baseline MUST be snapshotted before any biography act — it is the
|
|
||||||
// VFS root's own system-tier footprint, measured, never hardcoded.
|
|
||||||
await snapshotVfsBaseline(brain, model)
|
|
||||||
|
|
||||||
employees = []
|
|
||||||
for (let i = 0; i < 20; i++) {
|
|
||||||
employees.push(
|
|
||||||
await doAdd(`emp-${i}`, {
|
|
||||||
data: `employee record ${i}`,
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'employee',
|
|
||||||
metadata: { status: STATUSES[i % STATUSES.length], department: ['engineering', 'sales', 'support'][i % 3] }
|
|
||||||
})
|
|
||||||
)
|
|
||||||
}
|
|
||||||
customers = []
|
|
||||||
for (let i = 0; i < 20; i++) {
|
|
||||||
customers.push(
|
|
||||||
await doAdd(`cust-${i}`, {
|
|
||||||
data: `customer record ${i}`,
|
|
||||||
type: NounType.Person,
|
|
||||||
subtype: 'customer',
|
|
||||||
metadata: { status: STATUSES[i % STATUSES.length], tier: i % 2 === 0 ? 'gold' : 'standard' }
|
|
||||||
})
|
|
||||||
)
|
|
||||||
}
|
|
||||||
invoices = []
|
|
||||||
for (let i = 0; i < 30; i++) {
|
|
||||||
invoices.push(
|
|
||||||
await doAdd(`inv-${i}`, {
|
|
||||||
data: `invoice record ${i}`,
|
|
||||||
type: NounType.Document,
|
|
||||||
subtype: 'invoice',
|
|
||||||
metadata: { status: STATUSES[i % STATUSES.length], amount: 100 + i * 17 }
|
|
||||||
})
|
|
||||||
)
|
|
||||||
}
|
|
||||||
tasks = []
|
|
||||||
for (let i = 0; i < 25; i++) {
|
|
||||||
tasks.push(
|
|
||||||
await doAdd(`task-${i}`, {
|
|
||||||
data: `task record ${i}`,
|
|
||||||
type: NounType.Task,
|
|
||||||
subtype: 'milestone',
|
|
||||||
metadata: { status: STATUSES[i % STATUSES.length], priority: (i % 5) + 1 }
|
|
||||||
})
|
|
||||||
)
|
|
||||||
}
|
|
||||||
projects = []
|
|
||||||
for (let i = 0; i < 25; i++) {
|
|
||||||
projects.push(
|
|
||||||
await doAdd(`proj-${i}`, {
|
|
||||||
data: `project record ${i}`,
|
|
||||||
type: NounType.Project,
|
|
||||||
metadata: { status: STATUSES[i % STATUSES.length], budget: 1000 * (i + 1) }
|
|
||||||
})
|
|
||||||
)
|
|
||||||
}
|
|
||||||
expect(employees.length + customers.length + invoices.length + tasks.length + projects.length).toBe(120)
|
|
||||||
|
|
||||||
// Five hubs (proj-0..proj-4) fan out to tasks (Contains) and employees
|
|
||||||
// (WorksWith); a residual band of invoice->customer RelatedTo edges is
|
|
||||||
// unrelated to any hub. Hubs are never touched again for the rest of
|
|
||||||
// the biography, so they stay valid adjacency samples in every chapter.
|
|
||||||
const hubIds = projects.slice(0, 5)
|
|
||||||
for (let h = 0; h < 5; h++) {
|
|
||||||
for (let k = 0; k < 15; k++) {
|
|
||||||
const taskIdx = (h * 5 + k) % tasks.length
|
|
||||||
await doRelate({ from: hubIds[h], to: tasks[taskIdx], type: VerbType.Contains, subtype: 'delivers' })
|
|
||||||
}
|
|
||||||
for (let k = 0; k < 10; k++) {
|
|
||||||
const empIdx = (h * 4 + k) % employees.length
|
|
||||||
await doRelate({ from: hubIds[h], to: employees[empIdx], type: VerbType.WorksWith })
|
|
||||||
}
|
|
||||||
}
|
|
||||||
for (let j = 0; j < 25; j++) {
|
|
||||||
await doRelate({ from: invoices[j], to: customers[j % customers.length], type: VerbType.RelatedTo, subtype: 'billed-to' })
|
|
||||||
}
|
|
||||||
expect(model.relations.size).toBe(150)
|
|
||||||
|
|
||||||
// A handful of VFS files.
|
|
||||||
for (let i = 0; i < 5; i++) {
|
|
||||||
await doVfsWrite(`/report-${i}.txt`, `founding report ${i}`)
|
|
||||||
}
|
|
||||||
|
|
||||||
await brain.flush()
|
|
||||||
|
|
||||||
hubs = hubIds.map((id) => ({ id, typeFilters: [VerbType.Contains, VerbType.WorksWith] }))
|
|
||||||
await verifyChapter(brain, model, 'Ch1 FOUNDING', { hubs, bucketField: 'status' })
|
|
||||||
|
|
||||||
// =================================================================
|
|
||||||
// CHAPTER 2 — A WORKING DAY
|
|
||||||
// =================================================================
|
|
||||||
// Non-hub pool for every mutation below.
|
|
||||||
nonHub = [...employees, ...customers, ...invoices, ...tasks, ...projects.slice(5)]
|
|
||||||
|
|
||||||
// 40 updates that provably MOVE entities across find() status buckets.
|
|
||||||
const updateTargets = nonHub.slice(0, 40)
|
|
||||||
for (const id of updateTargets) {
|
|
||||||
const current = model.entities.get(id)!.metadata.status
|
|
||||||
await doUpdate(id, { metadata: { status: nextStatus(current) } })
|
|
||||||
}
|
|
||||||
|
|
||||||
// 10 visibility flips (public -> internal).
|
|
||||||
const visibilityTargets = nonHub.slice(40, 50)
|
|
||||||
for (const id of visibilityTargets) {
|
|
||||||
await doUpdate(id, { visibility: 'internal' })
|
|
||||||
}
|
|
||||||
|
|
||||||
// 15 deletes — some hub members (their edges cascade away), 3 of them
|
|
||||||
// earmarked for Ch6's resurrection.
|
|
||||||
const resurrectIds = [tasks[0], tasks[1], employees[0]]
|
|
||||||
const otherDeletes = [
|
|
||||||
tasks[2], tasks[3], tasks[4], tasks[5], tasks[6],
|
|
||||||
employees[1], employees[2], employees[3],
|
|
||||||
customers[0], customers[1], customers[2], customers[3]
|
|
||||||
]
|
|
||||||
const ch2DeleteTargets = [...resurrectIds, ...otherDeletes]
|
|
||||||
expect(ch2DeleteTargets.length).toBe(15)
|
|
||||||
for (const id of ch2DeleteTargets) {
|
|
||||||
await doRemove(id)
|
|
||||||
}
|
|
||||||
|
|
||||||
// 20 new adds.
|
|
||||||
const ch2NewTypes = [NounType.Person, NounType.Document, NounType.Task]
|
|
||||||
for (let i = 0; i < 20; i++) {
|
|
||||||
await doAdd(`ch2-new-${i}`, {
|
|
||||||
data: `working-day addition ${i}`,
|
|
||||||
type: ch2NewTypes[i % ch2NewTypes.length],
|
|
||||||
subtype: 'ad-hoc',
|
|
||||||
metadata: { status: STATUSES[i % STATUSES.length] }
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
// 10 updateRelation metadata patches — read AFTER the deletes above,
|
|
||||||
// so only relations the cascade left alive are ever targeted.
|
|
||||||
const survivingRelationIds = [...model.relations.keys()].slice(0, 10)
|
|
||||||
expect(survivingRelationIds.length).toBe(10)
|
|
||||||
for (const relId of survivingRelationIds) {
|
|
||||||
await doUpdateRelation(relId, { metadata: { reviewed: true } })
|
|
||||||
}
|
|
||||||
|
|
||||||
await brain.flush()
|
|
||||||
await verifyChapter(brain, model, 'Ch2 A WORKING DAY', { hubs, bucketField: 'status' })
|
|
||||||
|
|
||||||
// =================================================================
|
|
||||||
// CHAPTER 3 — CLEAN RESTART
|
|
||||||
// =================================================================
|
|
||||||
await brain.close()
|
|
||||||
brain = await openBrain(dir, { logAuthority: 'adopt' })
|
|
||||||
await verifyChapter(brain, model, 'Ch3 CLEAN RESTART', { hubs, bucketField: 'status' })
|
|
||||||
|
|
||||||
// Leave the brain closed and the directory intact for the next `it`
|
|
||||||
// (the biography continues there) — do NOT remove `dir` here.
|
|
||||||
await brain.close()
|
|
||||||
},
|
|
||||||
300000
|
|
||||||
)
|
|
||||||
|
|
||||||
it(
|
|
||||||
'Ch4 CRASH -> Ch5 REPAIR -> Ch6 SECOND LIFE: continues the Ch3 store',
|
|
||||||
async () => {
|
|
||||||
try {
|
|
||||||
brain = await openBrain(dir, { logAuthority: 'adopt' })
|
|
||||||
|
|
||||||
// ===============================================================
|
|
||||||
// CHAPTER 4 — CRASH
|
|
||||||
// ===============================================================
|
|
||||||
const ch4Types = [NounType.Person, NounType.Document, NounType.Task, NounType.Project]
|
|
||||||
for (let i = 0; i < 10; i++) {
|
|
||||||
await doAdd(`ch4-new-${i}`, {
|
|
||||||
data: `crash-window addition ${i}`,
|
|
||||||
type: ch4Types[i % ch4Types.length],
|
|
||||||
metadata: { status: STATUSES[i % STATUSES.length] }
|
|
||||||
})
|
|
||||||
}
|
|
||||||
const ch4UpdateTargets = nonHub.slice(50, 55) // invoices[10..14] — untouched so far
|
|
||||||
for (const id of ch4UpdateTargets) {
|
|
||||||
await doUpdate(id, { metadata: { status: 'active' } })
|
|
||||||
}
|
|
||||||
// NO flush — abandon exactly the way process death would (the
|
|
||||||
// at-ack law: every write already awaited above must survive).
|
|
||||||
await abandonAsCrashed(brain)
|
|
||||||
brain = await openBrain(dir, { logAuthority: 'adopt' })
|
|
||||||
await verifyChapter(brain, model, 'Ch4 CRASH', { hubs, bucketField: 'status' })
|
|
||||||
|
|
||||||
// ===============================================================
|
|
||||||
// CHAPTER 5 — REPAIR
|
|
||||||
// ===============================================================
|
|
||||||
const report = await brain.repairIndex()
|
|
||||||
for (const family of report.families) {
|
|
||||||
const accounted =
|
|
||||||
family.checked === true || (family.checked === false && typeof family.skipped === 'string' && family.skipped.length > 0)
|
|
||||||
expect(
|
|
||||||
accounted,
|
|
||||||
`[Ch5 REPAIR] family '${family.family}' must be checked or explicitly skipped with a reason; got ${JSON.stringify(family)}`
|
|
||||||
).toBe(true)
|
|
||||||
}
|
|
||||||
// A healthy store: repair must change nothing the model doesn't
|
|
||||||
// already expect — verifyChapter against the UNCHANGED model proves it.
|
|
||||||
await verifyChapter(brain, model, 'Ch5 REPAIR', { hubs, bucketField: 'status' })
|
|
||||||
|
|
||||||
// ===============================================================
|
|
||||||
// CHAPTER 6 — SECOND LIFE
|
|
||||||
// ===============================================================
|
|
||||||
const ch6Types = [NounType.Person, NounType.Document, NounType.Task, NounType.Project]
|
|
||||||
for (let i = 0; i < 10; i++) {
|
|
||||||
await doAdd(`ch6-new-${i}`, {
|
|
||||||
data: `second-life addition ${i}`,
|
|
||||||
type: ch6Types[i % ch6Types.length],
|
|
||||||
metadata: { status: STATUSES[i % STATUSES.length] }
|
|
||||||
})
|
|
||||||
}
|
|
||||||
const ch6UpdateTargets = nonHub.slice(55, 65) // invoices[15..24] — untouched so far
|
|
||||||
expect(ch6UpdateTargets.every((id) => model.entities.get(id)!.alive)).toBe(true)
|
|
||||||
for (const id of ch6UpdateTargets) {
|
|
||||||
await doUpdate(id, { metadata: { status: 'closed' } })
|
|
||||||
}
|
|
||||||
const ch6DeleteTargets = nonHub
|
|
||||||
.slice(65, 90) // invoices[25..29] + tasks[0..19] (some already dead — filtered below)
|
|
||||||
.filter((id) => model.entities.get(id)!.alive)
|
|
||||||
.slice(0, 7)
|
|
||||||
expect(ch6DeleteTargets.length).toBe(7)
|
|
||||||
for (const id of ch6DeleteTargets) {
|
|
||||||
await doRemove(id)
|
|
||||||
}
|
|
||||||
|
|
||||||
// Resurrection: the SAME three ids Ch2 deleted, reinserted with
|
|
||||||
// BRAND-NEW metadata — the model expects the new metadata only.
|
|
||||||
await doAdd('task-0', { data: 'resurrected task 0', type: NounType.Task, subtype: 'milestone', metadata: { status: 'active', resurrected: true } })
|
|
||||||
await doAdd('task-1', { data: 'resurrected task 1', type: NounType.Task, subtype: 'milestone', metadata: { status: 'pending', resurrected: true } })
|
|
||||||
await doAdd('emp-0', { data: 'resurrected employee 0', type: NounType.Person, subtype: 'employee', metadata: { status: 'active', resurrected: true } })
|
|
||||||
expect(tasks[0]).toBe(uid('task-0')) // same id as Ch1/Ch2 — the resurrection-adjacent shape
|
|
||||||
|
|
||||||
await brain.close()
|
|
||||||
brain = await openBrain(dir, { logAuthority: 'adopt' })
|
|
||||||
await verifyChapter(brain, model, 'Ch6 SECOND LIFE', { hubs, bucketField: 'status' })
|
|
||||||
|
|
||||||
// Final, standalone getCanonicalCounts() exactness check (beyond
|
|
||||||
// verifyChapter's own (f) leg) — the whole ledger, in one shot.
|
|
||||||
const finalCounts = await getCanonicalCountsFor(brain)
|
|
||||||
const aliveEntities = [...model.entities.values()].filter((e) => e.alive)
|
|
||||||
const alivePublicEntities = aliveEntities.filter((e) => (e.visibility ?? 'public') === 'public')
|
|
||||||
const aliveVerbs = model.relations.size
|
|
||||||
expect(finalCounts, 'final getCanonicalCounts() exactness — Ch6 SECOND LIFE').toEqual({
|
|
||||||
nouns: {
|
|
||||||
counted: alivePublicEntities.length + model.vfsFileNouns,
|
|
||||||
all: aliveEntities.length + model.vfsFileNouns + model.vfsBaselineNouns
|
|
||||||
},
|
|
||||||
verbs: {
|
|
||||||
counted: aliveVerbs + model.vfsContainsVerbs,
|
|
||||||
all: aliveVerbs + model.vfsContainsVerbs + model.vfsBaselineVerbs
|
|
||||||
},
|
|
||||||
suspect: false
|
|
||||||
})
|
|
||||||
} finally {
|
|
||||||
await brain.close().catch(() => {})
|
|
||||||
// Best-effort, retried: a still-draining background persistence
|
|
||||||
// write (e.g. count/index write-through) can race a single rmSync
|
|
||||||
// and leave a partial directory behind — retry a couple of times
|
|
||||||
// rather than let this temp dir leak.
|
|
||||||
for (let attempt = 0; attempt < 3; attempt++) {
|
|
||||||
try {
|
|
||||||
fs.rmSync(dir, { recursive: true, force: true })
|
|
||||||
if (!fs.existsSync(dir)) break
|
|
||||||
} catch {
|
|
||||||
// ignore and retry
|
|
||||||
}
|
|
||||||
await new Promise((resolve) => setTimeout(resolve, 100))
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
300000
|
|
||||||
)
|
|
||||||
})
|
|
||||||
|
|
@ -1,389 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/lifecycle/biographyHarness
|
|
||||||
* @description The referee for the LIFECYCLE LANE (see `biography.test.ts`):
|
|
||||||
* a plain in-memory SHADOW MODEL of a brain's contents, updated by every act
|
|
||||||
* the biography performs (add/update/remove/relate/updateRelation/vfs writes),
|
|
||||||
* plus `verifyChapter()`, which asserts the live brain agrees with the model
|
|
||||||
* after every chapter. No engine code runs inside the model — it is an
|
|
||||||
* independent ledger, not a mirror of the implementation under test.
|
|
||||||
*
|
|
||||||
* COUNT SEMANTICS this harness encodes (verified against the live engine,
|
|
||||||
* not assumed — see the module-level comments below for how each was
|
|
||||||
* confirmed):
|
|
||||||
*
|
|
||||||
* - `getNounCount()` / `getVerbCount()` count PUBLIC-tier alive records only
|
|
||||||
* (visibility absent or `'public'`) — `'internal'` and `'system'` are both
|
|
||||||
* excluded. `storage.getCanonicalCounts()` mirrors that same PUBLIC-only
|
|
||||||
* scalar as `counted`, and additionally reports `all` — every tier,
|
|
||||||
* unfiltered — as the coverage-ledger denominator (see
|
|
||||||
* tests/integration/canonical-count-ledger.test.ts).
|
|
||||||
* - `brain.vfs.writeFile()` for a brand-new file at a path directly under the
|
|
||||||
* VFS root creates exactly ONE new File noun plus ONE new `Contains` verb
|
|
||||||
* (root -> file), and BOTH are ordinary PUBLIC records (no visibility
|
|
||||||
* field is set) — so they count toward `getNounCount()`/`getVerbCount()`
|
|
||||||
* as well as the canonical `all` scalars. Only the VFS ROOT entity itself
|
|
||||||
* is `'system'`-tier (created once, at `init()`, before any biography
|
|
||||||
* chapter runs) — that lone record is the only hidden-tier footprint the
|
|
||||||
* model does not construct explicitly, so it is captured empirically via
|
|
||||||
* `snapshotVfsBaseline()` immediately after `init()` rather than hardcoded.
|
|
||||||
* - `related()` filters edges by the RELATION's own visibility tier, not by
|
|
||||||
* the visibility of the entities the edge connects — flipping an entity to
|
|
||||||
* `'internal'` does not hide its edges from `related()`. This lane never
|
|
||||||
* sets relation visibility, so every relation the model tracks is exactly
|
|
||||||
* as reachable as its presence in `model.relations` implies.
|
|
||||||
* - `remove()` cascades: every relation touching the removed entity (as
|
|
||||||
* `from` or `to`) is hard-deleted along with it. The model mirrors this by
|
|
||||||
* deleting the relation entirely from `model.relations` (no relation
|
|
||||||
* "alive" flag — presence in the map IS aliveness).
|
|
||||||
*/
|
|
||||||
import { expect } from 'vitest'
|
|
||||||
import type { Brainy } from '../../src/brainy.js'
|
|
||||||
import type { NounType, VerbType } from '../../src/types/graphTypes.js'
|
|
||||||
import type { EntityVisibility, StorageAdapter } from '../../src/coreTypes.js'
|
|
||||||
|
|
||||||
/**
|
|
||||||
* One entity's complete lifecycle-relevant state, as the biography's acts
|
|
||||||
* leave it. `alive: false` means the model believes the id has been removed
|
|
||||||
* — the entry is KEPT (never deleted from the map) so `verifyChapter` can
|
|
||||||
* assert the negative half of the contract: a dead id must read as `null`.
|
|
||||||
*/
|
|
||||||
export interface ShadowEntity {
|
|
||||||
type: NounType
|
|
||||||
subtype?: string
|
|
||||||
metadata: Record<string, unknown>
|
|
||||||
visibility?: EntityVisibility
|
|
||||||
alive: boolean
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* One relation's complete lifecycle-relevant state. There is no `alive`
|
|
||||||
* flag here — presence in {@link ShadowModel.relations} IS aliveness,
|
|
||||||
* mirroring the engine's hard delete of the canonical verb record on
|
|
||||||
* cascade (see the module header).
|
|
||||||
*/
|
|
||||||
export interface ShadowRelation {
|
|
||||||
from: string
|
|
||||||
to: string
|
|
||||||
type: VerbType
|
|
||||||
subtype?: string
|
|
||||||
metadata: Record<string, unknown>
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* The independent truth ledger the biography updates on every act it
|
|
||||||
* performs. `verifyChapter` checks the live brain against this — never the
|
|
||||||
* other way around.
|
|
||||||
*/
|
|
||||||
export interface ShadowModel {
|
|
||||||
entities: Map<string, ShadowEntity>
|
|
||||||
relations: Map<string, ShadowRelation>
|
|
||||||
/**
|
|
||||||
* `getCanonicalCounts()` nouns.all / verbs.all captured right after
|
|
||||||
* `init()`, before chapter 1 — the VFS root's own system-tier footprint.
|
|
||||||
* Set once via {@link snapshotVfsBaseline}; never hardcoded.
|
|
||||||
*/
|
|
||||||
vfsBaselineNouns: number
|
|
||||||
vfsBaselineVerbs: number
|
|
||||||
/**
|
|
||||||
* Public nouns/verbs created by `vfs.writeFile()` for a brand-new file at
|
|
||||||
* a flat top-level path: exactly one File noun + one Contains verb per
|
|
||||||
* call (see the module header). Bumped by {@link recordVfsFileWrite}.
|
|
||||||
*/
|
|
||||||
vfsFileNouns: number
|
|
||||||
vfsContainsVerbs: number
|
|
||||||
}
|
|
||||||
|
|
||||||
/** A fresh, empty shadow model — call once before chapter 1. */
|
|
||||||
export function createModel(): ShadowModel {
|
|
||||||
return {
|
|
||||||
entities: new Map(),
|
|
||||||
relations: new Map(),
|
|
||||||
vfsBaselineNouns: 0,
|
|
||||||
vfsBaselineVerbs: 0,
|
|
||||||
vfsFileNouns: 0,
|
|
||||||
vfsContainsVerbs: 0
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Narrow, documented private-storage access (the same style already used by
|
|
||||||
* `tests/helpers/durabilityKillMatrix.ts`'s `storeOf()`), needed because
|
|
||||||
* `getCanonicalCounts()` lives on the storage adapter, not on `Brainy`. */
|
|
||||||
function storageOf(brain: Brainy): StorageAdapter {
|
|
||||||
return (brain as unknown as { storage: StorageAdapter }).storage
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Public wrapper around the private-storage `getCanonicalCounts()` read, so
|
|
||||||
* callers never need their own private-access cast — used internally by
|
|
||||||
* {@link snapshotVfsBaseline} and {@link verifyChapter}, and by
|
|
||||||
* `biography.test.ts` for its final standalone exactness check. */
|
|
||||||
export async function getCanonicalCountsFor(brain: Brainy): ReturnType<NonNullable<StorageAdapter['getCanonicalCounts']>> {
|
|
||||||
const storage = storageOf(brain)
|
|
||||||
if (!storage.getCanonicalCounts) {
|
|
||||||
throw new Error(
|
|
||||||
'lifecycle lane: the storage adapter under test has no getCanonicalCounts() — the canonical-count-exactness leg of this lane is unrepresentable without it.'
|
|
||||||
)
|
|
||||||
}
|
|
||||||
return storage.getCanonicalCounts()
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Snapshot the VFS root's own hidden-tier footprint. Call exactly once,
|
|
||||||
* immediately after `init()` and before chapter 1 does anything — this is
|
|
||||||
* the ONE baseline offset the model does not construct by hand (see the
|
|
||||||
* module header for why: the root is `'system'`-tier plumbing the biography
|
|
||||||
* never explicitly creates).
|
|
||||||
*/
|
|
||||||
export async function snapshotVfsBaseline(brain: Brainy, model: ShadowModel): Promise<void> {
|
|
||||||
const counts = await getCanonicalCountsFor(brain)
|
|
||||||
model.vfsBaselineNouns = counts.nouns.all
|
|
||||||
model.vfsBaselineVerbs = counts.verbs.all
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Record one `brain.vfs.writeFile()` call for a brand-new file at a flat
|
|
||||||
* top-level path (no intermediate directories). Bumps both the noun and verb
|
|
||||||
* VFS counters by one, matching the engine's actual write path exactly (see
|
|
||||||
* the module header) — never call this for an overwrite of an existing path,
|
|
||||||
* a nested path (which would also vivify intermediate directory nouns/edges,
|
|
||||||
* a different, unmodeled shape), or the biography loses its exactness.
|
|
||||||
*/
|
|
||||||
export function recordVfsFileWrite(model: ShadowModel): void {
|
|
||||||
model.vfsFileNouns += 1
|
|
||||||
model.vfsContainsVerbs += 1
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Record a fresh `add()` (or a Ch6 resurrection — `Map.set` fully replaces
|
|
||||||
* whatever a prior dead entry held, which is exactly the "new metadata only"
|
|
||||||
* contract a resurrection must honor). */
|
|
||||||
export function modelAdd(
|
|
||||||
model: ShadowModel,
|
|
||||||
id: string,
|
|
||||||
entity: { type: NounType; subtype?: string; metadata: Record<string, unknown>; visibility?: EntityVisibility }
|
|
||||||
): void {
|
|
||||||
model.entities.set(id, {
|
|
||||||
type: entity.type,
|
|
||||||
subtype: entity.subtype,
|
|
||||||
metadata: { ...entity.metadata },
|
|
||||||
visibility: entity.visibility,
|
|
||||||
alive: true
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Record an `update()` — merges metadata by default, matching the engine's
|
|
||||||
* `merge: true` default; pass `merge: false` to mirror a full replace. */
|
|
||||||
export function modelUpdate(
|
|
||||||
model: ShadowModel,
|
|
||||||
id: string,
|
|
||||||
patch: { metadata?: Record<string, unknown>; merge?: boolean; visibility?: EntityVisibility }
|
|
||||||
): void {
|
|
||||||
const existing = model.entities.get(id)
|
|
||||||
if (!existing || !existing.alive) {
|
|
||||||
throw new Error(`shadow model: update() targeted ${id}, which the model does not have alive — biography sequencing bug`)
|
|
||||||
}
|
|
||||||
if (patch.metadata) {
|
|
||||||
existing.metadata = patch.merge === false ? { ...patch.metadata } : { ...existing.metadata, ...patch.metadata }
|
|
||||||
}
|
|
||||||
if (patch.visibility !== undefined) {
|
|
||||||
existing.visibility = patch.visibility
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Record a `remove()` — marks the entity dead (entry retained, per
|
|
||||||
* {@link ShadowEntity}) and cascades: every relation touching it, in either
|
|
||||||
* direction, is hard-deleted from the model too (matching the engine). */
|
|
||||||
export function modelDelete(model: ShadowModel, id: string): void {
|
|
||||||
const existing = model.entities.get(id)
|
|
||||||
if (!existing || !existing.alive) {
|
|
||||||
throw new Error(`shadow model: remove() targeted ${id}, which the model does not have alive — biography sequencing bug`)
|
|
||||||
}
|
|
||||||
existing.alive = false
|
|
||||||
for (const [relId, rel] of model.relations) {
|
|
||||||
if (rel.from === id || rel.to === id) model.relations.delete(relId)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Record a `relate()` — `id` is the relation id the real call returned. */
|
|
||||||
export function modelRelate(
|
|
||||||
model: ShadowModel,
|
|
||||||
id: string,
|
|
||||||
relation: { from: string; to: string; type: VerbType; subtype?: string; metadata?: Record<string, unknown> }
|
|
||||||
): void {
|
|
||||||
model.relations.set(id, {
|
|
||||||
from: relation.from,
|
|
||||||
to: relation.to,
|
|
||||||
type: relation.type,
|
|
||||||
subtype: relation.subtype,
|
|
||||||
metadata: { ...(relation.metadata ?? {}) }
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Record an `updateRelation()` metadata patch — merges by default. */
|
|
||||||
export function modelUpdateRelation(
|
|
||||||
model: ShadowModel,
|
|
||||||
id: string,
|
|
||||||
patch: { metadata?: Record<string, unknown>; merge?: boolean }
|
|
||||||
): void {
|
|
||||||
const existing = model.relations.get(id)
|
|
||||||
if (!existing) {
|
|
||||||
throw new Error(`shadow model: updateRelation() targeted ${id}, which the model does not have — biography sequencing bug`)
|
|
||||||
}
|
|
||||||
if (patch.metadata) {
|
|
||||||
existing.metadata = patch.merge === false ? { ...patch.metadata } : { ...existing.metadata, ...patch.metadata }
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Order-independent structural equality for plain JSON-shaped metadata. */
|
|
||||||
function deepEqual(a: unknown, b: unknown): boolean {
|
|
||||||
if (a === b) return true
|
|
||||||
if (typeof a !== typeof b) return false
|
|
||||||
if (a === null || b === null) return a === b
|
|
||||||
if (typeof a !== 'object') return false
|
|
||||||
const aKeys = Object.keys(a as Record<string, unknown>)
|
|
||||||
const bKeys = Object.keys(b as Record<string, unknown>)
|
|
||||||
if (aKeys.length !== bKeys.length) return false
|
|
||||||
for (const k of aKeys) {
|
|
||||||
if (!deepEqual((a as Record<string, unknown>)[k], (b as Record<string, unknown>)[k])) return false
|
|
||||||
}
|
|
||||||
return true
|
|
||||||
}
|
|
||||||
|
|
||||||
/** One hub entity to sample for the `related()` adjacency check, plus the
|
|
||||||
* verb type(s) it is known (by biography construction) to have OUT-edges
|
|
||||||
* of, so the type-filtered variant is exercised too. */
|
|
||||||
export interface HubCheck {
|
|
||||||
id: string
|
|
||||||
typeFilters: VerbType[]
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Options steering one `verifyChapter()` call. */
|
|
||||||
export interface VerifyOptions {
|
|
||||||
/** Hub entities to sample for the `related()` adjacency check. */
|
|
||||||
hubs: HubCheck[]
|
|
||||||
/** The metadata field `find()` bucket-checks against (a bare string field
|
|
||||||
* every alive entity may or may not carry — distinct values present among
|
|
||||||
* ALIVE model entities are discovered automatically each call, so a
|
|
||||||
* chapter that moves entities across buckets is re-checked exactly). */
|
|
||||||
bucketField: string
|
|
||||||
}
|
|
||||||
|
|
||||||
/**
|
|
||||||
* Assert the live brain agrees with the model, in full, after one chapter.
|
|
||||||
* Every failure message names the chapter `label`, the id (where
|
|
||||||
* applicable), and expected-vs-actual — a red here must be diagnosable from
|
|
||||||
* the assertion message alone, with no need to re-read this file.
|
|
||||||
*/
|
|
||||||
export async function verifyChapter(brain: Brainy, model: ShadowModel, label: string, opts: VerifyOptions): Promise<void> {
|
|
||||||
// (a) + (b): every alive entity reads back exactly as modeled; every dead
|
|
||||||
// entity reads as null.
|
|
||||||
for (const [id, entity] of model.entities) {
|
|
||||||
const live = await brain.get(id)
|
|
||||||
if (entity.alive) {
|
|
||||||
expect(live, `[${label}] alive entity ${id} (type=${entity.type}) must be readable via get(), got null`).not.toBeNull()
|
|
||||||
const e = live!
|
|
||||||
expect(e.type, `[${label}] entity ${id} .type mismatch: expected ${entity.type}, got ${e.type}`).toBe(entity.type)
|
|
||||||
expect(e.subtype, `[${label}] entity ${id} .subtype mismatch: expected ${JSON.stringify(entity.subtype)}, got ${JSON.stringify(e.subtype)}`).toBe(entity.subtype)
|
|
||||||
expect(
|
|
||||||
e.visibility,
|
|
||||||
`[${label}] entity ${id} .visibility mismatch: expected ${JSON.stringify(entity.visibility)}, got ${JSON.stringify(e.visibility)}`
|
|
||||||
).toBe(entity.visibility)
|
|
||||||
const metaMatches = deepEqual(e.metadata ?? {}, entity.metadata)
|
|
||||||
expect(
|
|
||||||
metaMatches,
|
|
||||||
`[${label}] entity ${id} .metadata mismatch: expected ${JSON.stringify(entity.metadata)}, got ${JSON.stringify(e.metadata)}`
|
|
||||||
).toBe(true)
|
|
||||||
} else {
|
|
||||||
expect(live, `[${label}] dead entity ${id} (type=${entity.type}) must read as null, got ${JSON.stringify(live)}`).toBeNull()
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// (c) find({ where: { <bucketField>: value } }) returns exactly the
|
|
||||||
// model's matching alive set, per distinct value currently present.
|
|
||||||
const bucketValues = new Set<string>()
|
|
||||||
for (const entity of model.entities.values()) {
|
|
||||||
if (!entity.alive) continue
|
|
||||||
const v = entity.metadata[opts.bucketField]
|
|
||||||
if (typeof v === 'string') bucketValues.add(v)
|
|
||||||
}
|
|
||||||
for (const value of bucketValues) {
|
|
||||||
const expectedIds = [...model.entities.entries()]
|
|
||||||
.filter(([, e]) => e.alive && e.metadata[opts.bucketField] === value)
|
|
||||||
.map(([id]) => id)
|
|
||||||
.sort()
|
|
||||||
const results = await brain.find({
|
|
||||||
where: { [opts.bucketField]: value } as Record<string, unknown>,
|
|
||||||
includeInternal: true,
|
|
||||||
limit: 100000
|
|
||||||
})
|
|
||||||
const actualIds = results.map((r) => r.id).sort()
|
|
||||||
expect(
|
|
||||||
actualIds,
|
|
||||||
`[${label}] find({ where: { ${opts.bucketField}: ${JSON.stringify(value)} } }) mismatch: expected ${expectedIds.length} ids ${JSON.stringify(expectedIds)}, got ${actualIds.length} ids ${JSON.stringify(actualIds)}`
|
|
||||||
).toEqual(expectedIds)
|
|
||||||
}
|
|
||||||
|
|
||||||
// (d) related(id) / related(id, { type }) for the hub sample matches the
|
|
||||||
// model's adjacency exactly (out-edges — related(id) is shorthand for
|
|
||||||
// { from: id }).
|
|
||||||
for (const hub of opts.hubs) {
|
|
||||||
const expectedAll = [...model.relations.entries()]
|
|
||||||
.filter(([, r]) => r.from === hub.id)
|
|
||||||
.map(([id]) => id)
|
|
||||||
.sort()
|
|
||||||
const liveAll = await brain.related({ from: hub.id, limit: 100000 })
|
|
||||||
const actualAllIds = liveAll.map((r) => r.id).sort()
|
|
||||||
expect(
|
|
||||||
actualAllIds,
|
|
||||||
`[${label}] related(${hub.id}) mismatch: expected ${expectedAll.length} ids ${JSON.stringify(expectedAll)}, got ${actualAllIds.length} ids ${JSON.stringify(actualAllIds)}`
|
|
||||||
).toEqual(expectedAll)
|
|
||||||
|
|
||||||
for (const typeFilter of hub.typeFilters) {
|
|
||||||
const expectedTyped = [...model.relations.entries()]
|
|
||||||
.filter(([, r]) => r.from === hub.id && r.type === typeFilter)
|
|
||||||
.map(([id]) => id)
|
|
||||||
.sort()
|
|
||||||
const liveTyped = await brain.related({ from: hub.id, type: typeFilter, limit: 100000 })
|
|
||||||
const actualTypedIds = liveTyped.map((r) => r.id).sort()
|
|
||||||
expect(
|
|
||||||
actualTypedIds,
|
|
||||||
`[${label}] related(${hub.id}, { type: '${typeFilter}' }) mismatch: expected ${expectedTyped.length} ids ${JSON.stringify(expectedTyped)}, got ${actualTypedIds.length} ids ${JSON.stringify(actualTypedIds)}`
|
|
||||||
).toEqual(expectedTyped)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// (e) getNounCount() / getVerbCount(): PUBLIC-tier alive records
|
|
||||||
// (visibility absent/'public'; 'internal' and 'system' both excluded — see
|
|
||||||
// the module header) plus the VFS's own public contributions.
|
|
||||||
const alivePublicNouns = [...model.entities.values()].filter((e) => e.alive && (e.visibility ?? 'public') === 'public').length
|
|
||||||
const aliveVerbs = model.relations.size
|
|
||||||
const expectedNounCount = alivePublicNouns + model.vfsFileNouns
|
|
||||||
const expectedVerbCount = aliveVerbs + model.vfsContainsVerbs
|
|
||||||
expect(
|
|
||||||
await brain.getNounCount(),
|
|
||||||
`[${label}] getNounCount() mismatch: expected ${expectedNounCount} (alive public entities ${alivePublicNouns} + vfs file nouns ${model.vfsFileNouns})`
|
|
||||||
).toBe(expectedNounCount)
|
|
||||||
expect(
|
|
||||||
await brain.getVerbCount(),
|
|
||||||
`[${label}] getVerbCount() mismatch: expected ${expectedVerbCount} (alive relations ${aliveVerbs} + vfs contains verbs ${model.vfsContainsVerbs})`
|
|
||||||
).toBe(expectedVerbCount)
|
|
||||||
|
|
||||||
// (f) getCanonicalCounts(): ALL-visibility scalars (every tier) equal the
|
|
||||||
// model's alive totals including hidden tiers, plus the VFS's own
|
|
||||||
// contributions (both file nouns/verbs AND the once-measured root
|
|
||||||
// baseline). suspect must be false — every delete in this biography goes
|
|
||||||
// through brain.remove(), which always proves the record it decrements.
|
|
||||||
const ledger = await getCanonicalCountsFor(brain)
|
|
||||||
const aliveAllNouns = [...model.entities.values()].filter((e) => e.alive).length
|
|
||||||
const expectedNounsAll = aliveAllNouns + model.vfsFileNouns + model.vfsBaselineNouns
|
|
||||||
const expectedVerbsAll = aliveVerbs + model.vfsContainsVerbs + model.vfsBaselineVerbs
|
|
||||||
expect(
|
|
||||||
ledger.nouns.all,
|
|
||||||
`[${label}] getCanonicalCounts().nouns.all mismatch: expected ${expectedNounsAll} (alive incl. internal ${aliveAllNouns} + vfs file nouns ${model.vfsFileNouns} + vfs root baseline ${model.vfsBaselineNouns})`
|
|
||||||
).toBe(expectedNounsAll)
|
|
||||||
expect(
|
|
||||||
ledger.verbs.all,
|
|
||||||
`[${label}] getCanonicalCounts().verbs.all mismatch: expected ${expectedVerbsAll} (alive relations ${aliveVerbs} + vfs contains verbs ${model.vfsContainsVerbs} + vfs root baseline ${model.vfsBaselineVerbs})`
|
|
||||||
).toBe(expectedVerbsAll)
|
|
||||||
expect(ledger.nouns.counted, `[${label}] getCanonicalCounts().nouns.counted mismatch (should mirror getNounCount())`).toBe(expectedNounCount)
|
|
||||||
expect(ledger.verbs.counted, `[${label}] getCanonicalCounts().verbs.counted mismatch (should mirror getVerbCount())`).toBe(expectedVerbCount)
|
|
||||||
expect(ledger.suspect, `[${label}] getCanonicalCounts().suspect must be false — every delete in this biography proves its record`).toBe(false)
|
|
||||||
}
|
|
||||||
|
|
@ -1,49 +1,38 @@
|
||||||
/**
|
/**
|
||||||
* @module tests/unit/brainy/lazy-notready-honor
|
* @module tests/unit/brainy/lazy-notready-honor
|
||||||
* @description THE SILENT-EMPTY TRAP pin (found during a fleet adoption,
|
* @description THE SILENT-EMPTY TRAP pin (found during a fleet adoption,
|
||||||
* SELF-ENGINE-PAIR-STANDARD): under `disableAutoRebuild: true`, the OLD lazy
|
* SELF-ENGINE-PAIR-STANDARD): under `disableAutoRebuild: true`, the lazy
|
||||||
* first-query path (`ensureIndexesLoaded`) assessed ONLY the vector index's
|
* first-query path (`ensureIndexesLoaded`) assessed ONLY the vector index's
|
||||||
* readiness — a native METADATA provider reporting not-ready (its strand
|
* readiness — a native METADATA provider reporting not-ready (its strand
|
||||||
* report) never blocked the completion latch, so the promised lazy rebuild
|
* report) never blocked the completion latch, so the promised lazy rebuild
|
||||||
* never fired and every `find()` silently returned `[]` on a populated store
|
* never fired and every `find()` silently returned `[]` on a populated
|
||||||
* (measured: 52 entities durable-but-unqueryable, first query 0ms/0 rows).
|
* store (measured: 52 entities durable-but-unqueryable, first query
|
||||||
*
|
* 0ms/0 rows). The law: a not-ready report from ANY provider falls through
|
||||||
* RE-POINTED to the health-gate law (a read never builds; a rebuild runs
|
* to the rebuild — never a silent empty.
|
||||||
* entirely at open): `ensureIndexesLoaded()` is now a pure CHECK. A not-ready
|
|
||||||
* report from ANY provider — metadata, vector, or graph — makes it THROW the
|
|
||||||
* matching typed `*NotReadyError` rather than silently letting the read
|
|
||||||
* proceed, and it NEVER calls `rebuildIndexesIfNeeded` (that is entirely
|
|
||||||
* open()'s job now — see the second describe block below). The spirit is
|
|
||||||
* unchanged: a not-ready report from any single provider can never be
|
|
||||||
* shadowed into a silent empty result.
|
|
||||||
*
|
*
|
||||||
* White-box provider-double pattern per tests/unit/brainy/migration-deference.
|
* White-box provider-double pattern per tests/unit/brainy/migration-deference.
|
||||||
*/
|
*/
|
||||||
import { describe, it, expect, afterEach, vi } from 'vitest'
|
import { describe, it, expect, afterEach, vi } from 'vitest'
|
||||||
import { mkdtempSync, rmSync } from 'node:fs'
|
import { Brainy } from '../../../src/index.js'
|
||||||
import { tmpdir } from 'node:os'
|
|
||||||
import { join } from 'node:path'
|
|
||||||
import { Brainy, MetadataIndexNotReadyError } from '../../../src/index.js'
|
|
||||||
import { NounType } from '../../../src/types/graphTypes.js'
|
import { NounType } from '../../../src/types/graphTypes.js'
|
||||||
import { createTestConfig } from '../../helpers/test-factory.js'
|
import { createTestConfig } from '../../helpers/test-factory.js'
|
||||||
|
|
||||||
interface BrainInternals {
|
interface BrainInternals {
|
||||||
index: { size(): number }
|
index: { size(): number }
|
||||||
metadataIndex: { isReady?: () => boolean }
|
metadataIndex: { isReady?: () => boolean }
|
||||||
ensureIndexesLoaded(): void
|
lazyRebuildCompleted: boolean
|
||||||
|
ensureIndexesLoaded(): Promise<void>
|
||||||
rebuildIndexesIfNeeded(force?: boolean): Promise<void>
|
rebuildIndexesIfNeeded(force?: boolean): Promise<void>
|
||||||
}
|
}
|
||||||
|
|
||||||
const brains: Brainy[] = []
|
const brains: Brainy[] = []
|
||||||
const dirs: string[] = []
|
|
||||||
|
|
||||||
afterEach(async () => {
|
afterEach(async () => {
|
||||||
for (const b of brains.splice(0)) await b.close().catch(() => {})
|
for (const b of brains.splice(0)) await b.close().catch(() => {})
|
||||||
for (const d of dirs.splice(0)) rmSync(d, { recursive: true, force: true })
|
|
||||||
vi.restoreAllMocks()
|
vi.restoreAllMocks()
|
||||||
})
|
})
|
||||||
|
|
||||||
async function warmBrain(): Promise<{ brain: Brainy; internals: BrainInternals }> {
|
async function warmLazyBrain(): Promise<{ brain: Brainy; internals: BrainInternals }> {
|
||||||
const brain = new Brainy(createTestConfig({ disableAutoRebuild: true }))
|
const brain = new Brainy(createTestConfig({ disableAutoRebuild: true }))
|
||||||
await brain.init()
|
await brain.init()
|
||||||
brains.push(brain)
|
brains.push(brain)
|
||||||
|
|
@ -51,59 +40,36 @@ async function warmBrain(): Promise<{ brain: Brainy; internals: BrainInternals }
|
||||||
await brain.add({ data: `row ${i}`, type: NounType.Document, metadata: { i } })
|
await brain.add({ data: `row ${i}`, type: NounType.Document, metadata: { i } })
|
||||||
}
|
}
|
||||||
const internals = brain as unknown as BrainInternals
|
const internals = brain as unknown as BrainInternals
|
||||||
|
internals.lazyRebuildCompleted = false // simulate the cold first query
|
||||||
return { brain, internals }
|
return { brain, internals }
|
||||||
}
|
}
|
||||||
|
|
||||||
describe('the read gate honors EVERY provider’s not-ready report', () => {
|
describe('lazy path honors EVERY provider’s not-ready report', () => {
|
||||||
it('a not-ready METADATA provider refuses loudly — it never lets a read proceed, and it never rebuilds', async () => {
|
it('a not-ready METADATA provider blocks the completion latch and fires the rebuild', async () => {
|
||||||
const { internals } = await warmBrain()
|
const { internals } = await warmLazyBrain()
|
||||||
|
|
||||||
// The trap's shape: vector side looks fine (populated), metadata
|
// The trap's shape: vector side looks fine (populated), metadata
|
||||||
// provider says NOT ready — the OLD gate silently latched complete here.
|
// provider says NOT ready — the old gate latched complete here.
|
||||||
// The new gate refuses loudly instead; a read never triggers a rebuild.
|
;(internals.metadataIndex as { isReady?: () => boolean }).isReady = () => false
|
||||||
internals.metadataIndex.isReady = () => false
|
const rebuildSpy = vi
|
||||||
const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded').mockResolvedValue(undefined)
|
.spyOn(internals, 'rebuildIndexesIfNeeded')
|
||||||
|
.mockResolvedValue(undefined)
|
||||||
|
|
||||||
expect(() => internals.ensureIndexesLoaded()).toThrow(MetadataIndexNotReadyError)
|
await internals.ensureIndexesLoaded()
|
||||||
expect(rebuildSpy, 'a read NEVER triggers a rebuild — building is entirely open()\'s job now').not.toHaveBeenCalled()
|
|
||||||
|
expect(rebuildSpy, 'not-ready metadata provider must fire the lazy rebuild').toHaveBeenCalledWith(true)
|
||||||
})
|
})
|
||||||
|
|
||||||
it('control: all providers ready/unknown+populated → the gate lets the read through, no rebuild', async () => {
|
it('control: all providers ready/unknown+populated → latch completes, no rebuild', async () => {
|
||||||
const { internals } = await warmBrain()
|
const { internals } = await warmLazyBrain()
|
||||||
internals.metadataIndex.isReady = () => true
|
;(internals.metadataIndex as { isReady?: () => boolean }).isReady = () => true
|
||||||
const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded').mockResolvedValue(undefined)
|
const rebuildSpy = vi
|
||||||
|
.spyOn(internals, 'rebuildIndexesIfNeeded')
|
||||||
|
.mockResolvedValue(undefined)
|
||||||
|
|
||||||
|
await internals.ensureIndexesLoaded()
|
||||||
|
|
||||||
expect(() => internals.ensureIndexesLoaded()).not.toThrow()
|
|
||||||
expect(rebuildSpy).not.toHaveBeenCalled()
|
expect(rebuildSpy).not.toHaveBeenCalled()
|
||||||
|
expect(internals.lazyRebuildCompleted).toBe(true)
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
||||||
describe('the open-time build honors the same law: a needed rebuild runs at open, never deferred to a read', () => {
|
|
||||||
it('disableAutoRebuild:true does not defer a needed rebuild past open() on a reopened, populated store', async () => {
|
|
||||||
const dir = mkdtempSync(join(tmpdir(), 'brainy-lazy-notready-honor-'))
|
|
||||||
dirs.push(dir)
|
|
||||||
|
|
||||||
const writer = new Brainy(createTestConfig({ disableAutoRebuild: true, storage: { type: 'filesystem', path: dir } }))
|
|
||||||
await writer.init()
|
|
||||||
for (let i = 0; i < 3; i++) {
|
|
||||||
await writer.add({ data: `row ${i}`, type: NounType.Document, metadata: { i } })
|
|
||||||
}
|
|
||||||
await writer.flush()
|
|
||||||
await writer.close()
|
|
||||||
|
|
||||||
// Fresh instance over the same store: its derived indexes start empty in
|
|
||||||
// memory, so open()'s rebuildIndexesIfNeeded MUST fire (and complete)
|
|
||||||
// before init() returns — even though disableAutoRebuild is true, there
|
|
||||||
// is no first-query lazy path left to defer to.
|
|
||||||
const reader = new Brainy(createTestConfig({ disableAutoRebuild: true, storage: { type: 'filesystem', path: dir } }))
|
|
||||||
const internals = reader as unknown as { rebuildIndexesIfNeeded(force?: boolean): Promise<void> }
|
|
||||||
const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded')
|
|
||||||
|
|
||||||
await reader.init()
|
|
||||||
brains.push(reader)
|
|
||||||
|
|
||||||
expect(rebuildSpy).toHaveBeenCalledTimes(1)
|
|
||||||
const rows = await reader.find({ where: { i: 1 } })
|
|
||||||
expect(rows.length).toBe(1)
|
|
||||||
}, 30000)
|
|
||||||
})
|
|
||||||
|
|
|
||||||
|
|
@ -1,19 +1,16 @@
|
||||||
/**
|
/**
|
||||||
* @module tests/unit/brainy/metadata-provider-contract
|
* @module tests/unit/brainy/metadata-provider-contract
|
||||||
* @description Brainy-side wiring of the metadata-provider contract.
|
* @description Brainy-side wiring of the two metadata-provider contract additions
|
||||||
|
* confirmed with cor for the lockstep:
|
||||||
*
|
*
|
||||||
* `getIdsForFilter(filter, opts?)` — brainy passes a page bound on the UNSORTED
|
* 1. `probeConsistency()` — an OPTIONAL O(1) cold-open consistency sampler. On the
|
||||||
|
* first read, brainy calls it once; on `false` it self-heals via
|
||||||
|
* `detectAndRepairCorruption()` (the metadata counterpart of the graph cold-load
|
||||||
|
* guard). The native provider implements it; the JS index omits it (no-op).
|
||||||
|
* 2. `getIdsForFilter(filter, opts?)` — brainy passes a page bound on the UNSORTED
|
||||||
* `find({ type, where, limit })` path so a native provider can early-stop. The JS
|
* `find({ type, where, limit })` path so a native provider can early-stop. The JS
|
||||||
* index ignores `opts`.
|
* index ignores `opts`.
|
||||||
*
|
*
|
||||||
* RETIRED (health-gate law): `probeConsistency()` / `ensureMetadataConsistencyProbed()`
|
|
||||||
* — a read-time consistency probe that launches `detectAndRepairCorruption()` on
|
|
||||||
* `false` was exactly the read-triggered dark rebuild the law forbids (a read must
|
|
||||||
* never start a store walk or a rebuild). The probe's diagnostic value lives on in
|
|
||||||
* `validateIndexConsistency()` / `repairIndex()`, which remain explicit, operator-invoked
|
|
||||||
* calls. The pin below confirms the retirement: `probeConsistency()` is never called by
|
|
||||||
* a read, even when a provider exposes it.
|
|
||||||
*
|
|
||||||
* These are unit tests of brainy's CALL behaviour (the real end-to-end honoring is
|
* These are unit tests of brainy's CALL behaviour (the real end-to-end honoring is
|
||||||
* exercised by cor's combined matrix); they inject probe/spy hooks onto the live JS
|
* exercised by cor's combined matrix); they inject probe/spy hooks onto the live JS
|
||||||
* metadata index, which has neither method by default.
|
* metadata index, which has neither method by default.
|
||||||
|
|
@ -22,7 +19,7 @@ import { describe, it, expect, beforeEach } from 'vitest'
|
||||||
import { Brainy } from '../../../src/brainy'
|
import { Brainy } from '../../../src/brainy'
|
||||||
import { NounType } from '../../../src/types/graphTypes'
|
import { NounType } from '../../../src/types/graphTypes'
|
||||||
|
|
||||||
describe('metadata-provider contract wiring (getIdsForFilter opts)', () => {
|
describe('metadata-provider contract wiring (probeConsistency + getIdsForFilter opts)', () => {
|
||||||
let brain: Brainy<any>
|
let brain: Brainy<any>
|
||||||
let mi: any
|
let mi: any
|
||||||
|
|
||||||
|
|
@ -32,23 +29,47 @@ describe('metadata-provider contract wiring (getIdsForFilter opts)', () => {
|
||||||
await brain.add({ data: 'a', type: NounType.Thing, metadata: { kind: 'x' } })
|
await brain.add({ data: 'a', type: NounType.Thing, metadata: { kind: 'x' } })
|
||||||
await brain.add({ data: 'b', type: NounType.Thing, metadata: { kind: 'y' } })
|
await brain.add({ data: 'b', type: NounType.Thing, metadata: { kind: 'y' } })
|
||||||
mi = (brain as any).metadataIndex
|
mi = (brain as any).metadataIndex
|
||||||
|
;(brain as any)._metadataConsistencyProbed = false // reset the one-shot guard
|
||||||
})
|
})
|
||||||
|
|
||||||
it('RETIRED: a read never calls probeConsistency() / self-heals via detectAndRepairCorruption — that is the read-triggered dark rebuild the health-gate law forbids', async () => {
|
it('calls probeConsistency once on cold open and self-heals via detectAndRepairCorruption on false', async () => {
|
||||||
let probes = 0
|
let probes = 0
|
||||||
let repairs = 0
|
let repairs = 0
|
||||||
mi.probeConsistency = async () => { probes++; return false } // would-be corrupt signal
|
mi.probeConsistency = async () => { probes++; return false } // corrupt → must repair
|
||||||
const origRepair = mi.detectAndRepairCorruption.bind(mi)
|
const origRepair = mi.detectAndRepairCorruption.bind(mi)
|
||||||
mi.detectAndRepairCorruption = async () => { repairs++; return origRepair() }
|
mi.detectAndRepairCorruption = async () => { repairs++; return origRepair() }
|
||||||
|
|
||||||
await brain.find({ where: { kind: 'x' } })
|
await brain.find({ where: { kind: 'x' } })
|
||||||
|
expect(probes).toBe(1)
|
||||||
|
expect(repairs).toBe(1)
|
||||||
|
|
||||||
|
// Second read must NOT re-probe (once per brain).
|
||||||
await brain.find({ where: { kind: 'y' } })
|
await brain.find({ where: { kind: 'y' } })
|
||||||
|
expect(probes).toBe(1)
|
||||||
|
expect(repairs).toBe(1)
|
||||||
|
})
|
||||||
|
|
||||||
expect(probes).toBe(0) // no read-time probe exists anymore
|
it('does NOT repair when the probe reports healthy', async () => {
|
||||||
expect(repairs).toBe(0) // and therefore no read-triggered self-heal either
|
let repairs = 0
|
||||||
|
mi.probeConsistency = async () => true // clean
|
||||||
|
const origRepair = mi.detectAndRepairCorruption.bind(mi)
|
||||||
|
mi.detectAndRepairCorruption = async () => { repairs++; return origRepair() }
|
||||||
|
|
||||||
delete mi.probeConsistency
|
await brain.find({ where: { kind: 'x' } })
|
||||||
mi.detectAndRepairCorruption = origRepair
|
expect(repairs).toBe(0)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a probe failure never breaks the read (best-effort, retried next time)', async () => {
|
||||||
|
let probes = 0
|
||||||
|
mi.probeConsistency = async () => { probes++; throw new Error('probe boom') }
|
||||||
|
|
||||||
|
// The read still succeeds despite the throwing probe.
|
||||||
|
const rows = await brain.find({ where: { kind: 'x' } })
|
||||||
|
expect(rows.length).toBe(1)
|
||||||
|
expect(probes).toBe(1)
|
||||||
|
// Guard reset on failure → the next read retries the probe.
|
||||||
|
await brain.find({ where: { kind: 'y' } })
|
||||||
|
expect(probes).toBe(2)
|
||||||
})
|
})
|
||||||
|
|
||||||
it('passes a page bound to getIdsForFilter on the unsorted find path (offset 0, brainy re-windows)', async () => {
|
it('passes a page bound to getIdsForFilter on the unsorted find path (offset 0, brainy re-windows)', async () => {
|
||||||
|
|
|
||||||
|
|
@ -25,7 +25,7 @@
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import { describe, it, expect, afterEach, vi } from 'vitest'
|
import { describe, it, expect, afterEach, vi } from 'vitest'
|
||||||
import { Brainy, VectorIndexNotReadyError } from '../../../src/index.js'
|
import { Brainy } from '../../../src/index.js'
|
||||||
import { NounType } from '../../../src/types/graphTypes.js'
|
import { NounType } from '../../../src/types/graphTypes.js'
|
||||||
import { createTestConfig } from '../../helpers/test-factory.js'
|
import { createTestConfig } from '../../helpers/test-factory.js'
|
||||||
import { BaseStorage } from '../../../src/storage/baseStorage.js'
|
import { BaseStorage } from '../../../src/storage/baseStorage.js'
|
||||||
|
|
@ -43,8 +43,9 @@ interface BrainInternals {
|
||||||
metadataIndex: { rebuild(...a: unknown[]): Promise<unknown> }
|
metadataIndex: { rebuild(...a: unknown[]): Promise<unknown> }
|
||||||
graphIndex: { size(): number; rebuild(...a: unknown[]): Promise<unknown> }
|
graphIndex: { size(): number; rebuild(...a: unknown[]): Promise<unknown> }
|
||||||
_indexEpochStale: boolean
|
_indexEpochStale: boolean
|
||||||
|
lazyRebuildCompleted: boolean
|
||||||
rebuildIndexesIfNeeded(force?: boolean): Promise<void>
|
rebuildIndexesIfNeeded(force?: boolean): Promise<void>
|
||||||
ensureIndexesLoaded(): void
|
ensureIndexesLoaded(): Promise<void>
|
||||||
storage: { readRawObject(p: string): Promise<unknown> }
|
storage: { readRawObject(p: string): Promise<unknown> }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -180,40 +181,40 @@ describe('rc.8 no-freeze migration deference (isMigrating / stampBrainFormat / b
|
||||||
expect(idxSpy).toHaveBeenCalledTimes(1)
|
expect(idxSpy).toHaveBeenCalledTimes(1)
|
||||||
})
|
})
|
||||||
|
|
||||||
// --- Hook 1: read-gate deference (RE-POINTED — the health-gate law retired
|
// --- Hook 1: large-path first-query lazy force-rebuild deference ----------
|
||||||
// the first-query lazy force-rebuild entirely: ensureIndexesLoaded() is now
|
|
||||||
// a pure CHECK that never calls rebuildIndexesIfNeeded, migrating or not.
|
|
||||||
// What survives from the original law is the DEFERENCE itself: a migrating
|
|
||||||
// provider's report is never judged by the gate — it neither throws nor
|
|
||||||
// rebuilds — while the exact same not-ready report on a NON-migrating
|
|
||||||
// provider throws the typed error instead of ever rebuilding.) ------------
|
|
||||||
|
|
||||||
it('the read gate defers to a migrating vector provider — a not-ready report neither throws nor rebuilds', async () => {
|
it('lazy first-query force-rebuild is SKIPPED when the vector provider isMigrating()', async () => {
|
||||||
|
// disableAutoRebuild routes first queries through ensureIndexesLoaded() (the
|
||||||
|
// large-brain lazy path that would otherwise force a blocking rebuild).
|
||||||
const brain = await makeWarmBrain(2, { disableAutoRebuild: true })
|
const brain = await makeWarmBrain(2, { disableAutoRebuild: true })
|
||||||
const internals = internalsOf(brain)
|
const internals = internalsOf(brain)
|
||||||
|
|
||||||
const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded').mockResolvedValue(undefined)
|
const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded').mockResolvedValue(undefined)
|
||||||
// Simulate a not-ready live vector index (cor is mid-swap, serving canonical).
|
// Simulate a cold/empty live vector index (cor is mid-swap, serving canonical).
|
||||||
;(internals.index as unknown as { isReady?: () => boolean }).isReady = () => false
|
vi.spyOn(internals.index, 'size').mockReturnValue(0)
|
||||||
|
internals.lazyRebuildCompleted = false
|
||||||
setMigrating(internals.index, true)
|
setMigrating(internals.index, true)
|
||||||
|
|
||||||
expect(() => internals.ensureIndexesLoaded()).not.toThrow()
|
await internals.ensureIndexesLoaded()
|
||||||
// A query during cor's background swap must not trigger brainy's own
|
|
||||||
// rebuild — reads never rebuild in any case, migrating or not.
|
// A query during cor's background swap must not trigger brainy's blocking rebuild.
|
||||||
expect(rebuildSpy).toHaveBeenCalledTimes(0)
|
expect(rebuildSpy).toHaveBeenCalledTimes(0)
|
||||||
})
|
})
|
||||||
|
|
||||||
it('the read gate THROWS for the same not-ready vector provider once migration clears (control)', async () => {
|
it('lazy first-query force-rebuild STILL fires when the vector provider is not migrating (control)', async () => {
|
||||||
const brain = await makeWarmBrain(2, { disableAutoRebuild: true })
|
const brain = await makeWarmBrain(2, { disableAutoRebuild: true })
|
||||||
const internals = internalsOf(brain)
|
const internals = internalsOf(brain)
|
||||||
|
|
||||||
const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded').mockResolvedValue(undefined)
|
const rebuildSpy = vi.spyOn(internals, 'rebuildIndexesIfNeeded').mockResolvedValue(undefined)
|
||||||
;(internals.index as unknown as { isReady?: () => boolean }).isReady = () => false
|
vi.spyOn(internals.index, 'size').mockReturnValue(0)
|
||||||
|
internals.lazyRebuildCompleted = false
|
||||||
// No isMigrating → not deferring.
|
// No isMigrating → not deferring.
|
||||||
|
|
||||||
expect(() => internals.ensureIndexesLoaded()).toThrow(VectorIndexNotReadyError)
|
await internals.ensureIndexesLoaded()
|
||||||
// Still never rebuilds — the gate refuses loudly instead.
|
|
||||||
expect(rebuildSpy).toHaveBeenCalledTimes(0)
|
// Without deference, the cold empty index drives the lazy force-rebuild.
|
||||||
|
expect(rebuildSpy).toHaveBeenCalledTimes(1)
|
||||||
|
expect(rebuildSpy).toHaveBeenCalledWith(true)
|
||||||
})
|
})
|
||||||
|
|
||||||
// --- Hook 2: public stampBrainFormat() -----------------------------------
|
// --- Hook 2: public stampBrainFormat() -----------------------------------
|
||||||
|
|
|
||||||
|
|
@ -3,14 +3,10 @@
|
||||||
* reported cold `find({ where })` returning a silent `[]` on a freshly-opened
|
* reported cold `find({ where })` returning a silent `[]` on a freshly-opened
|
||||||
* brain (a native metadata index that reports data but has not loaded its field
|
* brain (a native metadata index that reports data but has not loaded its field
|
||||||
* postings). This guard, the field-index counterpart of verifyGraphAdjacencyLive,
|
* postings). This guard, the field-index counterpart of verifyGraphAdjacencyLive,
|
||||||
* probes a known persisted value on the first filtered find().
|
* probes a known persisted value on the first filtered find(): if the index does
|
||||||
*
|
* not serve it, brainy rebuilds and re-probes, and raises a loud
|
||||||
* RE-POINTED to the health-gate law: the guard NEVER rebuilds and NEVER walks
|
* MetadataIndexNotReadyError only if the rebuild still can't serve — never a
|
||||||
* the store from a read — a read-path rebuild is exactly the dark-rebuild
|
* silent empty result that misrepresents existing data.
|
||||||
* failure mode the law retires (open() alone owns building). When the probe
|
|
||||||
* cannot serve the known value it raises a loud MetadataIndexNotReadyError
|
|
||||||
* IMMEDIATELY, with no rebuild attempt in between — never a silent empty
|
|
||||||
* result that misrepresents existing data.
|
|
||||||
*
|
*
|
||||||
* The 8.0 JS index cold-loads correctly, so we simulate the cold native failure
|
* The 8.0 JS index cold-loads correctly, so we simulate the cold native failure
|
||||||
* mode by intercepting the provider's getIdsForFilter/rebuild.
|
* mode by intercepting the provider's getIdsForFilter/rebuild.
|
||||||
|
|
@ -46,19 +42,37 @@ describe('Metadata cold-read guard (#venue silent-[])', () => {
|
||||||
mi.rebuild = origRebuild
|
mi.rebuild = origRebuild
|
||||||
})
|
})
|
||||||
|
|
||||||
it('cold index: verifyMetadataLive REFUSES immediately — find({where}) throws MetadataIndexNotReadyError, NEVER a silent [], and NEVER a rebuild attempt', async () => {
|
it('cold index: verifyMetadataLive self-heals via rebuild — find({where}) is correct, NOT silent []', async () => {
|
||||||
const mi = brain.metadataIndex
|
const mi = brain.metadataIndex
|
||||||
const origGetIds = mi.getIdsForFilter.bind(mi)
|
const origGetIds = mi.getIdsForFilter.bind(mi)
|
||||||
let rebuilds = 0
|
|
||||||
const origRebuild = mi.rebuild.bind(mi)
|
const origRebuild = mi.rebuild.bind(mi)
|
||||||
|
let cold = true
|
||||||
brain._metadataVerified = false // re-arm the one-shot for this scenario
|
brain._metadataVerified = false // re-arm the one-shot for this scenario
|
||||||
mi.getIdsForFilter = async () => [] // cold: the known value never resolves
|
mi.getIdsForFilter = async (...a: any[]) => (cold ? [] : origGetIds(...a))
|
||||||
mi.rebuild = async () => { rebuilds++; return origRebuild() }
|
mi.rebuild = async () => {
|
||||||
|
await origRebuild()
|
||||||
|
cold = false // the rebuild warms the postings
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const res = await brain.find({ where: { status: 'active' }, limit: 100 })
|
||||||
|
expect(res.length).toBe(1) // self-healed — the known entity is returned
|
||||||
|
} finally {
|
||||||
|
mi.getIdsForFilter = origGetIds
|
||||||
|
mi.rebuild = origRebuild
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('unrecoverably cold index: find({where}) throws MetadataIndexNotReadyError — never a silent []', async () => {
|
||||||
|
const mi = brain.metadataIndex
|
||||||
|
const origGetIds = mi.getIdsForFilter.bind(mi)
|
||||||
|
const origRebuild = mi.rebuild.bind(mi)
|
||||||
|
brain._metadataVerified = false
|
||||||
|
mi.getIdsForFilter = async () => [] // always cold; rebuild can't fix it
|
||||||
|
mi.rebuild = async () => {}
|
||||||
try {
|
try {
|
||||||
await expect(brain.find({ where: { status: 'active' }, limit: 100 })).rejects.toBeInstanceOf(
|
await expect(brain.find({ where: { status: 'active' }, limit: 100 })).rejects.toBeInstanceOf(
|
||||||
MetadataIndexNotReadyError
|
MetadataIndexNotReadyError
|
||||||
)
|
)
|
||||||
expect(rebuilds).toBe(0) // the guard never rebuilds from a read — it refuses loudly instead
|
|
||||||
} finally {
|
} finally {
|
||||||
mi.getIdsForFilter = origGetIds
|
mi.getIdsForFilter = origGetIds
|
||||||
mi.rebuild = origRebuild
|
mi.rebuild = origRebuild
|
||||||
|
|
|
||||||
|
|
@ -1,71 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/unit/plugin-activation-loudness
|
|
||||||
* @description The plugin-activation swallow closes. Two laws:
|
|
||||||
* (1) THE NOT-INSTALLED FREE PASS IS EXACT — a resolution failure earns the
|
|
||||||
* silent skip ONLY when it names the probed package itself, terminated
|
|
||||||
* where the name ends. A missing platform-binary SIBLING package
|
|
||||||
* ("<pkg>-linux-x64-gnu" — what a deploy replacing node_modules
|
|
||||||
* mid-restart leaves), an inner file path, or a dependency failure is a
|
|
||||||
* BROKEN install and must fail loud. A production storm ran 90s of
|
|
||||||
* throttled WASM behind this exact prefix-match hole.
|
|
||||||
* (2) A GRACEFUL DECLINE IS NARRATED ON THE ALWAYS-ON CHANNEL — activate()
|
|
||||||
* returning false warns via prodLog, which `silent: true` cannot patch
|
|
||||||
* away; a declined accelerator is never an invisible degrade.
|
|
||||||
*/
|
|
||||||
import { describe, it, expect, vi, afterEach } from 'vitest'
|
|
||||||
import { Brainy } from '../../src/brainy.js'
|
|
||||||
import { prodLog } from '../../src/utils/logger.js'
|
|
||||||
|
|
||||||
const isNotInstalled = (error: unknown, pkg: string): boolean =>
|
|
||||||
(Brainy as unknown as {
|
|
||||||
isPackageNotInstalledError(e: unknown, p: string): boolean
|
|
||||||
}).isPackageNotInstalledError(error, pkg)
|
|
||||||
|
|
||||||
const resolutionError = (message: string): Error => {
|
|
||||||
const e = new Error(message) as Error & { code?: string }
|
|
||||||
e.code = 'ERR_MODULE_NOT_FOUND'
|
|
||||||
return e
|
|
||||||
}
|
|
||||||
|
|
||||||
describe('the not-installed free pass is exact', () => {
|
|
||||||
const PKG = '@soulcraft/cor'
|
|
||||||
|
|
||||||
it('the package itself, quoted or bare → not-installed (the one free path)', () => {
|
|
||||||
expect(isNotInstalled(resolutionError(`Cannot find package '${PKG}' imported from /app/x.js`), PKG)).toBe(true)
|
|
||||||
expect(isNotInstalled(resolutionError(`Cannot find module ${PKG}`), PKG)).toBe(true)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('a missing platform-binary SIBLING package is a broken install, never not-installed', () => {
|
|
||||||
expect(isNotInstalled(resolutionError(`Cannot find package '${PKG}-linux-x64-gnu' imported from /app`), PKG)).toBe(false)
|
|
||||||
expect(isNotInstalled(resolutionError(`Failed to resolve ${PKG}-darwin-arm64`), PKG)).toBe(false)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('an inner file path or a non-resolution error is never not-installed', () => {
|
|
||||||
expect(isNotInstalled(resolutionError(`Cannot find module '/app/node_modules/${PKG}/native/b.node'`), PKG)).toBe(false)
|
|
||||||
expect(isNotInstalled(new Error(`dlopen failed: wrong ELF class in ${PKG}`), PKG)).toBe(false)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('a graceful decline is narrated on the always-on channel', () => {
|
|
||||||
afterEach(() => vi.restoreAllMocks())
|
|
||||||
|
|
||||||
it('activate() → false warns via prodLog even under silent: true', async () => {
|
|
||||||
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
|
|
||||||
const warn = vi.spyOn(prodLog, 'warn')
|
|
||||||
const brain: any = new Brainy({
|
|
||||||
requireSubtype: false,
|
|
||||||
storage: { type: 'memory' },
|
|
||||||
silent: true,
|
|
||||||
dimensions: 384
|
|
||||||
})
|
|
||||||
brain.use({ name: 'declining-accelerator', activate: async () => false })
|
|
||||||
await brain.init()
|
|
||||||
try {
|
|
||||||
expect(
|
|
||||||
warn.mock.calls.some((c) => String(c[0]).includes('"declining-accelerator" declined activation'))
|
|
||||||
).toBe(true)
|
|
||||||
} finally {
|
|
||||||
await brain.close().catch(() => {})
|
|
||||||
}
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
@ -63,9 +63,6 @@ function inGate(rel: string): boolean {
|
||||||
return (
|
return (
|
||||||
rel.startsWith('tests/unit/') ||
|
rel.startsWith('tests/unit/') ||
|
||||||
rel.startsWith('tests/integration/') ||
|
rel.startsWith('tests/integration/') ||
|
||||||
// The lifecycle biography lane — included by the integration config
|
|
||||||
// ('tests/lifecycle/**/*.test.ts'; see tests/lifecycle/README.md).
|
|
||||||
rel.startsWith('tests/lifecycle/') ||
|
|
||||||
rel.endsWith('.unit.test.ts') ||
|
rel.endsWith('.unit.test.ts') ||
|
||||||
rel.endsWith('.integration.test.ts')
|
rel.endsWith('.integration.test.ts')
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -1,157 +0,0 @@
|
||||||
/**
|
|
||||||
* @module tests/unit/utils/indexReadiness
|
|
||||||
* @description Pins for the read-gate authority, {@link assessProviderHealth}, and
|
|
||||||
* its older sibling {@link assessIndexReadiness}. The health-gate law: a provider's
|
|
||||||
* NAMED, synchronous, O(1) health report — when exposed — REPLACES the `isReady()`/
|
|
||||||
* size-heuristic fallback as the read gate's source of truth. A throw from
|
|
||||||
* `healthReport()` is a CONTRACT VIOLATION (never read as healthy, never swallowed
|
|
||||||
* into "unknown"); an `unledgered` family is UNKNOWN (never healthy, never broken —
|
|
||||||
* `serving` is always the provider's own verdict, verbatim).
|
|
||||||
*/
|
|
||||||
import { describe, it, expect } from 'vitest'
|
|
||||||
import { assessIndexReadiness, assessProviderHealth } from '../../../src/utils/indexReadiness.js'
|
|
||||||
import type { HealthReport, LedgerInvariantResult } from '../../../src/plugin.js'
|
|
||||||
|
|
||||||
function invariant(overrides: Partial<LedgerInvariantResult> = {}): LedgerInvariantResult {
|
|
||||||
return {
|
|
||||||
name: 'manifest-residency',
|
|
||||||
holds: true,
|
|
||||||
detail: 'ok',
|
|
||||||
heal: 'none',
|
|
||||||
source: 'ledger',
|
|
||||||
...overrides
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function report(overrides: Partial<HealthReport> = {}): HealthReport {
|
|
||||||
return {
|
|
||||||
provider: 'vector',
|
|
||||||
healthy: true,
|
|
||||||
serving: true,
|
|
||||||
invariants: [],
|
|
||||||
// A FIXED stamp, never Date.now(): the pin at :98 compares two
|
|
||||||
// independently-built reports, and a live clock made them differ by 1ms
|
|
||||||
// whenever the millisecond ticked between the two calls — a plant-lane
|
|
||||||
// red that had nothing to do with the code under test.
|
|
||||||
checkedAt: 1_700_000_000_000,
|
|
||||||
durationMs: 1,
|
|
||||||
generation: 1,
|
|
||||||
unledgered: [],
|
|
||||||
...overrides
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
describe('assessIndexReadiness (legacy isReady() classifier)', () => {
|
|
||||||
it('unknown when the provider is null/undefined', () => {
|
|
||||||
expect(assessIndexReadiness(null)).toBe('unknown')
|
|
||||||
expect(assessIndexReadiness(undefined)).toBe('unknown')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('unknown when isReady() is absent', () => {
|
|
||||||
expect(assessIndexReadiness({})).toBe('unknown')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('ready / not-ready mirror isReady()', () => {
|
|
||||||
expect(assessIndexReadiness({ isReady: () => true })).toBe('ready')
|
|
||||||
expect(assessIndexReadiness({ isReady: () => false })).toBe('not-ready')
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
||||||
describe('assessProviderHealth — the read-gate authority', () => {
|
|
||||||
it('via "none": no provider at all', () => {
|
|
||||||
const a = assessProviderHealth(null)
|
|
||||||
expect(a.via).toBe('none')
|
|
||||||
expect(a.readiness).toBe('unknown')
|
|
||||||
expect(a.report).toBeNull()
|
|
||||||
expect(a.reasons.length).toBeGreaterThan(0)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('via "size-heuristic": provider exposes neither healthReport() nor isReady()', () => {
|
|
||||||
const a = assessProviderHealth({})
|
|
||||||
expect(a.via).toBe('size-heuristic')
|
|
||||||
expect(a.readiness).toBe('unknown')
|
|
||||||
expect(a.report).toBeNull()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('via "is-ready": provider exposes isReady() but no healthReport() — ready', () => {
|
|
||||||
const a = assessProviderHealth({ isReady: () => true })
|
|
||||||
expect(a.via).toBe('is-ready')
|
|
||||||
expect(a.readiness).toBe('ready')
|
|
||||||
expect(a.reasons).toEqual([])
|
|
||||||
})
|
|
||||||
|
|
||||||
it('via "is-ready": isReady() === false — not-ready with a reason', () => {
|
|
||||||
const a = assessProviderHealth({ isReady: () => false })
|
|
||||||
expect(a.via).toBe('is-ready')
|
|
||||||
expect(a.readiness).toBe('not-ready')
|
|
||||||
expect(a.reasons.length).toBeGreaterThan(0)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('healthReport() present REPLACES isReady() — serving:true wins even if isReady() lies false', () => {
|
|
||||||
const p = { isReady: () => false, healthReport: () => report({ serving: true }) }
|
|
||||||
const a = assessProviderHealth(p)
|
|
||||||
expect(a.via).toBe('health-report')
|
|
||||||
expect(a.readiness).toBe('ready')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('serving:true, healthy:true, no invariants failing → ready, no reasons', () => {
|
|
||||||
const p = { healthReport: () => report({ serving: true, healthy: true }) }
|
|
||||||
const a = assessProviderHealth(p)
|
|
||||||
expect(a.readiness).toBe('ready')
|
|
||||||
expect(a.reasons).toEqual([])
|
|
||||||
expect(a.report).toEqual(report({ serving: true, healthy: true }))
|
|
||||||
})
|
|
||||||
|
|
||||||
it('serving:false with a named heal:"rebuild" failing invariant → not-ready, reason names it', () => {
|
|
||||||
const failing = invariant({ name: 'posted-count-floor', holds: false, heal: 'rebuild', detail: 'posted 10 < canonical 20' })
|
|
||||||
const p = { healthReport: () => report({ serving: false, healthy: false, invariants: [failing] }) }
|
|
||||||
const a = assessProviderHealth(p)
|
|
||||||
expect(a.readiness).toBe('not-ready')
|
|
||||||
expect(a.reasons.some((r) => r.includes('posted-count-floor') && r.includes('heal:rebuild') && r.includes('posted 10 < canonical 20'))).toBe(true)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('unledgered-only report (serving:true, no failing invariant) → ready, reason names the unledgered family', () => {
|
|
||||||
const p = { healthReport: () => report({ serving: true, healthy: true, unledgered: ['canonical-verb-coverage'] }) }
|
|
||||||
const a = assessProviderHealth(p)
|
|
||||||
expect(a.readiness).toBe('ready')
|
|
||||||
expect(a.reasons.some((r) => r.includes('unledgered') && r.includes('canonical-verb-coverage'))).toBe(true)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('UNLEDGERED IS UNKNOWN: an unledgered family never flips a NOT-serving provider to ready', () => {
|
|
||||||
const failing = invariant({ holds: false, heal: 'rebuild', name: 'x' })
|
|
||||||
const p = { healthReport: () => report({ serving: false, healthy: false, invariants: [failing], unledgered: ['some-family'] }) }
|
|
||||||
const a = assessProviderHealth(p)
|
|
||||||
expect(a.readiness).toBe('not-ready')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('serving:true, healthy:false with a heal:"repair" failure → still ready (degraded-but-serving)', () => {
|
|
||||||
const failing = invariant({ name: 'stale-counter', holds: false, heal: 'repair', detail: 'counter drift' })
|
|
||||||
const p = { healthReport: () => report({ serving: true, healthy: false, invariants: [failing] }) }
|
|
||||||
const a = assessProviderHealth(p)
|
|
||||||
expect(a.readiness).toBe('ready')
|
|
||||||
expect(a.reasons.some((r) => r.includes('stale-counter') && r.includes('heal:repair'))).toBe(true)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('healthReport() that THROWS is a CONTRACT VIOLATION: not-ready, via health-report, reason names the throw — never "unknown"', () => {
|
|
||||||
const p = { healthReport: () => { throw new Error('mmap window busy') } }
|
|
||||||
const a = assessProviderHealth(p)
|
|
||||||
expect(a.via).toBe('health-report')
|
|
||||||
expect(a.readiness).toBe('not-ready')
|
|
||||||
expect(a.report).toBeNull()
|
|
||||||
expect(a.reasons.some((r) => r.includes('mmap window busy'))).toBe(true)
|
|
||||||
expect(a.readiness).not.toBe('unknown')
|
|
||||||
})
|
|
||||||
|
|
||||||
it('healthReport() that throws a non-Error value still produces a named reason (String(err))', () => {
|
|
||||||
const p = { healthReport: () => { throw 'boom' } }
|
|
||||||
const a = assessProviderHealth(p)
|
|
||||||
expect(a.readiness).toBe('not-ready')
|
|
||||||
expect(a.reasons.some((r) => r.includes('boom'))).toBe(true)
|
|
||||||
})
|
|
||||||
|
|
||||||
it('the returned report carries the generation for narration dedup', () => {
|
|
||||||
const p = { healthReport: () => report({ generation: 42 }) }
|
|
||||||
const a = assessProviderHealth(p)
|
|
||||||
expect(a.report?.generation).toBe(42)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
@ -11,11 +11,8 @@
|
||||||
* Same rule, same verdict names as the shipped aggregation machinery
|
* Same rule, same verdict names as the shipped aggregation machinery
|
||||||
* (AggregationIndex.stateAdoptionVerdict).
|
* (AggregationIndex.stateAdoptionVerdict).
|
||||||
*
|
*
|
||||||
* The verdict is computed at init and consumed via
|
* The verdict is COMPUTED AND EXPOSED only — these pins assert no rebuild
|
||||||
* {@link MetadataIndexManager.applyWatermarkCatchup} — the coordinator
|
* trigger changed; acting on 'catchup' lands with the coordinator's wiring.
|
||||||
* (`Brainy.performInit`) calls it right after `init()`, with an open fact
|
|
||||||
* scan when the verdict is `'catchup'`. This file pins both halves: the
|
|
||||||
* verdict computation (above) and the fold/no-op/demotion behavior below.
|
|
||||||
*/
|
*/
|
||||||
import { describe, it, expect, vi, afterEach } from 'vitest'
|
import { describe, it, expect, vi, afterEach } from 'vitest'
|
||||||
import { v4 as uuidv4 } from 'uuid'
|
import { v4 as uuidv4 } from 'uuid'
|
||||||
|
|
@ -25,46 +22,6 @@ import {
|
||||||
} from '../../../src/utils/metadataIndex.js'
|
} from '../../../src/utils/metadataIndex.js'
|
||||||
import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js'
|
import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js'
|
||||||
import { prodLog } from '../../../src/utils/logger.js'
|
import { prodLog } from '../../../src/utils/logger.js'
|
||||||
import type { CommitFact, FactScanBatch, FactScanHandle } from '../../../src/db/factLog.js'
|
|
||||||
|
|
||||||
/** A fact scan handle over an in-memory list of facts — batches them one
|
|
||||||
* fact at a time (batch size is irrelevant to the fold, which reads
|
|
||||||
* `batch.facts` only). */
|
|
||||||
function fakeScan(facts: CommitFact[]): FactScanHandle {
|
|
||||||
return {
|
|
||||||
headGeneration: facts.length > 0 ? facts[facts.length - 1].generation : 0,
|
|
||||||
segmentCount: 1,
|
|
||||||
approxFactCount: facts.length,
|
|
||||||
async *batches(): AsyncGenerator<FactScanBatch> {
|
|
||||||
for (const fact of facts) {
|
|
||||||
yield {
|
|
||||||
facts: [fact],
|
|
||||||
firstGeneration: fact.generation,
|
|
||||||
lastGeneration: fact.generation,
|
|
||||||
factCount: 1,
|
|
||||||
byteSize: 0,
|
|
||||||
segmentId: 'fake'
|
|
||||||
}
|
|
||||||
}
|
|
||||||
},
|
|
||||||
summary: () => ({ factsYielded: facts.length, segmentsRead: 1 })
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** One noun after-image fact — the flat-record shape (no nested `metadata`
|
|
||||||
* key), matching this file's existing `writeArtifact` convention. */
|
|
||||||
function nounAdd(generation: number, id: string, metadata: Record<string, unknown>): CommitFact {
|
|
||||||
return {
|
|
||||||
generation,
|
|
||||||
timestamp: Date.now(),
|
|
||||||
ops: [{ kind: 'noun', id, record: { metadata, vector: null } }]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/** One noun tombstone fact. */
|
|
||||||
function nounDelete(generation: number, id: string): CommitFact {
|
|
||||||
return { generation, timestamp: Date.now(), ops: [{ kind: 'noun', id, record: null }] }
|
|
||||||
}
|
|
||||||
|
|
||||||
/** Fresh storage with a controllable committed generation. */
|
/** Fresh storage with a controllable committed generation. */
|
||||||
async function makeStorage(committed: number | null): Promise<MemoryStorage> {
|
async function makeStorage(committed: number | null): Promise<MemoryStorage> {
|
||||||
|
|
@ -212,106 +169,3 @@ describe('metadata index — watermark stamp + three-way load verdict', () => {
|
||||||
expect(await storage.getMetadata(METADATA_INDEX_STAMP_KEY)).toBeNull()
|
expect(await storage.getMetadata(METADATA_INDEX_STAMP_KEY)).toBeNull()
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
||||||
describe('metadata index — applyWatermarkCatchup (the coordinator door)', () => {
|
|
||||||
it("an 'adopt' verdict performs zero index writes", async () => {
|
|
||||||
const storage = await makeStorage(5)
|
|
||||||
await writeArtifact(storage, 5)
|
|
||||||
const index = await reopen(storage)
|
|
||||||
expect(index.watermarkVerdict()).toBe('adopt')
|
|
||||||
|
|
||||||
const addSpy = vi.spyOn(index, 'addToIndex')
|
|
||||||
const removeSpy = vi.spyOn(index, 'removeFromIndex')
|
|
||||||
|
|
||||||
const result = await index.applyWatermarkCatchup(null)
|
|
||||||
|
|
||||||
expect(result).toEqual({ action: 'noop' })
|
|
||||||
expect(addSpy).not.toHaveBeenCalled()
|
|
||||||
expect(removeSpy).not.toHaveBeenCalled()
|
|
||||||
})
|
|
||||||
|
|
||||||
it('a catchup window folding an add, an update (same id twice), and a delete → the index serves exactly the final state', async () => {
|
|
||||||
const storage = await makeStorage(5)
|
|
||||||
|
|
||||||
// Session 1: two pre-existing entities, stamped at generation 5.
|
|
||||||
const survivorId = uuidv4()
|
|
||||||
const deletedId = uuidv4()
|
|
||||||
{
|
|
||||||
const index = new MetadataIndexManager(storage)
|
|
||||||
await index.init()
|
|
||||||
await index.addToIndex(survivorId, { status: 'active' })
|
|
||||||
await index.addToIndex(deletedId, { status: 'active' })
|
|
||||||
index.stampWatermark(5)
|
|
||||||
await index.flush()
|
|
||||||
}
|
|
||||||
|
|
||||||
// The store advanced to generation 8 without another metadata flush —
|
|
||||||
// the exact shape a crash-then-adopt-reopen leaves behind.
|
|
||||||
setCommitted(storage, 8)
|
|
||||||
|
|
||||||
const index = await reopen(storage)
|
|
||||||
expect(index.watermarkVerdict()).toBe('catchup')
|
|
||||||
expect(index.watermarkGap()).toEqual({ from: 5, to: 8 })
|
|
||||||
|
|
||||||
const addedId = uuidv4()
|
|
||||||
const scan = fakeScan([
|
|
||||||
nounAdd(6, addedId, { status: 'new' }), // add
|
|
||||||
nounAdd(7, addedId, { status: 'updated' }), // update — same id twice
|
|
||||||
nounDelete(8, deletedId) // delete
|
|
||||||
])
|
|
||||||
|
|
||||||
const result = await index.applyWatermarkCatchup(scan)
|
|
||||||
|
|
||||||
expect(result.action).toBe('caught-up')
|
|
||||||
expect(result.window).toEqual({ from: 5, to: 8 })
|
|
||||||
expect(result.factsApplied).toBe(3)
|
|
||||||
expect(result.nounsApplied).toBe(3)
|
|
||||||
expect(result.verbsApplied).toBe(0)
|
|
||||||
|
|
||||||
// Final state: the added/updated id serves ONLY its final value...
|
|
||||||
expect(await index.getIds('status', 'updated')).toEqual([addedId])
|
|
||||||
expect(await index.getIds('status', 'new')).toEqual([]) // stale value gone
|
|
||||||
// ...the deleted id is gone...
|
|
||||||
expect(await index.getIds('status', 'active')).toEqual([survivorId])
|
|
||||||
// ...and the untouched survivor is unaffected.
|
|
||||||
expect(await index.getIds('status', 'active')).toContain(survivorId)
|
|
||||||
|
|
||||||
// The window is certified: watermark stamped at `to`, and a fresh
|
|
||||||
// reopen now verdicts 'adopt'.
|
|
||||||
expect(index.watermark()).toBe(8)
|
|
||||||
const reopened = await reopen(storage)
|
|
||||||
expect(reopened.watermarkVerdict()).toBe('adopt')
|
|
||||||
})
|
|
||||||
|
|
||||||
it("a 'rescan' verdict runs the existing rebuild path instead of folding", async () => {
|
|
||||||
const storage = await makeStorage(9)
|
|
||||||
await writeArtifact(storage, 9)
|
|
||||||
setCommitted(storage, 4) // a truncated log pulled the watermark back — stamp ABOVE committed → rescan
|
|
||||||
|
|
||||||
const index = await reopen(storage)
|
|
||||||
expect(index.watermarkVerdict()).toBe('rescan')
|
|
||||||
|
|
||||||
const rebuildSpy = vi.spyOn(index, 'rebuild')
|
|
||||||
const result = await index.applyWatermarkCatchup(null)
|
|
||||||
|
|
||||||
expect(result.action).toBe('rescan')
|
|
||||||
expect(result.reason).toBeTruthy()
|
|
||||||
expect(rebuildSpy).toHaveBeenCalledTimes(1)
|
|
||||||
})
|
|
||||||
|
|
||||||
it("a 'catchup' verdict with no fact log available demotes to rebuild, narrated", async () => {
|
|
||||||
const storage = await makeStorage(5)
|
|
||||||
await writeArtifact(storage, 5)
|
|
||||||
setCommitted(storage, 8)
|
|
||||||
|
|
||||||
const index = await reopen(storage)
|
|
||||||
expect(index.watermarkVerdict()).toBe('catchup')
|
|
||||||
|
|
||||||
const rebuildSpy = vi.spyOn(index, 'rebuild')
|
|
||||||
const result = await index.applyWatermarkCatchup(null) // no scan — no fact log
|
|
||||||
|
|
||||||
expect(result.action).toBe('rescan')
|
|
||||||
expect(result.reason).toContain('no fact log')
|
|
||||||
expect(rebuildSpy).toHaveBeenCalledTimes(1)
|
|
||||||
})
|
|
||||||
})
|
|
||||||
|
|
|
||||||
|
|
@ -3,14 +3,9 @@
|
||||||
* @description Pattern-A / Finding 1: a pure semantic find({ query }) has no
|
* @description Pattern-A / Finding 1: a pure semantic find({ query }) has no
|
||||||
* filter, so verifyMetadataLive never fires — nothing guarded the vector index.
|
* filter, so verifyMetadataLive never fires — nothing guarded the vector index.
|
||||||
* A cold native vector index that loaded its COUNT but not its serving structure
|
* A cold native vector index that loaded its COUNT but not its serving structure
|
||||||
* returned a silent []. verifyVectorLive() closes that: the health-report/isReady()
|
* returned a silent []. verifyVectorLive() closes that: honest isReady() first,
|
||||||
* authority first, else a known-vector self-match probe.
|
* else a known-vector self-match probe; self-heal (rebuild) or throw
|
||||||
*
|
* VectorIndexNotReadyError — never a silent empty result.
|
||||||
* RE-POINTED to the health-gate law: the guard NEVER rebuilds and NEVER walks
|
|
||||||
* the store from a read — a read-path rebuild is exactly the dark-rebuild
|
|
||||||
* failure mode the law retires (open() alone owns building). A not-serving
|
|
||||||
* signal (from either strategy) THROWS VectorIndexNotReadyError immediately,
|
|
||||||
* with no rebuild attempt in between — never a silent empty result.
|
|
||||||
*/
|
*/
|
||||||
import { describe, it, expect, beforeEach } from 'vitest'
|
import { describe, it, expect, beforeEach } from 'vitest'
|
||||||
import { Brainy, NounType, VectorIndexNotReadyError } from '../../src/index.js'
|
import { Brainy, NounType, VectorIndexNotReadyError } from '../../src/index.js'
|
||||||
|
|
@ -39,37 +34,50 @@ describe('Vector cold-read guard (verifyVectorLive) — silent-[] on cold semant
|
||||||
vi.rebuild = origRebuild
|
vi.rebuild = origRebuild
|
||||||
})
|
})
|
||||||
|
|
||||||
it('cold index (no isReady()): verifyVectorLive REFUSES immediately — throws VectorIndexNotReadyError, NEVER rebuilds', async () => {
|
it('cold index: verifyVectorLive self-heals via rebuild — semantic find is correct, NOT silent []', async () => {
|
||||||
const vi = brain.index
|
const vi = brain.index
|
||||||
const origSearch = vi.search.bind(vi)
|
const origSearch = vi.search.bind(vi)
|
||||||
let rebuilds = 0
|
|
||||||
const origRebuild = vi.rebuild.bind(vi)
|
const origRebuild = vi.rebuild.bind(vi)
|
||||||
|
let cold = true
|
||||||
brain._vectorVerified = false
|
brain._vectorVerified = false
|
||||||
// size()>0 (count present) but search never returns a hit for the known vector.
|
// size()>0 (count present) but search returns nothing until a rebuild warms it.
|
||||||
vi.search = async () => []
|
vi.search = async (...a: any[]) => (cold ? [] : origSearch(...a))
|
||||||
vi.rebuild = async (...a: any[]) => { rebuilds++; return origRebuild(...a) }
|
vi.rebuild = async (...a: any[]) => { await origRebuild(...a); cold = false }
|
||||||
try {
|
try {
|
||||||
await expect(
|
const res = await brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
|
||||||
brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
|
expect(res.length).toBeGreaterThan(0) // self-healed
|
||||||
).rejects.toBeInstanceOf(VectorIndexNotReadyError)
|
|
||||||
expect(rebuilds).toBe(0) // the guard never rebuilds from a read — it refuses loudly instead
|
|
||||||
} finally {
|
} finally {
|
||||||
vi.search = origSearch; vi.rebuild = origRebuild
|
vi.search = origSearch; vi.rebuild = origRebuild
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
|
|
||||||
it('native provider reporting isReady()===false THROWS immediately — never rebuilds', async () => {
|
it('unrecoverably cold index: semantic find throws VectorIndexNotReadyError', async () => {
|
||||||
const vi = brain.index
|
const vi = brain.index
|
||||||
let rebuilds = 0
|
const origSearch = vi.search.bind(vi)
|
||||||
const origRebuild = vi.rebuild.bind(vi)
|
const origRebuild = vi.rebuild.bind(vi)
|
||||||
brain._vectorVerified = false
|
brain._vectorVerified = false
|
||||||
vi.isReady = () => false
|
vi.search = async () => [] // always cold; rebuild can't fix it
|
||||||
vi.rebuild = async (...a: any[]) => { rebuilds++; return origRebuild(...a) }
|
vi.rebuild = async () => {}
|
||||||
try {
|
try {
|
||||||
await expect(
|
await expect(
|
||||||
brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
|
brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
|
||||||
).rejects.toBeInstanceOf(VectorIndexNotReadyError)
|
).rejects.toBeInstanceOf(VectorIndexNotReadyError)
|
||||||
expect(rebuilds).toBe(0) // a not-ready report throws immediately — it is never a rebuild trigger
|
} finally {
|
||||||
|
vi.search = origSearch; vi.rebuild = origRebuild
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('native provider reporting isReady()===false rebuilds, then serves', async () => {
|
||||||
|
const vi = brain.index
|
||||||
|
const origRebuild = vi.rebuild.bind(vi)
|
||||||
|
let ready = false
|
||||||
|
brain._vectorVerified = false
|
||||||
|
vi.isReady = () => ready
|
||||||
|
vi.rebuild = async (...a: any[]) => { await origRebuild(...a); ready = true }
|
||||||
|
try {
|
||||||
|
const res = await brain.find({ query: 'x', searchMode: 'semantic', limit: 100 })
|
||||||
|
expect(ready).toBe(true) // rebuild ran because isReady() was false
|
||||||
|
expect(res).toBeDefined()
|
||||||
} finally {
|
} finally {
|
||||||
delete vi.isReady; vi.rebuild = origRebuild
|
delete vi.isReady; vi.rebuild = origRebuild
|
||||||
}
|
}
|
||||||
|
|
|
||||||
Reference in a new issue