From 9a5a9cccbcab8d67e475df13458e5c9a4081e9a9 Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Wed, 22 Jul 2026 16:31:45 +0200
Subject: [PATCH 01/15] ci: run the pipeline on the forge
---
.forgejo/workflows/ci.yml | 40 +++++++++++++++++++++++++++++++++++++++
1 file changed, 40 insertions(+)
create mode 100644 .forgejo/workflows/ci.yml
diff --git a/.forgejo/workflows/ci.yml b/.forgejo/workflows/ci.yml
new file mode 100644
index 00000000..cdb2ab14
--- /dev/null
+++ b/.forgejo/workflows/ci.yml
@@ -0,0 +1,40 @@
+name: CI
+
+on:
+ push:
+ pull_request:
+
+jobs:
+ node:
+ name: Node ${{ matrix.node-version }}
+ runs-on: ubuntu-latest
+ strategy:
+ fail-fast: false
+ matrix:
+ node-version: ['22', '24']
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-node@v4
+ with:
+ node-version: ${{ matrix.node-version }}
+ cache: npm
+ - run: npm ci
+ - run: npm run test:unit
+
+ bun:
+ name: Bun (latest)
+ runs-on: ubuntu-latest
+ steps:
+ - uses: actions/checkout@v4
+ - uses: actions/setup-node@v4
+ with:
+ node-version: '22'
+ cache: npm
+ - uses: oven-sh/setup-bun@v2
+ with:
+ bun-version: latest
+ - run: npm ci
+ # test:bun imports the built dist/, so build first.
+ - run: npm run build
+ # Bun as a runtime is the supported Bun story (`bun add` / `bun run`).
+ - run: npm run test:bun
From 16832671fb5856294d79851f4feffa20790b3f76 Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Wed, 22 Jul 2026 16:42:26 -0700
Subject: [PATCH 02/15] feat: warm contract (warm/warmOnOpen/provider warm
hook), configurable transact budget floor, backend-neutral vector index op
names
Three pieces addressing the cold-restart-write incident where a production
deployment's first writes after every restart (33-35s each on a cold page
cache) blew the op-count-scaled transact budget mid-batch: every write is
itself a multi-op transaction, so one cold operation consumed the whole
budget, the gate before the next operation tripped, and the write rolled
back atomically - refused, retried, and refused again until the page cache
warmed passively.
- The budget's start-gating contract is now explicit and pinned: it gates
STARTING the next operation, never rolling back completed work for
elapsed time (the shipped schedule since 8.7.0, now stated in contract
JSDoc, guarded by code for operation 0, and enforced by regression
tests). The 30s floor is configurable via transactionBudgetFloorMs for
stores whose cold operations legitimately run long.
- New brain.warm() eagerly loads the vector index, metadata index, and
graph adjacency so first operations after a cold restart run at
steady-state cost. Returns a WarmReport with an honest per-surface
outcome (warmed / probed / unavailable) - never reports a probe as a
warm. warmOnOpen: true runs it during init(). New optional provider hook
warm() on the vector and graph plugin contracts.
- Vector-index transaction op classes renamed from the backend-specific
AddToHNSWOperation / RemoveFromHNSWOperation to backend-neutral
AddToVectorIndexOperation / RemoveFromVectorIndexOperation, stamping the
active backend into the emitted op-name string (AddToVectorIndex(js-hnsw)
vs a native provider's own identity) so journals never misdirect an
operator toward an index that isn't running.
---
RELEASES.md | 56 ++++
src/brainy.ts | 205 +++++++++++-
src/hnsw/hnswIndex.ts | 3 +
src/index.ts | 3 +
src/plugin.ts | 44 +++
src/transaction/Transaction.ts | 74 ++++-
src/transaction/operations/IndexOperations.ts | 79 +++--
src/transaction/operations/index.ts | 6 +-
src/types/brainy.types.ts | 36 ++
src/utils/metadataIndex.ts | 40 +++
tests/unit/brainy/warm.test.ts | 309 ++++++++++++++++++
.../budget-commit-completed-work.test.ts | 149 +++++++++
.../vectorIndexOperations-rename.test.ts | 152 +++++++++
13 files changed, 1099 insertions(+), 57 deletions(-)
create mode 100644 tests/unit/brainy/warm.test.ts
create mode 100644 tests/unit/transaction/budget-commit-completed-work.test.ts
create mode 100644 tests/unit/transaction/vectorIndexOperations-rename.test.ts
diff --git a/RELEASES.md b/RELEASES.md
index 7799c6f6..113b327d 100644
--- a/RELEASES.md
+++ b/RELEASES.md
@@ -31,6 +31,62 @@ is sometimes cited as a 7.x removal — those methods never existed on 7.x; the
---
+## Unreleased (the warm contract: cold-restart writes stop paying demand-load latency)
+
+From a production deployment's cold-restart incident: the FIRST writes after every
+restart on a large brain measured 33–35s each (page cache cold) against the transact
+apply budget — Brainy's own op-count-scaled budget, `max(30s, opCount × 2s)`, e.g.
+32,000ms for a 16-op batch. There is no external deadline in this story, and no
+post-completion veto either: every write is itself a multi-operation transaction (a
+single `add()` applies several operations — canonical writes, the vector-index insert,
+the metadata-index update), so ONE cold operation that legitimately runs ~33s consumes
+the whole budget, the gate before the NEXT operation trips, and the write rolls back
+atomically (zero loss, by design) — refused, retried, refused again, until the page
+cache warms passively (~30 minutes). The cure is not weaker atomicity; it is the two
+new knobs below — a budget floor sized for cold stores, and a warm contract that pays
+demand-load cost OFF the transaction path.
+
+- **The budget's start-gating contract is now explicit, documented, and pinned by
+ regression tests.** The budget gates STARTING the next operation — completed work is
+ never rolled back for elapsed time — and a transaction's first operation now
+ unconditionally starts by code, not merely because elapsed time happens to be ~0 when
+ it is checked. This has been the shipped schedule since 8.7.0 (no behavior change for
+ existing integrations); it is now stated in `Transaction.execute()`'s contract JSDoc
+ and enforced by tests so it cannot silently regress. Mid-batch atomicity is unchanged:
+ a trip before operation `i+1` still rolls back `0..i` and throws a retryable
+ `TransactionTimeoutError`.
+- **The budget's 30s floor is now configurable**: `new Brainy({ transactionBudgetFloorMs })`
+ raises (or lowers) the floor of `max(transactionBudgetFloorMs, opCount × 2000)` for every
+ internal transact batch. Useful for a store whose cold writes legitimately run past 30s
+ per operation, so a bulk batch gets a proportionally larger runway instead of tripping
+ mid-batch on cold-cache latency.
+- **New: `brain.warm()`** — eagerly loads/faults-in the vector index, metadata index, and
+ graph adjacency so the first real operation after a cold restart runs at steady-state
+ cost instead of paying demand-load latency on the critical path. Returns a `WarmReport`
+ with one honest outcome per surface — never conflate the first two:
+ - `'warmed'` — the surface's own provider `warm()` hook ran, or a full-hydration seam
+ loaded every shard/field/segment from storage. Steady-state cost is paid.
+ - `'probed'` — no `warm()` hook was available, so a best-effort read (one `search()` call
+ for the vector index) faulted in *some* backing storage as a side effect — real work,
+ but never reported as `'warmed'`.
+ - `'unavailable'` — nothing ran (no hook, no hydration seam, or nothing to probe).
+- **New config: `warmOnOpen: true`** makes `init()` await `brain.warm()` before it resolves
+ — a deliberate blocking trade-off: startup takes longer, the first request doesn't.
+ Default `false` (unchanged lazy behavior).
+- **New optional provider hook: `warm?(): Promise`** on the vector and graph
+ acceleration provider contracts (`src/plugin.ts`) — a native provider can implement it to
+ eagerly pretouch its own backing storage (e.g. mmap pretouch); absence means brainy falls
+ back to the probe/hydration behavior above.
+- **Transaction op-name strings changed in journals/timings**: the vector-index
+ transaction operations were renamed from `AddToHNSW`/`RemoveFromHNSW` to backend-neutral
+ `AddToVectorIndex(...)`/`RemoveFromVectorIndex(...)` — the old names hard-coded an
+ algorithm that may not be the one actually running (a non-HNSW native vector provider
+ emitting `"RemoveFromHNSW"` has sent an operator hunting an index that doesn't exist).
+ The parenthesized suffix names the ACTIVE backend: `js-hnsw` for the built-in engine, or
+ the native provider's own identity when it self-identifies. **If you parse these op-name
+ strings** (log processors, journal tooling), update your matcher from
+ `AddToHNSW`/`RemoveFromHNSW` to `AddToVectorIndex(`/`RemoveFromVectorIndex(`.
+
## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close())
The write path stops paying maintenance costs — the last structural piece of the
diff --git a/src/brainy.ts b/src/brainy.ts
index 9ab3acee..fd9e1ffa 100644
--- a/src/brainy.ts
+++ b/src/brainy.ts
@@ -63,7 +63,9 @@ import type {
PathOptions,
MetadataIndexProvider,
OpaqueIdSet,
- AtGenerationVectors
+ AtGenerationVectors,
+ VectorIndexProvider,
+ GraphIndexProvider
} from './plugin.js'
import type {
BrainyPlugin,
@@ -88,12 +90,12 @@ import { findCallerLocation } from './utils/callerLocation.js'
import {
SaveNounMetadataOperation,
SaveNounOperation,
- AddToHNSWOperation,
+ AddToVectorIndexOperation,
AddToMetadataIndexOperation,
SaveVerbMetadataOperation,
SaveVerbOperation,
AddToGraphIndexOperation,
- RemoveFromHNSWOperation,
+ RemoveFromVectorIndexOperation,
RemoveFromMetadataIndexOperation,
RemoveFromGraphIndexOperation,
UpdateNounMetadataOperation,
@@ -266,6 +268,7 @@ type ResolvedBrainyConfig = Required<
| 'retention'
| 'eagerEmbeddings'
| 'migrationWaitTimeoutMs'
+ | 'transactionBudgetFloorMs'
>
> &
Pick<
@@ -278,6 +281,7 @@ type ResolvedBrainyConfig = Required<
| 'retention'
| 'eagerEmbeddings'
| 'migrationWaitTimeoutMs'
+ | 'transactionBudgetFloorMs'
>
/**
@@ -388,6 +392,38 @@ class InsertPreconditionExistsSignal extends Error {
*/
export type IndexFamily = 'vector' | 'metadata' | 'graph'
+/**
+ * @description Honest per-surface outcome for {@link Brainy.warm}. Literal
+ * meanings — never conflate the first two:
+ * - `'warmed'` — the provider's own `warm?()` hook ran (vector/graph), or the
+ * surface's full-hydration seam loaded EVERY shard/field/segment from
+ * storage (metadata; graph's fallback path). The surface is genuinely at
+ * steady-state cost for the next operation.
+ * - `'probed'` — no `warm?()` hook was available, so a best-effort read
+ * (e.g. one `search()` call) faulted in *some* backing storage as a side
+ * effect. Real work happened, but it is NOT the same guarantee as
+ * `'warmed'` — never reported as `'warmed'`.
+ * - `'unavailable'` — nothing ran: no hook, no hydration seam, and (for the
+ * vector probe fallback) nothing to probe (an empty index or unknown
+ * vector dimension). The surface is unchanged by this `warm()` call.
+ */
+export type WarmOutcome = 'warmed' | 'probed' | 'unavailable'
+
+/**
+ * @description Result of {@link Brainy.warm}: one {@link WarmOutcome} +
+ * elapsed time per index surface, plus the total wall-clock time for the
+ * whole call. `durationMs` is measured around exactly the work described by
+ * that surface's `outcome` (e.g. the vector entry's `durationMs` times the
+ * provider `warm()` call OR the probe `search()` call — whichever ran).
+ */
+export interface WarmReport {
+ vector: { outcome: WarmOutcome; durationMs: number }
+ metadata: { outcome: WarmOutcome; durationMs: number }
+ graph: { outcome: WarmOutcome; durationMs: number }
+ /** Total wall-clock time for the whole `warm()` call (all three surfaces). */
+ totalDurationMs: number
+}
+
/**
* How long a failed aggregation-backfill walk suppresses fresh walk attempts.
* Within the window, queries rethrow the recorded failure instantly (loud,
@@ -1382,6 +1418,17 @@ export class Brainy implements BrainyInterface {
})
}
+ // Eager index warm (operator opt-in, `warmOnOpen: true`). Runs AFTER
+ // every step above — index construction, crash recovery, migrations,
+ // VFS bootstrap — never in place of any of it, so `warm()` always
+ // operates on a fully-initialized brain. Blocking BY DESIGN: the
+ // operator traded a longer startup for a first-request that runs at
+ // steady-state cost instead of paying demand-load latency on the
+ // critical path. See the `warmOnOpen` JSDoc in brainy.types.ts.
+ if (this.config.warmOnOpen) {
+ await this.warm()
+ }
+
// Resolve ready Promise - consumers awaiting brain.ready will now proceed
if (this._readyResolve) {
this._readyResolve()
@@ -1780,7 +1827,9 @@ export class Brainy implements BrainyInterface {
await this.generationStore.runWithoutGeneration(() =>
this.transactionManager.executeTransaction(run, {
timeout: transactTimeoutBudget(
- (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0)
+ (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0),
+ undefined,
+ this.config.transactionBudgetFloorMs
)
})
)
@@ -1797,7 +1846,9 @@ export class Brainy implements BrainyInterface {
execute: () =>
this.transactionManager.executeTransaction(run, {
timeout: transactTimeoutBudget(
- (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0)
+ (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0),
+ undefined,
+ this.config.transactionBudgetFloorMs
)
})
})
@@ -2109,7 +2160,7 @@ export class Brainy implements BrainyInterface {
// Operation 3: Add to HNSW index (after entity saved)
tx.addOperation(
- new AddToHNSWOperation(this.index, id, vector)
+ new AddToVectorIndexOperation(this.index, id, vector)
)
// Operation 4: Add to metadata index
@@ -3098,10 +3149,10 @@ export class Brainy implements BrainyInterface {
// Operation 3-4: Update HNSW index (remove and re-add if reindexing needed)
if (needsReindexing) {
tx.addOperation(
- new RemoveFromHNSWOperation(this.index, params.id, existing.vector)
+ new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector)
)
tx.addOperation(
- new AddToHNSWOperation(this.index, params.id, vector)
+ new AddToVectorIndexOperation(this.index, params.id, vector)
)
}
@@ -3217,7 +3268,7 @@ export class Brainy implements BrainyInterface {
// Operation 1: Remove from vector index
if (noun) {
tx.addOperation(
- new RemoveFromHNSWOperation(this.index, id, noun.vector)
+ new RemoveFromVectorIndexOperation(this.index, id, noun.vector)
)
}
@@ -7025,7 +7076,7 @@ export class Brainy implements BrainyInterface {
// Add delete operations to transaction
if (noun) {
tx.addOperation(
- new RemoveFromHNSWOperation(this.index, id, noun.vector)
+ new RemoveFromVectorIndexOperation(this.index, id, noun.vector)
)
}
@@ -7888,7 +7939,13 @@ export class Brainy implements BrainyInterface {
// Budget scales with the batch (or the caller's explicit
// timeoutMs): a flat 30s cap silently limited honest bulk work
// to ~15 ops on network disks (~2s/op measured in the field).
- { timeout: transactTimeoutBudget(plan.operations.length, options?.timeoutMs) }
+ {
+ timeout: transactTimeoutBudget(
+ plan.operations.length,
+ options?.timeoutMs,
+ this.config.transactionBudgetFloorMs
+ )
+ }
)
}
}))
@@ -9311,7 +9368,7 @@ export class Brainy implements BrainyInterface {
plan.operations.push(
new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew),
new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew),
- new AddToHNSWOperation(this.index, id, vector),
+ new AddToVectorIndexOperation(this.index, id, vector),
new AddToMetadataIndexOperation(this.metadataIndex, id, entityForIndexing)
)
plan.touchedNouns.push(id)
@@ -9479,8 +9536,8 @@ export class Brainy implements BrainyInterface {
)
if (needsReindexing) {
plan.operations.push(
- new RemoveFromHNSWOperation(this.index, params.id, existing.vector),
- new AddToHNSWOperation(this.index, params.id, vector)
+ new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector),
+ new AddToVectorIndexOperation(this.index, params.id, vector)
)
}
plan.operations.push(
@@ -9565,7 +9622,7 @@ export class Brainy implements BrainyInterface {
}
if (noun) {
- plan.operations.push(new RemoveFromHNSWOperation(this.index, id, noun.vector))
+ plan.operations.push(new RemoveFromVectorIndexOperation(this.index, id, noun.vector))
}
if (metadata) {
plan.operations.push(new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata))
@@ -14280,6 +14337,118 @@ export class Brainy implements BrainyInterface {
return this.embedder(textToEmbed)
}
+ /**
+ * Eagerly load/fault-in the vector index, metadata index, and graph
+ * adjacency so the FIRST real operation after this call runs at
+ * steady-state cost — no page-cache-miss / demand-load latency on the
+ * critical path. Complements {@link warmupEmbeddings} (which warms the
+ * embedding engine, not the storage indexes).
+ *
+ * Sequence per surface:
+ * - **Vector**: calls the provider's own `warm?()` when the active
+ * `'vector'` provider implements it (`'warmed'`). Otherwise runs a
+ * best-effort probe — one `search()` with a deterministic unit vector of
+ * the index's known dimension, `k = min(100, size())` — which faults in
+ * *some* backing storage as a side effect but is reported honestly as
+ * `'probed'`, never `'warmed'`. An empty index or unknown dimension has
+ * nothing to probe (`'unavailable'`).
+ * - **Metadata**: full hydration — every persisted field's sparse index is
+ * loaded from storage (`MetadataIndexManager.hydrateAll()`), not just the
+ * heuristic common-fields subset `init()` warms.
+ * - **Graph**: calls the provider's own `warm?()` when the active graph
+ * provider implements it; otherwise re-runs its existing eager cold-load
+ * `init()` seam (idempotent — the JS adjacency index's `init()` already
+ * loads the full LSM manifests + SSTables, so re-invoking it here is a
+ * genuine full hydration, not a stat call).
+ *
+ * A single surface's `'unavailable'`/probe outcome never fails the whole
+ * call — `warm()` is a best-effort readiness step, not a correctness gate;
+ * queries still demand-load normally regardless of what `warm()` achieved.
+ *
+ * @returns A {@link WarmReport}: honest per-surface outcome + timing.
+ * @example
+ * ```typescript
+ * const brain = new Brainy({ storage: { path: '/data' } })
+ * await brain.init()
+ * const report = await brain.warm() // blocking, explicit control over timing
+ * console.log(report.vector.outcome, report.vector.durationMs)
+ *
+ * // Or opt into the same work automatically during init():
+ * const eager = new Brainy({ storage: { path: '/data' }, warmOnOpen: true })
+ * await eager.init() // warm() already ran before this resolves
+ * ```
+ */
+ async warm(): Promise {
+ await this.ensureInitialized({ needs: ['vector', 'metadata', 'graph'] })
+
+ const totalStart = Date.now()
+
+ // --- Vector ---------------------------------------------------------
+ const vectorStart = Date.now()
+ let vectorOutcome: WarmOutcome
+ const vectorProvider = this.index as VectorIndexProvider & { warm?: () => Promise }
+ if (typeof vectorProvider.warm === 'function') {
+ await vectorProvider.warm()
+ vectorOutcome = 'warmed'
+ } else {
+ const size = this.index.size()
+ const dimension = this.dimensions
+ if (size > 0 && dimension && dimension > 0) {
+ // A deterministic unit vector (L2 norm 1) — same probe every call,
+ // no reliance on stored data shape.
+ const probeVector = new Array(dimension).fill(1 / Math.sqrt(dimension))
+ await this.index.search(probeVector, Math.min(100, size))
+ vectorOutcome = 'probed'
+ } else {
+ // Nothing to probe: an empty index, or the vector dimension is not
+ // yet known (no entity has ever been added).
+ vectorOutcome = 'unavailable'
+ }
+ }
+ const vectorDurationMs = Date.now() - vectorStart
+
+ // --- Metadata --------------------------------------------------------
+ const metadataStart = Date.now()
+ let metadataOutcome: WarmOutcome
+ const metadataWithHydrate = this.metadataIndex as unknown as { hydrateAll?: () => Promise }
+ if (typeof metadataWithHydrate.hydrateAll === 'function') {
+ await metadataWithHydrate.hydrateAll()
+ metadataOutcome = 'warmed'
+ } else {
+ // No hydration seam on this metadata provider — nothing to run.
+ metadataOutcome = 'unavailable'
+ }
+ const metadataDurationMs = Date.now() - metadataStart
+
+ // --- Graph -------------------------------------------------------------
+ const graphStart = Date.now()
+ let graphOutcome: WarmOutcome
+ const graphProvider = this.graphIndex as GraphIndexProvider & {
+ warm?: () => Promise
+ init?: () => Promise
+ }
+ if (typeof graphProvider.warm === 'function') {
+ await graphProvider.warm()
+ graphOutcome = 'warmed'
+ } else if (typeof graphProvider.init === 'function') {
+ // Existing eager cold-load seam (readiness contract): idempotent, and
+ // for the JS adjacency index it's already a genuine full load of the
+ // LSM manifests + SSTables — a real hydration, not a stat call.
+ await graphProvider.init()
+ graphOutcome = 'warmed'
+ } else {
+ graphOutcome = 'unavailable'
+ }
+ const graphDurationMs = Date.now() - graphStart
+
+ return {
+ vector: { outcome: vectorOutcome, durationMs: vectorDurationMs },
+ metadata: { outcome: metadataOutcome, durationMs: metadataDurationMs },
+ graph: { outcome: graphOutcome, durationMs: graphDurationMs },
+ totalDurationMs: Date.now() - totalStart
+ }
+ }
+
/**
* Explicitly warm up the embedding engine
*
@@ -14798,6 +14967,9 @@ export class Brainy implements BrainyInterface {
// Migration LOCK wait budget — left undefined when omitted so
// awaitMigrationLock() applies its 30 s default at the read site.
migrationWaitTimeoutMs: config?.migrationWaitTimeoutMs ?? undefined,
+ // Apply-phase transaction budget floor — left undefined when omitted so
+ // transactTimeoutBudget() applies its 30 s default at the read site.
+ transactionBudgetFloorMs: config?.transactionBudgetFloorMs ?? undefined,
// Pre-upgrade backup — default-on; opt out with `migrationBackup: false`.
migrationBackup: config?.migrationBackup ?? true,
// Vector index configuration (8.0) — algorithm-neutral surface with
@@ -14813,6 +14985,9 @@ export class Brainy implements BrainyInterface {
// is the active one on a non-reader writer outside unit tests). Explicit
// true/false always wins. See the eagerEmbeddings JSDoc in brainy.types.ts.
eagerEmbeddings: config?.eagerEmbeddings ?? undefined,
+ // Eager index warm at init() — default off (lazy demand-load), the
+ // pre-8.9 behavior. See the warmOnOpen JSDoc in brainy.types.ts.
+ warmOnOpen: config?.warmOnOpen ?? false,
// Plugin configuration - undefined = auto-detect
plugins: config?.plugins ?? undefined,
// Integration Hub - undefined/false = disabled
diff --git a/src/hnsw/hnswIndex.ts b/src/hnsw/hnswIndex.ts
index 605d2a0b..dcf3ed7b 100644
--- a/src/hnsw/hnswIndex.ts
+++ b/src/hnsw/hnswIndex.ts
@@ -54,6 +54,9 @@ export class HnswFlushError extends Error {
* acceleration provider).
*/
export class JsHnswVectorIndex implements VectorIndexProvider {
+ /** Self-identifies as the built-in JS fallback engine — see {@link VectorIndexProvider.providerId}. */
+ readonly providerId = 'js-hnsw'
+
private nouns: Map = new Map()
/**
* Reverse adjacency: `target id → (level → set of node ids that link TO target)`.
diff --git a/src/index.ts b/src/index.ts
index b978b9fd..bfab39da 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -28,6 +28,9 @@ export type { FileVersion } from './vfs/types.js'
// Export diagnostics result type
export type { DiagnosticsResult } from './brainy.js'
+// brain.warm() — eager index/storage readiness report (per-surface honest
+// outcome + timing). See the WarmReport JSDoc in brainy.ts.
+export type { WarmReport, WarmOutcome } from './brainy.js'
export type {
GraphAuditReport,
GraphAuditDiscrepancy
diff --git a/src/plugin.ts b/src/plugin.ts
index df7c7224..eeb2425a 100644
--- a/src/plugin.ts
+++ b/src/plugin.ts
@@ -381,6 +381,20 @@ export interface GraphIndexProvider {
*/
init?(): Promise
+ /**
+ * @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap
+ * pretouch) so first operations run at steady-state cost. Optional;
+ * absence means the provider demand-loads. Distinct from `init?()`: `init`
+ * is called once automatically during brain startup as part of the
+ * readiness contract (cold-load + rebuild gating); `warm` is an explicit,
+ * separate readiness step a caller opts into via `brain.warm()` (or
+ * `warmOnOpen`) specifically to pre-pay demand-load cost that `init` left
+ * lazy. A provider that already loads everything eagerly in `init?()` may
+ * implement `warm` as a no-op or omit it — `brain.warm()` falls back to a
+ * best-effort read-through probe when absent.
+ */
+ warm?(): Promise
+
/**
* @description OPTIONAL. A native provider returns true from the moment its
* `init()` detects a large epoch-drift until its background
@@ -947,6 +961,22 @@ export interface AtGenerationVectors {
* are retired and never looked up.
*/
export interface VectorIndexProvider {
+ /**
+ * @description OPTIONAL backend identity, stamped into the transaction
+ * op-name strings that surface in journals/timings (e.g.
+ * `AddToVectorIndex(js-hnsw)` vs `AddToVectorIndex()`) — see
+ * `src/transaction/operations/IndexOperations.ts`. Absent → the op-name
+ * string reports `unknown-provider` rather than guessing a backend name
+ * (guessing would resurrect the exact bug this field exists to prevent:
+ * an operator hunting an index that isn't the one actually running). The
+ * built-in JS index always sets this to `'js-hnsw'`; a native provider
+ * sets its own identity here (e.g. its plugin/package name) so operators
+ * reading a journal see the backend that actually ran, never a
+ * backend-specific fossil name from whichever engine wrote the original
+ * op classes.
+ */
+ readonly providerId?: string
+
addItem(item: VectorDocument): Promise
removeItem(id: string): Promise
search(
@@ -1009,6 +1039,20 @@ export interface VectorIndexProvider {
*/
init?(): Promise
+ /**
+ * @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap
+ * pretouch) so first operations run at steady-state cost. Optional;
+ * absence means the provider demand-loads. Distinct from `init?()`: `init`
+ * runs automatically once during brain startup (the cold-load + rebuild
+ * readiness contract above); `warm` is a separate, explicit step a caller
+ * opts into via `brain.warm()` (or `warmOnOpen`) to pre-pay demand-load
+ * cost `init` left lazy — e.g. touching every mmap page rather than just
+ * opening the file. A provider that already loads everything eagerly in
+ * `init?()` may implement `warm` as a no-op or omit it — `brain.warm()`
+ * falls back to a best-effort probe `search()` when absent.
+ */
+ warm?(): Promise
+
/**
* @description OPTIONAL honest durability signal (readiness contract,
* mirrors {@link GraphIndexProvider.isReady}). `true` ⇔ the persisted
diff --git a/src/transaction/Transaction.ts b/src/transaction/Transaction.ts
index 092a2a25..79b53006 100644
--- a/src/transaction/Transaction.ts
+++ b/src/transaction/Transaction.ts
@@ -37,22 +37,47 @@ const DEFAULT_OPTIONS: Required = {
maxRollbackRetries: 3
}
+/**
+ * The floor term of {@link transactTimeoutBudget}'s scaling formula when the
+ * caller supplies neither an explicit override nor `config.transactionBudgetFloorMs`.
+ */
+const DEFAULT_BUDGET_FLOOR_MS = 30_000
+
/**
* The apply-phase budget for a batch of `opCount` operations.
*
- * An explicit override wins untouched. Otherwise the budget SCALES with the
- * batch: `max(30 000 ms, opCount × 2 000 ms)`. The per-op term is calibrated
- * from field data — bulk imports on network-attached disks measure ~2 s per
- * operation (each op pays canonical writes + fsync + index maintenance) — so
- * a flat 30 s budget silently capped honest work at ~15 operations while
- * looking generous for small batches. Scaling keeps small transacts
- * fast-failing and gives bulk ones a budget proportional to the work they
- * actually asked for; a trip still rolls back atomically and throws a
- * retryable, fully-labeled TransactionTimeoutError.
+ * An explicit `override` wins untouched (a caller-specified deadline for this
+ * one batch). Otherwise the budget SCALES with the batch:
+ * `max(floorMs, opCount × 2 000)`, where `floorMs` defaults to `30 000` and
+ * is overridable via `BrainyConfig.transactionBudgetFloorMs` for the whole
+ * brain. The per-op term is calibrated from field data — bulk imports on
+ * network-attached disks measure ~2 s per operation (each op pays canonical
+ * writes + fsync + index maintenance) — so a flat 30 s budget silently capped
+ * honest work at ~15 operations while looking generous for small batches. A
+ * 16-op batch, for example, scales to `max(30 000, 16 × 2 000) = 32 000`.
+ * Scaling keeps small transacts fast-failing and gives bulk ones a budget
+ * proportional to the work they actually asked for.
+ *
+ * This is a **mid-batch** budget only: it governs whether the transaction's
+ * NEXT operation may start (see {@link Transaction.execute}), never whether
+ * already-completed work is rolled back after the fact. A trip mid-batch
+ * still rolls back every operation applied so far, atomically, and throws a
+ * retryable, fully-labeled TransactionTimeoutError — that zero-loss guarantee
+ * doesn't change; only the point at which the clock stops mattering does (at
+ * the last operation, not one check later).
+ *
+ * @param opCount - Number of operations in the batch.
+ * @param override - A full override for this call; wins over everything else.
+ * @param floorMs - The scaling floor for this call (e.g. `config.transactionBudgetFloorMs`).
+ * Ignored when `override` is set. Defaults to `30 000` when omitted.
*/
-export function transactTimeoutBudget(opCount: number, override?: number): number {
+export function transactTimeoutBudget(
+ opCount: number,
+ override?: number,
+ floorMs?: number
+): number {
if (override !== undefined) return override
- return Math.max(30_000, opCount * 2_000)
+ return Math.max(floorMs ?? DEFAULT_BUDGET_FLOOR_MS, opCount * 2_000)
}
/**
@@ -98,7 +123,20 @@ export class Transaction implements TransactionContext {
}
/**
- * Execute all operations atomically
+ * Execute all operations atomically.
+ *
+ * Budget semantics: the configured budget gates starting the next
+ * operation; completed work is never rolled back for elapsed time. Elapsed
+ * time is checked ONLY before starting operation `i` for `i ≥ 1` — never
+ * before operation 0 (an operation always gets to start) and never again
+ * after the final operation completes. Concretely: a single-op transaction
+ * can never time out post-hoc — it either runs (and its result stands, no
+ * matter how long it took) or a `TransactionTimeoutError` is thrown before
+ * it starts, which cannot happen since there is no operation before it. A
+ * multi-op transaction whose op `i` overruns the budget stops op `i+1` from
+ * starting, rolls back operations `0..i` (reverse order), and throws
+ * `TransactionTimeoutError` — that zero-loss rollback guarantee for
+ * mid-batch trips is unchanged; only the post-completion check is gone.
*/
async execute(): Promise {
if (this.state !== 'pending') {
@@ -128,10 +166,14 @@ export class Transaction implements TransactionContext {
// already-applied writes as torn, generation-less state in canonical
// storage.
for (let i = 0; i < this.operations.length; i++) {
- // Budget check BEFORE starting the next operation. A trip here throws
- // into the catch below and rolls back like any other failure — it must
- // never bypass rollback.
- if (Date.now() - this.startTime > this.options.timeout) {
+ // Budget gates STARTING the next operation; completed work is never
+ // rolled back for elapsed time. Skipped for i === 0 (operation 0
+ // always gets to start — nothing has run yet to have overrun) and
+ // never re-checked after the final operation completes (there is no
+ // "next operation" left to gate). A trip here throws into the catch
+ // below and rolls back like any other failure — it must never bypass
+ // rollback.
+ if (i > 0 && Date.now() - this.startTime > this.options.timeout) {
throw new TransactionTimeoutError(this.options.timeout, i, {
elapsedMs: Date.now() - this.startTime,
totalOperations: this.operations.length,
diff --git a/src/transaction/operations/IndexOperations.ts b/src/transaction/operations/IndexOperations.ts
index 97c39270..1c0995c6 100644
--- a/src/transaction/operations/IndexOperations.ts
+++ b/src/transaction/operations/IndexOperations.ts
@@ -2,34 +2,58 @@
* Index Operations with Rollback Support
*
* Provides transactional operations for all indexes:
- * - JsHnswVectorIndex (unified vector index)
+ * - VectorIndexProvider (the JS HNSW fallback, or a native acceleration provider)
* - MetadataIndexManager (roaring bitmap filtering)
* - GraphAdjacencyIndex (LSM-tree graph storage)
*
* Each operation can be executed and rolled back atomically.
*/
-import type { JsHnswVectorIndex } from '../../hnsw/hnswIndex.js'
+import type { VectorIndexProvider, GraphIndexProvider } from '../../plugin.js'
import type { MetadataIndexManager } from '../../utils/metadataIndex.js'
-import type { GraphIndexProvider } from '../../plugin.js'
import type { GraphVerb } from '../../coreTypes.js'
import type { Operation, RollbackAction } from '../types.js'
/**
- * Add to HNSW index with rollback support
+ * Backend identity stamped into an operation's emitted `name` string (e.g.
+ * `AddToVectorIndex(js-hnsw)`), resolved from the provider's own
+ * {@link VectorIndexProvider.providerId} when it self-identifies.
+ *
+ * These operation classes are backend-neutral (the vector index they wrap may
+ * be Brainy's own JS HNSW fallback OR a native acceleration provider), but
+ * their names surface directly in consumer-visible transaction journals and
+ * timings. A provider that hasn't set `providerId` resolves to
+ * `'unknown-provider'` rather than guessing — silently defaulting to the JS
+ * engine's name here is exactly the class of bug this stamping exists to
+ * prevent (an operator diagnosing a backend that isn't the one that ran).
+ */
+function resolveVectorProviderId(index: VectorIndexProvider): string {
+ return index.providerId ?? 'unknown-provider'
+}
+
+/**
+ * Add to the vector index with rollback support.
+ *
+ * Backend-neutral: `index` is whatever the `'vector'` provider factory
+ * returns — Brainy's own JS HNSW fallback, or a native acceleration provider
+ * (e.g. DiskANN). The emitted `name` stamps the active backend (see
+ * {@link resolveVectorProviderId}) so operators reading a transaction journal
+ * or timing trace see which engine actually ran, never a fossil name from
+ * whichever engine happened to be active when this op class was written.
*
* Rollback strategy:
* - Remove item from index
-
*/
-export class AddToHNSWOperation implements Operation {
- readonly name = 'AddToHNSW'
+export class AddToVectorIndexOperation implements Operation {
+ readonly name: string
constructor(
- private readonly index: JsHnswVectorIndex,
+ private readonly index: VectorIndexProvider,
private readonly id: string,
private readonly vector: number[]
- ) {}
+ ) {
+ this.name = `AddToVectorIndex(${resolveVectorProviderId(index)})`
+ }
async execute(): Promise {
// Check if item already exists (for rollback decision)
@@ -59,11 +83,12 @@ export class AddToHNSWOperation implements Operation {
* pre-existence as "existed" made every rollback skip removeItem, leaving
* phantom entries in the index after a failed transaction. The safe default
* is to remove what this operation added — update flows pair this op with a
- * RemoveFromHNSWOperation whose own rollback restores the prior vector, so
- * reverse-order rollback reconstructs the original state either way.
+ * RemoveFromVectorIndexOperation whose own rollback restores the prior
+ * vector, so reverse-order rollback reconstructs the original state either
+ * way.
*/
private async itemExists(id: string): Promise {
- const index = this.index as JsHnswVectorIndex & {
+ const index = this.index as VectorIndexProvider & {
getItem?: (id: string) => Promise
}
if (typeof index.getItem !== 'function') return false
@@ -77,21 +102,27 @@ export class AddToHNSWOperation implements Operation {
}
/**
- * Remove from HNSW index with rollback support
+ * Remove from the vector index with rollback support.
+ *
+ * Backend-neutral: see {@link AddToVectorIndexOperation} — `index` may be the
+ * JS HNSW fallback or a native acceleration provider; the emitted `name`
+ * stamps the active backend.
*
* Rollback strategy:
* - Re-add item to index with original vector
*
* Note: Requires storing the vector for rollback
*/
-export class RemoveFromHNSWOperation implements Operation {
- readonly name = 'RemoveFromHNSW'
+export class RemoveFromVectorIndexOperation implements Operation {
+ readonly name: string
constructor(
- private readonly index: JsHnswVectorIndex,
+ private readonly index: VectorIndexProvider,
private readonly id: string,
private readonly vector: number[] // Required for rollback
- ) {}
+ ) {
+ this.name = `RemoveFromVectorIndex(${resolveVectorProviderId(index)})`
+ }
async execute(): Promise {
// Remove from index
@@ -285,22 +316,24 @@ export class RemoveFromGraphIndexOperation implements Operation {
}
/**
- * Batch operation: Add multiple items to HNSW index
+ * Batch operation: Add multiple items to the vector index (backend-neutral —
+ * see {@link AddToVectorIndexOperation}).
*
* Useful for bulk imports with transaction support.
* Rolls back all items if any fail.
*/
-export class BatchAddToHNSWOperation implements Operation {
- readonly name = 'BatchAddToHNSW'
+export class BatchAddToVectorIndexOperation implements Operation {
+ readonly name: string
- private operations: AddToHNSWOperation[]
+ private operations: AddToVectorIndexOperation[]
constructor(
- index: JsHnswVectorIndex,
+ index: VectorIndexProvider,
items: Array<{ id: string; vector: number[] }>
) {
+ this.name = `BatchAddToVectorIndex(${resolveVectorProviderId(index)})`
this.operations = items.map(
- item => new AddToHNSWOperation(index, item.id, item.vector)
+ item => new AddToVectorIndexOperation(index, item.id, item.vector)
)
}
diff --git a/src/transaction/operations/index.ts b/src/transaction/operations/index.ts
index 422f6dad..c5548e70 100644
--- a/src/transaction/operations/index.ts
+++ b/src/transaction/operations/index.ts
@@ -21,12 +21,12 @@ export {
// Index Operations
export {
- AddToHNSWOperation,
- RemoveFromHNSWOperation,
+ AddToVectorIndexOperation,
+ RemoveFromVectorIndexOperation,
AddToMetadataIndexOperation,
RemoveFromMetadataIndexOperation,
AddToGraphIndexOperation,
RemoveFromGraphIndexOperation,
- BatchAddToHNSWOperation,
+ BatchAddToVectorIndexOperation,
BatchAddToMetadataIndexOperation
} from './IndexOperations.js'
diff --git a/src/types/brainy.types.ts b/src/types/brainy.types.ts
index 1ec4c4c3..8bede8d3 100644
--- a/src/types/brainy.types.ts
+++ b/src/types/brainy.types.ts
@@ -1651,6 +1651,25 @@ export interface BrainyConfig {
*/
disableAutoRebuild?: boolean
+ /**
+ * The floor (ms) of the apply-phase transaction budget's scaling formula:
+ * `max(transactionBudgetFloorMs, opCount × 2 000)`. The budget governs
+ * whether a `transact()` / single-op write's NEXT internal operation may
+ * **start** — never whether already-completed work gets rolled back after
+ * the fact (a single-op write can never time out post-hoc: it either runs
+ * or it commits). A trip mid-batch still rolls back every applied operation
+ * atomically and throws a retryable `TransactionTimeoutError`; only the
+ * floor of the formula is configurable here.
+ *
+ * Raise this when a cold store's first writes after a restart legitimately
+ * take longer than 30s per operation (e.g. page-cache-cold canonical writes
+ * on a large brain) so a bulk `transact()` batch gets a proportionally
+ * larger runway instead of tripping mid-batch. Lower it to fail faster on a
+ * latency-sensitive write path. Default: `30000` (30 s) — unchanged from
+ * pre-8.9 behavior.
+ */
+ transactionBudgetFloorMs?: number
+
/**
* How long (ms) an operation **waits** on the coordinated 7.x → 8.0 migration
* before throwing a retryable `MigrationInProgressError`.
@@ -1806,6 +1825,23 @@ export interface BrainyConfig {
*/
eagerEmbeddings?: boolean
+ /**
+ * When `true`, `init()` **awaits `warm()`** (see {@link Brainy.warm}) before
+ * it resolves — blocking startup until the vector index, metadata index,
+ * and graph adjacency have all faulted their backing storage in. This is a
+ * deliberate operator opt-in trade: startup takes longer, but the FIRST
+ * request after a cold restart runs at steady-state cost instead of paying
+ * page-cache-miss / demand-load latency on the critical path.
+ *
+ * Runs AFTER `init()`'s own sequence completes (index construction, crash
+ * recovery, migrations, VFS bootstrap) — never in place of any of it — so
+ * `warm()` always operates on a fully-initialized brain.
+ *
+ * Default: `false` (lazy — the pre-8.9 behavior: indexes demand-load on
+ * first access).
+ */
+ warmOnOpen?: boolean
+
// Plugin configuration
// Controls which plugins are loaded during init().
// - undefined (default): guarded auto-detection of the first-party
diff --git a/src/utils/metadataIndex.ts b/src/utils/metadataIndex.ts
index c772bce4..fdb17c22 100644
--- a/src/utils/metadataIndex.ts
+++ b/src/utils/metadataIndex.ts
@@ -423,6 +423,46 @@ export class MetadataIndexManager implements MetadataIndexProvider {
prodLog.debug('✅ Type-aware cache warming completed')
}
+ /**
+ * Full hydration — the {@link Brainy.warm} readiness seam for the metadata
+ * index. Unlike {@link warmCache} / {@link warmCacheForTopTypes} (which
+ * warm only a heuristic subset: common fields plus the top-N types' top
+ * fields), this loads EVERY field's sparse index the field registry knows
+ * about — a real read through {@link loadSparseIndex} into the unified
+ * cache for each field, not a stat/existence check. Idempotent: an
+ * already-cached field's `loadSparseIndex` call is a cheap cache hit.
+ *
+ * Re-reads the field registry first when `fieldIndexes` is empty (a warm()
+ * call issued before `init()` populated it would otherwise hydrate
+ * nothing), then loads every discovered field in parallel.
+ */
+ async hydrateAll(): Promise {
+ if (this.fieldIndexes.size === 0) {
+ await this.loadFieldRegistry()
+ }
+
+ const fields = Array.from(this.fieldIndexes.keys())
+ if (fields.length === 0) {
+ prodLog.debug('[MetadataIndex] hydrateAll: no persisted fields to hydrate')
+ return
+ }
+
+ prodLog.debug(`[MetadataIndex] hydrateAll: loading ${fields.length} field(s) — ${fields.join(', ')}`)
+
+ await Promise.all(
+ fields.map(async field => {
+ try {
+ await this.loadSparseIndex(field)
+ } catch (error) {
+ // A single field's load failure doesn't abort the rest of the
+ // hydration — warm() is a best-effort readiness step, never a
+ // correctness gate (queries still demand-load on miss).
+ prodLog.debug(`[MetadataIndex] hydrateAll: field '${field}' failed to load:`, error)
+ }
+ })
+ )
+ }
+
/**
* Acquire an in-memory lock for coordinating concurrent metadata index writes
* Uses in-memory locks since MetadataIndexManager doesn't have direct file system access
diff --git a/tests/unit/brainy/warm.test.ts b/tests/unit/brainy/warm.test.ts
new file mode 100644
index 00000000..437b7785
--- /dev/null
+++ b/tests/unit/brainy/warm.test.ts
@@ -0,0 +1,309 @@
+/**
+ * @module tests/unit/brainy/warm
+ * @description Coverage for `brain.warm()` / `warmOnOpen` / the provider
+ * `warm?()` contract (the cold-restart readiness fix): first operations after
+ * a cold restart should run at steady-state cost instead of paying
+ * demand-load latency on the critical path.
+ *
+ * Uses a real, in-process fake plugin provider implementing the actual
+ * `VectorIndexProvider` contract from `src/plugin.ts` — the real seam a
+ * native provider (e.g. a disk-native accelerator) plugs into. The real
+ * built-in JS metadata index and graph adjacency index run against real
+ * (filesystem or in-memory) storage with pre-existing data, so their
+ * hydration paths are exercised for real, not mocked.
+ */
+import { describe, it, expect, afterEach } from 'vitest'
+import * as fs from 'node:fs'
+import * as os from 'node:os'
+import * as path from 'node:path'
+import { Brainy } from '../../../src/brainy.js'
+import { NounType, VerbType } from '../../../src/types/graphTypes.js'
+import type { VectorIndexProvider } from '../../../src/plugin.js'
+import type { VectorDocument, Vector } from '../../../src/coreTypes.js'
+import { MetadataIndexManager } from '../../../src/utils/metadataIndex.js'
+import { GraphAdjacencyIndex } from '../../../src/graph/graphAdjacencyIndex.js'
+
+const tmpDirs: string[] = []
+function mkTmp(): string {
+ const d = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-warm-'))
+ tmpDirs.push(d)
+ return d
+}
+afterEach(() => {
+ for (const d of tmpDirs.splice(0)) fs.rmSync(d, { recursive: true, force: true })
+})
+
+// Brainy's ValidationConfig fixes vectors at exactly 384 dimensions
+// (src/utils/paramValidation.ts) — match it so `add()` doesn't reject test data.
+const DIM = 384
+const V = (seed = 1): number[] => Array.from({ length: DIM }, (_, i) => Math.sin(seed + i))
+
+/**
+ * A real (not mocked) VectorIndexProvider implementation, backed by a plain
+ * Map, that optionally implements `warm()` — the exact seam
+ * `AddToVectorIndexOperation` / `brain.warm()` call through.
+ */
+class FakeVectorProvider implements VectorIndexProvider {
+ readonly items = new Map()
+ warmCalls = 0
+ searchCalls: Array<{ k?: number }> = []
+ // Only present on the instance when the constructor is told to — mirrors a
+ // real provider that may or may not implement the optional hook. Assigned
+ // in the constructor BODY (not as a field initializer): under native ES
+ // class fields, field initializers run before constructor-body statements
+ // — including the parameter-property assignment — so referencing a
+ // parameter property from a field initializer would see it as still
+ // `undefined`.
+ warm?: () => Promise
+
+ constructor(hasWarm: boolean) {
+ if (hasWarm) {
+ this.warm = async (): Promise => {
+ this.warmCalls++
+ }
+ }
+ }
+
+ async addItem(item: VectorDocument): Promise {
+ this.items.set(item.id, item.vector)
+ return item.id
+ }
+ async removeItem(id: string): Promise {
+ return this.items.delete(id)
+ }
+ async search(_queryVector: Vector, k?: number): Promise> {
+ this.searchCalls.push({ k })
+ return [...this.items.keys()].slice(0, k ?? 10).map((id) => [id, 0])
+ }
+ size(): number {
+ return this.items.size
+ }
+ clear(): void {
+ this.items.clear()
+ }
+ async rebuild(): Promise {}
+ async flush(): Promise {
+ return 0
+ }
+ getPersistMode(): 'immediate' | 'deferred' {
+ return 'deferred'
+ }
+}
+
+/** Registers `provider` under the `'vector'` plugin key, before `init()`. */
+function useFakeVectorProvider(brain: Brainy, provider: FakeVectorProvider): void {
+ brain.use({
+ name: 'fake-vector-provider',
+ activate: async (ctx: any) => {
+ ctx.registerProvider('vector', () => provider)
+ return true
+ }
+ })
+}
+
+describe('brain.warm()', () => {
+ it('(a) calls the vector provider\'s warm() when present and reports "warmed"', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ const provider = new FakeVectorProvider(true)
+ useFakeVectorProvider(brain, provider)
+ await brain.init()
+ await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
+
+ const report = await brain.warm()
+
+ expect(provider.warmCalls).toBe(1)
+ expect(provider.searchCalls.length).toBe(0) // warm() ran — no probe fallback
+ expect(report.vector.outcome).toBe('warmed')
+ await brain.close()
+ })
+
+ it('(b) falls back to a probe search() when warm() is absent and reports "probed", never "warmed"', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ const provider = new FakeVectorProvider(false)
+ useFakeVectorProvider(brain, provider)
+ await brain.init()
+ await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
+ await brain.add({ data: 'b', type: NounType.Thing, vector: V(2) })
+
+ const report = await brain.warm()
+
+ expect(provider.warmCalls).toBe(0) // no warm() on this provider
+ expect(provider.searchCalls.length).toBe(1) // the probe ran
+ expect(provider.searchCalls[0].k).toBe(Math.min(100, provider.size())) // k = min(100, size)
+ expect(report.vector.outcome).toBe('probed')
+ expect(report.vector.outcome).not.toBe('warmed') // never conflated
+ await brain.close()
+ })
+
+ it('vector: reports "unavailable" when there is nothing to probe (empty index, unknown dimension)', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ // A disk-native provider MAY honestly report 0 resident entries while
+ // durable data exists on disk (the same posture documented on
+ // `VectorIndexProvider.isReady` — "an mmap/disk-native index may
+ // legitimately report 0 resident entries"). warm()'s probe fallback
+ // reads `size()`, so this is the real trigger for "nothing to probe":
+ // never a k=0 search, an honest skip.
+ const provider = new FakeVectorProvider(false)
+ provider.size = () => 0
+ useFakeVectorProvider(brain, provider)
+ await brain.init() // the VFS root bootstrap write sets `dimensions`, but size() still reports 0
+
+ const report = await brain.warm()
+
+ expect(provider.searchCalls.length).toBe(0) // nothing probed
+ expect(report.vector.outcome).toBe('unavailable')
+ await brain.close()
+ })
+
+ it('(c) metadata + graph hydration paths actually execute against filesystem storage with pre-existing data', async () => {
+ const dir = mkTmp()
+
+ // Build a brain with real data (including a field NOT in the metadata
+ // index's own common-fields warm subset — 'wave' — so hydrateAll()'s
+ // FULL hydration is distinguishable from init()'s partial warmCache()),
+ // and real graph edges, then close it (persisting everything).
+ const seed = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'filesystem', path: dir },
+ silent: true
+ })
+ await seed.init()
+ const ids: string[] = []
+ for (let i = 0; i < 6; i++) {
+ ids.push(
+ await seed.add({
+ data: `entity ${i}`,
+ type: NounType.Thing,
+ metadata: { wave: i % 3 },
+ vector: V(i + 1)
+ })
+ )
+ }
+ for (let i = 0; i + 1 < ids.length; i++) {
+ await seed.relate({ from: ids[i], to: ids[i + 1], type: VerbType.RelatedTo })
+ }
+ await seed.close()
+
+ // Cold-reopen a FRESH instance and spy on the real hydration seams before
+ // calling warm(), so we assert they actually ran (not just that the
+ // report claims they did).
+ const metaHydrateCalls: number[] = []
+ const loadedFields: string[] = []
+ const graphInitCalls: number[] = []
+
+ const origHydrateAll = MetadataIndexManager.prototype.hydrateAll
+ const origLoadSparseIndex = (MetadataIndexManager.prototype as any).loadSparseIndex
+ const origGraphInit = GraphAdjacencyIndex.prototype.init
+
+ MetadataIndexManager.prototype.hydrateAll = async function (...args: any[]) {
+ metaHydrateCalls.push(1)
+ return origHydrateAll.apply(this, args as any)
+ }
+ ;(MetadataIndexManager.prototype as any).loadSparseIndex = async function (
+ field: string,
+ ...args: any[]
+ ) {
+ loadedFields.push(field)
+ return origLoadSparseIndex.apply(this, [field, ...args] as any)
+ }
+ GraphAdjacencyIndex.prototype.init = async function (...args: any[]) {
+ graphInitCalls.push(1)
+ return origGraphInit.apply(this, args as any)
+ }
+
+ try {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'filesystem', path: dir },
+ silent: true
+ })
+ await brain.init()
+
+ const report = await brain.warm()
+
+ expect(metaHydrateCalls.length).toBe(1) // hydrateAll() actually ran
+ expect(loadedFields).toContain('wave') // a NON-common field was loaded — full hydration, not the heuristic subset
+ expect(report.metadata.outcome).toBe('warmed')
+
+ expect(graphInitCalls.length).toBeGreaterThanOrEqual(1) // graph's full-load seam ran (once at brain init, once via warm())
+ expect(report.graph.outcome).toBe('warmed')
+
+ // Correctness survives: the hydrated data still answers queries.
+ const byWhere = await brain.find({ type: NounType.Thing, where: { wave: 1 } })
+ expect(byWhere.length).toBe(2) // waves 1,4 of 0..5
+
+ await brain.close()
+ } finally {
+ MetadataIndexManager.prototype.hydrateAll = origHydrateAll
+ ;(MetadataIndexManager.prototype as any).loadSparseIndex = origLoadSparseIndex
+ GraphAdjacencyIndex.prototype.init = origGraphInit
+ }
+ })
+
+ it('(d) warmOnOpen: true runs warm() during init() — observable via the fake provider', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ warmOnOpen: true,
+ silent: true
+ })
+ const provider = new FakeVectorProvider(true)
+ useFakeVectorProvider(brain, provider)
+
+ // No explicit brain.warm() call — warmOnOpen must have run it as part of init().
+ await brain.init()
+
+ expect(provider.warmCalls).toBe(1)
+ await brain.close()
+ })
+
+ it('warmOnOpen defaults to false — init() does NOT run warm() unless opted in', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ const provider = new FakeVectorProvider(true)
+ useFakeVectorProvider(brain, provider)
+
+ await brain.init()
+
+ expect(provider.warmCalls).toBe(0)
+ await brain.close()
+ })
+
+ it('(e) WarmReport shape: outcome literal + durationMs per surface, plus totalDurationMs', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ const provider = new FakeVectorProvider(true)
+ useFakeVectorProvider(brain, provider)
+ await brain.init()
+ await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
+
+ const report = await brain.warm()
+
+ for (const surface of ['vector', 'metadata', 'graph'] as const) {
+ expect(['warmed', 'probed', 'unavailable']).toContain(report[surface].outcome)
+ expect(typeof report[surface].durationMs).toBe('number')
+ expect(report[surface].durationMs).toBeGreaterThanOrEqual(0)
+ }
+ expect(typeof report.totalDurationMs).toBe('number')
+ expect(report.totalDurationMs).toBeGreaterThanOrEqual(0)
+ await brain.close()
+ })
+})
diff --git a/tests/unit/transaction/budget-commit-completed-work.test.ts b/tests/unit/transaction/budget-commit-completed-work.test.ts
new file mode 100644
index 00000000..a32489de
--- /dev/null
+++ b/tests/unit/transaction/budget-commit-completed-work.test.ts
@@ -0,0 +1,149 @@
+/**
+ * @module tests/unit/transaction/budget-commit-completed-work
+ * @description Regression coverage for the "commit-completed-work" transaction
+ * budget semantics: the budget gates STARTING the next operation; it never
+ * converts already-completed work into a rollback. Concretely:
+ *
+ * - A single-op transaction can never time out post-hoc — its one operation
+ * either runs (and the transaction commits, however long it took) or it
+ * never gets to start (which cannot happen: there is no operation before it
+ * to have overrun the budget).
+ * - A multi-op transaction whose op `i` overruns the budget stops op `i+1`
+ * from starting: everything applied so far rolls back atomically and a
+ * `TransactionTimeoutError` is thrown — the mid-batch zero-loss guarantee is
+ * unchanged.
+ * - `transactTimeoutBudget`'s scaling floor (`max(floorMs, opCount * 2000)`)
+ * is overridable per call, the seam `BrainyConfig.transactionBudgetFloorMs`
+ * feeds at the brainy.ts read sites.
+ *
+ * Uses a real class implementing the `Operation` interface (not an anonymous
+ * object literal, not a mock of Transaction internals) so the exercised path
+ * is exactly what a real caller's operation looks like.
+ */
+import { describe, it, expect } from 'vitest'
+import { Transaction, transactTimeoutBudget } from '../../../src/transaction/Transaction.js'
+import type { Operation, RollbackAction } from '../../../src/transaction/types.js'
+import { TransactionTimeoutError } from '../../../src/transaction/errors.js'
+
+const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms))
+
+/**
+ * A real `Operation` implementation that sleeps for `delayMs` before applying
+ * a write to `log`, and returns a rollback action that removes it again. Used
+ * to deterministically make one operation "slow" relative to a transaction's
+ * budget without touching any Transaction internals.
+ */
+class SlowOperation implements Operation {
+ readonly name: string
+ executed = false
+ rolledBack = false
+
+ constructor(
+ private readonly log: string[],
+ label: string,
+ private readonly delayMs: number
+ ) {
+ this.name = label
+ }
+
+ async execute(): Promise {
+ this.executed = true
+ if (this.delayMs > 0) await sleep(this.delayMs)
+ this.log.push(this.name)
+ return async () => {
+ this.rolledBack = true
+ const idx = this.log.indexOf(this.name)
+ if (idx >= 0) this.log.splice(idx, 1)
+ }
+ }
+}
+
+describe('Transaction budget — commit-completed-work semantics', () => {
+ it('(a) single-op transaction whose op overruns the budget COMMITS — no rollback, no throw', async () => {
+ const log: string[] = []
+ const op = new SlowOperation(log, 'slow-single-op', 40)
+
+ // Budget (5ms) is far smaller than the op's 40ms — under the OLD
+ // (post-completion-check) semantics this would have thrown and rolled
+ // back after the op finished. Under the new semantics there is no
+ // operation after it to gate, so it commits.
+ const tx = new Transaction({ timeout: 5 })
+ tx.addOperation(op)
+
+ await expect(tx.execute()).resolves.toBeUndefined()
+
+ expect(tx.getState()).toBe('committed')
+ expect(op.executed).toBe(true)
+ expect(op.rolledBack).toBe(false)
+ expect(log).toEqual(['slow-single-op'])
+ })
+
+ it('(b) two-op transaction: op0 overruns the budget → op1 never starts, op0 rolls back, throws TransactionTimeoutError', async () => {
+ const log: string[] = []
+ const op0 = new SlowOperation(log, 'op0-overruns', 40)
+ const op1 = new SlowOperation(log, 'op1-never-starts', 0)
+
+ const tx = new Transaction({ timeout: 5 })
+ tx.addOperation(op0)
+ tx.addOperation(op1)
+
+ const error = await tx.execute().catch((e) => e)
+
+ expect(error).toBeInstanceOf(TransactionTimeoutError)
+ expect(op0.executed).toBe(true)
+ expect(op0.rolledBack).toBe(true)
+ expect(op1.executed).toBe(false)
+ expect(op1.rolledBack).toBe(false)
+ expect(log).toEqual([]) // op0's write was undone; op1 never wrote
+ expect(tx.getState()).toBe('rolled_back')
+ })
+
+ it('a single-op transaction never checks the budget before its only operation starts', async () => {
+ // Budget of 0ms: under the old "check before every op including 0" code
+ // this could theoretically trip before op 0 even started (if any
+ // measurable time elapsed between startTime capture and the check). The
+ // documented contract is stronger: operation 0 ALWAYS gets to start.
+ const log: string[] = []
+ const op = new SlowOperation(log, 'only-op', 5)
+
+ const tx = new Transaction({ timeout: 0 })
+ tx.addOperation(op)
+
+ await expect(tx.execute()).resolves.toBeUndefined()
+ expect(tx.getState()).toBe('committed')
+ expect(op.executed).toBe(true)
+ })
+
+ it('(c) transactTimeoutBudget: an explicit floorMs overrides the 30s default floor', () => {
+ // Below the per-op scaling term, a raised floor wins.
+ expect(transactTimeoutBudget(1, undefined, 5_000)).toBe(5_000) // 1 op × 2000 = 2000 < 5000 floor
+ expect(transactTimeoutBudget(1)).toBe(30_000) // unchanged default when floorMs omitted
+
+ // Above the floor, opCount * 2000 still wins over a smaller floor.
+ expect(transactTimeoutBudget(10, undefined, 5_000)).toBe(20_000) // 10 × 2000 = 20000 > 5000 floor
+
+ // A full `override` always wins, regardless of floorMs.
+ expect(transactTimeoutBudget(10, 999, 5_000)).toBe(999)
+ })
+
+ it('(c) a configured floor changes real Transaction behavior end-to-end via TransactionManager-style options', async () => {
+ // Simulates how brainy.ts wires `this.config.transactionBudgetFloorMs`
+ // into `transactTimeoutBudget()`: opCount 0 isolates the floor term
+ // (0 × 2000 = 0), so a lowered floor makes a normally-generous budget
+ // fail-fast for a real 2-op Transaction.
+ const lowFloorBudget = transactTimeoutBudget(0, undefined, 10) // 10ms floor
+ expect(lowFloorBudget).toBe(10)
+
+ const log: string[] = []
+ const op0 = new SlowOperation(log, 'op0', 30)
+ const op1 = new SlowOperation(log, 'op1-gated-by-lowered-floor', 0)
+
+ const tx = new Transaction({ timeout: lowFloorBudget })
+ tx.addOperation(op0)
+ tx.addOperation(op1)
+
+ const error = await tx.execute().catch((e) => e)
+ expect(error).toBeInstanceOf(TransactionTimeoutError)
+ expect(op1.executed).toBe(false)
+ })
+})
diff --git a/tests/unit/transaction/vectorIndexOperations-rename.test.ts b/tests/unit/transaction/vectorIndexOperations-rename.test.ts
new file mode 100644
index 00000000..80168346
--- /dev/null
+++ b/tests/unit/transaction/vectorIndexOperations-rename.test.ts
@@ -0,0 +1,152 @@
+/**
+ * @module tests/unit/transaction/vectorIndexOperations-rename
+ * @description Coverage for the backend-neutral vector-index transaction
+ * operations (formerly `AddToHNSWOperation` / `RemoveFromHNSWOperation`).
+ * These op-name strings surface directly in consumer-visible transaction
+ * journals and timings — a fossil "HNSW" name misdirects an operator running
+ * a native (non-HNSW) vector provider. Verifies:
+ * 1. rollback wiring stayed byte-identical under the rename,
+ * 2. the emitted `name` stamps the active backend — `'js-hnsw'` for
+ * Brainy's own JS index, a provider's own `providerId` when it
+ * self-identifies, and the honest `'unknown-provider'` literal when
+ * neither applies (never a silently-wrong guess).
+ */
+import { describe, it, expect } from 'vitest'
+import {
+ AddToVectorIndexOperation,
+ RemoveFromVectorIndexOperation,
+ BatchAddToVectorIndexOperation
+} from '../../../src/transaction/operations/IndexOperations.js'
+import type { VectorIndexProvider } from '../../../src/plugin.js'
+import type { VectorDocument, Vector } from '../../../src/coreTypes.js'
+import { JsHnswVectorIndex } from '../../../src/hnsw/hnswIndex.js'
+
+/**
+ * A minimal, real (not mocked) VectorIndexProvider implementation backed by
+ * a plain Map — exercises the exact contract the operations call through,
+ * without any Brainy internals.
+ */
+class FakeVectorProvider implements VectorIndexProvider {
+ readonly items = new Map()
+ constructor(readonly providerId?: string) {}
+
+ async addItem(item: VectorDocument): Promise {
+ this.items.set(item.id, item.vector)
+ return item.id
+ }
+ async removeItem(id: string): Promise {
+ return this.items.delete(id)
+ }
+ async getItem(id: string): Promise {
+ const vector = this.items.get(id)
+ return vector ? { id, vector } : undefined
+ }
+ async search(): Promise> {
+ return []
+ }
+ size(): number {
+ return this.items.size
+ }
+ clear(): void {
+ this.items.clear()
+ }
+ async rebuild(): Promise {}
+ async flush(): Promise {
+ return 0
+ }
+ getPersistMode(): 'immediate' | 'deferred' {
+ return 'immediate'
+ }
+}
+
+describe('Vector index transaction operations — backend-neutral rename', () => {
+ describe('rollback wiring (byte-identical behavior under the rename)', () => {
+ it('AddToVectorIndexOperation rollback removes a newly-added item', async () => {
+ const provider = new FakeVectorProvider('fake-provider')
+ const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+
+ const rollback = await op.execute()
+ expect(provider.items.has('id-1')).toBe(true)
+
+ await rollback()
+ expect(provider.items.has('id-1')).toBe(false)
+ })
+
+ it('AddToVectorIndexOperation rollback is a no-op when the item pre-existed (update semantics)', async () => {
+ const provider = new FakeVectorProvider('fake-provider')
+ await provider.addItem({ id: 'id-1', vector: [9, 9, 9] })
+
+ const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+ const rollback = await op.execute()
+ expect(provider.items.get('id-1')).toEqual([1, 2, 3])
+
+ await rollback()
+ // Pre-existing item is NOT removed by rollback — it stays (the update
+ // itself is not undone by this op; that's RemoveFromVectorIndexOperation's job).
+ expect(provider.items.has('id-1')).toBe(true)
+ })
+
+ it('RemoveFromVectorIndexOperation rollback re-adds the item with its original vector', async () => {
+ const provider = new FakeVectorProvider('fake-provider')
+ await provider.addItem({ id: 'id-1', vector: [4, 5, 6] })
+
+ const op = new RemoveFromVectorIndexOperation(provider, 'id-1', [4, 5, 6])
+ const rollback = await op.execute()
+ expect(provider.items.has('id-1')).toBe(false)
+
+ await rollback()
+ expect(provider.items.get('id-1')).toEqual([4, 5, 6])
+ })
+
+ it('BatchAddToVectorIndexOperation rolls back every item in reverse order on undo', async () => {
+ const provider = new FakeVectorProvider('fake-provider')
+ const op = new BatchAddToVectorIndexOperation(provider, [
+ { id: 'a', vector: [1] },
+ { id: 'b', vector: [2] },
+ { id: 'c', vector: [3] }
+ ])
+
+ const rollback = await op.execute()
+ expect([...provider.items.keys()].sort()).toEqual(['a', 'b', 'c'])
+
+ await rollback()
+ expect(provider.items.size).toBe(0)
+ })
+ })
+
+ describe('backend stamping in the emitted op name', () => {
+ it('stamps the real JS HNSW index as js-hnsw (self-identifying providerId)', () => {
+ const index = new JsHnswVectorIndex()
+ const addOp = new AddToVectorIndexOperation(index, 'id-1', [1, 2, 3])
+ const removeOp = new RemoveFromVectorIndexOperation(index, 'id-1', [1, 2, 3])
+
+ expect(addOp.name).toBe('AddToVectorIndex(js-hnsw)')
+ expect(removeOp.name).toBe('RemoveFromVectorIndex(js-hnsw)')
+ })
+
+ it('stamps a self-identifying provider\'s own providerId, never "HNSW"', () => {
+ const provider = new FakeVectorProvider('acme-diskann')
+ const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+ const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+ const batchOp = new BatchAddToVectorIndexOperation(provider, [{ id: 'a', vector: [1] }])
+
+ expect(addOp.name).toBe('AddToVectorIndex(acme-diskann)')
+ expect(removeOp.name).toBe('RemoveFromVectorIndex(acme-diskann)')
+ expect(batchOp.name).toBe('BatchAddToVectorIndex(acme-diskann)')
+ expect(addOp.name).not.toContain('HNSW')
+ expect(removeOp.name).not.toContain('HNSW')
+ })
+
+ it('honestly reports "unknown-provider" rather than guessing when a provider omits providerId', () => {
+ const provider = new FakeVectorProvider(undefined)
+ const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+ const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+
+ expect(addOp.name).toBe('AddToVectorIndex(unknown-provider)')
+ expect(removeOp.name).toBe('RemoveFromVectorIndex(unknown-provider)')
+ // Never silently falls back to the JS engine's name for a provider it
+ // knows nothing about — that's the exact fossil-naming bug being fixed.
+ expect(addOp.name).not.toContain('js-hnsw')
+ })
+ })
+})
From ec571747604868abe2aeda02352e836fd51b7f3d Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Thu, 23 Jul 2026 08:53:05 -0700
Subject: [PATCH 03/15] feat: vector provider identity is a required name field
(hnsw-js), rendered [vector-index:]
Reconciles the vector-index rename to the ruled three-layer naming: the
provider contract's identity field is now a REQUIRED readonly name (was
optional providerId), self-reported and truthful, rendered as
[vector-index:] where the index identifies itself and stamped into
the op-name strings journals already parse (AddToVectorIndex()).
The built-in JS engine names itself hnsw-js. A runtime provider instance
compiled against the previous optional contract is tolerated - never
crashed on, never silently mislabeled: it stamps unknown-provider and
emits one loud warning naming the missing field. Graph index operations
keep their static names (they never interpolate provider identity), and
no public API exports a provider-routed hnsw-carrying name, so no
deprecation shim is required.
---
RELEASES.md | 12 ++++-
src/hnsw/hnswIndex.ts | 4 +-
src/plugin.ts | 30 ++++++-----
src/transaction/operations/IndexOperations.ts | 29 +++++++---
tests/unit/brainy/warm.test.ts | 1 +
.../vectorIndexOperations-rename.test.ts | 54 +++++++++++++------
6 files changed, 92 insertions(+), 38 deletions(-)
diff --git a/RELEASES.md b/RELEASES.md
index 113b327d..89bf38e7 100644
--- a/RELEASES.md
+++ b/RELEASES.md
@@ -82,10 +82,20 @@ demand-load cost OFF the transaction path.
`AddToVectorIndex(...)`/`RemoveFromVectorIndex(...)` — the old names hard-coded an
algorithm that may not be the one actually running (a non-HNSW native vector provider
emitting `"RemoveFromHNSW"` has sent an operator hunting an index that doesn't exist).
- The parenthesized suffix names the ACTIVE backend: `js-hnsw` for the built-in engine, or
+ The parenthesized suffix names the ACTIVE backend: `hnsw-js` for the built-in engine, or
the native provider's own identity when it self-identifies. **If you parse these op-name
strings** (log processors, journal tooling), update your matcher from
`AddToHNSW`/`RemoveFromHNSW` to `AddToVectorIndex(`/`RemoveFromVectorIndex(`.
+- **Provider identity is now a REQUIRED `name` field** on the vector provider contract
+ (`VectorIndexProvider.name`, `src/plugin.ts`) — every implementation self-reports its own
+ identity truthfully (its algorithm/engine), never inheriting a default. It renders as the
+ op-name suffix above and, wherever the vector index identifies itself in prose log lines,
+ as the tag `[vector-index:]`. **Native provider adoption is a one-line change**:
+ declare `readonly name = ''`. A provider instance that still lacks
+ `name` at runtime (an older native build compiled against the previous, optional field) is
+ never crashed on and never silently mislabeled: it stamps `unknown-provider` and emits one
+ loud warning naming the missing field, so the gap is discoverable instead of a permanent
+ fossil label in every journal line.
## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close())
diff --git a/src/hnsw/hnswIndex.ts b/src/hnsw/hnswIndex.ts
index dcf3ed7b..eb2acd71 100644
--- a/src/hnsw/hnswIndex.ts
+++ b/src/hnsw/hnswIndex.ts
@@ -54,8 +54,8 @@ export class HnswFlushError extends Error {
* acceleration provider).
*/
export class JsHnswVectorIndex implements VectorIndexProvider {
- /** Self-identifies as the built-in JS fallback engine — see {@link VectorIndexProvider.providerId}. */
- readonly providerId = 'js-hnsw'
+ /** Self-identifies as the built-in JS fallback engine — see {@link VectorIndexProvider.name}. */
+ readonly name = 'hnsw-js'
private nouns: Map = new Map()
/**
diff --git a/src/plugin.ts b/src/plugin.ts
index eeb2425a..02639955 100644
--- a/src/plugin.ts
+++ b/src/plugin.ts
@@ -962,20 +962,24 @@ export interface AtGenerationVectors {
*/
export interface VectorIndexProvider {
/**
- * @description OPTIONAL backend identity, stamped into the transaction
- * op-name strings that surface in journals/timings (e.g.
- * `AddToVectorIndex(js-hnsw)` vs `AddToVectorIndex()`) — see
- * `src/transaction/operations/IndexOperations.ts`. Absent → the op-name
- * string reports `unknown-provider` rather than guessing a backend name
- * (guessing would resurrect the exact bug this field exists to prevent:
- * an operator hunting an index that isn't the one actually running). The
- * built-in JS index always sets this to `'js-hnsw'`; a native provider
- * sets its own identity here (e.g. its plugin/package name) so operators
- * reading a journal see the backend that actually ran, never a
- * backend-specific fossil name from whichever engine wrote the original
- * op classes.
+ * @description REQUIRED self-reported implementation identity, rendered as
+ * `[vector-index:]` in prose log lines and stamped into the
+ * transaction op-name strings that surface in journals/timings (e.g.
+ * `AddToVectorIndex(hnsw-js)`) — see
+ * `src/transaction/operations/IndexOperations.ts`. A provider must name
+ * itself TRUTHFULLY (its own algorithm/engine, e.g. its plugin/package
+ * name) and must never inherit a default — guessing a backend name would
+ * resurrect the exact bug this field exists to prevent: an operator
+ * hunting an index that isn't the one actually running. The built-in JS
+ * index always sets this to `'hnsw-js'`; a native provider picks its own
+ * string. At the TypeScript level this field is required; a runtime
+ * instance from an older provider compiled against the previous optional
+ * `providerId` contract is tolerated (never crashes, never silently
+ * mislabeled) — see `resolveVectorProviderId` in
+ * `src/transaction/operations/IndexOperations.ts`, which stamps
+ * `'unknown-provider'` and emits one loud warning for that case.
*/
- readonly providerId?: string
+ readonly name: string
addItem(item: VectorDocument): Promise
removeItem(id: string): Promise
diff --git a/src/transaction/operations/IndexOperations.ts b/src/transaction/operations/IndexOperations.ts
index 1c0995c6..d130bb3f 100644
--- a/src/transaction/operations/IndexOperations.ts
+++ b/src/transaction/operations/IndexOperations.ts
@@ -16,19 +16,34 @@ import type { Operation, RollbackAction } from '../types.js'
/**
* Backend identity stamped into an operation's emitted `name` string (e.g.
- * `AddToVectorIndex(js-hnsw)`), resolved from the provider's own
- * {@link VectorIndexProvider.providerId} when it self-identifies.
+ * `AddToVectorIndex(hnsw-js)`), resolved from the provider's own
+ * {@link VectorIndexProvider.name} self-report.
*
* These operation classes are backend-neutral (the vector index they wrap may
* be Brainy's own JS HNSW fallback OR a native acceleration provider), but
* their names surface directly in consumer-visible transaction journals and
- * timings. A provider that hasn't set `providerId` resolves to
- * `'unknown-provider'` rather than guessing — silently defaulting to the JS
- * engine's name here is exactly the class of bug this stamping exists to
- * prevent (an operator diagnosing a backend that isn't the one that ran).
+ * timings. `name` is REQUIRED at the TypeScript level — but a native provider
+ * instance compiled against the previous (pre-required) contract can still
+ * reach this function at runtime without it. That legacy case is tolerated,
+ * never crashed on and never silently mislabeled: it resolves to
+ * `'unknown-provider'` and emits ONE loud warning naming the missing contract
+ * field, so the fix (implement `name`) is discoverable rather than a silent
+ * fossil label in every journal line thereafter.
*/
+const warnedMissingName = new WeakSet()
+
function resolveVectorProviderId(index: VectorIndexProvider): string {
- return index.providerId ?? 'unknown-provider'
+ const name = (index as { name?: unknown }).name
+ if (typeof name === 'string') return name
+ if (!warnedMissingName.has(index)) {
+ warnedMissingName.add(index)
+ console.warn(
+ '[vector-index] provider is missing the required `name` field (VectorIndexProvider.name, ' +
+ 'required since 8.10.0) — stamping "unknown-provider" in transaction op names until the ' +
+ 'provider declares its own identity.'
+ )
+ }
+ return 'unknown-provider'
}
/**
diff --git a/tests/unit/brainy/warm.test.ts b/tests/unit/brainy/warm.test.ts
index 437b7785..ce213696 100644
--- a/tests/unit/brainy/warm.test.ts
+++ b/tests/unit/brainy/warm.test.ts
@@ -44,6 +44,7 @@ const V = (seed = 1): number[] => Array.from({ length: DIM }, (_, i) => Math.sin
* `AddToVectorIndexOperation` / `brain.warm()` call through.
*/
class FakeVectorProvider implements VectorIndexProvider {
+ readonly name = 'fake-vector-provider'
readonly items = new Map()
warmCalls = 0
searchCalls: Array<{ k?: number }> = []
diff --git a/tests/unit/transaction/vectorIndexOperations-rename.test.ts b/tests/unit/transaction/vectorIndexOperations-rename.test.ts
index 80168346..bdfe58a3 100644
--- a/tests/unit/transaction/vectorIndexOperations-rename.test.ts
+++ b/tests/unit/transaction/vectorIndexOperations-rename.test.ts
@@ -6,12 +6,14 @@
* journals and timings — a fossil "HNSW" name misdirects an operator running
* a native (non-HNSW) vector provider. Verifies:
* 1. rollback wiring stayed byte-identical under the rename,
- * 2. the emitted `name` stamps the active backend — `'js-hnsw'` for
- * Brainy's own JS index, a provider's own `providerId` when it
- * self-identifies, and the honest `'unknown-provider'` literal when
- * neither applies (never a silently-wrong guess).
+ * 2. the emitted `name` stamps the active backend — `'hnsw-js'` for
+ * Brainy's own JS index, a provider's own required `name` when it
+ * self-identifies, and the tolerant-loud `'unknown-provider'` literal
+ * (plus one console.warn) when a runtime instance lacks `name`
+ * altogether (an older native provider compiled against the previous,
+ * optional `providerId` contract) — never a silently-wrong guess.
*/
-import { describe, it, expect } from 'vitest'
+import { describe, it, expect, vi, afterEach } from 'vitest'
import {
AddToVectorIndexOperation,
RemoveFromVectorIndexOperation,
@@ -28,7 +30,7 @@ import { JsHnswVectorIndex } from '../../../src/hnsw/hnswIndex.js'
*/
class FakeVectorProvider implements VectorIndexProvider {
readonly items = new Map()
- constructor(readonly providerId?: string) {}
+ constructor(readonly name: string) {}
async addItem(item: VectorDocument): Promise {
this.items.set(item.id, item.vector)
@@ -115,16 +117,20 @@ describe('Vector index transaction operations — backend-neutral rename', () =>
})
describe('backend stamping in the emitted op name', () => {
- it('stamps the real JS HNSW index as js-hnsw (self-identifying providerId)', () => {
+ afterEach(() => {
+ vi.restoreAllMocks()
+ })
+
+ it('stamps the real JS HNSW index as hnsw-js (self-identifying name)', () => {
const index = new JsHnswVectorIndex()
const addOp = new AddToVectorIndexOperation(index, 'id-1', [1, 2, 3])
const removeOp = new RemoveFromVectorIndexOperation(index, 'id-1', [1, 2, 3])
- expect(addOp.name).toBe('AddToVectorIndex(js-hnsw)')
- expect(removeOp.name).toBe('RemoveFromVectorIndex(js-hnsw)')
+ expect(addOp.name).toBe('AddToVectorIndex(hnsw-js)')
+ expect(removeOp.name).toBe('RemoveFromVectorIndex(hnsw-js)')
})
- it('stamps a self-identifying provider\'s own providerId, never "HNSW"', () => {
+ it('stamps a self-identifying provider\'s own name, never "HNSW"', () => {
const provider = new FakeVectorProvider('acme-diskann')
const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3])
@@ -137,16 +143,34 @@ describe('Vector index transaction operations — backend-neutral rename', () =>
expect(removeOp.name).not.toContain('HNSW')
})
- it('honestly reports "unknown-provider" rather than guessing when a provider omits providerId', () => {
- const provider = new FakeVectorProvider(undefined)
- const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
- const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+ it('tolerant-loud: stamps "unknown-provider" and warns exactly once when a runtime provider lacks the required `name` (an older native provider compiled against the previous optional `providerId` contract)', () => {
+ const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
+ // Simulate a native provider compiled before `name` was required — the
+ // TypeScript contract requires `name`, but `as unknown as` mirrors what
+ // actually reaches this code at runtime from an un-rebuilt native addon.
+ const legacyProvider = {
+ addItem: async (item: VectorDocument) => item.id,
+ removeItem: async () => true,
+ search: async () => [],
+ size: () => 0,
+ clear: () => {},
+ rebuild: async () => {},
+ flush: async () => 0,
+ getPersistMode: () => 'immediate' as const
+ } as unknown as VectorIndexProvider
+
+ const addOp = new AddToVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3])
+ const removeOp = new RemoveFromVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3])
expect(addOp.name).toBe('AddToVectorIndex(unknown-provider)')
expect(removeOp.name).toBe('RemoveFromVectorIndex(unknown-provider)')
// Never silently falls back to the JS engine's name for a provider it
// knows nothing about — that's the exact fossil-naming bug being fixed.
- expect(addOp.name).not.toContain('js-hnsw')
+ expect(addOp.name).not.toContain('hnsw-js')
+ // Loud, not silent — but exactly once per provider instance, not once
+ // per op stamped against it.
+ expect(warnSpy).toHaveBeenCalledTimes(1)
+ expect(warnSpy.mock.calls[0][0]).toContain('name')
})
})
})
From 55b867c9984ee6cb92ee19df4f443725145eb91b Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Wed, 22 Jul 2026 16:42:26 -0700
Subject: [PATCH 04/15] feat: warm contract (warm/warmOnOpen/provider warm
hook), configurable transact budget floor, backend-neutral vector index op
names
Three pieces addressing the cold-restart-write incident where a production
deployment's first writes after every restart (33-35s each on a cold page
cache) blew the op-count-scaled transact budget mid-batch: every write is
itself a multi-op transaction, so one cold operation consumed the whole
budget, the gate before the next operation tripped, and the write rolled
back atomically - refused, retried, and refused again until the page cache
warmed passively.
- The budget's start-gating contract is now explicit and pinned: it gates
STARTING the next operation, never rolling back completed work for
elapsed time (the shipped schedule since 8.7.0, now stated in contract
JSDoc, guarded by code for operation 0, and enforced by regression
tests). The 30s floor is configurable via transactionBudgetFloorMs for
stores whose cold operations legitimately run long.
- New brain.warm() eagerly loads the vector index, metadata index, and
graph adjacency so first operations after a cold restart run at
steady-state cost. Returns a WarmReport with an honest per-surface
outcome (warmed / probed / unavailable) - never reports a probe as a
warm. warmOnOpen: true runs it during init(). New optional provider hook
warm() on the vector and graph plugin contracts.
- Vector-index transaction op classes renamed from the backend-specific
AddToHNSWOperation / RemoveFromHNSWOperation to backend-neutral
AddToVectorIndexOperation / RemoveFromVectorIndexOperation, stamping the
active backend into the emitted op-name string (AddToVectorIndex(js-hnsw)
vs a native provider's own identity) so journals never misdirect an
operator toward an index that isn't running.
---
RELEASES.md | 56 ++++
src/brainy.ts | 205 +++++++++++-
src/hnsw/hnswIndex.ts | 3 +
src/index.ts | 3 +
src/plugin.ts | 44 +++
src/transaction/Transaction.ts | 74 ++++-
src/transaction/operations/IndexOperations.ts | 79 +++--
src/transaction/operations/index.ts | 6 +-
src/types/brainy.types.ts | 36 ++
src/utils/metadataIndex.ts | 40 +++
tests/unit/brainy/warm.test.ts | 309 ++++++++++++++++++
.../budget-commit-completed-work.test.ts | 149 +++++++++
.../vectorIndexOperations-rename.test.ts | 152 +++++++++
13 files changed, 1099 insertions(+), 57 deletions(-)
create mode 100644 tests/unit/brainy/warm.test.ts
create mode 100644 tests/unit/transaction/budget-commit-completed-work.test.ts
create mode 100644 tests/unit/transaction/vectorIndexOperations-rename.test.ts
diff --git a/RELEASES.md b/RELEASES.md
index 7799c6f6..113b327d 100644
--- a/RELEASES.md
+++ b/RELEASES.md
@@ -31,6 +31,62 @@ is sometimes cited as a 7.x removal — those methods never existed on 7.x; the
---
+## Unreleased (the warm contract: cold-restart writes stop paying demand-load latency)
+
+From a production deployment's cold-restart incident: the FIRST writes after every
+restart on a large brain measured 33–35s each (page cache cold) against the transact
+apply budget — Brainy's own op-count-scaled budget, `max(30s, opCount × 2s)`, e.g.
+32,000ms for a 16-op batch. There is no external deadline in this story, and no
+post-completion veto either: every write is itself a multi-operation transaction (a
+single `add()` applies several operations — canonical writes, the vector-index insert,
+the metadata-index update), so ONE cold operation that legitimately runs ~33s consumes
+the whole budget, the gate before the NEXT operation trips, and the write rolls back
+atomically (zero loss, by design) — refused, retried, refused again, until the page
+cache warms passively (~30 minutes). The cure is not weaker atomicity; it is the two
+new knobs below — a budget floor sized for cold stores, and a warm contract that pays
+demand-load cost OFF the transaction path.
+
+- **The budget's start-gating contract is now explicit, documented, and pinned by
+ regression tests.** The budget gates STARTING the next operation — completed work is
+ never rolled back for elapsed time — and a transaction's first operation now
+ unconditionally starts by code, not merely because elapsed time happens to be ~0 when
+ it is checked. This has been the shipped schedule since 8.7.0 (no behavior change for
+ existing integrations); it is now stated in `Transaction.execute()`'s contract JSDoc
+ and enforced by tests so it cannot silently regress. Mid-batch atomicity is unchanged:
+ a trip before operation `i+1` still rolls back `0..i` and throws a retryable
+ `TransactionTimeoutError`.
+- **The budget's 30s floor is now configurable**: `new Brainy({ transactionBudgetFloorMs })`
+ raises (or lowers) the floor of `max(transactionBudgetFloorMs, opCount × 2000)` for every
+ internal transact batch. Useful for a store whose cold writes legitimately run past 30s
+ per operation, so a bulk batch gets a proportionally larger runway instead of tripping
+ mid-batch on cold-cache latency.
+- **New: `brain.warm()`** — eagerly loads/faults-in the vector index, metadata index, and
+ graph adjacency so the first real operation after a cold restart runs at steady-state
+ cost instead of paying demand-load latency on the critical path. Returns a `WarmReport`
+ with one honest outcome per surface — never conflate the first two:
+ - `'warmed'` — the surface's own provider `warm()` hook ran, or a full-hydration seam
+ loaded every shard/field/segment from storage. Steady-state cost is paid.
+ - `'probed'` — no `warm()` hook was available, so a best-effort read (one `search()` call
+ for the vector index) faulted in *some* backing storage as a side effect — real work,
+ but never reported as `'warmed'`.
+ - `'unavailable'` — nothing ran (no hook, no hydration seam, or nothing to probe).
+- **New config: `warmOnOpen: true`** makes `init()` await `brain.warm()` before it resolves
+ — a deliberate blocking trade-off: startup takes longer, the first request doesn't.
+ Default `false` (unchanged lazy behavior).
+- **New optional provider hook: `warm?(): Promise`** on the vector and graph
+ acceleration provider contracts (`src/plugin.ts`) — a native provider can implement it to
+ eagerly pretouch its own backing storage (e.g. mmap pretouch); absence means brainy falls
+ back to the probe/hydration behavior above.
+- **Transaction op-name strings changed in journals/timings**: the vector-index
+ transaction operations were renamed from `AddToHNSW`/`RemoveFromHNSW` to backend-neutral
+ `AddToVectorIndex(...)`/`RemoveFromVectorIndex(...)` — the old names hard-coded an
+ algorithm that may not be the one actually running (a non-HNSW native vector provider
+ emitting `"RemoveFromHNSW"` has sent an operator hunting an index that doesn't exist).
+ The parenthesized suffix names the ACTIVE backend: `js-hnsw` for the built-in engine, or
+ the native provider's own identity when it self-identifies. **If you parse these op-name
+ strings** (log processors, journal tooling), update your matcher from
+ `AddToHNSW`/`RemoveFromHNSW` to `AddToVectorIndex(`/`RemoveFromVectorIndex(`.
+
## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close())
The write path stops paying maintenance costs — the last structural piece of the
diff --git a/src/brainy.ts b/src/brainy.ts
index 8ba991dd..37b6e491 100644
--- a/src/brainy.ts
+++ b/src/brainy.ts
@@ -63,7 +63,9 @@ import type {
PathOptions,
MetadataIndexProvider,
OpaqueIdSet,
- AtGenerationVectors
+ AtGenerationVectors,
+ VectorIndexProvider,
+ GraphIndexProvider
} from './plugin.js'
import type {
BrainyPlugin,
@@ -88,12 +90,12 @@ import { findCallerLocation } from './utils/callerLocation.js'
import {
SaveNounMetadataOperation,
SaveNounOperation,
- AddToHNSWOperation,
+ AddToVectorIndexOperation,
AddToMetadataIndexOperation,
SaveVerbMetadataOperation,
SaveVerbOperation,
AddToGraphIndexOperation,
- RemoveFromHNSWOperation,
+ RemoveFromVectorIndexOperation,
RemoveFromMetadataIndexOperation,
RemoveFromGraphIndexOperation,
UpdateNounMetadataOperation,
@@ -266,6 +268,7 @@ type ResolvedBrainyConfig = Required<
| 'retention'
| 'eagerEmbeddings'
| 'migrationWaitTimeoutMs'
+ | 'transactionBudgetFloorMs'
>
> &
Pick<
@@ -278,6 +281,7 @@ type ResolvedBrainyConfig = Required<
| 'retention'
| 'eagerEmbeddings'
| 'migrationWaitTimeoutMs'
+ | 'transactionBudgetFloorMs'
>
/**
@@ -388,6 +392,38 @@ class InsertPreconditionExistsSignal extends Error {
*/
export type IndexFamily = 'vector' | 'metadata' | 'graph'
+/**
+ * @description Honest per-surface outcome for {@link Brainy.warm}. Literal
+ * meanings — never conflate the first two:
+ * - `'warmed'` — the provider's own `warm?()` hook ran (vector/graph), or the
+ * surface's full-hydration seam loaded EVERY shard/field/segment from
+ * storage (metadata; graph's fallback path). The surface is genuinely at
+ * steady-state cost for the next operation.
+ * - `'probed'` — no `warm?()` hook was available, so a best-effort read
+ * (e.g. one `search()` call) faulted in *some* backing storage as a side
+ * effect. Real work happened, but it is NOT the same guarantee as
+ * `'warmed'` — never reported as `'warmed'`.
+ * - `'unavailable'` — nothing ran: no hook, no hydration seam, and (for the
+ * vector probe fallback) nothing to probe (an empty index or unknown
+ * vector dimension). The surface is unchanged by this `warm()` call.
+ */
+export type WarmOutcome = 'warmed' | 'probed' | 'unavailable'
+
+/**
+ * @description Result of {@link Brainy.warm}: one {@link WarmOutcome} +
+ * elapsed time per index surface, plus the total wall-clock time for the
+ * whole call. `durationMs` is measured around exactly the work described by
+ * that surface's `outcome` (e.g. the vector entry's `durationMs` times the
+ * provider `warm()` call OR the probe `search()` call — whichever ran).
+ */
+export interface WarmReport {
+ vector: { outcome: WarmOutcome; durationMs: number }
+ metadata: { outcome: WarmOutcome; durationMs: number }
+ graph: { outcome: WarmOutcome; durationMs: number }
+ /** Total wall-clock time for the whole `warm()` call (all three surfaces). */
+ totalDurationMs: number
+}
+
/**
* How long a failed aggregation-backfill walk suppresses fresh walk attempts.
* Within the window, queries rethrow the recorded failure instantly (loud,
@@ -1382,6 +1418,17 @@ export class Brainy implements BrainyInterface {
})
}
+ // Eager index warm (operator opt-in, `warmOnOpen: true`). Runs AFTER
+ // every step above — index construction, crash recovery, migrations,
+ // VFS bootstrap — never in place of any of it, so `warm()` always
+ // operates on a fully-initialized brain. Blocking BY DESIGN: the
+ // operator traded a longer startup for a first-request that runs at
+ // steady-state cost instead of paying demand-load latency on the
+ // critical path. See the `warmOnOpen` JSDoc in brainy.types.ts.
+ if (this.config.warmOnOpen) {
+ await this.warm()
+ }
+
// Resolve ready Promise - consumers awaiting brain.ready will now proceed
if (this._readyResolve) {
this._readyResolve()
@@ -1780,7 +1827,9 @@ export class Brainy implements BrainyInterface {
await this.generationStore.runWithoutGeneration(() =>
this.transactionManager.executeTransaction(run, {
timeout: transactTimeoutBudget(
- (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0)
+ (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0),
+ undefined,
+ this.config.transactionBudgetFloorMs
)
})
)
@@ -1797,7 +1846,9 @@ export class Brainy implements BrainyInterface {
execute: () =>
this.transactionManager.executeTransaction(run, {
timeout: transactTimeoutBudget(
- (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0)
+ (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0),
+ undefined,
+ this.config.transactionBudgetFloorMs
)
})
})
@@ -2109,7 +2160,7 @@ export class Brainy implements BrainyInterface {
// Operation 3: Add to HNSW index (after entity saved)
tx.addOperation(
- new AddToHNSWOperation(this.index, id, vector)
+ new AddToVectorIndexOperation(this.index, id, vector)
)
// Operation 4: Add to metadata index
@@ -3098,10 +3149,10 @@ export class Brainy implements BrainyInterface {
// Operation 3-4: Update HNSW index (remove and re-add if reindexing needed)
if (needsReindexing) {
tx.addOperation(
- new RemoveFromHNSWOperation(this.index, params.id, existing.vector)
+ new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector)
)
tx.addOperation(
- new AddToHNSWOperation(this.index, params.id, vector)
+ new AddToVectorIndexOperation(this.index, params.id, vector)
)
}
@@ -3217,7 +3268,7 @@ export class Brainy implements BrainyInterface {
// Operation 1: Remove from vector index
if (noun) {
tx.addOperation(
- new RemoveFromHNSWOperation(this.index, id, noun.vector)
+ new RemoveFromVectorIndexOperation(this.index, id, noun.vector)
)
}
@@ -7025,7 +7076,7 @@ export class Brainy implements BrainyInterface {
// Add delete operations to transaction
if (noun) {
tx.addOperation(
- new RemoveFromHNSWOperation(this.index, id, noun.vector)
+ new RemoveFromVectorIndexOperation(this.index, id, noun.vector)
)
}
@@ -7888,7 +7939,13 @@ export class Brainy implements BrainyInterface {
// Budget scales with the batch (or the caller's explicit
// timeoutMs): a flat 30s cap silently limited honest bulk work
// to ~15 ops on network disks (~2s/op measured in the field).
- { timeout: transactTimeoutBudget(plan.operations.length, options?.timeoutMs) }
+ {
+ timeout: transactTimeoutBudget(
+ plan.operations.length,
+ options?.timeoutMs,
+ this.config.transactionBudgetFloorMs
+ )
+ }
)
}
}))
@@ -9272,7 +9329,7 @@ export class Brainy implements BrainyInterface {
plan.operations.push(
new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew),
new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew),
- new AddToHNSWOperation(this.index, id, vector),
+ new AddToVectorIndexOperation(this.index, id, vector),
new AddToMetadataIndexOperation(this.metadataIndex, id, entityForIndexing)
)
plan.touchedNouns.push(id)
@@ -9440,8 +9497,8 @@ export class Brainy implements BrainyInterface {
)
if (needsReindexing) {
plan.operations.push(
- new RemoveFromHNSWOperation(this.index, params.id, existing.vector),
- new AddToHNSWOperation(this.index, params.id, vector)
+ new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector),
+ new AddToVectorIndexOperation(this.index, params.id, vector)
)
}
plan.operations.push(
@@ -9526,7 +9583,7 @@ export class Brainy implements BrainyInterface {
}
if (noun) {
- plan.operations.push(new RemoveFromHNSWOperation(this.index, id, noun.vector))
+ plan.operations.push(new RemoveFromVectorIndexOperation(this.index, id, noun.vector))
}
if (metadata) {
plan.operations.push(new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata))
@@ -14241,6 +14298,118 @@ export class Brainy implements BrainyInterface {
return this.embedder(textToEmbed)
}
+ /**
+ * Eagerly load/fault-in the vector index, metadata index, and graph
+ * adjacency so the FIRST real operation after this call runs at
+ * steady-state cost — no page-cache-miss / demand-load latency on the
+ * critical path. Complements {@link warmupEmbeddings} (which warms the
+ * embedding engine, not the storage indexes).
+ *
+ * Sequence per surface:
+ * - **Vector**: calls the provider's own `warm?()` when the active
+ * `'vector'` provider implements it (`'warmed'`). Otherwise runs a
+ * best-effort probe — one `search()` with a deterministic unit vector of
+ * the index's known dimension, `k = min(100, size())` — which faults in
+ * *some* backing storage as a side effect but is reported honestly as
+ * `'probed'`, never `'warmed'`. An empty index or unknown dimension has
+ * nothing to probe (`'unavailable'`).
+ * - **Metadata**: full hydration — every persisted field's sparse index is
+ * loaded from storage (`MetadataIndexManager.hydrateAll()`), not just the
+ * heuristic common-fields subset `init()` warms.
+ * - **Graph**: calls the provider's own `warm?()` when the active graph
+ * provider implements it; otherwise re-runs its existing eager cold-load
+ * `init()` seam (idempotent — the JS adjacency index's `init()` already
+ * loads the full LSM manifests + SSTables, so re-invoking it here is a
+ * genuine full hydration, not a stat call).
+ *
+ * A single surface's `'unavailable'`/probe outcome never fails the whole
+ * call — `warm()` is a best-effort readiness step, not a correctness gate;
+ * queries still demand-load normally regardless of what `warm()` achieved.
+ *
+ * @returns A {@link WarmReport}: honest per-surface outcome + timing.
+ * @example
+ * ```typescript
+ * const brain = new Brainy({ storage: { path: '/data' } })
+ * await brain.init()
+ * const report = await brain.warm() // blocking, explicit control over timing
+ * console.log(report.vector.outcome, report.vector.durationMs)
+ *
+ * // Or opt into the same work automatically during init():
+ * const eager = new Brainy({ storage: { path: '/data' }, warmOnOpen: true })
+ * await eager.init() // warm() already ran before this resolves
+ * ```
+ */
+ async warm(): Promise {
+ await this.ensureInitialized({ needs: ['vector', 'metadata', 'graph'] })
+
+ const totalStart = Date.now()
+
+ // --- Vector ---------------------------------------------------------
+ const vectorStart = Date.now()
+ let vectorOutcome: WarmOutcome
+ const vectorProvider = this.index as VectorIndexProvider & { warm?: () => Promise }
+ if (typeof vectorProvider.warm === 'function') {
+ await vectorProvider.warm()
+ vectorOutcome = 'warmed'
+ } else {
+ const size = this.index.size()
+ const dimension = this.dimensions
+ if (size > 0 && dimension && dimension > 0) {
+ // A deterministic unit vector (L2 norm 1) — same probe every call,
+ // no reliance on stored data shape.
+ const probeVector = new Array(dimension).fill(1 / Math.sqrt(dimension))
+ await this.index.search(probeVector, Math.min(100, size))
+ vectorOutcome = 'probed'
+ } else {
+ // Nothing to probe: an empty index, or the vector dimension is not
+ // yet known (no entity has ever been added).
+ vectorOutcome = 'unavailable'
+ }
+ }
+ const vectorDurationMs = Date.now() - vectorStart
+
+ // --- Metadata --------------------------------------------------------
+ const metadataStart = Date.now()
+ let metadataOutcome: WarmOutcome
+ const metadataWithHydrate = this.metadataIndex as unknown as { hydrateAll?: () => Promise }
+ if (typeof metadataWithHydrate.hydrateAll === 'function') {
+ await metadataWithHydrate.hydrateAll()
+ metadataOutcome = 'warmed'
+ } else {
+ // No hydration seam on this metadata provider — nothing to run.
+ metadataOutcome = 'unavailable'
+ }
+ const metadataDurationMs = Date.now() - metadataStart
+
+ // --- Graph -------------------------------------------------------------
+ const graphStart = Date.now()
+ let graphOutcome: WarmOutcome
+ const graphProvider = this.graphIndex as GraphIndexProvider & {
+ warm?: () => Promise
+ init?: () => Promise
+ }
+ if (typeof graphProvider.warm === 'function') {
+ await graphProvider.warm()
+ graphOutcome = 'warmed'
+ } else if (typeof graphProvider.init === 'function') {
+ // Existing eager cold-load seam (readiness contract): idempotent, and
+ // for the JS adjacency index it's already a genuine full load of the
+ // LSM manifests + SSTables — a real hydration, not a stat call.
+ await graphProvider.init()
+ graphOutcome = 'warmed'
+ } else {
+ graphOutcome = 'unavailable'
+ }
+ const graphDurationMs = Date.now() - graphStart
+
+ return {
+ vector: { outcome: vectorOutcome, durationMs: vectorDurationMs },
+ metadata: { outcome: metadataOutcome, durationMs: metadataDurationMs },
+ graph: { outcome: graphOutcome, durationMs: graphDurationMs },
+ totalDurationMs: Date.now() - totalStart
+ }
+ }
+
/**
* Explicitly warm up the embedding engine
*
@@ -14759,6 +14928,9 @@ export class Brainy implements BrainyInterface {
// Migration LOCK wait budget — left undefined when omitted so
// awaitMigrationLock() applies its 30 s default at the read site.
migrationWaitTimeoutMs: config?.migrationWaitTimeoutMs ?? undefined,
+ // Apply-phase transaction budget floor — left undefined when omitted so
+ // transactTimeoutBudget() applies its 30 s default at the read site.
+ transactionBudgetFloorMs: config?.transactionBudgetFloorMs ?? undefined,
// Pre-upgrade backup — default-on; opt out with `migrationBackup: false`.
migrationBackup: config?.migrationBackup ?? true,
// Vector index configuration (8.0) — algorithm-neutral surface with
@@ -14774,6 +14946,9 @@ export class Brainy implements BrainyInterface {
// is the active one on a non-reader writer outside unit tests). Explicit
// true/false always wins. See the eagerEmbeddings JSDoc in brainy.types.ts.
eagerEmbeddings: config?.eagerEmbeddings ?? undefined,
+ // Eager index warm at init() — default off (lazy demand-load), the
+ // pre-8.9 behavior. See the warmOnOpen JSDoc in brainy.types.ts.
+ warmOnOpen: config?.warmOnOpen ?? false,
// Plugin configuration - undefined = auto-detect
plugins: config?.plugins ?? undefined,
// Integration Hub - undefined/false = disabled
diff --git a/src/hnsw/hnswIndex.ts b/src/hnsw/hnswIndex.ts
index 605d2a0b..dcf3ed7b 100644
--- a/src/hnsw/hnswIndex.ts
+++ b/src/hnsw/hnswIndex.ts
@@ -54,6 +54,9 @@ export class HnswFlushError extends Error {
* acceleration provider).
*/
export class JsHnswVectorIndex implements VectorIndexProvider {
+ /** Self-identifies as the built-in JS fallback engine — see {@link VectorIndexProvider.providerId}. */
+ readonly providerId = 'js-hnsw'
+
private nouns: Map = new Map()
/**
* Reverse adjacency: `target id → (level → set of node ids that link TO target)`.
diff --git a/src/index.ts b/src/index.ts
index ee01d885..00adc191 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -28,6 +28,9 @@ export type { FileVersion } from './vfs/types.js'
// Export diagnostics result type
export type { DiagnosticsResult } from './brainy.js'
+// brain.warm() — eager index/storage readiness report (per-surface honest
+// outcome + timing). See the WarmReport JSDoc in brainy.ts.
+export type { WarmReport, WarmOutcome } from './brainy.js'
export type {
GraphAuditReport,
GraphAuditDiscrepancy
diff --git a/src/plugin.ts b/src/plugin.ts
index df7c7224..eeb2425a 100644
--- a/src/plugin.ts
+++ b/src/plugin.ts
@@ -381,6 +381,20 @@ export interface GraphIndexProvider {
*/
init?(): Promise
+ /**
+ * @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap
+ * pretouch) so first operations run at steady-state cost. Optional;
+ * absence means the provider demand-loads. Distinct from `init?()`: `init`
+ * is called once automatically during brain startup as part of the
+ * readiness contract (cold-load + rebuild gating); `warm` is an explicit,
+ * separate readiness step a caller opts into via `brain.warm()` (or
+ * `warmOnOpen`) specifically to pre-pay demand-load cost that `init` left
+ * lazy. A provider that already loads everything eagerly in `init?()` may
+ * implement `warm` as a no-op or omit it — `brain.warm()` falls back to a
+ * best-effort read-through probe when absent.
+ */
+ warm?(): Promise
+
/**
* @description OPTIONAL. A native provider returns true from the moment its
* `init()` detects a large epoch-drift until its background
@@ -947,6 +961,22 @@ export interface AtGenerationVectors {
* are retired and never looked up.
*/
export interface VectorIndexProvider {
+ /**
+ * @description OPTIONAL backend identity, stamped into the transaction
+ * op-name strings that surface in journals/timings (e.g.
+ * `AddToVectorIndex(js-hnsw)` vs `AddToVectorIndex()`) — see
+ * `src/transaction/operations/IndexOperations.ts`. Absent → the op-name
+ * string reports `unknown-provider` rather than guessing a backend name
+ * (guessing would resurrect the exact bug this field exists to prevent:
+ * an operator hunting an index that isn't the one actually running). The
+ * built-in JS index always sets this to `'js-hnsw'`; a native provider
+ * sets its own identity here (e.g. its plugin/package name) so operators
+ * reading a journal see the backend that actually ran, never a
+ * backend-specific fossil name from whichever engine wrote the original
+ * op classes.
+ */
+ readonly providerId?: string
+
addItem(item: VectorDocument): Promise
removeItem(id: string): Promise
search(
@@ -1009,6 +1039,20 @@ export interface VectorIndexProvider {
*/
init?(): Promise
+ /**
+ * @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap
+ * pretouch) so first operations run at steady-state cost. Optional;
+ * absence means the provider demand-loads. Distinct from `init?()`: `init`
+ * runs automatically once during brain startup (the cold-load + rebuild
+ * readiness contract above); `warm` is a separate, explicit step a caller
+ * opts into via `brain.warm()` (or `warmOnOpen`) to pre-pay demand-load
+ * cost `init` left lazy — e.g. touching every mmap page rather than just
+ * opening the file. A provider that already loads everything eagerly in
+ * `init?()` may implement `warm` as a no-op or omit it — `brain.warm()`
+ * falls back to a best-effort probe `search()` when absent.
+ */
+ warm?(): Promise
+
/**
* @description OPTIONAL honest durability signal (readiness contract,
* mirrors {@link GraphIndexProvider.isReady}). `true` ⇔ the persisted
diff --git a/src/transaction/Transaction.ts b/src/transaction/Transaction.ts
index 092a2a25..79b53006 100644
--- a/src/transaction/Transaction.ts
+++ b/src/transaction/Transaction.ts
@@ -37,22 +37,47 @@ const DEFAULT_OPTIONS: Required = {
maxRollbackRetries: 3
}
+/**
+ * The floor term of {@link transactTimeoutBudget}'s scaling formula when the
+ * caller supplies neither an explicit override nor `config.transactionBudgetFloorMs`.
+ */
+const DEFAULT_BUDGET_FLOOR_MS = 30_000
+
/**
* The apply-phase budget for a batch of `opCount` operations.
*
- * An explicit override wins untouched. Otherwise the budget SCALES with the
- * batch: `max(30 000 ms, opCount × 2 000 ms)`. The per-op term is calibrated
- * from field data — bulk imports on network-attached disks measure ~2 s per
- * operation (each op pays canonical writes + fsync + index maintenance) — so
- * a flat 30 s budget silently capped honest work at ~15 operations while
- * looking generous for small batches. Scaling keeps small transacts
- * fast-failing and gives bulk ones a budget proportional to the work they
- * actually asked for; a trip still rolls back atomically and throws a
- * retryable, fully-labeled TransactionTimeoutError.
+ * An explicit `override` wins untouched (a caller-specified deadline for this
+ * one batch). Otherwise the budget SCALES with the batch:
+ * `max(floorMs, opCount × 2 000)`, where `floorMs` defaults to `30 000` and
+ * is overridable via `BrainyConfig.transactionBudgetFloorMs` for the whole
+ * brain. The per-op term is calibrated from field data — bulk imports on
+ * network-attached disks measure ~2 s per operation (each op pays canonical
+ * writes + fsync + index maintenance) — so a flat 30 s budget silently capped
+ * honest work at ~15 operations while looking generous for small batches. A
+ * 16-op batch, for example, scales to `max(30 000, 16 × 2 000) = 32 000`.
+ * Scaling keeps small transacts fast-failing and gives bulk ones a budget
+ * proportional to the work they actually asked for.
+ *
+ * This is a **mid-batch** budget only: it governs whether the transaction's
+ * NEXT operation may start (see {@link Transaction.execute}), never whether
+ * already-completed work is rolled back after the fact. A trip mid-batch
+ * still rolls back every operation applied so far, atomically, and throws a
+ * retryable, fully-labeled TransactionTimeoutError — that zero-loss guarantee
+ * doesn't change; only the point at which the clock stops mattering does (at
+ * the last operation, not one check later).
+ *
+ * @param opCount - Number of operations in the batch.
+ * @param override - A full override for this call; wins over everything else.
+ * @param floorMs - The scaling floor for this call (e.g. `config.transactionBudgetFloorMs`).
+ * Ignored when `override` is set. Defaults to `30 000` when omitted.
*/
-export function transactTimeoutBudget(opCount: number, override?: number): number {
+export function transactTimeoutBudget(
+ opCount: number,
+ override?: number,
+ floorMs?: number
+): number {
if (override !== undefined) return override
- return Math.max(30_000, opCount * 2_000)
+ return Math.max(floorMs ?? DEFAULT_BUDGET_FLOOR_MS, opCount * 2_000)
}
/**
@@ -98,7 +123,20 @@ export class Transaction implements TransactionContext {
}
/**
- * Execute all operations atomically
+ * Execute all operations atomically.
+ *
+ * Budget semantics: the configured budget gates starting the next
+ * operation; completed work is never rolled back for elapsed time. Elapsed
+ * time is checked ONLY before starting operation `i` for `i ≥ 1` — never
+ * before operation 0 (an operation always gets to start) and never again
+ * after the final operation completes. Concretely: a single-op transaction
+ * can never time out post-hoc — it either runs (and its result stands, no
+ * matter how long it took) or a `TransactionTimeoutError` is thrown before
+ * it starts, which cannot happen since there is no operation before it. A
+ * multi-op transaction whose op `i` overruns the budget stops op `i+1` from
+ * starting, rolls back operations `0..i` (reverse order), and throws
+ * `TransactionTimeoutError` — that zero-loss rollback guarantee for
+ * mid-batch trips is unchanged; only the post-completion check is gone.
*/
async execute(): Promise {
if (this.state !== 'pending') {
@@ -128,10 +166,14 @@ export class Transaction implements TransactionContext {
// already-applied writes as torn, generation-less state in canonical
// storage.
for (let i = 0; i < this.operations.length; i++) {
- // Budget check BEFORE starting the next operation. A trip here throws
- // into the catch below and rolls back like any other failure — it must
- // never bypass rollback.
- if (Date.now() - this.startTime > this.options.timeout) {
+ // Budget gates STARTING the next operation; completed work is never
+ // rolled back for elapsed time. Skipped for i === 0 (operation 0
+ // always gets to start — nothing has run yet to have overrun) and
+ // never re-checked after the final operation completes (there is no
+ // "next operation" left to gate). A trip here throws into the catch
+ // below and rolls back like any other failure — it must never bypass
+ // rollback.
+ if (i > 0 && Date.now() - this.startTime > this.options.timeout) {
throw new TransactionTimeoutError(this.options.timeout, i, {
elapsedMs: Date.now() - this.startTime,
totalOperations: this.operations.length,
diff --git a/src/transaction/operations/IndexOperations.ts b/src/transaction/operations/IndexOperations.ts
index 97c39270..1c0995c6 100644
--- a/src/transaction/operations/IndexOperations.ts
+++ b/src/transaction/operations/IndexOperations.ts
@@ -2,34 +2,58 @@
* Index Operations with Rollback Support
*
* Provides transactional operations for all indexes:
- * - JsHnswVectorIndex (unified vector index)
+ * - VectorIndexProvider (the JS HNSW fallback, or a native acceleration provider)
* - MetadataIndexManager (roaring bitmap filtering)
* - GraphAdjacencyIndex (LSM-tree graph storage)
*
* Each operation can be executed and rolled back atomically.
*/
-import type { JsHnswVectorIndex } from '../../hnsw/hnswIndex.js'
+import type { VectorIndexProvider, GraphIndexProvider } from '../../plugin.js'
import type { MetadataIndexManager } from '../../utils/metadataIndex.js'
-import type { GraphIndexProvider } from '../../plugin.js'
import type { GraphVerb } from '../../coreTypes.js'
import type { Operation, RollbackAction } from '../types.js'
/**
- * Add to HNSW index with rollback support
+ * Backend identity stamped into an operation's emitted `name` string (e.g.
+ * `AddToVectorIndex(js-hnsw)`), resolved from the provider's own
+ * {@link VectorIndexProvider.providerId} when it self-identifies.
+ *
+ * These operation classes are backend-neutral (the vector index they wrap may
+ * be Brainy's own JS HNSW fallback OR a native acceleration provider), but
+ * their names surface directly in consumer-visible transaction journals and
+ * timings. A provider that hasn't set `providerId` resolves to
+ * `'unknown-provider'` rather than guessing — silently defaulting to the JS
+ * engine's name here is exactly the class of bug this stamping exists to
+ * prevent (an operator diagnosing a backend that isn't the one that ran).
+ */
+function resolveVectorProviderId(index: VectorIndexProvider): string {
+ return index.providerId ?? 'unknown-provider'
+}
+
+/**
+ * Add to the vector index with rollback support.
+ *
+ * Backend-neutral: `index` is whatever the `'vector'` provider factory
+ * returns — Brainy's own JS HNSW fallback, or a native acceleration provider
+ * (e.g. DiskANN). The emitted `name` stamps the active backend (see
+ * {@link resolveVectorProviderId}) so operators reading a transaction journal
+ * or timing trace see which engine actually ran, never a fossil name from
+ * whichever engine happened to be active when this op class was written.
*
* Rollback strategy:
* - Remove item from index
-
*/
-export class AddToHNSWOperation implements Operation {
- readonly name = 'AddToHNSW'
+export class AddToVectorIndexOperation implements Operation {
+ readonly name: string
constructor(
- private readonly index: JsHnswVectorIndex,
+ private readonly index: VectorIndexProvider,
private readonly id: string,
private readonly vector: number[]
- ) {}
+ ) {
+ this.name = `AddToVectorIndex(${resolveVectorProviderId(index)})`
+ }
async execute(): Promise {
// Check if item already exists (for rollback decision)
@@ -59,11 +83,12 @@ export class AddToHNSWOperation implements Operation {
* pre-existence as "existed" made every rollback skip removeItem, leaving
* phantom entries in the index after a failed transaction. The safe default
* is to remove what this operation added — update flows pair this op with a
- * RemoveFromHNSWOperation whose own rollback restores the prior vector, so
- * reverse-order rollback reconstructs the original state either way.
+ * RemoveFromVectorIndexOperation whose own rollback restores the prior
+ * vector, so reverse-order rollback reconstructs the original state either
+ * way.
*/
private async itemExists(id: string): Promise {
- const index = this.index as JsHnswVectorIndex & {
+ const index = this.index as VectorIndexProvider & {
getItem?: (id: string) => Promise
}
if (typeof index.getItem !== 'function') return false
@@ -77,21 +102,27 @@ export class AddToHNSWOperation implements Operation {
}
/**
- * Remove from HNSW index with rollback support
+ * Remove from the vector index with rollback support.
+ *
+ * Backend-neutral: see {@link AddToVectorIndexOperation} — `index` may be the
+ * JS HNSW fallback or a native acceleration provider; the emitted `name`
+ * stamps the active backend.
*
* Rollback strategy:
* - Re-add item to index with original vector
*
* Note: Requires storing the vector for rollback
*/
-export class RemoveFromHNSWOperation implements Operation {
- readonly name = 'RemoveFromHNSW'
+export class RemoveFromVectorIndexOperation implements Operation {
+ readonly name: string
constructor(
- private readonly index: JsHnswVectorIndex,
+ private readonly index: VectorIndexProvider,
private readonly id: string,
private readonly vector: number[] // Required for rollback
- ) {}
+ ) {
+ this.name = `RemoveFromVectorIndex(${resolveVectorProviderId(index)})`
+ }
async execute(): Promise {
// Remove from index
@@ -285,22 +316,24 @@ export class RemoveFromGraphIndexOperation implements Operation {
}
/**
- * Batch operation: Add multiple items to HNSW index
+ * Batch operation: Add multiple items to the vector index (backend-neutral —
+ * see {@link AddToVectorIndexOperation}).
*
* Useful for bulk imports with transaction support.
* Rolls back all items if any fail.
*/
-export class BatchAddToHNSWOperation implements Operation {
- readonly name = 'BatchAddToHNSW'
+export class BatchAddToVectorIndexOperation implements Operation {
+ readonly name: string
- private operations: AddToHNSWOperation[]
+ private operations: AddToVectorIndexOperation[]
constructor(
- index: JsHnswVectorIndex,
+ index: VectorIndexProvider,
items: Array<{ id: string; vector: number[] }>
) {
+ this.name = `BatchAddToVectorIndex(${resolveVectorProviderId(index)})`
this.operations = items.map(
- item => new AddToHNSWOperation(index, item.id, item.vector)
+ item => new AddToVectorIndexOperation(index, item.id, item.vector)
)
}
diff --git a/src/transaction/operations/index.ts b/src/transaction/operations/index.ts
index 422f6dad..c5548e70 100644
--- a/src/transaction/operations/index.ts
+++ b/src/transaction/operations/index.ts
@@ -21,12 +21,12 @@ export {
// Index Operations
export {
- AddToHNSWOperation,
- RemoveFromHNSWOperation,
+ AddToVectorIndexOperation,
+ RemoveFromVectorIndexOperation,
AddToMetadataIndexOperation,
RemoveFromMetadataIndexOperation,
AddToGraphIndexOperation,
RemoveFromGraphIndexOperation,
- BatchAddToHNSWOperation,
+ BatchAddToVectorIndexOperation,
BatchAddToMetadataIndexOperation
} from './IndexOperations.js'
diff --git a/src/types/brainy.types.ts b/src/types/brainy.types.ts
index 1ec4c4c3..8bede8d3 100644
--- a/src/types/brainy.types.ts
+++ b/src/types/brainy.types.ts
@@ -1651,6 +1651,25 @@ export interface BrainyConfig {
*/
disableAutoRebuild?: boolean
+ /**
+ * The floor (ms) of the apply-phase transaction budget's scaling formula:
+ * `max(transactionBudgetFloorMs, opCount × 2 000)`. The budget governs
+ * whether a `transact()` / single-op write's NEXT internal operation may
+ * **start** — never whether already-completed work gets rolled back after
+ * the fact (a single-op write can never time out post-hoc: it either runs
+ * or it commits). A trip mid-batch still rolls back every applied operation
+ * atomically and throws a retryable `TransactionTimeoutError`; only the
+ * floor of the formula is configurable here.
+ *
+ * Raise this when a cold store's first writes after a restart legitimately
+ * take longer than 30s per operation (e.g. page-cache-cold canonical writes
+ * on a large brain) so a bulk `transact()` batch gets a proportionally
+ * larger runway instead of tripping mid-batch. Lower it to fail faster on a
+ * latency-sensitive write path. Default: `30000` (30 s) — unchanged from
+ * pre-8.9 behavior.
+ */
+ transactionBudgetFloorMs?: number
+
/**
* How long (ms) an operation **waits** on the coordinated 7.x → 8.0 migration
* before throwing a retryable `MigrationInProgressError`.
@@ -1806,6 +1825,23 @@ export interface BrainyConfig {
*/
eagerEmbeddings?: boolean
+ /**
+ * When `true`, `init()` **awaits `warm()`** (see {@link Brainy.warm}) before
+ * it resolves — blocking startup until the vector index, metadata index,
+ * and graph adjacency have all faulted their backing storage in. This is a
+ * deliberate operator opt-in trade: startup takes longer, but the FIRST
+ * request after a cold restart runs at steady-state cost instead of paying
+ * page-cache-miss / demand-load latency on the critical path.
+ *
+ * Runs AFTER `init()`'s own sequence completes (index construction, crash
+ * recovery, migrations, VFS bootstrap) — never in place of any of it — so
+ * `warm()` always operates on a fully-initialized brain.
+ *
+ * Default: `false` (lazy — the pre-8.9 behavior: indexes demand-load on
+ * first access).
+ */
+ warmOnOpen?: boolean
+
// Plugin configuration
// Controls which plugins are loaded during init().
// - undefined (default): guarded auto-detection of the first-party
diff --git a/src/utils/metadataIndex.ts b/src/utils/metadataIndex.ts
index c772bce4..fdb17c22 100644
--- a/src/utils/metadataIndex.ts
+++ b/src/utils/metadataIndex.ts
@@ -423,6 +423,46 @@ export class MetadataIndexManager implements MetadataIndexProvider {
prodLog.debug('✅ Type-aware cache warming completed')
}
+ /**
+ * Full hydration — the {@link Brainy.warm} readiness seam for the metadata
+ * index. Unlike {@link warmCache} / {@link warmCacheForTopTypes} (which
+ * warm only a heuristic subset: common fields plus the top-N types' top
+ * fields), this loads EVERY field's sparse index the field registry knows
+ * about — a real read through {@link loadSparseIndex} into the unified
+ * cache for each field, not a stat/existence check. Idempotent: an
+ * already-cached field's `loadSparseIndex` call is a cheap cache hit.
+ *
+ * Re-reads the field registry first when `fieldIndexes` is empty (a warm()
+ * call issued before `init()` populated it would otherwise hydrate
+ * nothing), then loads every discovered field in parallel.
+ */
+ async hydrateAll(): Promise {
+ if (this.fieldIndexes.size === 0) {
+ await this.loadFieldRegistry()
+ }
+
+ const fields = Array.from(this.fieldIndexes.keys())
+ if (fields.length === 0) {
+ prodLog.debug('[MetadataIndex] hydrateAll: no persisted fields to hydrate')
+ return
+ }
+
+ prodLog.debug(`[MetadataIndex] hydrateAll: loading ${fields.length} field(s) — ${fields.join(', ')}`)
+
+ await Promise.all(
+ fields.map(async field => {
+ try {
+ await this.loadSparseIndex(field)
+ } catch (error) {
+ // A single field's load failure doesn't abort the rest of the
+ // hydration — warm() is a best-effort readiness step, never a
+ // correctness gate (queries still demand-load on miss).
+ prodLog.debug(`[MetadataIndex] hydrateAll: field '${field}' failed to load:`, error)
+ }
+ })
+ )
+ }
+
/**
* Acquire an in-memory lock for coordinating concurrent metadata index writes
* Uses in-memory locks since MetadataIndexManager doesn't have direct file system access
diff --git a/tests/unit/brainy/warm.test.ts b/tests/unit/brainy/warm.test.ts
new file mode 100644
index 00000000..437b7785
--- /dev/null
+++ b/tests/unit/brainy/warm.test.ts
@@ -0,0 +1,309 @@
+/**
+ * @module tests/unit/brainy/warm
+ * @description Coverage for `brain.warm()` / `warmOnOpen` / the provider
+ * `warm?()` contract (the cold-restart readiness fix): first operations after
+ * a cold restart should run at steady-state cost instead of paying
+ * demand-load latency on the critical path.
+ *
+ * Uses a real, in-process fake plugin provider implementing the actual
+ * `VectorIndexProvider` contract from `src/plugin.ts` — the real seam a
+ * native provider (e.g. a disk-native accelerator) plugs into. The real
+ * built-in JS metadata index and graph adjacency index run against real
+ * (filesystem or in-memory) storage with pre-existing data, so their
+ * hydration paths are exercised for real, not mocked.
+ */
+import { describe, it, expect, afterEach } from 'vitest'
+import * as fs from 'node:fs'
+import * as os from 'node:os'
+import * as path from 'node:path'
+import { Brainy } from '../../../src/brainy.js'
+import { NounType, VerbType } from '../../../src/types/graphTypes.js'
+import type { VectorIndexProvider } from '../../../src/plugin.js'
+import type { VectorDocument, Vector } from '../../../src/coreTypes.js'
+import { MetadataIndexManager } from '../../../src/utils/metadataIndex.js'
+import { GraphAdjacencyIndex } from '../../../src/graph/graphAdjacencyIndex.js'
+
+const tmpDirs: string[] = []
+function mkTmp(): string {
+ const d = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-warm-'))
+ tmpDirs.push(d)
+ return d
+}
+afterEach(() => {
+ for (const d of tmpDirs.splice(0)) fs.rmSync(d, { recursive: true, force: true })
+})
+
+// Brainy's ValidationConfig fixes vectors at exactly 384 dimensions
+// (src/utils/paramValidation.ts) — match it so `add()` doesn't reject test data.
+const DIM = 384
+const V = (seed = 1): number[] => Array.from({ length: DIM }, (_, i) => Math.sin(seed + i))
+
+/**
+ * A real (not mocked) VectorIndexProvider implementation, backed by a plain
+ * Map, that optionally implements `warm()` — the exact seam
+ * `AddToVectorIndexOperation` / `brain.warm()` call through.
+ */
+class FakeVectorProvider implements VectorIndexProvider {
+ readonly items = new Map()
+ warmCalls = 0
+ searchCalls: Array<{ k?: number }> = []
+ // Only present on the instance when the constructor is told to — mirrors a
+ // real provider that may or may not implement the optional hook. Assigned
+ // in the constructor BODY (not as a field initializer): under native ES
+ // class fields, field initializers run before constructor-body statements
+ // — including the parameter-property assignment — so referencing a
+ // parameter property from a field initializer would see it as still
+ // `undefined`.
+ warm?: () => Promise
+
+ constructor(hasWarm: boolean) {
+ if (hasWarm) {
+ this.warm = async (): Promise => {
+ this.warmCalls++
+ }
+ }
+ }
+
+ async addItem(item: VectorDocument): Promise {
+ this.items.set(item.id, item.vector)
+ return item.id
+ }
+ async removeItem(id: string): Promise {
+ return this.items.delete(id)
+ }
+ async search(_queryVector: Vector, k?: number): Promise> {
+ this.searchCalls.push({ k })
+ return [...this.items.keys()].slice(0, k ?? 10).map((id) => [id, 0])
+ }
+ size(): number {
+ return this.items.size
+ }
+ clear(): void {
+ this.items.clear()
+ }
+ async rebuild(): Promise {}
+ async flush(): Promise {
+ return 0
+ }
+ getPersistMode(): 'immediate' | 'deferred' {
+ return 'deferred'
+ }
+}
+
+/** Registers `provider` under the `'vector'` plugin key, before `init()`. */
+function useFakeVectorProvider(brain: Brainy, provider: FakeVectorProvider): void {
+ brain.use({
+ name: 'fake-vector-provider',
+ activate: async (ctx: any) => {
+ ctx.registerProvider('vector', () => provider)
+ return true
+ }
+ })
+}
+
+describe('brain.warm()', () => {
+ it('(a) calls the vector provider\'s warm() when present and reports "warmed"', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ const provider = new FakeVectorProvider(true)
+ useFakeVectorProvider(brain, provider)
+ await brain.init()
+ await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
+
+ const report = await brain.warm()
+
+ expect(provider.warmCalls).toBe(1)
+ expect(provider.searchCalls.length).toBe(0) // warm() ran — no probe fallback
+ expect(report.vector.outcome).toBe('warmed')
+ await brain.close()
+ })
+
+ it('(b) falls back to a probe search() when warm() is absent and reports "probed", never "warmed"', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ const provider = new FakeVectorProvider(false)
+ useFakeVectorProvider(brain, provider)
+ await brain.init()
+ await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
+ await brain.add({ data: 'b', type: NounType.Thing, vector: V(2) })
+
+ const report = await brain.warm()
+
+ expect(provider.warmCalls).toBe(0) // no warm() on this provider
+ expect(provider.searchCalls.length).toBe(1) // the probe ran
+ expect(provider.searchCalls[0].k).toBe(Math.min(100, provider.size())) // k = min(100, size)
+ expect(report.vector.outcome).toBe('probed')
+ expect(report.vector.outcome).not.toBe('warmed') // never conflated
+ await brain.close()
+ })
+
+ it('vector: reports "unavailable" when there is nothing to probe (empty index, unknown dimension)', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ // A disk-native provider MAY honestly report 0 resident entries while
+ // durable data exists on disk (the same posture documented on
+ // `VectorIndexProvider.isReady` — "an mmap/disk-native index may
+ // legitimately report 0 resident entries"). warm()'s probe fallback
+ // reads `size()`, so this is the real trigger for "nothing to probe":
+ // never a k=0 search, an honest skip.
+ const provider = new FakeVectorProvider(false)
+ provider.size = () => 0
+ useFakeVectorProvider(brain, provider)
+ await brain.init() // the VFS root bootstrap write sets `dimensions`, but size() still reports 0
+
+ const report = await brain.warm()
+
+ expect(provider.searchCalls.length).toBe(0) // nothing probed
+ expect(report.vector.outcome).toBe('unavailable')
+ await brain.close()
+ })
+
+ it('(c) metadata + graph hydration paths actually execute against filesystem storage with pre-existing data', async () => {
+ const dir = mkTmp()
+
+ // Build a brain with real data (including a field NOT in the metadata
+ // index's own common-fields warm subset — 'wave' — so hydrateAll()'s
+ // FULL hydration is distinguishable from init()'s partial warmCache()),
+ // and real graph edges, then close it (persisting everything).
+ const seed = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'filesystem', path: dir },
+ silent: true
+ })
+ await seed.init()
+ const ids: string[] = []
+ for (let i = 0; i < 6; i++) {
+ ids.push(
+ await seed.add({
+ data: `entity ${i}`,
+ type: NounType.Thing,
+ metadata: { wave: i % 3 },
+ vector: V(i + 1)
+ })
+ )
+ }
+ for (let i = 0; i + 1 < ids.length; i++) {
+ await seed.relate({ from: ids[i], to: ids[i + 1], type: VerbType.RelatedTo })
+ }
+ await seed.close()
+
+ // Cold-reopen a FRESH instance and spy on the real hydration seams before
+ // calling warm(), so we assert they actually ran (not just that the
+ // report claims they did).
+ const metaHydrateCalls: number[] = []
+ const loadedFields: string[] = []
+ const graphInitCalls: number[] = []
+
+ const origHydrateAll = MetadataIndexManager.prototype.hydrateAll
+ const origLoadSparseIndex = (MetadataIndexManager.prototype as any).loadSparseIndex
+ const origGraphInit = GraphAdjacencyIndex.prototype.init
+
+ MetadataIndexManager.prototype.hydrateAll = async function (...args: any[]) {
+ metaHydrateCalls.push(1)
+ return origHydrateAll.apply(this, args as any)
+ }
+ ;(MetadataIndexManager.prototype as any).loadSparseIndex = async function (
+ field: string,
+ ...args: any[]
+ ) {
+ loadedFields.push(field)
+ return origLoadSparseIndex.apply(this, [field, ...args] as any)
+ }
+ GraphAdjacencyIndex.prototype.init = async function (...args: any[]) {
+ graphInitCalls.push(1)
+ return origGraphInit.apply(this, args as any)
+ }
+
+ try {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'filesystem', path: dir },
+ silent: true
+ })
+ await brain.init()
+
+ const report = await brain.warm()
+
+ expect(metaHydrateCalls.length).toBe(1) // hydrateAll() actually ran
+ expect(loadedFields).toContain('wave') // a NON-common field was loaded — full hydration, not the heuristic subset
+ expect(report.metadata.outcome).toBe('warmed')
+
+ expect(graphInitCalls.length).toBeGreaterThanOrEqual(1) // graph's full-load seam ran (once at brain init, once via warm())
+ expect(report.graph.outcome).toBe('warmed')
+
+ // Correctness survives: the hydrated data still answers queries.
+ const byWhere = await brain.find({ type: NounType.Thing, where: { wave: 1 } })
+ expect(byWhere.length).toBe(2) // waves 1,4 of 0..5
+
+ await brain.close()
+ } finally {
+ MetadataIndexManager.prototype.hydrateAll = origHydrateAll
+ ;(MetadataIndexManager.prototype as any).loadSparseIndex = origLoadSparseIndex
+ GraphAdjacencyIndex.prototype.init = origGraphInit
+ }
+ })
+
+ it('(d) warmOnOpen: true runs warm() during init() — observable via the fake provider', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ warmOnOpen: true,
+ silent: true
+ })
+ const provider = new FakeVectorProvider(true)
+ useFakeVectorProvider(brain, provider)
+
+ // No explicit brain.warm() call — warmOnOpen must have run it as part of init().
+ await brain.init()
+
+ expect(provider.warmCalls).toBe(1)
+ await brain.close()
+ })
+
+ it('warmOnOpen defaults to false — init() does NOT run warm() unless opted in', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ const provider = new FakeVectorProvider(true)
+ useFakeVectorProvider(brain, provider)
+
+ await brain.init()
+
+ expect(provider.warmCalls).toBe(0)
+ await brain.close()
+ })
+
+ it('(e) WarmReport shape: outcome literal + durationMs per surface, plus totalDurationMs', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ const provider = new FakeVectorProvider(true)
+ useFakeVectorProvider(brain, provider)
+ await brain.init()
+ await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
+
+ const report = await brain.warm()
+
+ for (const surface of ['vector', 'metadata', 'graph'] as const) {
+ expect(['warmed', 'probed', 'unavailable']).toContain(report[surface].outcome)
+ expect(typeof report[surface].durationMs).toBe('number')
+ expect(report[surface].durationMs).toBeGreaterThanOrEqual(0)
+ }
+ expect(typeof report.totalDurationMs).toBe('number')
+ expect(report.totalDurationMs).toBeGreaterThanOrEqual(0)
+ await brain.close()
+ })
+})
diff --git a/tests/unit/transaction/budget-commit-completed-work.test.ts b/tests/unit/transaction/budget-commit-completed-work.test.ts
new file mode 100644
index 00000000..a32489de
--- /dev/null
+++ b/tests/unit/transaction/budget-commit-completed-work.test.ts
@@ -0,0 +1,149 @@
+/**
+ * @module tests/unit/transaction/budget-commit-completed-work
+ * @description Regression coverage for the "commit-completed-work" transaction
+ * budget semantics: the budget gates STARTING the next operation; it never
+ * converts already-completed work into a rollback. Concretely:
+ *
+ * - A single-op transaction can never time out post-hoc — its one operation
+ * either runs (and the transaction commits, however long it took) or it
+ * never gets to start (which cannot happen: there is no operation before it
+ * to have overrun the budget).
+ * - A multi-op transaction whose op `i` overruns the budget stops op `i+1`
+ * from starting: everything applied so far rolls back atomically and a
+ * `TransactionTimeoutError` is thrown — the mid-batch zero-loss guarantee is
+ * unchanged.
+ * - `transactTimeoutBudget`'s scaling floor (`max(floorMs, opCount * 2000)`)
+ * is overridable per call, the seam `BrainyConfig.transactionBudgetFloorMs`
+ * feeds at the brainy.ts read sites.
+ *
+ * Uses a real class implementing the `Operation` interface (not an anonymous
+ * object literal, not a mock of Transaction internals) so the exercised path
+ * is exactly what a real caller's operation looks like.
+ */
+import { describe, it, expect } from 'vitest'
+import { Transaction, transactTimeoutBudget } from '../../../src/transaction/Transaction.js'
+import type { Operation, RollbackAction } from '../../../src/transaction/types.js'
+import { TransactionTimeoutError } from '../../../src/transaction/errors.js'
+
+const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms))
+
+/**
+ * A real `Operation` implementation that sleeps for `delayMs` before applying
+ * a write to `log`, and returns a rollback action that removes it again. Used
+ * to deterministically make one operation "slow" relative to a transaction's
+ * budget without touching any Transaction internals.
+ */
+class SlowOperation implements Operation {
+ readonly name: string
+ executed = false
+ rolledBack = false
+
+ constructor(
+ private readonly log: string[],
+ label: string,
+ private readonly delayMs: number
+ ) {
+ this.name = label
+ }
+
+ async execute(): Promise {
+ this.executed = true
+ if (this.delayMs > 0) await sleep(this.delayMs)
+ this.log.push(this.name)
+ return async () => {
+ this.rolledBack = true
+ const idx = this.log.indexOf(this.name)
+ if (idx >= 0) this.log.splice(idx, 1)
+ }
+ }
+}
+
+describe('Transaction budget — commit-completed-work semantics', () => {
+ it('(a) single-op transaction whose op overruns the budget COMMITS — no rollback, no throw', async () => {
+ const log: string[] = []
+ const op = new SlowOperation(log, 'slow-single-op', 40)
+
+ // Budget (5ms) is far smaller than the op's 40ms — under the OLD
+ // (post-completion-check) semantics this would have thrown and rolled
+ // back after the op finished. Under the new semantics there is no
+ // operation after it to gate, so it commits.
+ const tx = new Transaction({ timeout: 5 })
+ tx.addOperation(op)
+
+ await expect(tx.execute()).resolves.toBeUndefined()
+
+ expect(tx.getState()).toBe('committed')
+ expect(op.executed).toBe(true)
+ expect(op.rolledBack).toBe(false)
+ expect(log).toEqual(['slow-single-op'])
+ })
+
+ it('(b) two-op transaction: op0 overruns the budget → op1 never starts, op0 rolls back, throws TransactionTimeoutError', async () => {
+ const log: string[] = []
+ const op0 = new SlowOperation(log, 'op0-overruns', 40)
+ const op1 = new SlowOperation(log, 'op1-never-starts', 0)
+
+ const tx = new Transaction({ timeout: 5 })
+ tx.addOperation(op0)
+ tx.addOperation(op1)
+
+ const error = await tx.execute().catch((e) => e)
+
+ expect(error).toBeInstanceOf(TransactionTimeoutError)
+ expect(op0.executed).toBe(true)
+ expect(op0.rolledBack).toBe(true)
+ expect(op1.executed).toBe(false)
+ expect(op1.rolledBack).toBe(false)
+ expect(log).toEqual([]) // op0's write was undone; op1 never wrote
+ expect(tx.getState()).toBe('rolled_back')
+ })
+
+ it('a single-op transaction never checks the budget before its only operation starts', async () => {
+ // Budget of 0ms: under the old "check before every op including 0" code
+ // this could theoretically trip before op 0 even started (if any
+ // measurable time elapsed between startTime capture and the check). The
+ // documented contract is stronger: operation 0 ALWAYS gets to start.
+ const log: string[] = []
+ const op = new SlowOperation(log, 'only-op', 5)
+
+ const tx = new Transaction({ timeout: 0 })
+ tx.addOperation(op)
+
+ await expect(tx.execute()).resolves.toBeUndefined()
+ expect(tx.getState()).toBe('committed')
+ expect(op.executed).toBe(true)
+ })
+
+ it('(c) transactTimeoutBudget: an explicit floorMs overrides the 30s default floor', () => {
+ // Below the per-op scaling term, a raised floor wins.
+ expect(transactTimeoutBudget(1, undefined, 5_000)).toBe(5_000) // 1 op × 2000 = 2000 < 5000 floor
+ expect(transactTimeoutBudget(1)).toBe(30_000) // unchanged default when floorMs omitted
+
+ // Above the floor, opCount * 2000 still wins over a smaller floor.
+ expect(transactTimeoutBudget(10, undefined, 5_000)).toBe(20_000) // 10 × 2000 = 20000 > 5000 floor
+
+ // A full `override` always wins, regardless of floorMs.
+ expect(transactTimeoutBudget(10, 999, 5_000)).toBe(999)
+ })
+
+ it('(c) a configured floor changes real Transaction behavior end-to-end via TransactionManager-style options', async () => {
+ // Simulates how brainy.ts wires `this.config.transactionBudgetFloorMs`
+ // into `transactTimeoutBudget()`: opCount 0 isolates the floor term
+ // (0 × 2000 = 0), so a lowered floor makes a normally-generous budget
+ // fail-fast for a real 2-op Transaction.
+ const lowFloorBudget = transactTimeoutBudget(0, undefined, 10) // 10ms floor
+ expect(lowFloorBudget).toBe(10)
+
+ const log: string[] = []
+ const op0 = new SlowOperation(log, 'op0', 30)
+ const op1 = new SlowOperation(log, 'op1-gated-by-lowered-floor', 0)
+
+ const tx = new Transaction({ timeout: lowFloorBudget })
+ tx.addOperation(op0)
+ tx.addOperation(op1)
+
+ const error = await tx.execute().catch((e) => e)
+ expect(error).toBeInstanceOf(TransactionTimeoutError)
+ expect(op1.executed).toBe(false)
+ })
+})
diff --git a/tests/unit/transaction/vectorIndexOperations-rename.test.ts b/tests/unit/transaction/vectorIndexOperations-rename.test.ts
new file mode 100644
index 00000000..80168346
--- /dev/null
+++ b/tests/unit/transaction/vectorIndexOperations-rename.test.ts
@@ -0,0 +1,152 @@
+/**
+ * @module tests/unit/transaction/vectorIndexOperations-rename
+ * @description Coverage for the backend-neutral vector-index transaction
+ * operations (formerly `AddToHNSWOperation` / `RemoveFromHNSWOperation`).
+ * These op-name strings surface directly in consumer-visible transaction
+ * journals and timings — a fossil "HNSW" name misdirects an operator running
+ * a native (non-HNSW) vector provider. Verifies:
+ * 1. rollback wiring stayed byte-identical under the rename,
+ * 2. the emitted `name` stamps the active backend — `'js-hnsw'` for
+ * Brainy's own JS index, a provider's own `providerId` when it
+ * self-identifies, and the honest `'unknown-provider'` literal when
+ * neither applies (never a silently-wrong guess).
+ */
+import { describe, it, expect } from 'vitest'
+import {
+ AddToVectorIndexOperation,
+ RemoveFromVectorIndexOperation,
+ BatchAddToVectorIndexOperation
+} from '../../../src/transaction/operations/IndexOperations.js'
+import type { VectorIndexProvider } from '../../../src/plugin.js'
+import type { VectorDocument, Vector } from '../../../src/coreTypes.js'
+import { JsHnswVectorIndex } from '../../../src/hnsw/hnswIndex.js'
+
+/**
+ * A minimal, real (not mocked) VectorIndexProvider implementation backed by
+ * a plain Map — exercises the exact contract the operations call through,
+ * without any Brainy internals.
+ */
+class FakeVectorProvider implements VectorIndexProvider {
+ readonly items = new Map()
+ constructor(readonly providerId?: string) {}
+
+ async addItem(item: VectorDocument): Promise {
+ this.items.set(item.id, item.vector)
+ return item.id
+ }
+ async removeItem(id: string): Promise {
+ return this.items.delete(id)
+ }
+ async getItem(id: string): Promise {
+ const vector = this.items.get(id)
+ return vector ? { id, vector } : undefined
+ }
+ async search(): Promise> {
+ return []
+ }
+ size(): number {
+ return this.items.size
+ }
+ clear(): void {
+ this.items.clear()
+ }
+ async rebuild(): Promise {}
+ async flush(): Promise {
+ return 0
+ }
+ getPersistMode(): 'immediate' | 'deferred' {
+ return 'immediate'
+ }
+}
+
+describe('Vector index transaction operations — backend-neutral rename', () => {
+ describe('rollback wiring (byte-identical behavior under the rename)', () => {
+ it('AddToVectorIndexOperation rollback removes a newly-added item', async () => {
+ const provider = new FakeVectorProvider('fake-provider')
+ const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+
+ const rollback = await op.execute()
+ expect(provider.items.has('id-1')).toBe(true)
+
+ await rollback()
+ expect(provider.items.has('id-1')).toBe(false)
+ })
+
+ it('AddToVectorIndexOperation rollback is a no-op when the item pre-existed (update semantics)', async () => {
+ const provider = new FakeVectorProvider('fake-provider')
+ await provider.addItem({ id: 'id-1', vector: [9, 9, 9] })
+
+ const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+ const rollback = await op.execute()
+ expect(provider.items.get('id-1')).toEqual([1, 2, 3])
+
+ await rollback()
+ // Pre-existing item is NOT removed by rollback — it stays (the update
+ // itself is not undone by this op; that's RemoveFromVectorIndexOperation's job).
+ expect(provider.items.has('id-1')).toBe(true)
+ })
+
+ it('RemoveFromVectorIndexOperation rollback re-adds the item with its original vector', async () => {
+ const provider = new FakeVectorProvider('fake-provider')
+ await provider.addItem({ id: 'id-1', vector: [4, 5, 6] })
+
+ const op = new RemoveFromVectorIndexOperation(provider, 'id-1', [4, 5, 6])
+ const rollback = await op.execute()
+ expect(provider.items.has('id-1')).toBe(false)
+
+ await rollback()
+ expect(provider.items.get('id-1')).toEqual([4, 5, 6])
+ })
+
+ it('BatchAddToVectorIndexOperation rolls back every item in reverse order on undo', async () => {
+ const provider = new FakeVectorProvider('fake-provider')
+ const op = new BatchAddToVectorIndexOperation(provider, [
+ { id: 'a', vector: [1] },
+ { id: 'b', vector: [2] },
+ { id: 'c', vector: [3] }
+ ])
+
+ const rollback = await op.execute()
+ expect([...provider.items.keys()].sort()).toEqual(['a', 'b', 'c'])
+
+ await rollback()
+ expect(provider.items.size).toBe(0)
+ })
+ })
+
+ describe('backend stamping in the emitted op name', () => {
+ it('stamps the real JS HNSW index as js-hnsw (self-identifying providerId)', () => {
+ const index = new JsHnswVectorIndex()
+ const addOp = new AddToVectorIndexOperation(index, 'id-1', [1, 2, 3])
+ const removeOp = new RemoveFromVectorIndexOperation(index, 'id-1', [1, 2, 3])
+
+ expect(addOp.name).toBe('AddToVectorIndex(js-hnsw)')
+ expect(removeOp.name).toBe('RemoveFromVectorIndex(js-hnsw)')
+ })
+
+ it('stamps a self-identifying provider\'s own providerId, never "HNSW"', () => {
+ const provider = new FakeVectorProvider('acme-diskann')
+ const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+ const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+ const batchOp = new BatchAddToVectorIndexOperation(provider, [{ id: 'a', vector: [1] }])
+
+ expect(addOp.name).toBe('AddToVectorIndex(acme-diskann)')
+ expect(removeOp.name).toBe('RemoveFromVectorIndex(acme-diskann)')
+ expect(batchOp.name).toBe('BatchAddToVectorIndex(acme-diskann)')
+ expect(addOp.name).not.toContain('HNSW')
+ expect(removeOp.name).not.toContain('HNSW')
+ })
+
+ it('honestly reports "unknown-provider" rather than guessing when a provider omits providerId', () => {
+ const provider = new FakeVectorProvider(undefined)
+ const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+ const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+
+ expect(addOp.name).toBe('AddToVectorIndex(unknown-provider)')
+ expect(removeOp.name).toBe('RemoveFromVectorIndex(unknown-provider)')
+ // Never silently falls back to the JS engine's name for a provider it
+ // knows nothing about — that's the exact fossil-naming bug being fixed.
+ expect(addOp.name).not.toContain('js-hnsw')
+ })
+ })
+})
From 3be4ba96c299b04ce12ffc57898a629c0e8eb02d Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Thu, 23 Jul 2026 08:53:05 -0700
Subject: [PATCH 05/15] feat: vector provider identity is a required name field
(hnsw-js), rendered [vector-index:]
Reconciles the vector-index rename to the ruled three-layer naming: the
provider contract's identity field is now a REQUIRED readonly name (was
optional providerId), self-reported and truthful, rendered as
[vector-index:] where the index identifies itself and stamped into
the op-name strings journals already parse (AddToVectorIndex()).
The built-in JS engine names itself hnsw-js. A runtime provider instance
compiled against the previous optional contract is tolerated - never
crashed on, never silently mislabeled: it stamps unknown-provider and
emits one loud warning naming the missing field. Graph index operations
keep their static names (they never interpolate provider identity), and
no public API exports a provider-routed hnsw-carrying name, so no
deprecation shim is required.
---
RELEASES.md | 12 ++++-
src/hnsw/hnswIndex.ts | 4 +-
src/plugin.ts | 30 ++++++-----
src/transaction/operations/IndexOperations.ts | 29 +++++++---
tests/unit/brainy/warm.test.ts | 1 +
.../vectorIndexOperations-rename.test.ts | 54 +++++++++++++------
6 files changed, 92 insertions(+), 38 deletions(-)
diff --git a/RELEASES.md b/RELEASES.md
index 113b327d..89bf38e7 100644
--- a/RELEASES.md
+++ b/RELEASES.md
@@ -82,10 +82,20 @@ demand-load cost OFF the transaction path.
`AddToVectorIndex(...)`/`RemoveFromVectorIndex(...)` — the old names hard-coded an
algorithm that may not be the one actually running (a non-HNSW native vector provider
emitting `"RemoveFromHNSW"` has sent an operator hunting an index that doesn't exist).
- The parenthesized suffix names the ACTIVE backend: `js-hnsw` for the built-in engine, or
+ The parenthesized suffix names the ACTIVE backend: `hnsw-js` for the built-in engine, or
the native provider's own identity when it self-identifies. **If you parse these op-name
strings** (log processors, journal tooling), update your matcher from
`AddToHNSW`/`RemoveFromHNSW` to `AddToVectorIndex(`/`RemoveFromVectorIndex(`.
+- **Provider identity is now a REQUIRED `name` field** on the vector provider contract
+ (`VectorIndexProvider.name`, `src/plugin.ts`) — every implementation self-reports its own
+ identity truthfully (its algorithm/engine), never inheriting a default. It renders as the
+ op-name suffix above and, wherever the vector index identifies itself in prose log lines,
+ as the tag `[vector-index:]`. **Native provider adoption is a one-line change**:
+ declare `readonly name = ''`. A provider instance that still lacks
+ `name` at runtime (an older native build compiled against the previous, optional field) is
+ never crashed on and never silently mislabeled: it stamps `unknown-provider` and emits one
+ loud warning naming the missing field, so the gap is discoverable instead of a permanent
+ fossil label in every journal line.
## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close())
diff --git a/src/hnsw/hnswIndex.ts b/src/hnsw/hnswIndex.ts
index dcf3ed7b..eb2acd71 100644
--- a/src/hnsw/hnswIndex.ts
+++ b/src/hnsw/hnswIndex.ts
@@ -54,8 +54,8 @@ export class HnswFlushError extends Error {
* acceleration provider).
*/
export class JsHnswVectorIndex implements VectorIndexProvider {
- /** Self-identifies as the built-in JS fallback engine — see {@link VectorIndexProvider.providerId}. */
- readonly providerId = 'js-hnsw'
+ /** Self-identifies as the built-in JS fallback engine — see {@link VectorIndexProvider.name}. */
+ readonly name = 'hnsw-js'
private nouns: Map = new Map()
/**
diff --git a/src/plugin.ts b/src/plugin.ts
index eeb2425a..02639955 100644
--- a/src/plugin.ts
+++ b/src/plugin.ts
@@ -962,20 +962,24 @@ export interface AtGenerationVectors {
*/
export interface VectorIndexProvider {
/**
- * @description OPTIONAL backend identity, stamped into the transaction
- * op-name strings that surface in journals/timings (e.g.
- * `AddToVectorIndex(js-hnsw)` vs `AddToVectorIndex()`) — see
- * `src/transaction/operations/IndexOperations.ts`. Absent → the op-name
- * string reports `unknown-provider` rather than guessing a backend name
- * (guessing would resurrect the exact bug this field exists to prevent:
- * an operator hunting an index that isn't the one actually running). The
- * built-in JS index always sets this to `'js-hnsw'`; a native provider
- * sets its own identity here (e.g. its plugin/package name) so operators
- * reading a journal see the backend that actually ran, never a
- * backend-specific fossil name from whichever engine wrote the original
- * op classes.
+ * @description REQUIRED self-reported implementation identity, rendered as
+ * `[vector-index:]` in prose log lines and stamped into the
+ * transaction op-name strings that surface in journals/timings (e.g.
+ * `AddToVectorIndex(hnsw-js)`) — see
+ * `src/transaction/operations/IndexOperations.ts`. A provider must name
+ * itself TRUTHFULLY (its own algorithm/engine, e.g. its plugin/package
+ * name) and must never inherit a default — guessing a backend name would
+ * resurrect the exact bug this field exists to prevent: an operator
+ * hunting an index that isn't the one actually running. The built-in JS
+ * index always sets this to `'hnsw-js'`; a native provider picks its own
+ * string. At the TypeScript level this field is required; a runtime
+ * instance from an older provider compiled against the previous optional
+ * `providerId` contract is tolerated (never crashes, never silently
+ * mislabeled) — see `resolveVectorProviderId` in
+ * `src/transaction/operations/IndexOperations.ts`, which stamps
+ * `'unknown-provider'` and emits one loud warning for that case.
*/
- readonly providerId?: string
+ readonly name: string
addItem(item: VectorDocument): Promise
removeItem(id: string): Promise
diff --git a/src/transaction/operations/IndexOperations.ts b/src/transaction/operations/IndexOperations.ts
index 1c0995c6..d130bb3f 100644
--- a/src/transaction/operations/IndexOperations.ts
+++ b/src/transaction/operations/IndexOperations.ts
@@ -16,19 +16,34 @@ import type { Operation, RollbackAction } from '../types.js'
/**
* Backend identity stamped into an operation's emitted `name` string (e.g.
- * `AddToVectorIndex(js-hnsw)`), resolved from the provider's own
- * {@link VectorIndexProvider.providerId} when it self-identifies.
+ * `AddToVectorIndex(hnsw-js)`), resolved from the provider's own
+ * {@link VectorIndexProvider.name} self-report.
*
* These operation classes are backend-neutral (the vector index they wrap may
* be Brainy's own JS HNSW fallback OR a native acceleration provider), but
* their names surface directly in consumer-visible transaction journals and
- * timings. A provider that hasn't set `providerId` resolves to
- * `'unknown-provider'` rather than guessing — silently defaulting to the JS
- * engine's name here is exactly the class of bug this stamping exists to
- * prevent (an operator diagnosing a backend that isn't the one that ran).
+ * timings. `name` is REQUIRED at the TypeScript level — but a native provider
+ * instance compiled against the previous (pre-required) contract can still
+ * reach this function at runtime without it. That legacy case is tolerated,
+ * never crashed on and never silently mislabeled: it resolves to
+ * `'unknown-provider'` and emits ONE loud warning naming the missing contract
+ * field, so the fix (implement `name`) is discoverable rather than a silent
+ * fossil label in every journal line thereafter.
*/
+const warnedMissingName = new WeakSet()
+
function resolveVectorProviderId(index: VectorIndexProvider): string {
- return index.providerId ?? 'unknown-provider'
+ const name = (index as { name?: unknown }).name
+ if (typeof name === 'string') return name
+ if (!warnedMissingName.has(index)) {
+ warnedMissingName.add(index)
+ console.warn(
+ '[vector-index] provider is missing the required `name` field (VectorIndexProvider.name, ' +
+ 'required since 8.10.0) — stamping "unknown-provider" in transaction op names until the ' +
+ 'provider declares its own identity.'
+ )
+ }
+ return 'unknown-provider'
}
/**
diff --git a/tests/unit/brainy/warm.test.ts b/tests/unit/brainy/warm.test.ts
index 437b7785..ce213696 100644
--- a/tests/unit/brainy/warm.test.ts
+++ b/tests/unit/brainy/warm.test.ts
@@ -44,6 +44,7 @@ const V = (seed = 1): number[] => Array.from({ length: DIM }, (_, i) => Math.sin
* `AddToVectorIndexOperation` / `brain.warm()` call through.
*/
class FakeVectorProvider implements VectorIndexProvider {
+ readonly name = 'fake-vector-provider'
readonly items = new Map()
warmCalls = 0
searchCalls: Array<{ k?: number }> = []
diff --git a/tests/unit/transaction/vectorIndexOperations-rename.test.ts b/tests/unit/transaction/vectorIndexOperations-rename.test.ts
index 80168346..bdfe58a3 100644
--- a/tests/unit/transaction/vectorIndexOperations-rename.test.ts
+++ b/tests/unit/transaction/vectorIndexOperations-rename.test.ts
@@ -6,12 +6,14 @@
* journals and timings — a fossil "HNSW" name misdirects an operator running
* a native (non-HNSW) vector provider. Verifies:
* 1. rollback wiring stayed byte-identical under the rename,
- * 2. the emitted `name` stamps the active backend — `'js-hnsw'` for
- * Brainy's own JS index, a provider's own `providerId` when it
- * self-identifies, and the honest `'unknown-provider'` literal when
- * neither applies (never a silently-wrong guess).
+ * 2. the emitted `name` stamps the active backend — `'hnsw-js'` for
+ * Brainy's own JS index, a provider's own required `name` when it
+ * self-identifies, and the tolerant-loud `'unknown-provider'` literal
+ * (plus one console.warn) when a runtime instance lacks `name`
+ * altogether (an older native provider compiled against the previous,
+ * optional `providerId` contract) — never a silently-wrong guess.
*/
-import { describe, it, expect } from 'vitest'
+import { describe, it, expect, vi, afterEach } from 'vitest'
import {
AddToVectorIndexOperation,
RemoveFromVectorIndexOperation,
@@ -28,7 +30,7 @@ import { JsHnswVectorIndex } from '../../../src/hnsw/hnswIndex.js'
*/
class FakeVectorProvider implements VectorIndexProvider {
readonly items = new Map()
- constructor(readonly providerId?: string) {}
+ constructor(readonly name: string) {}
async addItem(item: VectorDocument): Promise {
this.items.set(item.id, item.vector)
@@ -115,16 +117,20 @@ describe('Vector index transaction operations — backend-neutral rename', () =>
})
describe('backend stamping in the emitted op name', () => {
- it('stamps the real JS HNSW index as js-hnsw (self-identifying providerId)', () => {
+ afterEach(() => {
+ vi.restoreAllMocks()
+ })
+
+ it('stamps the real JS HNSW index as hnsw-js (self-identifying name)', () => {
const index = new JsHnswVectorIndex()
const addOp = new AddToVectorIndexOperation(index, 'id-1', [1, 2, 3])
const removeOp = new RemoveFromVectorIndexOperation(index, 'id-1', [1, 2, 3])
- expect(addOp.name).toBe('AddToVectorIndex(js-hnsw)')
- expect(removeOp.name).toBe('RemoveFromVectorIndex(js-hnsw)')
+ expect(addOp.name).toBe('AddToVectorIndex(hnsw-js)')
+ expect(removeOp.name).toBe('RemoveFromVectorIndex(hnsw-js)')
})
- it('stamps a self-identifying provider\'s own providerId, never "HNSW"', () => {
+ it('stamps a self-identifying provider\'s own name, never "HNSW"', () => {
const provider = new FakeVectorProvider('acme-diskann')
const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3])
@@ -137,16 +143,34 @@ describe('Vector index transaction operations — backend-neutral rename', () =>
expect(removeOp.name).not.toContain('HNSW')
})
- it('honestly reports "unknown-provider" rather than guessing when a provider omits providerId', () => {
- const provider = new FakeVectorProvider(undefined)
- const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
- const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3])
+ it('tolerant-loud: stamps "unknown-provider" and warns exactly once when a runtime provider lacks the required `name` (an older native provider compiled against the previous optional `providerId` contract)', () => {
+ const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
+ // Simulate a native provider compiled before `name` was required — the
+ // TypeScript contract requires `name`, but `as unknown as` mirrors what
+ // actually reaches this code at runtime from an un-rebuilt native addon.
+ const legacyProvider = {
+ addItem: async (item: VectorDocument) => item.id,
+ removeItem: async () => true,
+ search: async () => [],
+ size: () => 0,
+ clear: () => {},
+ rebuild: async () => {},
+ flush: async () => 0,
+ getPersistMode: () => 'immediate' as const
+ } as unknown as VectorIndexProvider
+
+ const addOp = new AddToVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3])
+ const removeOp = new RemoveFromVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3])
expect(addOp.name).toBe('AddToVectorIndex(unknown-provider)')
expect(removeOp.name).toBe('RemoveFromVectorIndex(unknown-provider)')
// Never silently falls back to the JS engine's name for a provider it
// knows nothing about — that's the exact fossil-naming bug being fixed.
- expect(addOp.name).not.toContain('js-hnsw')
+ expect(addOp.name).not.toContain('hnsw-js')
+ // Loud, not silent — but exactly once per provider instance, not once
+ // per op stamped against it.
+ expect(warnSpy).toHaveBeenCalledTimes(1)
+ expect(warnSpy.mock.calls[0][0]).toContain('name')
})
})
})
From 3a1efc9460617a4b7f37236772f922e408684eb6 Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Thu, 23 Jul 2026 08:56:57 -0700
Subject: [PATCH 06/15] docs: project guide version line points at npm instead
of a hardcoded stale number
---
CLAUDE.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/CLAUDE.md b/CLAUDE.md
index 568c10db..c7336a18 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -12,7 +12,7 @@ Handoff file: `/home/dpsifr/.strategy/PLATFORM-HANDOFF.md`
**Brainy's current open actions:** None. MIT open-source — no platform-specific actions.
-**Current version:** `@soulcraft/brainy@7.31.5` (latest published; 8.0.0 release candidate on `feat/8.0-u64-ids`)
+**Current version:** run `npm view @soulcraft/brainy version` (never trust a hardcoded number here — this line went stale for months); consumer-facing changes tracked in `RELEASES.md`
---
From 6ba94c8c432551575c16c4b758386f1f5912b40d Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Thu, 23 Jul 2026 10:01:28 -0700
Subject: [PATCH 07/15] fix(release): push the public mirror explicitly and
verify the tag lands at the right commit before publishing
Origin moved to the private forge; the public repo is now a mirror with an
unknown sync cadence. The release flow creates the GitHub release directly,
and a missing tag there would be silently created at the default-branch
head - the wrong commit. The script now pushes branch+tag to the mirror
itself and hard-stops if the tag's commit on the mirror differs from local,
before anything irreversible (npm publish) happens.
---
scripts/release.sh | 22 +++++++++++++++++++---
1 file changed, 19 insertions(+), 3 deletions(-)
diff --git a/scripts/release.sh b/scripts/release.sh
index 7d860564..42f5b345 100755
--- a/scripts/release.sh
+++ b/scripts/release.sh
@@ -175,10 +175,26 @@ echo -e "${BLUE}7️⃣ Creating git tag v${NEW_VERSION}...${NC}"
git tag -a "v${NEW_VERSION}" -m "Release v${NEW_VERSION}"
echo -e "${GREEN}✅ Tag created${NC}\n"
-# Step 9: Push to GitHub
-echo -e "${BLUE}8️⃣ Pushing to GitHub...${NC}"
+# Step 9: Push to origin (source of truth) and the public GitHub mirror
+echo -e "${BLUE}8️⃣ Pushing to origin...${NC}"
git push --follow-tags origin "$CURRENT_BRANCH"
-echo -e "${GREEN}✅ Pushed to GitHub${NC}\n"
+echo -e "${GREEN}✅ Pushed to origin${NC}\n"
+
+# The public GitHub repo is a mirror of origin with an unknown sync cadence.
+# `gh release create` below targets GitHub directly: if the new tag hasn't
+# reached GitHub yet, gh would CREATE it — pointed at GitHub's default-branch
+# head, i.e. the wrong commit. Push branch+tag to GitHub explicitly, then
+# verify the tag resolves there to the same commit before any release is cut.
+GITHUB_URL="https://github.com/soulcraftlabs/brainy.git"
+echo -e "${BLUE}8️⃣½ Pushing to the public GitHub mirror...${NC}"
+git push --follow-tags "$GITHUB_URL" "$CURRENT_BRANCH"
+LOCAL_TAG_SHA="$(git rev-parse "v${NEW_VERSION}^{}")"
+GITHUB_TAG_SHA="$(git ls-remote --tags "$GITHUB_URL" "v${NEW_VERSION}^{}" | cut -f1)"
+if [ "$LOCAL_TAG_SHA" != "$GITHUB_TAG_SHA" ]; then
+ echo -e "${RED}❌ Tag v${NEW_VERSION} on GitHub (${GITHUB_TAG_SHA:-absent}) does not match local (${LOCAL_TAG_SHA}) — aborting before npm publish. Fix the mirror, then re-run.${NC}"
+ exit 1
+fi
+echo -e "${GREEN}✅ GitHub mirror has the tag at the right commit${NC}\n"
# Step 10: Publish to npm
echo -e "${BLUE}9️⃣ Publishing to npm (dist-tag: ${NPM_TAG})...${NC}"
From 9a99a7b96210e7bf7fd87e86a05876ddd0459b60 Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Thu, 23 Jul 2026 10:22:34 -0700
Subject: [PATCH 08/15] =?UTF-8?q?docs:=20adoption=20storefront=20=E2=80=94?=
=?UTF-8?q?=20contributing=20guide,=20security=20policy,=20README=20suppor?=
=?UTF-8?q?t=20+=20cor=20section?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
---
CONTRIBUTING.md | 320 +++++++-----------------------------------------
README.md | 24 ++--
SECURITY.md | 36 ++++++
3 files changed, 98 insertions(+), 282 deletions(-)
create mode 100644 SECURITY.md
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index ab8a0246..ef9c4a51 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -1,298 +1,70 @@
# Contributing to Brainy
-Thank you for your interest in contributing to Brainy! This document provides guidelines and instructions for contributing to the project.
+Brainy is MIT-licensed and genuinely open to outside contributions. This page
+is the honest, current path — please don't rely on older instructions you
+may find elsewhere in the repo's history.
-## Code of Conduct
+## Where the project lives
-By participating in this project, you agree to abide by our Code of Conduct:
-- Be respectful and inclusive
-- Welcome newcomers and help them get started
-- Focus on constructive criticism
-- Respect differing viewpoints and experiences
+The source of truth is a self-hosted forge: **source.soulcraft.com/soulcraft/brainy**.
+It's anonymously readable and cloneable — no account needed to browse, clone,
+or build.
-## How to Contribute
+**github.com/soulcraftlabs/brainy** is a public read-only mirror. It's a fine
+place to read code or star the project, but issues and pull requests opened
+there won't be picked up — please use one of the paths below instead.
-### Reporting Issues
+## How to contribute
-Before creating an issue, please check existing issues to avoid duplicates.
+**Found a bug, or have an idea?** Email **brainy@soulcraft.com**. No account,
+no ceremony — you'll get a receipt, and it goes to a human.
-When creating an issue, include:
-- Clear, descriptive title
-- Detailed description of the problem
-- Steps to reproduce
-- Expected vs actual behavior
-- System information (OS, Node version, Brainy version)
-- Code examples if applicable
+**Want to send a patch?** Two ways, both first-class:
-### Suggesting Features
+- **Email a patch.** Run `git format-patch` against your change and email the
+ output to **brainy@soulcraft.com**. This is a genuinely supported path, not
+ a fallback — plenty of good contributions arrive this way.
+- **Open a pull request on the forge.** Request an account at
+ **source.soulcraft.com** (registration is request-with-approval, so allow
+ a little lag), clone, push a branch, and open a PR there. Maintainers
+ review and land it.
-Feature requests are welcome! Please provide:
-- Clear use case
-- Proposed API/interface
-- Examples of how it would work
-- Any potential challenges or considerations
+Either way, for anything beyond a small fix, opening an issue first (email is
+fine) to talk through the approach saves everyone rework.
-### Pull Requests
+## Development setup
-#### Before Starting
-
-1. Check existing issues and PRs
-2. Open an issue to discuss significant changes
-3. Fork the repository
-4. Create a feature branch from `main`
-
-#### Development Setup
-
-**Quick Setup (Recommended):**
```bash
-# Clone your fork
-git clone https://github.com/your-username/brainy.git
+git clone https://source.soulcraft.com/soulcraft/brainy.git
cd brainy
-
-# Run setup script (installs all dependencies including Rust)
-./scripts/setup-dev.sh
-```
-
-**Manual Setup:**
-```bash
-# Clone your fork
-git clone https://github.com/your-username/brainy.git
-cd brainy
-
-# Install system dependencies (Ubuntu/Debian)
-sudo apt-get install -y build-essential pkg-config libssl-dev
-
-# Install Rust (for WASM embedding engine)
-curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
-source ~/.cargo/env
-rustup target add wasm32-unknown-unknown
-cargo install wasm-pack
-
-# Install Node.js dependencies
npm install
-
-# Build Candle WASM embedding engine
-npm run build:candle
-
-# Build TypeScript
npm run build
-
-# Run tests
npm test
```
-#### Making Changes
+Tests run on [Vitest](https://vitest.dev/). `npm test` runs the unit suite;
+see `package.json` for `test:integration`, `test:coverage`, and friends.
-1. **Follow the code style**
- - TypeScript for all source code
- - Clear variable and function names
- - Comments for complex logic
- - JSDoc for public APIs
+## Standards
-2. **Write tests**
- - Add tests for new features
- - Update tests for changes
- - Ensure all tests pass
+- **Strict TypeScript.** No `any` escape hatches to dodge the type checker.
+- **Tests exercise real behavior.** No mocking away the thing you're supposed
+ to be testing.
+- **No stubs, no TODO-code.** If something can't be finished, say so and
+ leave it out — don't merge a placeholder.
+- **JSDoc on every exported function, class, and type.**
+- **[Conventional Commits](https://www.conventionalcommits.org/).** `feat:`,
+ `fix:`, `docs:`, `perf:`, `refactor:`, `test:`, `chore:`. Never
+ `BREAKING CHANGE` in a commit message — major version bumps are a separate,
+ deliberate decision.
+- **Performance claims are measured or labeled projected.** If a PR or its
+ description states a number, cite the benchmark that produced it (see
+ [docs/performance-envelopes.md](docs/performance-envelopes.md) for the
+ pattern). Don't state an estimate as if it were measured.
-3. **Update documentation**
- - Update README if needed
- - Add/update API documentation
- - Include examples
+## License
-#### Commit Guidelines
+Brainy is [MIT licensed](LICENSE). Contributions are accepted under the same
+license — there's no CLA to sign.
-Follow conventional commits format:
-
-```
-type(scope): description
-
-[optional body]
-
-[optional footer]
-```
-
-Types:
-- `feat`: New feature
-- `fix`: Bug fix
-- `docs`: Documentation changes
-- `style`: Code style changes
-- `refactor`: Code refactoring
-- `perf`: Performance improvements
-- `test`: Test changes
-- `chore`: Build/tooling changes
-
-Examples:
-```bash
-feat(triple): add graph traversal depth limit
-fix(storage): handle concurrent write conflicts
-docs(api): update search method documentation
-```
-
-#### Submitting PR
-
-1. Push to your fork
-2. Create PR against `main` branch
-3. Fill out PR template
-4. Ensure CI checks pass
-5. Wait for review
-
-### Testing
-
-#### Running Tests
-
-```bash
-# Run all tests
-npm test
-
-# Run specific test file
-npm test tests/core.test.ts
-
-# Run with coverage
-npm run test:coverage
-
-# Watch mode
-npm run test:watch
-```
-
-#### Writing Tests
-
-```typescript
-import { describe, it, expect } from 'vitest'
-import { Brainy } from '../src'
-
-describe('Feature Name', () => {
- it('should do something specific', async () => {
- const brain = new Brainy()
- await brain.init()
-
- // Test implementation
- const result = await brain.search("test")
-
- expect(result).toBeDefined()
- expect(result.length).toBeGreaterThan(0)
- })
-})
-```
-
-## Architecture Guidelines
-
-### Adding New Features
-
-1. **Check existing functionality**
- - Review `ARCHITECTURE.md`
- - Check if similar features exist
- - Consider if it should be an augmentation
-
-2. **Design considerations**
- - Maintain backward compatibility
- - Consider performance impact
- - Think about all storage adapters
- - Plan for extensibility
-
-3. **Implementation checklist**
- - [ ] Core functionality
- - [ ] Tests (unit and integration)
- - [ ] Documentation
- - [ ] TypeScript types
- - [ ] Examples
- - [ ] Performance benchmarks (if applicable)
-
-### Creating Augmentations
-
-Augmentations extend Brainy's functionality:
-
-```typescript
-import { BrainyAugmentation } from '../types'
-
-export class MyAugmentation extends BrainyAugmentation {
- name = 'MyAugmentation'
-
- async onInit(brain: Brainy): Promise {
- // Initialize augmentation
- }
-
- async onAdd(item: any, brain: Brainy): Promise {
- // Process before adding
- return item
- }
-
- async onSearch(query: any, results: any[], brain: Brainy): Promise {
- // Process search results
- return results
- }
-}
-```
-
-### Performance Considerations
-
-- Use batch operations where possible
-- Implement caching strategically
-- Consider memory usage
-- Profile performance impacts
-- Add benchmarks for critical paths
-
-## Documentation
-
-### API Documentation
-
-Use JSDoc for all public APIs:
-
-```typescript
-/**
- * Searches for similar items using vector similarity
- * @param query - Search query (text or vector)
- * @param options - Search options
- * @returns Array of search results with scores
- * @example
- * ```typescript
- * const results = await brain.search("machine learning", { limit: 10 })
- * ```
- */
-async search(query: string | Vector, options?: SearchOptions): Promise {
- // Implementation
-}
-```
-
-### Examples
-
-Add examples for new features:
-
-```typescript
-// examples/feature-name.ts
-import { Brainy } from 'brainy'
-
-async function exampleUsage() {
- const brain = new Brainy()
- await brain.init()
-
- // Show feature usage
- // Include comments explaining what's happening
- // Handle errors appropriately
-}
-
-exampleUsage().catch(console.error)
-```
-
-## Release Process
-
-1. **Version bump**: Follow semantic versioning
-2. **Update CHANGELOG**: Document all changes
-3. **Run tests**: Ensure all tests pass
-4. **Build**: Generate distribution files
-5. **Tag**: Create git tag for version
-6. **Publish**: Release to npm
-
-## Getting Help
-
-- **Discord**: Join our community
-- **Issues**: Ask questions on GitHub
-- **Discussions**: Share ideas and get feedback
-
-## Recognition
-
-Contributors will be recognized in:
-- CHANGELOG.md for their contributions
-- README.md contributors section
-- GitHub contributors page
-
-Thank you for contributing to Brainy! 🧠
\ No newline at end of file
+Thank you for considering a contribution.
diff --git a/README.md b/README.md
index d1342f52..2fc42060 100644
--- a/README.md
+++ b/README.md
@@ -23,8 +23,9 @@
Quick start ·
One query ·
Features ·
- Scale with Cor ·
- Docs
+ Scale with Cor ·
+ Docs ·
+ Support
---
@@ -172,9 +173,11 @@ await brain.vfs.search('React components with hooks') // semantic file
**[Multi-process model](docs/concepts/multi-process.md)** · **[Inspection guide](docs/guides/inspection.md)**
-## From laptop to hundreds of millions
+## When you outgrow Brainy
-Brainy's TypeScript engines take you a long way. When you outgrow them, add the native engine — **the API doesn't change**:
+Brainy's pure-TypeScript engines carry real workloads a long way on their own — see the measured, per-operation numbers (not marketing figures) in **[docs/performance-envelopes.md](docs/performance-envelopes.md)** for what to expect, unaccelerated, on plain filesystem storage.
+
+When a deployment needs native-scale vector/graph performance — memory-mapped indexes that don't need your dataset in RAM, billion-scale ambitions — add the native engine. **The API doesn't change:**
```bash
npm install @soulcraft/cor
@@ -187,13 +190,14 @@ await brain.init() // @soulcraft/cor detected — same code, native engines un
Installing the package is the opt-in: if `@soulcraft/cor` is present, it loads and announces itself in the init log; if it's present but broken, `init()` **throws** — an installed accelerator never silently vanishes behind the JS engines. Opt out with `plugins: []`, or pin exactly what loads with `plugins: ['@soulcraft/cor']`. [`@soulcraft/cor`](https://www.npmjs.com/package/@soulcraft/cor) (Brainy 8.x ↔ Cor 3.x, version-matched) registers Rust implementations behind every provider seam: SIMD distance kernels, memory-mapped storage, a disk-native vector index that doesn't need your dataset in RAM, durable LSM field/graph indexes that serve cold opens instantly, and native aggregation. Recall@10 measured **0.99 / 0.96 / 0.96 at 1M / 10M / 100M vectors** in Cor's release gate.
-Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is licensed and funds both.
+Open core, commercial accelerator: Brainy is MIT and complete on its own — Cor is more headroom for when you need it, not capability held back to sell you later. Licensing and support: **cor@soulcraft.com**.
## Performance
+- Per-operation p50/p95 at 1k and 10k entities, pure-JS floor, measured and re-run every release that touches a measured path: **[docs/performance-envelopes.md](docs/performance-envelopes.md)**.
- JS distance kernels: **~6× faster cosine, ~1.4× euclidean** than 7.x (measured: [`tests/benchmarks/distance-microbench.mjs`](tests/benchmarks/distance-microbench.mjs), 384-dim, median of 41).
- Whole-graph reads are single **O(N + E)** cursor walks — a consumer-measured 19k-edge export dropped from ~27 s of per-node calls to one scan.
-- Full numbers and capacity planning: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)**
+- Capacity planning and architecture: **[docs/PERFORMANCE.md](docs/PERFORMANCE.md)** · **[docs/SCALING.md](docs/SCALING.md)**
## Use cases
@@ -212,6 +216,10 @@ Open core, commercial accelerator: Brainy is MIT and complete on its own; Cor is
**Bun ≥ 1.1** (recommended) or **Node.js ≥ 22**. Brainy 8.x is server-only; the 7.x line remains on npm for browser use.
-## Contributing & license
+## Support & community
-Contributions welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)**. MIT © Brainy Contributors.
+- **Bugs and ideas** → **brainy@soulcraft.com** — no account needed, you'll get a receipt.
+- **Security reports** → **security@soulcraft.com** — see **[SECURITY.md](SECURITY.md)**.
+- **Contributing** → see **[CONTRIBUTING.md](CONTRIBUTING.md)**.
+
+MIT © Brainy Contributors.
diff --git a/SECURITY.md b/SECURITY.md
new file mode 100644
index 00000000..1f3c4732
--- /dev/null
+++ b/SECURITY.md
@@ -0,0 +1,36 @@
+# Security Policy
+
+## Reporting a vulnerability
+
+Email **security@soulcraft.com**. That's the one door for security reports
+across the company, and it works the same way for Brainy: every report is
+read by a human, you'll get a private receipt, and we'll work with you on
+coordinated disclosure — please don't open a public issue for anything
+that isn't already public.
+
+Include what you'd want if you were on the other end: affected version,
+how to reproduce, and what you think the impact is. If you have a patch or
+a suggested fix, send it along — it's welcome but not required.
+
+There is no bounty program today. We're saying that plainly so you know
+what to expect going in.
+
+## Response time
+
+We respond as fast as truth allows. That means: no fixed SLA, no promise of
+a reply within a specific number of hours — but a real report from a real
+person gets read promptly and taken seriously. If you haven't heard anything
+in a reasonable stretch, a follow-up email is completely fine.
+
+## Supported versions
+
+The latest `8.x` minor release line receives security fixes. If you're
+running an older major version, please upgrade before reporting — we can't
+commit to backporting fixes to unsupported lines.
+
+## Scope
+
+This policy covers the `@soulcraft/brainy` package itself — the code in
+this repository. If you're evaluating a deployment that also uses
+`@soulcraft/cor`, report issues in that package the same way, to the same
+address; we'll route internally.
From 4c8e384bc8271044828eff721abfda2334d9a911 Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Thu, 23 Jul 2026 10:46:18 -0700
Subject: [PATCH 09/15] chore(release): 8.10.0
---
CHANGELOG.md | 9 +++++++++
package-lock.json | 4 ++--
package.json | 2 +-
3 files changed, 12 insertions(+), 3 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index fd0de54e..b6dd91fd 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,15 @@
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
+### [8.10.0](https://github.com/soulcraftlabs/brainy/compare/v8.9.0...v8.10.0) (2026-07-23)
+
+- docs: adoption storefront — contributing guide, security policy, README support + cor section (9a99a7b)
+- fix(release): push the public mirror explicitly and verify the tag lands at the right commit before publishing (6ba94c8)
+- docs: project guide version line points at npm instead of a hardcoded stale number (3a1efc9)
+- feat: vector provider identity is a required name field (hnsw-js), rendered [vector-index:] (3be4ba9)
+- feat: warm contract (warm/warmOnOpen/provider warm hook), configurable transact budget floor, backend-neutral vector index op names (55b867c)
+
+
### [8.9.0](https://github.com/soulcraftlabs/brainy/compare/v8.8.2...v8.9.0) (2026-07-19)
- docs: measured performance envelopes v1 (per-op p50/p95 at 1k and 10k, pure-JS floor) (5cabd78)
diff --git a/package-lock.json b/package-lock.json
index fb9262e9..37aeb81d 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "@soulcraft/brainy",
- "version": "8.9.0",
+ "version": "8.10.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@soulcraft/brainy",
- "version": "8.9.0",
+ "version": "8.10.0",
"license": "MIT",
"dependencies": {
"@msgpack/msgpack": "^3.1.2",
diff --git a/package.json b/package.json
index 7366ce98..e4bc8144 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "@soulcraft/brainy",
- "version": "8.9.0",
+ "version": "8.10.0",
"description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.",
"main": "dist/index.js",
"module": "dist/index.js",
From 415e824a1da44a1109f54818721289f5e9a42af2 Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Thu, 23 Jul 2026 11:09:43 -0700
Subject: [PATCH 10/15] =?UTF-8?q?chore:=20the=20forge=20is=20the=20address?=
=?UTF-8?q?=20=E2=80=94=20retire=20the=20archived=20mirror=20from=20every?=
=?UTF-8?q?=20live=20surface?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Ruled today: the project's one public home is source.soulcraft.com. The
old public repo is archived history and no longer part of any release.
- package.json repository/homepage/bugs now point at the forge (this is
what the npm page links as Repository/Homepage/Issues)
- README CI badge reads the forge pipeline; CONTRIBUTING drops the
mirror paragraph (forge account or email patch were already the ruled
contribution paths)
- release.sh: mirror push + external release step removed; publishes go
forge-first (box-held write token, temp userconfig so the token never
hits argv; a forge-publish failure aborts before the storefront so the
pair can never diverge), then npmjs with the scope-override pin (the
fleet npmrc maps @soulcraft to the forge and scope mappings beat
--registry); release page created via forge API when a token is
present, loud skip otherwise; changelog compare links point home
- dead external CI workflow removed (.forgejo/workflows/ci.yml is the
live pipeline)
Historical CHANGELOG links to the archive stay as written - history is
history and the archive serves them read-only.
---
.github/workflows/ci.yml | 40 ----------------------
CONTRIBUTING.md | 4 ---
README.md | 2 +-
package.json | 6 ++--
scripts/release.sh | 71 +++++++++++++++++++++++++---------------
5 files changed, 48 insertions(+), 75 deletions(-)
delete mode 100644 .github/workflows/ci.yml
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
deleted file mode 100644
index cdb2ab14..00000000
--- a/.github/workflows/ci.yml
+++ /dev/null
@@ -1,40 +0,0 @@
-name: CI
-
-on:
- push:
- pull_request:
-
-jobs:
- node:
- name: Node ${{ matrix.node-version }}
- runs-on: ubuntu-latest
- strategy:
- fail-fast: false
- matrix:
- node-version: ['22', '24']
- steps:
- - uses: actions/checkout@v4
- - uses: actions/setup-node@v4
- with:
- node-version: ${{ matrix.node-version }}
- cache: npm
- - run: npm ci
- - run: npm run test:unit
-
- bun:
- name: Bun (latest)
- runs-on: ubuntu-latest
- steps:
- - uses: actions/checkout@v4
- - uses: actions/setup-node@v4
- with:
- node-version: '22'
- cache: npm
- - uses: oven-sh/setup-bun@v2
- with:
- bun-version: latest
- - run: npm ci
- # test:bun imports the built dist/, so build first.
- - run: npm run build
- # Bun as a runtime is the supported Bun story (`bun add` / `bun run`).
- - run: npm run test:bun
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index ef9c4a51..d277091d 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -10,10 +10,6 @@ The source of truth is a self-hosted forge: **source.soulcraft.com/soulcraft/bra
It's anonymously readable and cloneable — no account needed to browse, clone,
or build.
-**github.com/soulcraftlabs/brainy** is a public read-only mirror. It's a fine
-place to read code or star the project, but issues and pull requests opened
-there won't be picked up — please use one of the paths below instead.
-
## How to contribute
**Found a bug, or have an idea?** Email **brainy@soulcraft.com**. No account,
diff --git a/README.md b/README.md
index 2fc42060..2caf6493 100644
--- a/README.md
+++ b/README.md
@@ -13,7 +13,7 @@
-
+
diff --git a/package.json b/package.json
index e4bc8144..a3ece83c 100644
--- a/package.json
+++ b/package.json
@@ -128,13 +128,13 @@
"publishConfig": {
"access": "public"
},
- "homepage": "https://github.com/soulcraftlabs/brainy",
+ "homepage": "https://source.soulcraft.com/soulcraft/brainy",
"bugs": {
- "url": "https://github.com/soulcraftlabs/brainy/issues"
+ "url": "https://source.soulcraft.com/soulcraft/brainy/issues"
},
"repository": {
"type": "git",
- "url": "git+https://github.com/soulcraftlabs/brainy.git"
+ "url": "git+https://source.soulcraft.com/soulcraft/brainy.git"
},
"files": [
"dist/**/*.js",
diff --git a/scripts/release.sh b/scripts/release.sh
index 42f5b345..43fa50bd 100755
--- a/scripts/release.sh
+++ b/scripts/release.sh
@@ -142,7 +142,7 @@ else
fi
# Create new changelog entry
-CHANGELOG_ENTRY="### [${NEW_VERSION}](https://github.com/soulcraftlabs/brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d))
+CHANGELOG_ENTRY="### [${NEW_VERSION}](https://source.soulcraft.com/soulcraft/brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d))
${COMMITS}
"
@@ -175,42 +175,59 @@ echo -e "${BLUE}7️⃣ Creating git tag v${NEW_VERSION}...${NC}"
git tag -a "v${NEW_VERSION}" -m "Release v${NEW_VERSION}"
echo -e "${GREEN}✅ Tag created${NC}\n"
-# Step 9: Push to origin (source of truth) and the public GitHub mirror
+# Step 9: Push to origin — the forge is the one home (ruled 2026-07-23; the
+# old public GitHub repo is archived history, no longer part of any release).
echo -e "${BLUE}8️⃣ Pushing to origin...${NC}"
git push --follow-tags origin "$CURRENT_BRANCH"
echo -e "${GREEN}✅ Pushed to origin${NC}\n"
-# The public GitHub repo is a mirror of origin with an unknown sync cadence.
-# `gh release create` below targets GitHub directly: if the new tag hasn't
-# reached GitHub yet, gh would CREATE it — pointed at GitHub's default-branch
-# head, i.e. the wrong commit. Push branch+tag to GitHub explicitly, then
-# verify the tag resolves there to the same commit before any release is cut.
-GITHUB_URL="https://github.com/soulcraftlabs/brainy.git"
-echo -e "${BLUE}8️⃣½ Pushing to the public GitHub mirror...${NC}"
-git push --follow-tags "$GITHUB_URL" "$CURRENT_BRANCH"
-LOCAL_TAG_SHA="$(git rev-parse "v${NEW_VERSION}^{}")"
-GITHUB_TAG_SHA="$(git ls-remote --tags "$GITHUB_URL" "v${NEW_VERSION}^{}" | cut -f1)"
-if [ "$LOCAL_TAG_SHA" != "$GITHUB_TAG_SHA" ]; then
- echo -e "${RED}❌ Tag v${NEW_VERSION} on GitHub (${GITHUB_TAG_SHA:-absent}) does not match local (${LOCAL_TAG_SHA}) — aborting before npm publish. Fix the mirror, then re-run.${NC}"
+# Step 10: Publish — forge FIRST (home), npmjs second (the world's storefront).
+# The fleet-wide ~/.npmrc maps the @soulcraft scope to the forge registry, and
+# a scope mapping BEATS `--registry` on the command line — so each publish
+# names its registry via the scope override explicitly. Nothing implicit.
+FORGE_NPM_REG="https://source.soulcraft.com/api/packages/soulcraft/npm/"
+FORGE_NPM_TOKEN_FILE="$HOME/.config/soulcraft/npm-publish-brainy.token"
+echo -e "${BLUE}9️⃣ Publishing to the forge registry (home)...${NC}"
+if [ -f "$FORGE_NPM_TOKEN_FILE" ]; then
+ TMPRC="$(mktemp)"
+ chmod 600 "$TMPRC"
+ {
+ echo "@soulcraft:registry=${FORGE_NPM_REG}"
+ echo "//source.soulcraft.com/api/packages/soulcraft/npm/:_authToken=$(cat "$FORGE_NPM_TOKEN_FILE")"
+ } > "$TMPRC"
+ if npm publish --tag "$NPM_TAG" --userconfig "$TMPRC"; then
+ echo -e "${GREEN}✅ Published to the forge${NC}\n"
+ else
+ rm -f "$TMPRC"
+ echo -e "${RED}❌ Forge publish FAILED — aborting before npmjs so the pair never diverges. Fix and re-run.${NC}"
+ exit 1
+ fi
+ rm -f "$TMPRC"
+else
+ echo -e "${RED}❌ Forge publish token missing (${FORGE_NPM_TOKEN_FILE}) — aborting. The forge is home; publish it first or restage the token.${NC}"
exit 1
fi
-echo -e "${GREEN}✅ GitHub mirror has the tag at the right commit${NC}\n"
-# Step 10: Publish to npm
-echo -e "${BLUE}9️⃣ Publishing to npm (dist-tag: ${NPM_TAG})...${NC}"
-npm publish --tag "$NPM_TAG"
+echo -e "${BLUE}9️⃣½ Publishing to npmjs (storefront, dist-tag: ${NPM_TAG})...${NC}"
+npm publish --tag "$NPM_TAG" "--@soulcraft:registry=https://registry.npmjs.org/"
# Brainy is the only PUBLIC @soulcraft package — verify visibility after every publish.
-npm access get status @soulcraft/brainy || true
-echo -e "${GREEN}✅ Published to npm${NC}\n"
+npm access get status @soulcraft/brainy "--@soulcraft:registry=https://registry.npmjs.org/" || true
+echo -e "${GREEN}✅ Published to npmjs${NC}\n"
-# Step 11: Create GitHub release
-echo -e "${BLUE}🔟 Creating GitHub release...${NC}"
-if [ "$PRERELEASE" = true ]; then
- gh release create "v${NEW_VERSION}" --generate-notes --prerelease
+# Step 11: Release object on the forge (presentational — the tag, CHANGELOG,
+# and RELEASES.md are the record; this just gives the forge UI a release page).
+echo -e "${BLUE}🔟 Creating forge release...${NC}"
+if [ -n "${FORGEJO_RELEASE_TOKEN:-}" ]; then
+ if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraft/brainy/releases" \
+ -H "Authorization: token ${FORGEJO_RELEASE_TOKEN}" -H "Content-Type: application/json" \
+ -d "{\"tag_name\":\"v${NEW_VERSION}\",\"name\":\"v${NEW_VERSION}\",\"prerelease\":${PRERELEASE}}" >/dev/null; then
+ echo -e "${GREEN}✅ Forge release created${NC}\n"
+ else
+ echo -e "${RED}⚠️ Forge release API call failed — tag + CHANGELOG remain the record; create the release page via the forge UI if wanted${NC}\n"
+ fi
else
- gh release create "v${NEW_VERSION}" --generate-notes
+ echo -e "${RED}⚠️ FORGEJO_RELEASE_TOKEN unset — no release page created; tag + CHANGELOG remain the record${NC}\n"
fi
-echo -e "${GREEN}✅ GitHub release created${NC}\n"
# Step 12: Push public docs to the soulcraft.com docs ingest door
# (VENUE-DOCS-RELEASE-PUSH). Skips with a loud warning when
@@ -229,4 +246,4 @@ echo -e "${GREEN}🎉 Release ${NEW_VERSION} complete!${NC}"
echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""
echo -e "📦 npm: ${BLUE}https://www.npmjs.com/package/@soulcraft/brainy/v/${NEW_VERSION}${NC}"
-echo -e "🐙 GitHub: ${BLUE}https://github.com/soulcraftlabs/brainy/releases/tag/v${NEW_VERSION}${NC}"
+echo -e "🏠 Forge: ${BLUE}https://source.soulcraft.com/soulcraft/brainy/releases/tag/v${NEW_VERSION}${NC}"
From 22702b81c0557df6e0f10713f958beb956d06044 Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Thu, 23 Jul 2026 11:09:43 -0700
Subject: [PATCH 11/15] =?UTF-8?q?chore:=20the=20forge=20is=20the=20address?=
=?UTF-8?q?=20=E2=80=94=20retire=20the=20archived=20mirror=20from=20every?=
=?UTF-8?q?=20live=20surface?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Ruled today: the project's one public home is source.soulcraft.com. The
old public repo is archived history and no longer part of any release.
- package.json repository/homepage/bugs now point at the forge (this is
what the npm page links as Repository/Homepage/Issues)
- README CI badge reads the forge pipeline; CONTRIBUTING drops the
mirror paragraph (forge account or email patch were already the ruled
contribution paths)
- release.sh: mirror push + external release step removed; publishes go
forge-first (box-held write token, temp userconfig so the token never
hits argv; a forge-publish failure aborts before the storefront so the
pair can never diverge), then npmjs with the scope-override pin (the
fleet npmrc maps @soulcraft to the forge and scope mappings beat
--registry); release page created via forge API when a token is
present, loud skip otherwise; changelog compare links point home
- dead external CI workflow removed (.forgejo/workflows/ci.yml is the
live pipeline)
Historical CHANGELOG links to the archive stay as written - history is
history and the archive serves them read-only.
---
.github/workflows/ci.yml | 40 ----------------------
CONTRIBUTING.md | 4 ---
README.md | 2 +-
package.json | 6 ++--
scripts/release.sh | 71 +++++++++++++++++++++++++---------------
5 files changed, 48 insertions(+), 75 deletions(-)
delete mode 100644 .github/workflows/ci.yml
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
deleted file mode 100644
index cdb2ab14..00000000
--- a/.github/workflows/ci.yml
+++ /dev/null
@@ -1,40 +0,0 @@
-name: CI
-
-on:
- push:
- pull_request:
-
-jobs:
- node:
- name: Node ${{ matrix.node-version }}
- runs-on: ubuntu-latest
- strategy:
- fail-fast: false
- matrix:
- node-version: ['22', '24']
- steps:
- - uses: actions/checkout@v4
- - uses: actions/setup-node@v4
- with:
- node-version: ${{ matrix.node-version }}
- cache: npm
- - run: npm ci
- - run: npm run test:unit
-
- bun:
- name: Bun (latest)
- runs-on: ubuntu-latest
- steps:
- - uses: actions/checkout@v4
- - uses: actions/setup-node@v4
- with:
- node-version: '22'
- cache: npm
- - uses: oven-sh/setup-bun@v2
- with:
- bun-version: latest
- - run: npm ci
- # test:bun imports the built dist/, so build first.
- - run: npm run build
- # Bun as a runtime is the supported Bun story (`bun add` / `bun run`).
- - run: npm run test:bun
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index ef9c4a51..d277091d 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -10,10 +10,6 @@ The source of truth is a self-hosted forge: **source.soulcraft.com/soulcraft/bra
It's anonymously readable and cloneable — no account needed to browse, clone,
or build.
-**github.com/soulcraftlabs/brainy** is a public read-only mirror. It's a fine
-place to read code or star the project, but issues and pull requests opened
-there won't be picked up — please use one of the paths below instead.
-
## How to contribute
**Found a bug, or have an idea?** Email **brainy@soulcraft.com**. No account,
diff --git a/README.md b/README.md
index 2fc42060..2caf6493 100644
--- a/README.md
+++ b/README.md
@@ -13,7 +13,7 @@
-
+
diff --git a/package.json b/package.json
index e4bc8144..a3ece83c 100644
--- a/package.json
+++ b/package.json
@@ -128,13 +128,13 @@
"publishConfig": {
"access": "public"
},
- "homepage": "https://github.com/soulcraftlabs/brainy",
+ "homepage": "https://source.soulcraft.com/soulcraft/brainy",
"bugs": {
- "url": "https://github.com/soulcraftlabs/brainy/issues"
+ "url": "https://source.soulcraft.com/soulcraft/brainy/issues"
},
"repository": {
"type": "git",
- "url": "git+https://github.com/soulcraftlabs/brainy.git"
+ "url": "git+https://source.soulcraft.com/soulcraft/brainy.git"
},
"files": [
"dist/**/*.js",
diff --git a/scripts/release.sh b/scripts/release.sh
index 42f5b345..43fa50bd 100755
--- a/scripts/release.sh
+++ b/scripts/release.sh
@@ -142,7 +142,7 @@ else
fi
# Create new changelog entry
-CHANGELOG_ENTRY="### [${NEW_VERSION}](https://github.com/soulcraftlabs/brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d))
+CHANGELOG_ENTRY="### [${NEW_VERSION}](https://source.soulcraft.com/soulcraft/brainy/compare/v${CURRENT_VERSION}...v${NEW_VERSION}) ($(date +%Y-%m-%d))
${COMMITS}
"
@@ -175,42 +175,59 @@ echo -e "${BLUE}7️⃣ Creating git tag v${NEW_VERSION}...${NC}"
git tag -a "v${NEW_VERSION}" -m "Release v${NEW_VERSION}"
echo -e "${GREEN}✅ Tag created${NC}\n"
-# Step 9: Push to origin (source of truth) and the public GitHub mirror
+# Step 9: Push to origin — the forge is the one home (ruled 2026-07-23; the
+# old public GitHub repo is archived history, no longer part of any release).
echo -e "${BLUE}8️⃣ Pushing to origin...${NC}"
git push --follow-tags origin "$CURRENT_BRANCH"
echo -e "${GREEN}✅ Pushed to origin${NC}\n"
-# The public GitHub repo is a mirror of origin with an unknown sync cadence.
-# `gh release create` below targets GitHub directly: if the new tag hasn't
-# reached GitHub yet, gh would CREATE it — pointed at GitHub's default-branch
-# head, i.e. the wrong commit. Push branch+tag to GitHub explicitly, then
-# verify the tag resolves there to the same commit before any release is cut.
-GITHUB_URL="https://github.com/soulcraftlabs/brainy.git"
-echo -e "${BLUE}8️⃣½ Pushing to the public GitHub mirror...${NC}"
-git push --follow-tags "$GITHUB_URL" "$CURRENT_BRANCH"
-LOCAL_TAG_SHA="$(git rev-parse "v${NEW_VERSION}^{}")"
-GITHUB_TAG_SHA="$(git ls-remote --tags "$GITHUB_URL" "v${NEW_VERSION}^{}" | cut -f1)"
-if [ "$LOCAL_TAG_SHA" != "$GITHUB_TAG_SHA" ]; then
- echo -e "${RED}❌ Tag v${NEW_VERSION} on GitHub (${GITHUB_TAG_SHA:-absent}) does not match local (${LOCAL_TAG_SHA}) — aborting before npm publish. Fix the mirror, then re-run.${NC}"
+# Step 10: Publish — forge FIRST (home), npmjs second (the world's storefront).
+# The fleet-wide ~/.npmrc maps the @soulcraft scope to the forge registry, and
+# a scope mapping BEATS `--registry` on the command line — so each publish
+# names its registry via the scope override explicitly. Nothing implicit.
+FORGE_NPM_REG="https://source.soulcraft.com/api/packages/soulcraft/npm/"
+FORGE_NPM_TOKEN_FILE="$HOME/.config/soulcraft/npm-publish-brainy.token"
+echo -e "${BLUE}9️⃣ Publishing to the forge registry (home)...${NC}"
+if [ -f "$FORGE_NPM_TOKEN_FILE" ]; then
+ TMPRC="$(mktemp)"
+ chmod 600 "$TMPRC"
+ {
+ echo "@soulcraft:registry=${FORGE_NPM_REG}"
+ echo "//source.soulcraft.com/api/packages/soulcraft/npm/:_authToken=$(cat "$FORGE_NPM_TOKEN_FILE")"
+ } > "$TMPRC"
+ if npm publish --tag "$NPM_TAG" --userconfig "$TMPRC"; then
+ echo -e "${GREEN}✅ Published to the forge${NC}\n"
+ else
+ rm -f "$TMPRC"
+ echo -e "${RED}❌ Forge publish FAILED — aborting before npmjs so the pair never diverges. Fix and re-run.${NC}"
+ exit 1
+ fi
+ rm -f "$TMPRC"
+else
+ echo -e "${RED}❌ Forge publish token missing (${FORGE_NPM_TOKEN_FILE}) — aborting. The forge is home; publish it first or restage the token.${NC}"
exit 1
fi
-echo -e "${GREEN}✅ GitHub mirror has the tag at the right commit${NC}\n"
-# Step 10: Publish to npm
-echo -e "${BLUE}9️⃣ Publishing to npm (dist-tag: ${NPM_TAG})...${NC}"
-npm publish --tag "$NPM_TAG"
+echo -e "${BLUE}9️⃣½ Publishing to npmjs (storefront, dist-tag: ${NPM_TAG})...${NC}"
+npm publish --tag "$NPM_TAG" "--@soulcraft:registry=https://registry.npmjs.org/"
# Brainy is the only PUBLIC @soulcraft package — verify visibility after every publish.
-npm access get status @soulcraft/brainy || true
-echo -e "${GREEN}✅ Published to npm${NC}\n"
+npm access get status @soulcraft/brainy "--@soulcraft:registry=https://registry.npmjs.org/" || true
+echo -e "${GREEN}✅ Published to npmjs${NC}\n"
-# Step 11: Create GitHub release
-echo -e "${BLUE}🔟 Creating GitHub release...${NC}"
-if [ "$PRERELEASE" = true ]; then
- gh release create "v${NEW_VERSION}" --generate-notes --prerelease
+# Step 11: Release object on the forge (presentational — the tag, CHANGELOG,
+# and RELEASES.md are the record; this just gives the forge UI a release page).
+echo -e "${BLUE}🔟 Creating forge release...${NC}"
+if [ -n "${FORGEJO_RELEASE_TOKEN:-}" ]; then
+ if curl -sf -X POST "https://source.soulcraft.com/api/v1/repos/soulcraft/brainy/releases" \
+ -H "Authorization: token ${FORGEJO_RELEASE_TOKEN}" -H "Content-Type: application/json" \
+ -d "{\"tag_name\":\"v${NEW_VERSION}\",\"name\":\"v${NEW_VERSION}\",\"prerelease\":${PRERELEASE}}" >/dev/null; then
+ echo -e "${GREEN}✅ Forge release created${NC}\n"
+ else
+ echo -e "${RED}⚠️ Forge release API call failed — tag + CHANGELOG remain the record; create the release page via the forge UI if wanted${NC}\n"
+ fi
else
- gh release create "v${NEW_VERSION}" --generate-notes
+ echo -e "${RED}⚠️ FORGEJO_RELEASE_TOKEN unset — no release page created; tag + CHANGELOG remain the record${NC}\n"
fi
-echo -e "${GREEN}✅ GitHub release created${NC}\n"
# Step 12: Push public docs to the soulcraft.com docs ingest door
# (VENUE-DOCS-RELEASE-PUSH). Skips with a loud warning when
@@ -229,4 +246,4 @@ echo -e "${GREEN}🎉 Release ${NEW_VERSION} complete!${NC}"
echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}"
echo ""
echo -e "📦 npm: ${BLUE}https://www.npmjs.com/package/@soulcraft/brainy/v/${NEW_VERSION}${NC}"
-echo -e "🐙 GitHub: ${BLUE}https://github.com/soulcraftlabs/brainy/releases/tag/v${NEW_VERSION}${NC}"
+echo -e "🏠 Forge: ${BLUE}https://source.soulcraft.com/soulcraft/brainy/releases/tag/v${NEW_VERSION}${NC}"
From 003e2a74ea7dd2598022b2ca6ee3784d85214662 Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Fri, 24 Jul 2026 16:01:41 -0700
Subject: [PATCH 12/15] fix: transaction timeouts are a typed no-hot-retry
contract; engine-side non-retry pinned; dead transaction path removed
A production incident: a native-provider op ground 38-40s inside a transaction,
blew the apply-phase budget, rolled back, and a downstream pipeline hot-retried
the identical operation into a 6-minute CPU storm. Brainy itself never
auto-retried the timeout; the gap was that TransactionTimeoutError only said
"retryable" in prose, with nothing machine-readable for a caller to branch on.
- TransactionTimeoutError gains two typed, always-true fields: retryable
(a later attempt may succeed once the slowness resolves or the budget is
raised) and hotRetryUnsafe (an immediate identical retry re-pays the full
cost that just timed out and can cascade into a CPU storm -- callers must
latch and back off, never loop). context's existing telemetry fields
(timeoutMs, operationIndex, elapsedMs, totalOperations, operationName) are
now documented as the caller's backoff inputs.
- Updated the "retryable" doc-prose sites (transact()'s timeoutMs option,
transactionBudgetFloorMs, Transaction.execute()'s contract) to point at
the new fields instead of bare prose.
- Regression pin (tests/unit/transaction/timeout-never-internally-retried.test.ts):
an execution counter proves the engine never re-drives a timed-out
operation, through both the single-op engine TransactionManager/Transaction
drives for every single-record write, and add()'s upsert-race retry loop
(which must exit on the first TransactionTimeoutError, never treat it like
the lost-insert-race signal it retries on).
- Removed TransactionManager.executeTransactionWithResult -- zero callers
anywhere in the codebase.
---
src/db/types.ts | 10 +-
src/transaction/Transaction.ts | 7 +-
src/transaction/TransactionManager.ts | 29 ----
src/transaction/errors.ts | 45 +++++-
src/types/brainy.types.ts | 6 +-
.../TransactionManager.unit.test.ts | 37 -----
.../timeout-never-internally-retried.test.ts | 141 ++++++++++++++++++
7 files changed, 199 insertions(+), 76 deletions(-)
create mode 100644 tests/unit/transaction/timeout-never-internally-retried.test.ts
diff --git a/src/db/types.ts b/src/db/types.ts
index 4c8a4957..a7f86a89 100644
--- a/src/db/types.ts
+++ b/src/db/types.ts
@@ -121,9 +121,13 @@ export interface TransactOptions {
* with the batch: `max(30 000, opCount × 2 000)` — production imports on
* network-attached disks measure ~2 s per operation, so a flat 30 s budget
* silently capped honest bulk work at ~15 operations. A tripped budget
- * rolls the whole batch back and throws a retryable
- * `TransactionTimeoutError` naming the operation it stopped at, the batch
- * size, and the elapsed/budget times.
+ * rolls the whole batch back and throws a `TransactionTimeoutError` naming
+ * the operation it stopped at, the batch size, and the elapsed/budget
+ * times. That error is retryable-with-latch, never hot-retry: its
+ * `retryable` field says a later attempt may succeed, its
+ * `hotRetryUnsafe` field says an immediate identical retry re-pays the
+ * full cost that just timed out — callers must latch and back off, never
+ * loop.
*/
timeoutMs?: number
}
diff --git a/src/transaction/Transaction.ts b/src/transaction/Transaction.ts
index 79b53006..ede65e83 100644
--- a/src/transaction/Transaction.ts
+++ b/src/transaction/Transaction.ts
@@ -62,9 +62,10 @@ const DEFAULT_BUDGET_FLOOR_MS = 30_000
* NEXT operation may start (see {@link Transaction.execute}), never whether
* already-completed work is rolled back after the fact. A trip mid-batch
* still rolls back every operation applied so far, atomically, and throws a
- * retryable, fully-labeled TransactionTimeoutError — that zero-loss guarantee
- * doesn't change; only the point at which the clock stops mattering does (at
- * the last operation, not one check later).
+ * fully-labeled `TransactionTimeoutError` — retryable-with-latch, never
+ * hot-retry (see its `retryable` and `hotRetryUnsafe` fields) — that
+ * zero-loss guarantee doesn't change; only the point at which the clock
+ * stops mattering does (at the last operation, not one check later).
*
* @param opCount - Number of operations in the batch.
* @param override - A full override for this call; wins over everything else.
diff --git a/src/transaction/TransactionManager.ts b/src/transaction/TransactionManager.ts
index 0f13b6a3..5abf48ba 100644
--- a/src/transaction/TransactionManager.ts
+++ b/src/transaction/TransactionManager.ts
@@ -19,7 +19,6 @@
import { Transaction } from './Transaction.js'
import {
TransactionFunction,
- TransactionResult,
TransactionOptions
} from './types.js'
import { TransactionError } from './errors.js'
@@ -105,34 +104,6 @@ export class TransactionManager {
}
}
- /**
- * Execute a transaction and return detailed result
- */
- async executeTransactionWithResult(
- fn: TransactionFunction,
- options?: TransactionOptions
- ): Promise> {
- const startTime = Date.now()
- const transaction = new Transaction(options)
-
- try {
- const value = await fn(transaction)
- await transaction.execute()
-
- const executionTimeMs = Date.now() - startTime
-
- return {
- value,
- operationCount: transaction.getOperationCount(),
- executionTimeMs
- }
-
- } catch (error) {
- // Transaction failed
- throw error
- }
- }
-
/**
* Get transaction statistics
*/
diff --git a/src/transaction/errors.ts b/src/transaction/errors.ts
index c270d0ed..d8382a4b 100644
--- a/src/transaction/errors.ts
+++ b/src/transaction/errors.ts
@@ -73,14 +73,47 @@ export class InvalidTransactionStateError extends TransactionError {
/**
* Error for transaction timeout
+ *
+ * Machine-readable no-hot-retry contract: {@link retryable} and
+ * {@link hotRetryUnsafe} are both always `true` on this class — they exist
+ * so a caller can branch on the *shape* of the error instead of parsing
+ * message text. Read them together: the operation may eventually succeed,
+ * but never by looping on it immediately.
+ *
+ * `context` (inherited from {@link TransactionError}) carries the caller's
+ * backoff inputs — see the field docs below.
*/
export class TransactionTimeoutError extends TransactionError {
+ /**
+ * The failed operation MAY succeed on a later attempt — once the
+ * underlying slowness resolves (e.g. a cold page cache warms up) or the
+ * budget is deliberately raised (`transactionBudgetFloorMs`, or a larger
+ * `timeoutMs` override on the batch). This is a statement about eventual
+ * retryability, not a license to retry now — see {@link hotRetryUnsafe}.
+ */
+ public readonly retryable = true
+
+ /**
+ * An immediate, identical retry re-pays the FULL cost of the work that
+ * just timed out — it does not resume partway. Looping on this error
+ * (hot-retrying) repeats that full cost every attempt and can cascade
+ * into a CPU/resource storm on the caller's side. Callers MUST latch: on
+ * this error, record `{ at: Date.now(), error }`, surface one loud
+ * failure to their own caller, and hold a cooldown window before any
+ * re-attempt (clearing the latch only on success). Never retry this error
+ * in a tight loop.
+ */
+ public readonly hotRetryUnsafe = true
+
constructor(
timeoutMs: number,
operationIndex: number,
telemetry?: {
+ /** Milliseconds elapsed in the transaction when the budget tripped. */
elapsedMs?: number
+ /** Total number of operations in the batch that timed out. */
totalOperations?: number
+ /** Name of the operation the batch was about to start when it tripped, if named. */
operationName?: string
}
) {
@@ -93,8 +126,16 @@ export class TransactionTimeoutError extends TransactionError {
telemetry?.elapsedMs !== undefined ? `${telemetry.elapsedMs}ms elapsed, ` : ''
super(
`Transaction timed out at operation ${progress}${name} — ${elapsed}budget ${timeoutMs}ms. ` +
- `The batch rolled back atomically; retry with a higher timeoutMs or a smaller batch.`,
- { timeoutMs, operationIndex, ...telemetry }
+ `The batch rolled back atomically; retryable after the underlying slowness resolves or ` +
+ `the budget is raised, but hot-retry-unsafe — latch and back off, never loop.`,
+ {
+ // Caller backoff inputs — all present on every instance:
+ /** Configured budget (ms) that was exceeded. */
+ timeoutMs,
+ /** Index of the operation the batch was about to start when it tripped. */
+ operationIndex,
+ ...telemetry
+ }
)
this.name = 'TransactionTimeoutError'
}
diff --git a/src/types/brainy.types.ts b/src/types/brainy.types.ts
index 8bede8d3..b5f286ed 100644
--- a/src/types/brainy.types.ts
+++ b/src/types/brainy.types.ts
@@ -1658,8 +1658,10 @@ export interface BrainyConfig {
* **start** — never whether already-completed work gets rolled back after
* the fact (a single-op write can never time out post-hoc: it either runs
* or it commits). A trip mid-batch still rolls back every applied operation
- * atomically and throws a retryable `TransactionTimeoutError`; only the
- * floor of the formula is configurable here.
+ * atomically and throws a `TransactionTimeoutError` that is
+ * retryable-with-latch, never hot-retry (see its `retryable` and
+ * `hotRetryUnsafe` fields); only the floor of the formula is configurable
+ * here.
*
* Raise this when a cold store's first writes after a restart legitimately
* take longer than 30s per operation (e.g. page-cache-cold canonical writes
diff --git a/tests/transaction/TransactionManager.unit.test.ts b/tests/transaction/TransactionManager.unit.test.ts
index 86b1692c..29e7f7ae 100644
--- a/tests/transaction/TransactionManager.unit.test.ts
+++ b/tests/transaction/TransactionManager.unit.test.ts
@@ -5,7 +5,6 @@
* - High-level transaction API
* - Statistics tracking
* - Error handling
- * - Result wrapping
*/
import { describe, it, expect, beforeEach } from 'vitest'
@@ -84,42 +83,6 @@ describe('TransactionManager', () => {
})
})
- describe('executeTransactionWithResult', () => {
- it('should return detailed result', async () => {
- const result = await manager.executeTransactionWithResult(async (tx) => {
- tx.addOperation({
- execute: async () => {
- await new Promise(resolve => setTimeout(resolve, 1))
- return async () => {}
- }
- })
- tx.addOperation({ execute: async () => undefined })
- return 'success'
- })
-
- expect(result.value).toBe('success')
- expect(result.operationCount).toBe(2)
- expect(result.executionTimeMs).toBeGreaterThanOrEqual(0)
- })
-
- it('should measure execution time', async () => {
- const result = await manager.executeTransactionWithResult(async (tx) => {
- tx.addOperation({
- execute: async () => {
- await new Promise(resolve => setTimeout(resolve, 25))
- return async () => {}
- }
- })
- return 'done'
- })
-
- // Timer coalescing can fire a setTimeout up to a few ms EARLY under
- // load, so assert well below the sleep — this tests that time is
- // MEASURED, not the OS timer's precision.
- expect(result.executionTimeMs).toBeGreaterThanOrEqual(20)
- })
- })
-
describe('Statistics Tracking', () => {
it('should track total transactions', async () => {
await manager.executeTransaction(async (tx) => {
diff --git a/tests/unit/transaction/timeout-never-internally-retried.test.ts b/tests/unit/transaction/timeout-never-internally-retried.test.ts
new file mode 100644
index 00000000..a43a81a8
--- /dev/null
+++ b/tests/unit/transaction/timeout-never-internally-retried.test.ts
@@ -0,0 +1,141 @@
+/**
+ * @module tests/unit/transaction/timeout-never-internally-retried
+ * @description Regression pin for the no-hot-retry contract (8.10.1).
+ *
+ * A production incident: a native-provider op ground 38-40s inside a
+ * transaction, blew the ~32s budget, was rolled back, and a CONSUMER pipeline
+ * hot-retried the identical operation into a 6-minute 100%-CPU storm.
+ * Investigation established brainy itself never auto-retries a
+ * `TransactionTimeoutError` — the storm was entirely the consumer's hot-retry
+ * loop, driven by a "retryable" doc-prose claim with no machine-readable
+ * contract. This file pins the brainy-side half of that story so it can never
+ * regress silently:
+ *
+ * (i) the underlying engine (`TransactionManager.executeTransaction()` →
+ * `Transaction.execute()`) — the exact machinery every single-record
+ * write (`add`/`update`/`remove`/...) drives via
+ * `Brainy.persistSingleOp()` — never internally re-executes a timed-out
+ * operation, and the error it surfaces carries `retryable === true` and
+ * `hotRetryUnsafe === true` (see `src/transaction/errors.ts`).
+ *
+ * Constructed directly (mirrors the existing
+ * `tests/unit/transaction/timeout-rollback.test.ts` pattern) rather than
+ * through a real `brain.add()` call: `transactTimeoutBudget()` floors
+ * every single-op write's budget at `opCount * 2000`ms with NO override
+ * seam (`transactionBudgetFloorMs` only RAISES that floor — it cannot
+ * lower it below the per-op-count term), so getting a real `add()` to
+ * time out requires a multi-second sleep. The engine-level
+ * `options.timeout` override used here is the exact same
+ * `TransactionManager`/`Transaction` code `persistSingleOp` calls —
+ * pinning it here pins add()'s guarantee without paying that wall-clock
+ * cost.
+ *
+ * (ii) `Brainy.add()`'s upsert-race retry loop (src/brainy.ts,
+ * `MAX_UPSERT_ATTEMPTS = 10`) — proving the loop's `catch` treats a
+ * `TransactionTimeoutError` as terminal (immediate rethrow) rather than
+ * the `InsertPreconditionExistsSignal` it retries on, so a mid-flight
+ * timeout can never be silently swallowed and re-attempted up to 10
+ * times.
+ */
+import { describe, it, expect } from 'vitest'
+import { TransactionManager } from '../../../src/transaction/TransactionManager.js'
+import type { Operation, RollbackAction } from '../../../src/transaction/types.js'
+import { TransactionTimeoutError } from '../../../src/transaction/errors.js'
+import { Brainy } from '../../../src/brainy.js'
+import { NounType } from '../../../src/types/graphTypes.js'
+
+const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms))
+
+// Brainy's ValidationConfig fixes vectors at exactly 384 dimensions
+// (src/utils/paramValidation.ts) — match it so add() doesn't reject test data.
+const DIM = 384
+const V = (): number[] => Array(DIM).fill(0.1)
+
+/** An operation that counts every `execute()` invocation — the re-drive detector. */
+function countingOp(opts: { delayMs?: number; name: string }): Operation & { calls: number } {
+ const op = {
+ name: opts.name,
+ calls: 0,
+ async execute(): Promise {
+ op.calls++
+ if (opts.delayMs) await sleep(opts.delayMs)
+ return async () => {}
+ }
+ }
+ return op
+}
+
+describe('transaction timeouts are never internally re-driven (8.10.1 no-hot-retry contract)', () => {
+ it('(i) the single-op engine (TransactionManager.executeTransaction / Transaction.execute — what add() drives via persistSingleOp) runs the overrun operation EXACTLY once and surfaces ONE retryable+hotRetryUnsafe error', async () => {
+ const manager = new TransactionManager()
+ const op0 = countingOp({ name: 'op0-overruns-budget', delayMs: 30 })
+ const op1 = countingOp({ name: 'op1-must-never-start' })
+
+ let caught: unknown
+ try {
+ await manager.executeTransaction(
+ async (tx) => {
+ tx.addOperation(op0)
+ tx.addOperation(op1)
+ },
+ // Tiny explicit override — the same override seam `transact()`
+ // exposes as `options.timeoutMs`; wins outright over the
+ // opCount*2000 floor that gates every real single-op write
+ // (transactTimeoutBudget()'s override semantics).
+ { timeout: 5 }
+ )
+ } catch (err) {
+ caught = err
+ }
+
+ expect(caught).toBeInstanceOf(TransactionTimeoutError)
+ const err = caught as TransactionTimeoutError
+ // The machine-readable contract callers branch on instead of parsing
+ // message text (src/transaction/errors.ts).
+ expect(err.retryable).toBe(true)
+ expect(err.hotRetryUnsafe).toBe(true)
+
+ // The re-drive assertion: op0 (the one that overran) executed EXACTLY
+ // once — nothing inside TransactionManager/Transaction looped back and
+ // re-ran it — and op1 never started at all (the budget gate stopped it
+ // before it began, per Transaction.execute()'s per-operation loop).
+ expect(op0.calls).toBe(1)
+ expect(op1.calls).toBe(0)
+ })
+
+ it('(ii) add()\'s upsert-race retry loop (MAX_UPSERT_ATTEMPTS=10) exits on the FIRST TransactionTimeoutError — attempt counter stays at 1, never mistaken for the lost-insert-race signal it retries on', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ await brain.init()
+
+ let persistSingleOpCalls = 0
+ const timeoutError = new TransactionTimeoutError(5, 1, {
+ elapsedMs: 6,
+ totalOperations: 2,
+ operationName: 'SaveNounMetadata'
+ })
+ // Stub the private commit seam add() drives (persistSingleOp) to throw
+ // the exact error type the real engine surfaces on a mid-flight timeout.
+ // This test pins the upsert loop's EXCEPTION-HANDLING contract (does it
+ // retry a TransactionTimeoutError like it retries
+ // InsertPreconditionExistsSignal?), not the timing mechanics of a real
+ // timeout — those are pinned by test (i) and by
+ // tests/unit/transaction/timeout-rollback.test.ts.
+ ;(brain as any).persistSingleOp = async (): Promise => {
+ persistSingleOpCalls++
+ throw timeoutError
+ }
+
+ await expect(
+ brain.add({ data: 'a', type: NounType.Thing, vector: V() })
+ ).rejects.toBe(timeoutError)
+
+ // The loop's attempt counter: exactly one call, never retried up to
+ // MAX_UPSERT_ATTEMPTS.
+ expect(persistSingleOpCalls).toBe(1)
+ await brain.close()
+ })
+})
From 5b2cbf74e568d4bb1f89cd7019d8d70c05999188 Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Fri, 24 Jul 2026 16:02:01 -0700
Subject: [PATCH 13/15] fix: warm() metadata surface routes through the active
provider (warm hook added to the metadata contract); add maintenanceDebt()
observability surface
A production deployment's warm report showed metadata: 'unavailable' under a
native metadata provider. brain.warm()'s metadata leg only duck-typed the
built-in JS manager's hydrateAll() method, which a native provider has no
reason to implement.
- MetadataIndexProvider (src/plugin.ts) gains an optional warm?(): Promise
hook, mirroring the existing vector and graph provider hooks. brain.warm()
now checks the active provider's own warm() FIRST, falls back to the JS
manager's hydrateAll() when absent, and reports 'unavailable' only when
neither exists -- never init() as a stand-in, since a native provider's
init() may be a cheap verify rather than a real warm.
- Tests (tests/unit/brainy/warm.test.ts): a live provider instance shaped to
have warm() reports 'warmed' and the hook called with no hydrateAll
fallback; shaped to have neither hook reports 'unavailable' (pins the
honest branch); the unmodified built-in JS manager still reports 'warmed'
via hydrateAll(), unchanged.
Additive scope agreed mid-flight with the native-provider team: a
maintenance-debt observability seam so an operator sees a grind coming
instead of discovering it as a CPU storm.
- New optional maintenanceDebt?(): Promise hook on
all three provider contracts (vector, metadata, graph -- the same three
warm?() lives on). ProviderMaintenanceDebt is fields-all-optional: a
provider reports only what it truly measures (pendingBytes, pendingItems,
lastPassCompletedAt, lastPassOutcome, converging), never an estimate
dressed as fact.
- New public brain.maintenanceDebt(): a pure passthrough -- for each surface
it calls only the active provider's own hook and reports the payload
verbatim, or 'unavailable' when absent. No thresholds, no polling, no
JS-side estimation; the provider owns the numbers, the operator owns the
policy.
- ProviderMaintenanceDebt, MaintenanceDebtReport, and MaintenanceDebtOutcome
are exported from the package root.
- Tests (tests/unit/brainy/maintenance-debt.test.ts): hook present reports
'reported' with the exact payload passed through; hook absent reports
'unavailable' on every surface; mixed surfaces resolve independently of
each other.
RELEASES.md gains the 8.10.1 entry covering both fixes above and this
feature, including the no-hot-retry contract from the prior commit.
---
RELEASES.md | 62 +++++++++++
src/brainy.ts | 112 ++++++++++++++++++-
src/index.ts | 10 ++
src/plugin.ts | 83 ++++++++++++++
tests/unit/brainy/maintenance-debt.test.ts | 122 +++++++++++++++++++++
tests/unit/brainy/warm.test.ts | 88 +++++++++++++++
6 files changed, 471 insertions(+), 6 deletions(-)
create mode 100644 tests/unit/brainy/maintenance-debt.test.ts
diff --git a/RELEASES.md b/RELEASES.md
index 89bf38e7..03d7283a 100644
--- a/RELEASES.md
+++ b/RELEASES.md
@@ -31,6 +31,68 @@ is sometimes cited as a 7.x removal — those methods never existed on 7.x; the
---
+## v8.10.1 — 2026-07-24 (the no-hot-retry contract + warm()'s metadata surface under native providers)
+
+From a production incident: a native-provider op ground 38-40s inside a transaction,
+blew the ~32s apply-phase budget, was rolled back (zero loss, by design), and a
+downstream pipeline hot-retried the identical operation into a 6-minute, 100%-CPU
+storm. Investigation confirmed Brainy itself never auto-retries a timed-out
+transaction — the storm was entirely the consumer's own retry loop, driven by a
+"retryable" doc-prose claim with no machine-readable contract to branch on. This
+release closes that contract gap and, separately, fixes a real `warm()` reporting gap
+surfaced by the same investigation.
+
+- **`TransactionTimeoutError` is now a machine-readable no-hot-retry contract.** Two
+ new typed, always-`true` fields replace prose-only guidance:
+ - `retryable: true` — the operation MAY succeed on a later attempt, once the
+ underlying slowness resolves or the budget is deliberately raised
+ (`transactionBudgetFloorMs`, or a batch's own `timeoutMs` override).
+ - `hotRetryUnsafe: true` — an immediate, identical retry re-pays the FULL cost of
+ the work that just timed out (it does not resume partway) and can cascade into
+ exactly the CPU storm above. **Never loop on this error.** The documented pattern
+ is a latch, not a retry loop:
+ ```
+ on TransactionTimeoutError:
+ record { at: Date.now(), error }
+ rethrow loudly to your own caller
+ hold a cooldown window before any re-attempt
+ clear the latch only on a subsequent success
+ ```
+ - `context` (unchanged, now fully documented) carries the backoff inputs:
+ `timeoutMs`, `operationIndex`, `elapsedMs`, `totalOperations`, `operationName`.
+ - Every "retryable" doc-prose site referencing this error (`transact()`'s
+ `timeoutMs` option, `transactionBudgetFloorMs`, `Transaction.execute()`) now
+ points at these fields instead of bare prose.
+ - Regression-pinned: the engine never internally re-drives a timed-out operation
+ (verified via an execution counter through both the single-op write path and
+ `add()`'s upsert-race retry loop), so this has always been true — it is now
+ provable and typed.
+- **Dead code removed**: `TransactionManager.executeTransactionWithResult()` had zero
+ callers in this codebase and is deleted.
+- **`brain.warm()`'s metadata surface now routes through the ACTIVE provider.** A
+ production deployment's warm report showed `metadata: 'unavailable'` under a native
+ metadata provider — the previous logic only duck-typed the built-in JS manager's
+ `hydrateAll()` method, which a native provider has no reason to implement. The
+ metadata provider contract (`MetadataIndexProvider`, `src/plugin.ts`) gains an
+ optional `warm?(): Promise` hook, mirroring the existing vector and graph
+ provider hooks. `brain.warm()` now checks the active provider's own `warm()` FIRST,
+ falls back to the JS manager's `hydrateAll()` when absent, and only reports
+ `'unavailable'` when neither exists — never `init()` as a stand-in, since a native
+ provider's `init()` may be a cheap verify rather than a real warm. A native
+ provider lights this surface up the same way `@soulcraft/cor` already lights the
+ vector and graph surfaces: implement `warm()` on its metadata provider.
+- **New: `brain.maintenanceDebt()`** — the observability seam so an operator sees a
+ provider's outstanding background maintenance work (pending bytes/items, last pass
+ outcome, whether it's converging) BEFORE it grinds into the kind of budget-busting
+ op this release's timeout contract exists for, instead of discovering it as a CPU
+ storm. It is a pure passthrough: brainy applies no thresholds, no polling, and no
+ estimation — it calls each active provider's own optional `maintenanceDebt?()` hook
+ (vector, metadata, graph — the same three contracts `warm?()` lives on) and reports
+ the payload verbatim, or `'unavailable'` when a surface's provider doesn't track
+ debt. Useful as a pre-warm/post-warm check or a boot gate. `@soulcraft/cor` does not
+ yet implement the hook as of this release — expect it on cor's next release; until
+ then all three surfaces honestly report `'unavailable'`.
+
## Unreleased (the warm contract: cold-restart writes stop paying demand-load latency)
From a production deployment's cold-restart incident: the FIRST writes after every
diff --git a/src/brainy.ts b/src/brainy.ts
index 37b6e491..007cdb32 100644
--- a/src/brainy.ts
+++ b/src/brainy.ts
@@ -65,7 +65,8 @@ import type {
OpaqueIdSet,
AtGenerationVectors,
VectorIndexProvider,
- GraphIndexProvider
+ GraphIndexProvider,
+ ProviderMaintenanceDebt
} from './plugin.js'
import type {
BrainyPlugin,
@@ -424,6 +425,30 @@ export interface WarmReport {
totalDurationMs: number
}
+/**
+ * @description Result of {@link Brainy.maintenanceDebt}: one outcome per
+ * index surface, mirroring {@link WarmReport}'s shape.
+ * - `'reported'` — the active provider for this surface implements
+ * `maintenanceDebt?()` and its {@link ProviderMaintenanceDebt} payload is
+ * attached verbatim under `debt`.
+ * - `'unavailable'` — the active provider does not implement the hook, so
+ * nothing is known; brainy never estimates or infers a payload on its
+ * behalf.
+ */
+export type MaintenanceDebtOutcome = 'reported' | 'unavailable'
+
+/**
+ * @description Per-surface result of {@link Brainy.maintenanceDebt}. Brainy
+ * performs no thresholding, polling, or estimation over this data — it is a
+ * pure passthrough of each active provider's own self-report (the provider
+ * owns the numbers; the operator owns the policy).
+ */
+export interface MaintenanceDebtReport {
+ vector: { outcome: MaintenanceDebtOutcome; debt?: ProviderMaintenanceDebt }
+ metadata: { outcome: MaintenanceDebtOutcome; debt?: ProviderMaintenanceDebt }
+ graph: { outcome: MaintenanceDebtOutcome; debt?: ProviderMaintenanceDebt }
+}
+
/**
* How long a failed aggregation-backfill walk suppresses fresh walk attempts.
* Within the window, queries rethrow the recorded failure instantly (loud,
@@ -14313,9 +14338,16 @@ export class Brainy implements BrainyInterface {
* *some* backing storage as a side effect but is reported honestly as
* `'probed'`, never `'warmed'`. An empty index or unknown dimension has
* nothing to probe (`'unavailable'`).
- * - **Metadata**: full hydration — every persisted field's sparse index is
- * loaded from storage (`MetadataIndexManager.hydrateAll()`), not just the
- * heuristic common-fields subset `init()` warms.
+ * - **Metadata**: calls the provider's own `warm?()` when the active
+ * `'metadataIndex'` provider implements it (`'warmed'`) — the seam a
+ * native metadata provider lights up so it is not duck-typed against the
+ * JS manager's method. Otherwise falls back to full hydration on the
+ * built-in JS manager — every persisted field's sparse index is loaded
+ * from storage (`MetadataIndexManager.hydrateAll()`), not just the
+ * heuristic common-fields subset `init()` warms — and reports `'warmed'`.
+ * Neither seam present → `'unavailable'` (honest: `init()` is never used
+ * as a substitute here, since a native provider's `init()` may be a
+ * cheap verify rather than a real warm).
* - **Graph**: calls the provider's own `warm?()` when the active graph
* provider implements it; otherwise re-runs its existing eager cold-load
* `init()` seam (idempotent — the JS adjacency index's `init()` already
@@ -14371,12 +14403,23 @@ export class Brainy implements BrainyInterface {
// --- Metadata --------------------------------------------------------
const metadataStart = Date.now()
let metadataOutcome: WarmOutcome
+ const metadataProvider = this.metadataIndex as unknown as MetadataIndexProvider
const metadataWithHydrate = this.metadataIndex as unknown as { hydrateAll?: () => Promise }
- if (typeof metadataWithHydrate.hydrateAll === 'function') {
+ if (typeof metadataProvider.warm === 'function') {
+ // Active provider (e.g. a native metadata index) declares its own warm
+ // seam — route through it FIRST so a native provider's warmth is
+ // reported honestly instead of being duck-typed against the JS
+ // manager's hydrateAll(), which a native provider does not implement.
+ await metadataProvider.warm()
+ metadataOutcome = 'warmed'
+ } else if (typeof metadataWithHydrate.hydrateAll === 'function') {
+ // Built-in JS manager path — full sparse-index hydration.
await metadataWithHydrate.hydrateAll()
metadataOutcome = 'warmed'
} else {
- // No hydration seam on this metadata provider — nothing to run.
+ // No hydration seam on this metadata provider — nothing to run. (No
+ // init() fallback here: init() on a native provider may be a cheap
+ // verify, and reporting that as warmth would lie.)
metadataOutcome = 'unavailable'
}
const metadataDurationMs = Date.now() - metadataStart
@@ -14410,6 +14453,63 @@ export class Brainy implements BrainyInterface {
}
}
+ /**
+ * Read each index surface's self-reported outstanding maintenance work —
+ * the observability seam so an operator sees a grind coming (rising
+ * pending bytes/items, a stalled background pass) instead of discovering
+ * it as a CPU storm or a transaction blowing its budget mid-flight (see
+ * {@link TransactionTimeoutError}).
+ *
+ * PURE PASSTHROUGH: for each of vector/metadata/graph, this calls ONLY the
+ * ACTIVE provider's own `maintenanceDebt?()` hook (the same per-surface
+ * provider resolution {@link Brainy.warm} uses) and reports its
+ * {@link ProviderMaintenanceDebt} payload verbatim. There is no JS-side
+ * fallback computation, no threshold evaluation, and no polling — brainy
+ * surfaces the truth the provider measured; the provider owns the numbers
+ * and the operator owns the policy (what threshold matters, what action to
+ * take). A surface whose active provider does not implement the hook
+ * reports `'unavailable'` — never a guessed or zeroed payload.
+ *
+ * @returns A {@link MaintenanceDebtReport}: per-surface outcome + payload.
+ * @example
+ * ```typescript
+ * const debt = await brain.maintenanceDebt()
+ * if (debt.metadata.outcome === 'reported' && debt.metadata.debt?.pendingBytes) {
+ * console.log('metadata pending bytes:', debt.metadata.debt.pendingBytes)
+ * }
+ * ```
+ */
+ async maintenanceDebt(): Promise {
+ await this.ensureInitialized({ needs: ['vector', 'metadata', 'graph'] })
+
+ // --- Vector ---------------------------------------------------------
+ const vectorProvider = this.index as VectorIndexProvider & {
+ maintenanceDebt?: () => Promise
+ }
+ const vector =
+ typeof vectorProvider.maintenanceDebt === 'function'
+ ? { outcome: 'reported' as const, debt: await vectorProvider.maintenanceDebt() }
+ : { outcome: 'unavailable' as const }
+
+ // --- Metadata --------------------------------------------------------
+ const metadataProvider = this.metadataIndex as unknown as MetadataIndexProvider
+ const metadata =
+ typeof metadataProvider.maintenanceDebt === 'function'
+ ? { outcome: 'reported' as const, debt: await metadataProvider.maintenanceDebt() }
+ : { outcome: 'unavailable' as const }
+
+ // --- Graph -------------------------------------------------------------
+ const graphProvider = this.graphIndex as GraphIndexProvider & {
+ maintenanceDebt?: () => Promise
+ }
+ const graph =
+ typeof graphProvider.maintenanceDebt === 'function'
+ ? { outcome: 'reported' as const, debt: await graphProvider.maintenanceDebt() }
+ : { outcome: 'unavailable' as const }
+
+ return { vector, metadata, graph }
+ }
+
/**
* Explicitly warm up the embedding engine
*
diff --git a/src/index.ts b/src/index.ts
index 00adc191..e60739f7 100644
--- a/src/index.ts
+++ b/src/index.ts
@@ -31,6 +31,11 @@ export type { DiagnosticsResult } from './brainy.js'
// brain.warm() — eager index/storage readiness report (per-surface honest
// outcome + timing). See the WarmReport JSDoc in brainy.ts.
export type { WarmReport, WarmOutcome } from './brainy.js'
+// brain.maintenanceDebt() — per-surface passthrough of each active
+// provider's self-reported background maintenance debt. See the
+// MaintenanceDebtReport JSDoc in brainy.ts and ProviderMaintenanceDebt in
+// plugin.ts for the measure-only-what-you-track contract.
+export type { MaintenanceDebtReport, MaintenanceDebtOutcome } from './brainy.js'
export type {
GraphAuditReport,
GraphAuditDiscrepancy
@@ -227,6 +232,11 @@ export type { FamilyStamp, StampMembers, StampVerdict } from './db/familyStamp.j
export { isVersionedIndexProvider } from './plugin.js'
export type { VersionedIndexProvider } from './plugin.js'
export type { ProviderInvariantReport, InvariantResult, InvariantHeal } from './plugin.js'
+// Optional provider self-report of outstanding background maintenance work
+// (compaction, deferred writes, etc.) — the payload type for
+// brain.maintenanceDebt(). See the measure-only-what-you-track contract on
+// ProviderMaintenanceDebt in plugin.ts.
+export type { ProviderMaintenanceDebt } from './plugin.js'
// Optional native graph-acceleration engine (cor 3.0) — the published provider
// contract + its columnar wire types. Brainy feature-detects an implementation
// and falls back to its pure-TS adjacency when absent.
diff --git a/src/plugin.ts b/src/plugin.ts
index 02639955..ce973386 100644
--- a/src/plugin.ts
+++ b/src/plugin.ts
@@ -171,6 +171,37 @@ export interface ProviderInvariantReport {
durationMs: number
}
+/**
+ * @description A provider's self-report of its own outstanding background
+ * maintenance work (compaction, deferred writes, a build-new→verify→swap in
+ * flight, etc.) — the observability seam so an operator sees a grind coming
+ * (rising pending bytes/items, a stalled pass) instead of discovering it as a
+ * CPU storm or a timeout under transaction budget pressure. Every field is
+ * OPTIONAL and every field is a MEASUREMENT: a provider reports ONLY what it
+ * actually tracks, never an estimate dressed up as a fact. Absence of the
+ * {@link VectorIndexProvider.maintenanceDebt} /
+ * {@link GraphIndexProvider.maintenanceDebt} /
+ * {@link MetadataIndexProvider.maintenanceDebt} hook itself means the
+ * provider does not track debt at all — brainy reports that surface
+ * `'unavailable'` rather than inventing zeros. Brainy performs NO threshold
+ * checks, NO polling, and NO JS-side estimation over this payload — it is a
+ * pure passthrough via {@link Brainy.maintenanceDebt}; the provider owns the
+ * numbers and the operator owns the policy (what threshold matters, what to
+ * do about it).
+ */
+export interface ProviderMaintenanceDebt {
+ /** Bytes of outstanding/unmerged work, if the provider measures it (e.g. unflushed writes, unmerged segments). */
+ pendingBytes?: number
+ /** Count of outstanding items (records, segments, nodes) awaiting the provider's background pass. */
+ pendingItems?: number
+ /** Epoch millis when the provider's last maintenance pass finished, if it tracks one. */
+ lastPassCompletedAt?: number
+ /** How the last pass ended, if the provider tracks pass outcomes. */
+ lastPassOutcome?: 'completed' | 'partial' | 'failed'
+ /** `true` if the provider's own measurements show debt trending down (making progress); `false` if flat or growing; omitted if the provider can't tell. */
+ converging?: boolean
+}
+
/**
* The `'metadataIndex'` provider — a drop-in for `MetadataIndexManager`.
* Brainy calls this surface via `this.metadataIndex.*` (see `brainy.ts`) and
@@ -181,6 +212,34 @@ export interface MetadataIndexProvider {
flush(): Promise
rebuild(): Promise
+ /**
+ * @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap
+ * pretouch, full sparse-index hydration) so first queries run at
+ * steady-state cost. Optional; absence means the provider demand-loads.
+ * Mirrors {@link GraphIndexProvider.warm} / the vector provider's `warm?()`
+ * (`src/plugin.ts` VectorIndexProvider). Distinct from `init()`: `init` is
+ * required and runs once automatically during brain startup; `warm` is a
+ * separate, explicit step a caller opts into via `brain.warm()` (or
+ * `warmOnOpen`) to pre-pay demand-load cost `init` left lazy. Idempotent —
+ * calling it more than once must be safe and cheap on a brain that is
+ * already warm. A provider that already loads everything eagerly in
+ * `init()` may implement `warm` as a no-op or omit it — `brain.warm()`
+ * falls back to the built-in JS manager's `hydrateAll()` duck-type when
+ * absent, and to an honest `'unavailable'` when neither exists.
+ */
+ warm?(): Promise
+
+ /**
+ * @description OPTIONAL self-reported {@link ProviderMaintenanceDebt} —
+ * the observability seam so an operator sees outstanding background
+ * maintenance work (e.g. unmerged postings) BEFORE it grinds a transaction
+ * into a budget-busting op. Absence means this provider does not track
+ * debt; `brain.maintenanceDebt()` reports this surface `'unavailable'`
+ * rather than guessing. See {@link ProviderMaintenanceDebt} for the
+ * measure-only-what-you-track contract.
+ */
+ maintenanceDebt?(): Promise
+
/**
* @description OPTIONAL honest durability signal (readiness contract,
* mirrors `isReady?()` on the graph and vector providers). `true` ⇔ the
@@ -395,6 +454,18 @@ export interface GraphIndexProvider {
*/
warm?(): Promise
+ /**
+ * @description OPTIONAL self-reported {@link ProviderMaintenanceDebt} —
+ * the observability seam so an operator sees outstanding background
+ * maintenance work (e.g. a build-new→verify→swap in flight, unmerged
+ * adjacency segments) BEFORE it grinds a transaction into a
+ * budget-busting op. Absence means this provider does not track debt;
+ * `brain.maintenanceDebt()` reports this surface `'unavailable'` rather
+ * than guessing. See {@link ProviderMaintenanceDebt} for the
+ * measure-only-what-you-track contract.
+ */
+ maintenanceDebt?(): Promise
+
/**
* @description OPTIONAL. A native provider returns true from the moment its
* `init()` detects a large epoch-drift until its background
@@ -1057,6 +1128,18 @@ export interface VectorIndexProvider {
*/
warm?(): Promise
+ /**
+ * @description OPTIONAL self-reported {@link ProviderMaintenanceDebt} —
+ * the observability seam so an operator sees outstanding background
+ * maintenance work (e.g. unflushed writes, a pending rebuild) BEFORE it
+ * grinds a transaction into a budget-busting op. Absence means this
+ * provider does not track debt; `brain.maintenanceDebt()` reports this
+ * surface `'unavailable'` rather than guessing. See
+ * {@link ProviderMaintenanceDebt} for the measure-only-what-you-track
+ * contract.
+ */
+ maintenanceDebt?(): Promise
+
/**
* @description OPTIONAL honest durability signal (readiness contract,
* mirrors {@link GraphIndexProvider.isReady}). `true` ⇔ the persisted
diff --git a/tests/unit/brainy/maintenance-debt.test.ts b/tests/unit/brainy/maintenance-debt.test.ts
new file mode 100644
index 00000000..4026d655
--- /dev/null
+++ b/tests/unit/brainy/maintenance-debt.test.ts
@@ -0,0 +1,122 @@
+/**
+ * @module tests/unit/brainy/maintenance-debt
+ * @description Coverage for `brain.maintenanceDebt()` (8.10.1) — the
+ * observability seam so an operator sees a provider's outstanding background
+ * maintenance work (compaction, deferred writes, a build-new→verify→swap in
+ * flight, ...) BEFORE it grinds a transaction into a budget-busting op, the
+ * same failure class documented on `TransactionTimeoutError`
+ * (src/transaction/errors.ts). Sibling to tests/unit/brainy/warm.test.ts,
+ * which establishes this file's technique: shape the probe points brain.ts
+ * reads (`typeof provider.maintenanceDebt === 'function'`) directly on the
+ * REAL, live provider instances rather than hand-rolling full fakes for the
+ * larger `MetadataIndexProvider` / `GraphIndexProvider` interfaces.
+ *
+ * `brain.maintenanceDebt()` is a PURE PASSTHROUGH: no thresholds, no
+ * polling, no JS-side estimation — these tests pin exactly that by asserting
+ * the returned payload is the provider's object, verbatim.
+ */
+import { describe, it, expect } from 'vitest'
+import { Brainy } from '../../../src/brainy.js'
+import { NounType } from '../../../src/types/graphTypes.js'
+import type { ProviderMaintenanceDebt } from '../../../src/plugin.js'
+
+// Brainy's ValidationConfig fixes vectors at exactly 384 dimensions
+// (src/utils/paramValidation.ts) — match it so add() doesn't reject test data.
+const DIM = 384
+const V = (seed = 1): number[] => Array.from({ length: DIM }, (_, i) => Math.sin(seed + i))
+
+async function freshBrain(): Promise> {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ await brain.init()
+ await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
+ return brain
+}
+
+describe('brain.maintenanceDebt()', () => {
+ it('reports "unavailable" for every surface when no active provider implements maintenanceDebt() (the built-in JS stack today)', async () => {
+ const brain = await freshBrain()
+
+ const report = await brain.maintenanceDebt()
+
+ expect(report.vector.outcome).toBe('unavailable')
+ expect(report.vector.debt).toBeUndefined()
+ expect(report.metadata.outcome).toBe('unavailable')
+ expect(report.metadata.debt).toBeUndefined()
+ expect(report.graph.outcome).toBe('unavailable')
+ expect(report.graph.debt).toBeUndefined()
+
+ await brain.close()
+ })
+
+ it('reports "reported" + the exact payload when the active provider implements maintenanceDebt() (verbatim passthrough, no thresholding)', async () => {
+ const brain = await freshBrain()
+
+ const vectorDebt: ProviderMaintenanceDebt = {
+ pendingBytes: 4_096,
+ pendingItems: 12,
+ lastPassCompletedAt: 1_700_000_000_000,
+ lastPassOutcome: 'completed',
+ converging: true
+ }
+ ;(brain as any).index.maintenanceDebt = async () => vectorDebt
+
+ const report = await brain.maintenanceDebt()
+
+ expect(report.vector.outcome).toBe('reported')
+ // Verbatim passthrough — the exact object, not a re-derived copy.
+ expect(report.vector.debt).toBe(vectorDebt)
+ // Untouched surfaces stay honestly 'unavailable'.
+ expect(report.metadata.outcome).toBe('unavailable')
+ expect(report.graph.outcome).toBe('unavailable')
+
+ await brain.close()
+ })
+
+ it('mixed surfaces: each surface\'s outcome depends ONLY on its OWN active provider — one surface reporting never leaks into another', async () => {
+ const brain = await freshBrain()
+
+ const metadataDebt: ProviderMaintenanceDebt = {
+ pendingItems: 3,
+ lastPassOutcome: 'partial',
+ converging: false
+ }
+ const graphDebt: ProviderMaintenanceDebt = {
+ pendingBytes: 0,
+ converging: true
+ }
+ ;(brain as any).metadataIndex.maintenanceDebt = async () => metadataDebt
+ ;(brain as any).graphIndex.maintenanceDebt = async () => graphDebt
+ // Vector is deliberately left unpatched.
+
+ const report = await brain.maintenanceDebt()
+
+ expect(report.vector.outcome).toBe('unavailable')
+ expect(report.vector.debt).toBeUndefined()
+
+ expect(report.metadata.outcome).toBe('reported')
+ expect(report.metadata.debt).toBe(metadataDebt)
+
+ expect(report.graph.outcome).toBe('reported')
+ expect(report.graph.debt).toBe(graphDebt)
+
+ await brain.close()
+ })
+
+ it('an empty ProviderMaintenanceDebt object (every field omitted) is still honestly "reported" — presence of the hook, not the payload\'s richness, drives the outcome', async () => {
+ const brain = await freshBrain()
+
+ const emptyDebt: ProviderMaintenanceDebt = {}
+ ;(brain as any).graphIndex.maintenanceDebt = async () => emptyDebt
+
+ const report = await brain.maintenanceDebt()
+
+ expect(report.graph.outcome).toBe('reported')
+ expect(report.graph.debt).toEqual({})
+
+ await brain.close()
+ })
+})
diff --git a/tests/unit/brainy/warm.test.ts b/tests/unit/brainy/warm.test.ts
index ce213696..8a0bb4da 100644
--- a/tests/unit/brainy/warm.test.ts
+++ b/tests/unit/brainy/warm.test.ts
@@ -307,4 +307,92 @@ describe('brain.warm()', () => {
expect(report.totalDurationMs).toBeGreaterThanOrEqual(0)
await brain.close()
})
+
+ // --- Metadata leg routes through the ACTIVE provider (8.10.1) -----------
+ //
+ // `MetadataIndexProvider` is a ~50-method interface (src/plugin.ts) — far
+ // too large to hand-write a compliant fake class the way `FakeVectorProvider`
+ // fakes the ~8-method `VectorIndexProvider` above. Test (c) already
+ // establishes this file's pattern for the metadata leg: exercise the REAL
+ // `MetadataIndexManager` instance and shape just the probe points brain.ts
+ // reads (`typeof provider.warm === 'function'` /
+ // `typeof provider.hydrateAll === 'function'`) directly on that instance.
+ // Shadowing an own property on the live object stands in for "a different
+ // provider implementation" without needing a hand-rolled full fake — the
+ // rest of the real manager (used by add()/init() above) is untouched.
+ describe('metadata leg — warm() routes through the active provider', () => {
+ it('(f) calls the ACTIVE metadata provider\'s warm() when present and reports "warmed", never falling back to hydrateAll', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ await brain.init()
+ await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
+
+ const metadataIndex = (brain as any).metadataIndex
+ let warmCalls = 0
+ let hydrateAllCalls = 0
+ const origHydrateAll = metadataIndex.hydrateAll.bind(metadataIndex)
+ metadataIndex.hydrateAll = async (...args: unknown[]) => {
+ hydrateAllCalls++
+ return origHydrateAll(...args)
+ }
+ // Simulates a native metadata provider declaring the optional `warm()`
+ // hook added to `MetadataIndexProvider` (src/plugin.ts) in 8.10.1.
+ metadataIndex.warm = async () => {
+ warmCalls++
+ }
+
+ const report = await brain.warm()
+
+ expect(warmCalls).toBe(1)
+ expect(hydrateAllCalls).toBe(0) // warm() ran — no hydrateAll fallback
+ expect(report.metadata.outcome).toBe('warmed')
+ await brain.close()
+ })
+
+ it('reports "unavailable" when the active metadata provider implements neither warm() nor hydrateAll() (the honest branch a native provider without either hook must hit)', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ await brain.init()
+ await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
+
+ const metadataIndex = (brain as any).metadataIndex
+ // Shadow away BOTH optional hooks — models a genuinely native provider
+ // that (unlike the built-in JS manager) offers neither seam. This must
+ // never fall back to calling init() as a stand-in for warmth.
+ metadataIndex.warm = undefined
+ metadataIndex.hydrateAll = undefined
+
+ const report = await brain.warm()
+
+ expect(report.metadata.outcome).toBe('unavailable')
+ await brain.close()
+ })
+
+ it('the built-in JS manager (no warm()) still reports "warmed" via its existing hydrateAll() duck-type — unchanged by the new provider hook', async () => {
+ const brain = new Brainy({
+ requireSubtype: false,
+ storage: { type: 'memory' },
+ silent: true
+ })
+ await brain.init()
+ await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
+
+ // No patching at all — the default built-in MetadataIndexManager has
+ // hydrateAll() but no warm(), exactly as it did before this change.
+ const metadataIndex = (brain as any).metadataIndex
+ expect(typeof metadataIndex.warm).not.toBe('function')
+ expect(typeof metadataIndex.hydrateAll).toBe('function')
+
+ const report = await brain.warm()
+
+ expect(report.metadata.outcome).toBe('warmed')
+ await brain.close()
+ })
+ })
})
From edf123a5e232919881ae9d5bfaa4877c7ee457ee Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Fri, 24 Jul 2026 16:04:41 -0700
Subject: [PATCH 14/15] refactor: remove the orphaned transaction-result type
left behind by the dead-path removal
---
src/transaction/types.ts | 20 --------------------
1 file changed, 20 deletions(-)
diff --git a/src/transaction/types.ts b/src/transaction/types.ts
index 9a3a2eaa..6cbc56ca 100644
--- a/src/transaction/types.ts
+++ b/src/transaction/types.ts
@@ -66,26 +66,6 @@ export interface TransactionContext {
*/
export type TransactionFunction = (ctx: TransactionContext) => Promise
-/**
- * Transaction execution result
- */
-export interface TransactionResult {
- /**
- * Result value from user function
- */
- value: T
-
- /**
- * Number of operations executed
- */
- operationCount: number
-
- /**
- * Execution time in milliseconds
- */
- executionTimeMs: number
-}
-
/**
* Transaction execution options
*/
From d9cc7b9024aff3fbffeb2fee658543d81fb4c0c9 Mon Sep 17 00:00:00 2001
From: David Snelling
Date: Fri, 24 Jul 2026 16:09:47 -0700
Subject: [PATCH 15/15] chore(release): 8.10.1
---
CHANGELOG.md | 8 ++++++++
package-lock.json | 4 ++--
package.json | 2 +-
3 files changed, 11 insertions(+), 3 deletions(-)
diff --git a/CHANGELOG.md b/CHANGELOG.md
index b6dd91fd..a5f344cc 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,14 @@
All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines.
+### [8.10.1](https://source.soulcraft.com/soulcraft/brainy/compare/v8.10.0...v8.10.1) (2026-07-24)
+
+- refactor: remove the orphaned transaction-result type left behind by the dead-path removal (edf123a5)
+- fix: warm() metadata surface routes through the active provider (warm hook added to the metadata contract); add maintenanceDebt() observability surface (5b2cbf74)
+- fix: transaction timeouts are a typed no-hot-retry contract; engine-side non-retry pinned; dead transaction path removed (003e2a74)
+- chore: the forge is the address — retire the archived mirror from every live surface (22702b81)
+
+
### [8.10.0](https://github.com/soulcraftlabs/brainy/compare/v8.9.0...v8.10.0) (2026-07-23)
- docs: adoption storefront — contributing guide, security policy, README support + cor section (9a99a7b)
diff --git a/package-lock.json b/package-lock.json
index 37aeb81d..d0c7b9d9 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -1,12 +1,12 @@
{
"name": "@soulcraft/brainy",
- "version": "8.10.0",
+ "version": "8.10.1",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "@soulcraft/brainy",
- "version": "8.10.0",
+ "version": "8.10.1",
"license": "MIT",
"dependencies": {
"@msgpack/msgpack": "^3.1.2",
diff --git a/package.json b/package.json
index a3ece83c..ce670369 100644
--- a/package.json
+++ b/package.json
@@ -1,6 +1,6 @@
{
"name": "@soulcraft/brainy",
- "version": "8.10.0",
+ "version": "8.10.1",
"description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.",
"main": "dist/index.js",
"module": "dist/index.js",