Compare commits
2 commits
main
...
feat/warm-
| Author | SHA1 | Date | |
|---|---|---|---|
| ec57174760 | |||
| 16832671fb |
13 changed files with 1153 additions and 57 deletions
66
RELEASES.md
66
RELEASES.md
|
|
@ -31,6 +31,72 @@ is sometimes cited as a 7.x removal — those methods never existed on 7.x; the
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## Unreleased (the warm contract: cold-restart writes stop paying demand-load latency)
|
||||||
|
|
||||||
|
From a production deployment's cold-restart incident: the FIRST writes after every
|
||||||
|
restart on a large brain measured 33–35s each (page cache cold) against the transact
|
||||||
|
apply budget — Brainy's own op-count-scaled budget, `max(30s, opCount × 2s)`, e.g.
|
||||||
|
32,000ms for a 16-op batch. There is no external deadline in this story, and no
|
||||||
|
post-completion veto either: every write is itself a multi-operation transaction (a
|
||||||
|
single `add()` applies several operations — canonical writes, the vector-index insert,
|
||||||
|
the metadata-index update), so ONE cold operation that legitimately runs ~33s consumes
|
||||||
|
the whole budget, the gate before the NEXT operation trips, and the write rolls back
|
||||||
|
atomically (zero loss, by design) — refused, retried, refused again, until the page
|
||||||
|
cache warms passively (~30 minutes). The cure is not weaker atomicity; it is the two
|
||||||
|
new knobs below — a budget floor sized for cold stores, and a warm contract that pays
|
||||||
|
demand-load cost OFF the transaction path.
|
||||||
|
|
||||||
|
- **The budget's start-gating contract is now explicit, documented, and pinned by
|
||||||
|
regression tests.** The budget gates STARTING the next operation — completed work is
|
||||||
|
never rolled back for elapsed time — and a transaction's first operation now
|
||||||
|
unconditionally starts by code, not merely because elapsed time happens to be ~0 when
|
||||||
|
it is checked. This has been the shipped schedule since 8.7.0 (no behavior change for
|
||||||
|
existing integrations); it is now stated in `Transaction.execute()`'s contract JSDoc
|
||||||
|
and enforced by tests so it cannot silently regress. Mid-batch atomicity is unchanged:
|
||||||
|
a trip before operation `i+1` still rolls back `0..i` and throws a retryable
|
||||||
|
`TransactionTimeoutError`.
|
||||||
|
- **The budget's 30s floor is now configurable**: `new Brainy({ transactionBudgetFloorMs })`
|
||||||
|
raises (or lowers) the floor of `max(transactionBudgetFloorMs, opCount × 2000)` for every
|
||||||
|
internal transact batch. Useful for a store whose cold writes legitimately run past 30s
|
||||||
|
per operation, so a bulk batch gets a proportionally larger runway instead of tripping
|
||||||
|
mid-batch on cold-cache latency.
|
||||||
|
- **New: `brain.warm()`** — eagerly loads/faults-in the vector index, metadata index, and
|
||||||
|
graph adjacency so the first real operation after a cold restart runs at steady-state
|
||||||
|
cost instead of paying demand-load latency on the critical path. Returns a `WarmReport`
|
||||||
|
with one honest outcome per surface — never conflate the first two:
|
||||||
|
- `'warmed'` — the surface's own provider `warm()` hook ran, or a full-hydration seam
|
||||||
|
loaded every shard/field/segment from storage. Steady-state cost is paid.
|
||||||
|
- `'probed'` — no `warm()` hook was available, so a best-effort read (one `search()` call
|
||||||
|
for the vector index) faulted in *some* backing storage as a side effect — real work,
|
||||||
|
but never reported as `'warmed'`.
|
||||||
|
- `'unavailable'` — nothing ran (no hook, no hydration seam, or nothing to probe).
|
||||||
|
- **New config: `warmOnOpen: true`** makes `init()` await `brain.warm()` before it resolves
|
||||||
|
— a deliberate blocking trade-off: startup takes longer, the first request doesn't.
|
||||||
|
Default `false` (unchanged lazy behavior).
|
||||||
|
- **New optional provider hook: `warm?(): Promise<void>`** on the vector and graph
|
||||||
|
acceleration provider contracts (`src/plugin.ts`) — a native provider can implement it to
|
||||||
|
eagerly pretouch its own backing storage (e.g. mmap pretouch); absence means brainy falls
|
||||||
|
back to the probe/hydration behavior above.
|
||||||
|
- **Transaction op-name strings changed in journals/timings**: the vector-index
|
||||||
|
transaction operations were renamed from `AddToHNSW`/`RemoveFromHNSW` to backend-neutral
|
||||||
|
`AddToVectorIndex(...)`/`RemoveFromVectorIndex(...)` — the old names hard-coded an
|
||||||
|
algorithm that may not be the one actually running (a non-HNSW native vector provider
|
||||||
|
emitting `"RemoveFromHNSW"` has sent an operator hunting an index that doesn't exist).
|
||||||
|
The parenthesized suffix names the ACTIVE backend: `hnsw-js` for the built-in engine, or
|
||||||
|
the native provider's own identity when it self-identifies. **If you parse these op-name
|
||||||
|
strings** (log processors, journal tooling), update your matcher from
|
||||||
|
`AddToHNSW`/`RemoveFromHNSW` to `AddToVectorIndex(`/`RemoveFromVectorIndex(`.
|
||||||
|
- **Provider identity is now a REQUIRED `name` field** on the vector provider contract
|
||||||
|
(`VectorIndexProvider.name`, `src/plugin.ts`) — every implementation self-reports its own
|
||||||
|
identity truthfully (its algorithm/engine), never inheriting a default. It renders as the
|
||||||
|
op-name suffix above and, wherever the vector index identifies itself in prose log lines,
|
||||||
|
as the tag `[vector-index:<name>]`. **Native provider adoption is a one-line change**:
|
||||||
|
declare `readonly name = '<your-provider-name>'`. A provider instance that still lacks
|
||||||
|
`name` at runtime (an older native build compiled against the previous, optional field) is
|
||||||
|
never crashed on and never silently mislabeled: it stamps `unknown-provider` and emits one
|
||||||
|
loud warning naming the missing field, so the gap is discoverable instead of a permanent
|
||||||
|
fossil label in every journal line.
|
||||||
|
|
||||||
## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close())
|
## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close())
|
||||||
|
|
||||||
The write path stops paying maintenance costs — the last structural piece of the
|
The write path stops paying maintenance costs — the last structural piece of the
|
||||||
|
|
|
||||||
205
src/brainy.ts
205
src/brainy.ts
|
|
@ -63,7 +63,9 @@ import type {
|
||||||
PathOptions,
|
PathOptions,
|
||||||
MetadataIndexProvider,
|
MetadataIndexProvider,
|
||||||
OpaqueIdSet,
|
OpaqueIdSet,
|
||||||
AtGenerationVectors
|
AtGenerationVectors,
|
||||||
|
VectorIndexProvider,
|
||||||
|
GraphIndexProvider
|
||||||
} from './plugin.js'
|
} from './plugin.js'
|
||||||
import type {
|
import type {
|
||||||
BrainyPlugin,
|
BrainyPlugin,
|
||||||
|
|
@ -88,12 +90,12 @@ import { findCallerLocation } from './utils/callerLocation.js'
|
||||||
import {
|
import {
|
||||||
SaveNounMetadataOperation,
|
SaveNounMetadataOperation,
|
||||||
SaveNounOperation,
|
SaveNounOperation,
|
||||||
AddToHNSWOperation,
|
AddToVectorIndexOperation,
|
||||||
AddToMetadataIndexOperation,
|
AddToMetadataIndexOperation,
|
||||||
SaveVerbMetadataOperation,
|
SaveVerbMetadataOperation,
|
||||||
SaveVerbOperation,
|
SaveVerbOperation,
|
||||||
AddToGraphIndexOperation,
|
AddToGraphIndexOperation,
|
||||||
RemoveFromHNSWOperation,
|
RemoveFromVectorIndexOperation,
|
||||||
RemoveFromMetadataIndexOperation,
|
RemoveFromMetadataIndexOperation,
|
||||||
RemoveFromGraphIndexOperation,
|
RemoveFromGraphIndexOperation,
|
||||||
UpdateNounMetadataOperation,
|
UpdateNounMetadataOperation,
|
||||||
|
|
@ -266,6 +268,7 @@ type ResolvedBrainyConfig = Required<
|
||||||
| 'retention'
|
| 'retention'
|
||||||
| 'eagerEmbeddings'
|
| 'eagerEmbeddings'
|
||||||
| 'migrationWaitTimeoutMs'
|
| 'migrationWaitTimeoutMs'
|
||||||
|
| 'transactionBudgetFloorMs'
|
||||||
>
|
>
|
||||||
> &
|
> &
|
||||||
Pick<
|
Pick<
|
||||||
|
|
@ -278,6 +281,7 @@ type ResolvedBrainyConfig = Required<
|
||||||
| 'retention'
|
| 'retention'
|
||||||
| 'eagerEmbeddings'
|
| 'eagerEmbeddings'
|
||||||
| 'migrationWaitTimeoutMs'
|
| 'migrationWaitTimeoutMs'
|
||||||
|
| 'transactionBudgetFloorMs'
|
||||||
>
|
>
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -388,6 +392,38 @@ class InsertPreconditionExistsSignal extends Error {
|
||||||
*/
|
*/
|
||||||
export type IndexFamily = 'vector' | 'metadata' | 'graph'
|
export type IndexFamily = 'vector' | 'metadata' | 'graph'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Honest per-surface outcome for {@link Brainy.warm}. Literal
|
||||||
|
* meanings — never conflate the first two:
|
||||||
|
* - `'warmed'` — the provider's own `warm?()` hook ran (vector/graph), or the
|
||||||
|
* surface's full-hydration seam loaded EVERY shard/field/segment from
|
||||||
|
* storage (metadata; graph's fallback path). The surface is genuinely at
|
||||||
|
* steady-state cost for the next operation.
|
||||||
|
* - `'probed'` — no `warm?()` hook was available, so a best-effort read
|
||||||
|
* (e.g. one `search()` call) faulted in *some* backing storage as a side
|
||||||
|
* effect. Real work happened, but it is NOT the same guarantee as
|
||||||
|
* `'warmed'` — never reported as `'warmed'`.
|
||||||
|
* - `'unavailable'` — nothing ran: no hook, no hydration seam, and (for the
|
||||||
|
* vector probe fallback) nothing to probe (an empty index or unknown
|
||||||
|
* vector dimension). The surface is unchanged by this `warm()` call.
|
||||||
|
*/
|
||||||
|
export type WarmOutcome = 'warmed' | 'probed' | 'unavailable'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Result of {@link Brainy.warm}: one {@link WarmOutcome} +
|
||||||
|
* elapsed time per index surface, plus the total wall-clock time for the
|
||||||
|
* whole call. `durationMs` is measured around exactly the work described by
|
||||||
|
* that surface's `outcome` (e.g. the vector entry's `durationMs` times the
|
||||||
|
* provider `warm()` call OR the probe `search()` call — whichever ran).
|
||||||
|
*/
|
||||||
|
export interface WarmReport {
|
||||||
|
vector: { outcome: WarmOutcome; durationMs: number }
|
||||||
|
metadata: { outcome: WarmOutcome; durationMs: number }
|
||||||
|
graph: { outcome: WarmOutcome; durationMs: number }
|
||||||
|
/** Total wall-clock time for the whole `warm()` call (all three surfaces). */
|
||||||
|
totalDurationMs: number
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* How long a failed aggregation-backfill walk suppresses fresh walk attempts.
|
* How long a failed aggregation-backfill walk suppresses fresh walk attempts.
|
||||||
* Within the window, queries rethrow the recorded failure instantly (loud,
|
* Within the window, queries rethrow the recorded failure instantly (loud,
|
||||||
|
|
@ -1382,6 +1418,17 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Eager index warm (operator opt-in, `warmOnOpen: true`). Runs AFTER
|
||||||
|
// every step above — index construction, crash recovery, migrations,
|
||||||
|
// VFS bootstrap — never in place of any of it, so `warm()` always
|
||||||
|
// operates on a fully-initialized brain. Blocking BY DESIGN: the
|
||||||
|
// operator traded a longer startup for a first-request that runs at
|
||||||
|
// steady-state cost instead of paying demand-load latency on the
|
||||||
|
// critical path. See the `warmOnOpen` JSDoc in brainy.types.ts.
|
||||||
|
if (this.config.warmOnOpen) {
|
||||||
|
await this.warm()
|
||||||
|
}
|
||||||
|
|
||||||
// Resolve ready Promise - consumers awaiting brain.ready will now proceed
|
// Resolve ready Promise - consumers awaiting brain.ready will now proceed
|
||||||
if (this._readyResolve) {
|
if (this._readyResolve) {
|
||||||
this._readyResolve()
|
this._readyResolve()
|
||||||
|
|
@ -1780,7 +1827,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
await this.generationStore.runWithoutGeneration(() =>
|
await this.generationStore.runWithoutGeneration(() =>
|
||||||
this.transactionManager.executeTransaction(run, {
|
this.transactionManager.executeTransaction(run, {
|
||||||
timeout: transactTimeoutBudget(
|
timeout: transactTimeoutBudget(
|
||||||
(touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0)
|
(touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0),
|
||||||
|
undefined,
|
||||||
|
this.config.transactionBudgetFloorMs
|
||||||
)
|
)
|
||||||
})
|
})
|
||||||
)
|
)
|
||||||
|
|
@ -1797,7 +1846,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
execute: () =>
|
execute: () =>
|
||||||
this.transactionManager.executeTransaction(run, {
|
this.transactionManager.executeTransaction(run, {
|
||||||
timeout: transactTimeoutBudget(
|
timeout: transactTimeoutBudget(
|
||||||
(touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0)
|
(touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0),
|
||||||
|
undefined,
|
||||||
|
this.config.transactionBudgetFloorMs
|
||||||
)
|
)
|
||||||
})
|
})
|
||||||
})
|
})
|
||||||
|
|
@ -2109,7 +2160,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
|
|
||||||
// Operation 3: Add to HNSW index (after entity saved)
|
// Operation 3: Add to HNSW index (after entity saved)
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new AddToHNSWOperation(this.index, id, vector)
|
new AddToVectorIndexOperation(this.index, id, vector)
|
||||||
)
|
)
|
||||||
|
|
||||||
// Operation 4: Add to metadata index
|
// Operation 4: Add to metadata index
|
||||||
|
|
@ -3098,10 +3149,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Operation 3-4: Update HNSW index (remove and re-add if reindexing needed)
|
// Operation 3-4: Update HNSW index (remove and re-add if reindexing needed)
|
||||||
if (needsReindexing) {
|
if (needsReindexing) {
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new RemoveFromHNSWOperation(this.index, params.id, existing.vector)
|
new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector)
|
||||||
)
|
)
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new AddToHNSWOperation(this.index, params.id, vector)
|
new AddToVectorIndexOperation(this.index, params.id, vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -3217,7 +3268,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Operation 1: Remove from vector index
|
// Operation 1: Remove from vector index
|
||||||
if (noun) {
|
if (noun) {
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new RemoveFromHNSWOperation(this.index, id, noun.vector)
|
new RemoveFromVectorIndexOperation(this.index, id, noun.vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -7025,7 +7076,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Add delete operations to transaction
|
// Add delete operations to transaction
|
||||||
if (noun) {
|
if (noun) {
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new RemoveFromHNSWOperation(this.index, id, noun.vector)
|
new RemoveFromVectorIndexOperation(this.index, id, noun.vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -7888,7 +7939,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Budget scales with the batch (or the caller's explicit
|
// Budget scales with the batch (or the caller's explicit
|
||||||
// timeoutMs): a flat 30s cap silently limited honest bulk work
|
// timeoutMs): a flat 30s cap silently limited honest bulk work
|
||||||
// to ~15 ops on network disks (~2s/op measured in the field).
|
// to ~15 ops on network disks (~2s/op measured in the field).
|
||||||
{ timeout: transactTimeoutBudget(plan.operations.length, options?.timeoutMs) }
|
{
|
||||||
|
timeout: transactTimeoutBudget(
|
||||||
|
plan.operations.length,
|
||||||
|
options?.timeoutMs,
|
||||||
|
this.config.transactionBudgetFloorMs
|
||||||
|
)
|
||||||
|
}
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
}))
|
}))
|
||||||
|
|
@ -9311,7 +9368,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
plan.operations.push(
|
plan.operations.push(
|
||||||
new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew),
|
new SaveNounMetadataOperation(this.storage, id, storageMetadata, isNew),
|
||||||
new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew),
|
new SaveNounOperation(this.storage, { id, vector, connections: new Map(), level: 0 }, isNew),
|
||||||
new AddToHNSWOperation(this.index, id, vector),
|
new AddToVectorIndexOperation(this.index, id, vector),
|
||||||
new AddToMetadataIndexOperation(this.metadataIndex, id, entityForIndexing)
|
new AddToMetadataIndexOperation(this.metadataIndex, id, entityForIndexing)
|
||||||
)
|
)
|
||||||
plan.touchedNouns.push(id)
|
plan.touchedNouns.push(id)
|
||||||
|
|
@ -9479,8 +9536,8 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
)
|
)
|
||||||
if (needsReindexing) {
|
if (needsReindexing) {
|
||||||
plan.operations.push(
|
plan.operations.push(
|
||||||
new RemoveFromHNSWOperation(this.index, params.id, existing.vector),
|
new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector),
|
||||||
new AddToHNSWOperation(this.index, params.id, vector)
|
new AddToVectorIndexOperation(this.index, params.id, vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
plan.operations.push(
|
plan.operations.push(
|
||||||
|
|
@ -9565,7 +9622,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
}
|
}
|
||||||
|
|
||||||
if (noun) {
|
if (noun) {
|
||||||
plan.operations.push(new RemoveFromHNSWOperation(this.index, id, noun.vector))
|
plan.operations.push(new RemoveFromVectorIndexOperation(this.index, id, noun.vector))
|
||||||
}
|
}
|
||||||
if (metadata) {
|
if (metadata) {
|
||||||
plan.operations.push(new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata))
|
plan.operations.push(new RemoveFromMetadataIndexOperation(this.metadataIndex, id, metadata))
|
||||||
|
|
@ -14280,6 +14337,118 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
return this.embedder(textToEmbed)
|
return this.embedder(textToEmbed)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Eagerly load/fault-in the vector index, metadata index, and graph
|
||||||
|
* adjacency so the FIRST real operation after this call runs at
|
||||||
|
* steady-state cost — no page-cache-miss / demand-load latency on the
|
||||||
|
* critical path. Complements {@link warmupEmbeddings} (which warms the
|
||||||
|
* embedding engine, not the storage indexes).
|
||||||
|
*
|
||||||
|
* Sequence per surface:
|
||||||
|
* - **Vector**: calls the provider's own `warm?()` when the active
|
||||||
|
* `'vector'` provider implements it (`'warmed'`). Otherwise runs a
|
||||||
|
* best-effort probe — one `search()` with a deterministic unit vector of
|
||||||
|
* the index's known dimension, `k = min(100, size())` — which faults in
|
||||||
|
* *some* backing storage as a side effect but is reported honestly as
|
||||||
|
* `'probed'`, never `'warmed'`. An empty index or unknown dimension has
|
||||||
|
* nothing to probe (`'unavailable'`).
|
||||||
|
* - **Metadata**: full hydration — every persisted field's sparse index is
|
||||||
|
* loaded from storage (`MetadataIndexManager.hydrateAll()`), not just the
|
||||||
|
* heuristic common-fields subset `init()` warms.
|
||||||
|
* - **Graph**: calls the provider's own `warm?()` when the active graph
|
||||||
|
* provider implements it; otherwise re-runs its existing eager cold-load
|
||||||
|
* `init()` seam (idempotent — the JS adjacency index's `init()` already
|
||||||
|
* loads the full LSM manifests + SSTables, so re-invoking it here is a
|
||||||
|
* genuine full hydration, not a stat call).
|
||||||
|
*
|
||||||
|
* A single surface's `'unavailable'`/probe outcome never fails the whole
|
||||||
|
* call — `warm()` is a best-effort readiness step, not a correctness gate;
|
||||||
|
* queries still demand-load normally regardless of what `warm()` achieved.
|
||||||
|
*
|
||||||
|
* @returns A {@link WarmReport}: honest per-surface outcome + timing.
|
||||||
|
* @example
|
||||||
|
* ```typescript
|
||||||
|
* const brain = new Brainy({ storage: { path: '/data' } })
|
||||||
|
* await brain.init()
|
||||||
|
* const report = await brain.warm() // blocking, explicit control over timing
|
||||||
|
* console.log(report.vector.outcome, report.vector.durationMs)
|
||||||
|
*
|
||||||
|
* // Or opt into the same work automatically during init():
|
||||||
|
* const eager = new Brainy({ storage: { path: '/data' }, warmOnOpen: true })
|
||||||
|
* await eager.init() // warm() already ran before this resolves
|
||||||
|
* ```
|
||||||
|
*/
|
||||||
|
async warm(): Promise<WarmReport> {
|
||||||
|
await this.ensureInitialized({ needs: ['vector', 'metadata', 'graph'] })
|
||||||
|
|
||||||
|
const totalStart = Date.now()
|
||||||
|
|
||||||
|
// --- Vector ---------------------------------------------------------
|
||||||
|
const vectorStart = Date.now()
|
||||||
|
let vectorOutcome: WarmOutcome
|
||||||
|
const vectorProvider = this.index as VectorIndexProvider & { warm?: () => Promise<void> }
|
||||||
|
if (typeof vectorProvider.warm === 'function') {
|
||||||
|
await vectorProvider.warm()
|
||||||
|
vectorOutcome = 'warmed'
|
||||||
|
} else {
|
||||||
|
const size = this.index.size()
|
||||||
|
const dimension = this.dimensions
|
||||||
|
if (size > 0 && dimension && dimension > 0) {
|
||||||
|
// A deterministic unit vector (L2 norm 1) — same probe every call,
|
||||||
|
// no reliance on stored data shape.
|
||||||
|
const probeVector = new Array(dimension).fill(1 / Math.sqrt(dimension))
|
||||||
|
await this.index.search(probeVector, Math.min(100, size))
|
||||||
|
vectorOutcome = 'probed'
|
||||||
|
} else {
|
||||||
|
// Nothing to probe: an empty index, or the vector dimension is not
|
||||||
|
// yet known (no entity has ever been added).
|
||||||
|
vectorOutcome = 'unavailable'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const vectorDurationMs = Date.now() - vectorStart
|
||||||
|
|
||||||
|
// --- Metadata --------------------------------------------------------
|
||||||
|
const metadataStart = Date.now()
|
||||||
|
let metadataOutcome: WarmOutcome
|
||||||
|
const metadataWithHydrate = this.metadataIndex as unknown as { hydrateAll?: () => Promise<void> }
|
||||||
|
if (typeof metadataWithHydrate.hydrateAll === 'function') {
|
||||||
|
await metadataWithHydrate.hydrateAll()
|
||||||
|
metadataOutcome = 'warmed'
|
||||||
|
} else {
|
||||||
|
// No hydration seam on this metadata provider — nothing to run.
|
||||||
|
metadataOutcome = 'unavailable'
|
||||||
|
}
|
||||||
|
const metadataDurationMs = Date.now() - metadataStart
|
||||||
|
|
||||||
|
// --- Graph -------------------------------------------------------------
|
||||||
|
const graphStart = Date.now()
|
||||||
|
let graphOutcome: WarmOutcome
|
||||||
|
const graphProvider = this.graphIndex as GraphIndexProvider & {
|
||||||
|
warm?: () => Promise<void>
|
||||||
|
init?: () => Promise<void>
|
||||||
|
}
|
||||||
|
if (typeof graphProvider.warm === 'function') {
|
||||||
|
await graphProvider.warm()
|
||||||
|
graphOutcome = 'warmed'
|
||||||
|
} else if (typeof graphProvider.init === 'function') {
|
||||||
|
// Existing eager cold-load seam (readiness contract): idempotent, and
|
||||||
|
// for the JS adjacency index it's already a genuine full load of the
|
||||||
|
// LSM manifests + SSTables — a real hydration, not a stat call.
|
||||||
|
await graphProvider.init()
|
||||||
|
graphOutcome = 'warmed'
|
||||||
|
} else {
|
||||||
|
graphOutcome = 'unavailable'
|
||||||
|
}
|
||||||
|
const graphDurationMs = Date.now() - graphStart
|
||||||
|
|
||||||
|
return {
|
||||||
|
vector: { outcome: vectorOutcome, durationMs: vectorDurationMs },
|
||||||
|
metadata: { outcome: metadataOutcome, durationMs: metadataDurationMs },
|
||||||
|
graph: { outcome: graphOutcome, durationMs: graphDurationMs },
|
||||||
|
totalDurationMs: Date.now() - totalStart
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Explicitly warm up the embedding engine
|
* Explicitly warm up the embedding engine
|
||||||
*
|
*
|
||||||
|
|
@ -14798,6 +14967,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// Migration LOCK wait budget — left undefined when omitted so
|
// Migration LOCK wait budget — left undefined when omitted so
|
||||||
// awaitMigrationLock() applies its 30 s default at the read site.
|
// awaitMigrationLock() applies its 30 s default at the read site.
|
||||||
migrationWaitTimeoutMs: config?.migrationWaitTimeoutMs ?? undefined,
|
migrationWaitTimeoutMs: config?.migrationWaitTimeoutMs ?? undefined,
|
||||||
|
// Apply-phase transaction budget floor — left undefined when omitted so
|
||||||
|
// transactTimeoutBudget() applies its 30 s default at the read site.
|
||||||
|
transactionBudgetFloorMs: config?.transactionBudgetFloorMs ?? undefined,
|
||||||
// Pre-upgrade backup — default-on; opt out with `migrationBackup: false`.
|
// Pre-upgrade backup — default-on; opt out with `migrationBackup: false`.
|
||||||
migrationBackup: config?.migrationBackup ?? true,
|
migrationBackup: config?.migrationBackup ?? true,
|
||||||
// Vector index configuration (8.0) — algorithm-neutral surface with
|
// Vector index configuration (8.0) — algorithm-neutral surface with
|
||||||
|
|
@ -14813,6 +14985,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
// is the active one on a non-reader writer outside unit tests). Explicit
|
// is the active one on a non-reader writer outside unit tests). Explicit
|
||||||
// true/false always wins. See the eagerEmbeddings JSDoc in brainy.types.ts.
|
// true/false always wins. See the eagerEmbeddings JSDoc in brainy.types.ts.
|
||||||
eagerEmbeddings: config?.eagerEmbeddings ?? undefined,
|
eagerEmbeddings: config?.eagerEmbeddings ?? undefined,
|
||||||
|
// Eager index warm at init() — default off (lazy demand-load), the
|
||||||
|
// pre-8.9 behavior. See the warmOnOpen JSDoc in brainy.types.ts.
|
||||||
|
warmOnOpen: config?.warmOnOpen ?? false,
|
||||||
// Plugin configuration - undefined = auto-detect
|
// Plugin configuration - undefined = auto-detect
|
||||||
plugins: config?.plugins ?? undefined,
|
plugins: config?.plugins ?? undefined,
|
||||||
// Integration Hub - undefined/false = disabled
|
// Integration Hub - undefined/false = disabled
|
||||||
|
|
|
||||||
|
|
@ -54,6 +54,9 @@ export class HnswFlushError extends Error {
|
||||||
* acceleration provider).
|
* acceleration provider).
|
||||||
*/
|
*/
|
||||||
export class JsHnswVectorIndex implements VectorIndexProvider {
|
export class JsHnswVectorIndex implements VectorIndexProvider {
|
||||||
|
/** Self-identifies as the built-in JS fallback engine — see {@link VectorIndexProvider.name}. */
|
||||||
|
readonly name = 'hnsw-js'
|
||||||
|
|
||||||
private nouns: Map<string, HNSWNoun> = new Map()
|
private nouns: Map<string, HNSWNoun> = new Map()
|
||||||
/**
|
/**
|
||||||
* Reverse adjacency: `target id → (level → set of node ids that link TO target)`.
|
* Reverse adjacency: `target id → (level → set of node ids that link TO target)`.
|
||||||
|
|
|
||||||
|
|
@ -28,6 +28,9 @@ export type { FileVersion } from './vfs/types.js'
|
||||||
|
|
||||||
// Export diagnostics result type
|
// Export diagnostics result type
|
||||||
export type { DiagnosticsResult } from './brainy.js'
|
export type { DiagnosticsResult } from './brainy.js'
|
||||||
|
// brain.warm() — eager index/storage readiness report (per-surface honest
|
||||||
|
// outcome + timing). See the WarmReport JSDoc in brainy.ts.
|
||||||
|
export type { WarmReport, WarmOutcome } from './brainy.js'
|
||||||
export type {
|
export type {
|
||||||
GraphAuditReport,
|
GraphAuditReport,
|
||||||
GraphAuditDiscrepancy
|
GraphAuditDiscrepancy
|
||||||
|
|
|
||||||
|
|
@ -381,6 +381,20 @@ export interface GraphIndexProvider {
|
||||||
*/
|
*/
|
||||||
init?(): Promise<void>
|
init?(): Promise<void>
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap
|
||||||
|
* pretouch) so first operations run at steady-state cost. Optional;
|
||||||
|
* absence means the provider demand-loads. Distinct from `init?()`: `init`
|
||||||
|
* is called once automatically during brain startup as part of the
|
||||||
|
* readiness contract (cold-load + rebuild gating); `warm` is an explicit,
|
||||||
|
* separate readiness step a caller opts into via `brain.warm()` (or
|
||||||
|
* `warmOnOpen`) specifically to pre-pay demand-load cost that `init` left
|
||||||
|
* lazy. A provider that already loads everything eagerly in `init?()` may
|
||||||
|
* implement `warm` as a no-op or omit it — `brain.warm()` falls back to a
|
||||||
|
* best-effort read-through probe when absent.
|
||||||
|
*/
|
||||||
|
warm?(): Promise<void>
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description OPTIONAL. A native provider returns true from the moment its
|
* @description OPTIONAL. A native provider returns true from the moment its
|
||||||
* `init()` detects a large epoch-drift until its background
|
* `init()` detects a large epoch-drift until its background
|
||||||
|
|
@ -947,6 +961,26 @@ export interface AtGenerationVectors {
|
||||||
* are retired and never looked up.
|
* are retired and never looked up.
|
||||||
*/
|
*/
|
||||||
export interface VectorIndexProvider {
|
export interface VectorIndexProvider {
|
||||||
|
/**
|
||||||
|
* @description REQUIRED self-reported implementation identity, rendered as
|
||||||
|
* `[vector-index:<name>]` in prose log lines and stamped into the
|
||||||
|
* transaction op-name strings that surface in journals/timings (e.g.
|
||||||
|
* `AddToVectorIndex(hnsw-js)`) — see
|
||||||
|
* `src/transaction/operations/IndexOperations.ts`. A provider must name
|
||||||
|
* itself TRUTHFULLY (its own algorithm/engine, e.g. its plugin/package
|
||||||
|
* name) and must never inherit a default — guessing a backend name would
|
||||||
|
* resurrect the exact bug this field exists to prevent: an operator
|
||||||
|
* hunting an index that isn't the one actually running. The built-in JS
|
||||||
|
* index always sets this to `'hnsw-js'`; a native provider picks its own
|
||||||
|
* string. At the TypeScript level this field is required; a runtime
|
||||||
|
* instance from an older provider compiled against the previous optional
|
||||||
|
* `providerId` contract is tolerated (never crashes, never silently
|
||||||
|
* mislabeled) — see `resolveVectorProviderId` in
|
||||||
|
* `src/transaction/operations/IndexOperations.ts`, which stamps
|
||||||
|
* `'unknown-provider'` and emits one loud warning for that case.
|
||||||
|
*/
|
||||||
|
readonly name: string
|
||||||
|
|
||||||
addItem(item: VectorDocument): Promise<string>
|
addItem(item: VectorDocument): Promise<string>
|
||||||
removeItem(id: string): Promise<boolean>
|
removeItem(id: string): Promise<boolean>
|
||||||
search(
|
search(
|
||||||
|
|
@ -1009,6 +1043,20 @@ export interface VectorIndexProvider {
|
||||||
*/
|
*/
|
||||||
init?(): Promise<void>
|
init?(): Promise<void>
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description OPTIONAL. Eagerly load/fault-in backing storage (e.g. mmap
|
||||||
|
* pretouch) so first operations run at steady-state cost. Optional;
|
||||||
|
* absence means the provider demand-loads. Distinct from `init?()`: `init`
|
||||||
|
* runs automatically once during brain startup (the cold-load + rebuild
|
||||||
|
* readiness contract above); `warm` is a separate, explicit step a caller
|
||||||
|
* opts into via `brain.warm()` (or `warmOnOpen`) to pre-pay demand-load
|
||||||
|
* cost `init` left lazy — e.g. touching every mmap page rather than just
|
||||||
|
* opening the file. A provider that already loads everything eagerly in
|
||||||
|
* `init?()` may implement `warm` as a no-op or omit it — `brain.warm()`
|
||||||
|
* falls back to a best-effort probe `search()` when absent.
|
||||||
|
*/
|
||||||
|
warm?(): Promise<void>
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* @description OPTIONAL honest durability signal (readiness contract,
|
* @description OPTIONAL honest durability signal (readiness contract,
|
||||||
* mirrors {@link GraphIndexProvider.isReady}). `true` ⇔ the persisted
|
* mirrors {@link GraphIndexProvider.isReady}). `true` ⇔ the persisted
|
||||||
|
|
|
||||||
|
|
@ -37,22 +37,47 @@ const DEFAULT_OPTIONS: Required<TransactionOptions> = {
|
||||||
maxRollbackRetries: 3
|
maxRollbackRetries: 3
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The floor term of {@link transactTimeoutBudget}'s scaling formula when the
|
||||||
|
* caller supplies neither an explicit override nor `config.transactionBudgetFloorMs`.
|
||||||
|
*/
|
||||||
|
const DEFAULT_BUDGET_FLOOR_MS = 30_000
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* The apply-phase budget for a batch of `opCount` operations.
|
* The apply-phase budget for a batch of `opCount` operations.
|
||||||
*
|
*
|
||||||
* An explicit override wins untouched. Otherwise the budget SCALES with the
|
* An explicit `override` wins untouched (a caller-specified deadline for this
|
||||||
* batch: `max(30 000 ms, opCount × 2 000 ms)`. The per-op term is calibrated
|
* one batch). Otherwise the budget SCALES with the batch:
|
||||||
* from field data — bulk imports on network-attached disks measure ~2 s per
|
* `max(floorMs, opCount × 2 000)`, where `floorMs` defaults to `30 000` and
|
||||||
* operation (each op pays canonical writes + fsync + index maintenance) — so
|
* is overridable via `BrainyConfig.transactionBudgetFloorMs` for the whole
|
||||||
* a flat 30 s budget silently capped honest work at ~15 operations while
|
* brain. The per-op term is calibrated from field data — bulk imports on
|
||||||
* looking generous for small batches. Scaling keeps small transacts
|
* network-attached disks measure ~2 s per operation (each op pays canonical
|
||||||
* fast-failing and gives bulk ones a budget proportional to the work they
|
* writes + fsync + index maintenance) — so a flat 30 s budget silently capped
|
||||||
* actually asked for; a trip still rolls back atomically and throws a
|
* honest work at ~15 operations while looking generous for small batches. A
|
||||||
* retryable, fully-labeled TransactionTimeoutError.
|
* 16-op batch, for example, scales to `max(30 000, 16 × 2 000) = 32 000`.
|
||||||
|
* Scaling keeps small transacts fast-failing and gives bulk ones a budget
|
||||||
|
* proportional to the work they actually asked for.
|
||||||
|
*
|
||||||
|
* This is a **mid-batch** budget only: it governs whether the transaction's
|
||||||
|
* NEXT operation may start (see {@link Transaction.execute}), never whether
|
||||||
|
* already-completed work is rolled back after the fact. A trip mid-batch
|
||||||
|
* still rolls back every operation applied so far, atomically, and throws a
|
||||||
|
* retryable, fully-labeled TransactionTimeoutError — that zero-loss guarantee
|
||||||
|
* doesn't change; only the point at which the clock stops mattering does (at
|
||||||
|
* the last operation, not one check later).
|
||||||
|
*
|
||||||
|
* @param opCount - Number of operations in the batch.
|
||||||
|
* @param override - A full override for this call; wins over everything else.
|
||||||
|
* @param floorMs - The scaling floor for this call (e.g. `config.transactionBudgetFloorMs`).
|
||||||
|
* Ignored when `override` is set. Defaults to `30 000` when omitted.
|
||||||
*/
|
*/
|
||||||
export function transactTimeoutBudget(opCount: number, override?: number): number {
|
export function transactTimeoutBudget(
|
||||||
|
opCount: number,
|
||||||
|
override?: number,
|
||||||
|
floorMs?: number
|
||||||
|
): number {
|
||||||
if (override !== undefined) return override
|
if (override !== undefined) return override
|
||||||
return Math.max(30_000, opCount * 2_000)
|
return Math.max(floorMs ?? DEFAULT_BUDGET_FLOOR_MS, opCount * 2_000)
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -98,7 +123,20 @@ export class Transaction implements TransactionContext {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Execute all operations atomically
|
* Execute all operations atomically.
|
||||||
|
*
|
||||||
|
* Budget semantics: the configured budget gates starting the next
|
||||||
|
* operation; completed work is never rolled back for elapsed time. Elapsed
|
||||||
|
* time is checked ONLY before starting operation `i` for `i ≥ 1` — never
|
||||||
|
* before operation 0 (an operation always gets to start) and never again
|
||||||
|
* after the final operation completes. Concretely: a single-op transaction
|
||||||
|
* can never time out post-hoc — it either runs (and its result stands, no
|
||||||
|
* matter how long it took) or a `TransactionTimeoutError` is thrown before
|
||||||
|
* it starts, which cannot happen since there is no operation before it. A
|
||||||
|
* multi-op transaction whose op `i` overruns the budget stops op `i+1` from
|
||||||
|
* starting, rolls back operations `0..i` (reverse order), and throws
|
||||||
|
* `TransactionTimeoutError` — that zero-loss rollback guarantee for
|
||||||
|
* mid-batch trips is unchanged; only the post-completion check is gone.
|
||||||
*/
|
*/
|
||||||
async execute(): Promise<void> {
|
async execute(): Promise<void> {
|
||||||
if (this.state !== 'pending') {
|
if (this.state !== 'pending') {
|
||||||
|
|
@ -128,10 +166,14 @@ export class Transaction implements TransactionContext {
|
||||||
// already-applied writes as torn, generation-less state in canonical
|
// already-applied writes as torn, generation-less state in canonical
|
||||||
// storage.
|
// storage.
|
||||||
for (let i = 0; i < this.operations.length; i++) {
|
for (let i = 0; i < this.operations.length; i++) {
|
||||||
// Budget check BEFORE starting the next operation. A trip here throws
|
// Budget gates STARTING the next operation; completed work is never
|
||||||
// into the catch below and rolls back like any other failure — it must
|
// rolled back for elapsed time. Skipped for i === 0 (operation 0
|
||||||
// never bypass rollback.
|
// always gets to start — nothing has run yet to have overrun) and
|
||||||
if (Date.now() - this.startTime > this.options.timeout) {
|
// never re-checked after the final operation completes (there is no
|
||||||
|
// "next operation" left to gate). A trip here throws into the catch
|
||||||
|
// below and rolls back like any other failure — it must never bypass
|
||||||
|
// rollback.
|
||||||
|
if (i > 0 && Date.now() - this.startTime > this.options.timeout) {
|
||||||
throw new TransactionTimeoutError(this.options.timeout, i, {
|
throw new TransactionTimeoutError(this.options.timeout, i, {
|
||||||
elapsedMs: Date.now() - this.startTime,
|
elapsedMs: Date.now() - this.startTime,
|
||||||
totalOperations: this.operations.length,
|
totalOperations: this.operations.length,
|
||||||
|
|
|
||||||
|
|
@ -2,34 +2,73 @@
|
||||||
* Index Operations with Rollback Support
|
* Index Operations with Rollback Support
|
||||||
*
|
*
|
||||||
* Provides transactional operations for all indexes:
|
* Provides transactional operations for all indexes:
|
||||||
* - JsHnswVectorIndex (unified vector index)
|
* - VectorIndexProvider (the JS HNSW fallback, or a native acceleration provider)
|
||||||
* - MetadataIndexManager (roaring bitmap filtering)
|
* - MetadataIndexManager (roaring bitmap filtering)
|
||||||
* - GraphAdjacencyIndex (LSM-tree graph storage)
|
* - GraphAdjacencyIndex (LSM-tree graph storage)
|
||||||
*
|
*
|
||||||
* Each operation can be executed and rolled back atomically.
|
* Each operation can be executed and rolled back atomically.
|
||||||
*/
|
*/
|
||||||
|
|
||||||
import type { JsHnswVectorIndex } from '../../hnsw/hnswIndex.js'
|
import type { VectorIndexProvider, GraphIndexProvider } from '../../plugin.js'
|
||||||
import type { MetadataIndexManager } from '../../utils/metadataIndex.js'
|
import type { MetadataIndexManager } from '../../utils/metadataIndex.js'
|
||||||
import type { GraphIndexProvider } from '../../plugin.js'
|
|
||||||
import type { GraphVerb } from '../../coreTypes.js'
|
import type { GraphVerb } from '../../coreTypes.js'
|
||||||
import type { Operation, RollbackAction } from '../types.js'
|
import type { Operation, RollbackAction } from '../types.js'
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Add to HNSW index with rollback support
|
* Backend identity stamped into an operation's emitted `name` string (e.g.
|
||||||
|
* `AddToVectorIndex(hnsw-js)`), resolved from the provider's own
|
||||||
|
* {@link VectorIndexProvider.name} self-report.
|
||||||
|
*
|
||||||
|
* These operation classes are backend-neutral (the vector index they wrap may
|
||||||
|
* be Brainy's own JS HNSW fallback OR a native acceleration provider), but
|
||||||
|
* their names surface directly in consumer-visible transaction journals and
|
||||||
|
* timings. `name` is REQUIRED at the TypeScript level — but a native provider
|
||||||
|
* instance compiled against the previous (pre-required) contract can still
|
||||||
|
* reach this function at runtime without it. That legacy case is tolerated,
|
||||||
|
* never crashed on and never silently mislabeled: it resolves to
|
||||||
|
* `'unknown-provider'` and emits ONE loud warning naming the missing contract
|
||||||
|
* field, so the fix (implement `name`) is discoverable rather than a silent
|
||||||
|
* fossil label in every journal line thereafter.
|
||||||
|
*/
|
||||||
|
const warnedMissingName = new WeakSet<VectorIndexProvider>()
|
||||||
|
|
||||||
|
function resolveVectorProviderId(index: VectorIndexProvider): string {
|
||||||
|
const name = (index as { name?: unknown }).name
|
||||||
|
if (typeof name === 'string') return name
|
||||||
|
if (!warnedMissingName.has(index)) {
|
||||||
|
warnedMissingName.add(index)
|
||||||
|
console.warn(
|
||||||
|
'[vector-index] provider is missing the required `name` field (VectorIndexProvider.name, ' +
|
||||||
|
'required since 8.10.0) — stamping "unknown-provider" in transaction op names until the ' +
|
||||||
|
'provider declares its own identity.'
|
||||||
|
)
|
||||||
|
}
|
||||||
|
return 'unknown-provider'
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Add to the vector index with rollback support.
|
||||||
|
*
|
||||||
|
* Backend-neutral: `index` is whatever the `'vector'` provider factory
|
||||||
|
* returns — Brainy's own JS HNSW fallback, or a native acceleration provider
|
||||||
|
* (e.g. DiskANN). The emitted `name` stamps the active backend (see
|
||||||
|
* {@link resolveVectorProviderId}) so operators reading a transaction journal
|
||||||
|
* or timing trace see which engine actually ran, never a fossil name from
|
||||||
|
* whichever engine happened to be active when this op class was written.
|
||||||
*
|
*
|
||||||
* Rollback strategy:
|
* Rollback strategy:
|
||||||
* - Remove item from index
|
* - Remove item from index
|
||||||
|
|
||||||
*/
|
*/
|
||||||
export class AddToHNSWOperation implements Operation {
|
export class AddToVectorIndexOperation implements Operation {
|
||||||
readonly name = 'AddToHNSW'
|
readonly name: string
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
private readonly index: JsHnswVectorIndex,
|
private readonly index: VectorIndexProvider,
|
||||||
private readonly id: string,
|
private readonly id: string,
|
||||||
private readonly vector: number[]
|
private readonly vector: number[]
|
||||||
) {}
|
) {
|
||||||
|
this.name = `AddToVectorIndex(${resolveVectorProviderId(index)})`
|
||||||
|
}
|
||||||
|
|
||||||
async execute(): Promise<RollbackAction> {
|
async execute(): Promise<RollbackAction> {
|
||||||
// Check if item already exists (for rollback decision)
|
// Check if item already exists (for rollback decision)
|
||||||
|
|
@ -59,11 +98,12 @@ export class AddToHNSWOperation implements Operation {
|
||||||
* pre-existence as "existed" made every rollback skip removeItem, leaving
|
* pre-existence as "existed" made every rollback skip removeItem, leaving
|
||||||
* phantom entries in the index after a failed transaction. The safe default
|
* phantom entries in the index after a failed transaction. The safe default
|
||||||
* is to remove what this operation added — update flows pair this op with a
|
* is to remove what this operation added — update flows pair this op with a
|
||||||
* RemoveFromHNSWOperation whose own rollback restores the prior vector, so
|
* RemoveFromVectorIndexOperation whose own rollback restores the prior
|
||||||
* reverse-order rollback reconstructs the original state either way.
|
* vector, so reverse-order rollback reconstructs the original state either
|
||||||
|
* way.
|
||||||
*/
|
*/
|
||||||
private async itemExists(id: string): Promise<boolean> {
|
private async itemExists(id: string): Promise<boolean> {
|
||||||
const index = this.index as JsHnswVectorIndex & {
|
const index = this.index as VectorIndexProvider & {
|
||||||
getItem?: (id: string) => Promise<unknown>
|
getItem?: (id: string) => Promise<unknown>
|
||||||
}
|
}
|
||||||
if (typeof index.getItem !== 'function') return false
|
if (typeof index.getItem !== 'function') return false
|
||||||
|
|
@ -77,21 +117,27 @@ export class AddToHNSWOperation implements Operation {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Remove from HNSW index with rollback support
|
* Remove from the vector index with rollback support.
|
||||||
|
*
|
||||||
|
* Backend-neutral: see {@link AddToVectorIndexOperation} — `index` may be the
|
||||||
|
* JS HNSW fallback or a native acceleration provider; the emitted `name`
|
||||||
|
* stamps the active backend.
|
||||||
*
|
*
|
||||||
* Rollback strategy:
|
* Rollback strategy:
|
||||||
* - Re-add item to index with original vector
|
* - Re-add item to index with original vector
|
||||||
*
|
*
|
||||||
* Note: Requires storing the vector for rollback
|
* Note: Requires storing the vector for rollback
|
||||||
*/
|
*/
|
||||||
export class RemoveFromHNSWOperation implements Operation {
|
export class RemoveFromVectorIndexOperation implements Operation {
|
||||||
readonly name = 'RemoveFromHNSW'
|
readonly name: string
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
private readonly index: JsHnswVectorIndex,
|
private readonly index: VectorIndexProvider,
|
||||||
private readonly id: string,
|
private readonly id: string,
|
||||||
private readonly vector: number[] // Required for rollback
|
private readonly vector: number[] // Required for rollback
|
||||||
) {}
|
) {
|
||||||
|
this.name = `RemoveFromVectorIndex(${resolveVectorProviderId(index)})`
|
||||||
|
}
|
||||||
|
|
||||||
async execute(): Promise<RollbackAction> {
|
async execute(): Promise<RollbackAction> {
|
||||||
// Remove from index
|
// Remove from index
|
||||||
|
|
@ -285,22 +331,24 @@ export class RemoveFromGraphIndexOperation implements Operation {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Batch operation: Add multiple items to HNSW index
|
* Batch operation: Add multiple items to the vector index (backend-neutral —
|
||||||
|
* see {@link AddToVectorIndexOperation}).
|
||||||
*
|
*
|
||||||
* Useful for bulk imports with transaction support.
|
* Useful for bulk imports with transaction support.
|
||||||
* Rolls back all items if any fail.
|
* Rolls back all items if any fail.
|
||||||
*/
|
*/
|
||||||
export class BatchAddToHNSWOperation implements Operation {
|
export class BatchAddToVectorIndexOperation implements Operation {
|
||||||
readonly name = 'BatchAddToHNSW'
|
readonly name: string
|
||||||
|
|
||||||
private operations: AddToHNSWOperation[]
|
private operations: AddToVectorIndexOperation[]
|
||||||
|
|
||||||
constructor(
|
constructor(
|
||||||
index: JsHnswVectorIndex,
|
index: VectorIndexProvider,
|
||||||
items: Array<{ id: string; vector: number[] }>
|
items: Array<{ id: string; vector: number[] }>
|
||||||
) {
|
) {
|
||||||
|
this.name = `BatchAddToVectorIndex(${resolveVectorProviderId(index)})`
|
||||||
this.operations = items.map(
|
this.operations = items.map(
|
||||||
item => new AddToHNSWOperation(index, item.id, item.vector)
|
item => new AddToVectorIndexOperation(index, item.id, item.vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -21,12 +21,12 @@ export {
|
||||||
|
|
||||||
// Index Operations
|
// Index Operations
|
||||||
export {
|
export {
|
||||||
AddToHNSWOperation,
|
AddToVectorIndexOperation,
|
||||||
RemoveFromHNSWOperation,
|
RemoveFromVectorIndexOperation,
|
||||||
AddToMetadataIndexOperation,
|
AddToMetadataIndexOperation,
|
||||||
RemoveFromMetadataIndexOperation,
|
RemoveFromMetadataIndexOperation,
|
||||||
AddToGraphIndexOperation,
|
AddToGraphIndexOperation,
|
||||||
RemoveFromGraphIndexOperation,
|
RemoveFromGraphIndexOperation,
|
||||||
BatchAddToHNSWOperation,
|
BatchAddToVectorIndexOperation,
|
||||||
BatchAddToMetadataIndexOperation
|
BatchAddToMetadataIndexOperation
|
||||||
} from './IndexOperations.js'
|
} from './IndexOperations.js'
|
||||||
|
|
|
||||||
|
|
@ -1651,6 +1651,25 @@ export interface BrainyConfig {
|
||||||
*/
|
*/
|
||||||
disableAutoRebuild?: boolean
|
disableAutoRebuild?: boolean
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The floor (ms) of the apply-phase transaction budget's scaling formula:
|
||||||
|
* `max(transactionBudgetFloorMs, opCount × 2 000)`. The budget governs
|
||||||
|
* whether a `transact()` / single-op write's NEXT internal operation may
|
||||||
|
* **start** — never whether already-completed work gets rolled back after
|
||||||
|
* the fact (a single-op write can never time out post-hoc: it either runs
|
||||||
|
* or it commits). A trip mid-batch still rolls back every applied operation
|
||||||
|
* atomically and throws a retryable `TransactionTimeoutError`; only the
|
||||||
|
* floor of the formula is configurable here.
|
||||||
|
*
|
||||||
|
* Raise this when a cold store's first writes after a restart legitimately
|
||||||
|
* take longer than 30s per operation (e.g. page-cache-cold canonical writes
|
||||||
|
* on a large brain) so a bulk `transact()` batch gets a proportionally
|
||||||
|
* larger runway instead of tripping mid-batch. Lower it to fail faster on a
|
||||||
|
* latency-sensitive write path. Default: `30000` (30 s) — unchanged from
|
||||||
|
* pre-8.9 behavior.
|
||||||
|
*/
|
||||||
|
transactionBudgetFloorMs?: number
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* How long (ms) an operation **waits** on the coordinated 7.x → 8.0 migration
|
* How long (ms) an operation **waits** on the coordinated 7.x → 8.0 migration
|
||||||
* before throwing a retryable `MigrationInProgressError`.
|
* before throwing a retryable `MigrationInProgressError`.
|
||||||
|
|
@ -1806,6 +1825,23 @@ export interface BrainyConfig {
|
||||||
*/
|
*/
|
||||||
eagerEmbeddings?: boolean
|
eagerEmbeddings?: boolean
|
||||||
|
|
||||||
|
/**
|
||||||
|
* When `true`, `init()` **awaits `warm()`** (see {@link Brainy.warm}) before
|
||||||
|
* it resolves — blocking startup until the vector index, metadata index,
|
||||||
|
* and graph adjacency have all faulted their backing storage in. This is a
|
||||||
|
* deliberate operator opt-in trade: startup takes longer, but the FIRST
|
||||||
|
* request after a cold restart runs at steady-state cost instead of paying
|
||||||
|
* page-cache-miss / demand-load latency on the critical path.
|
||||||
|
*
|
||||||
|
* Runs AFTER `init()`'s own sequence completes (index construction, crash
|
||||||
|
* recovery, migrations, VFS bootstrap) — never in place of any of it — so
|
||||||
|
* `warm()` always operates on a fully-initialized brain.
|
||||||
|
*
|
||||||
|
* Default: `false` (lazy — the pre-8.9 behavior: indexes demand-load on
|
||||||
|
* first access).
|
||||||
|
*/
|
||||||
|
warmOnOpen?: boolean
|
||||||
|
|
||||||
// Plugin configuration
|
// Plugin configuration
|
||||||
// Controls which plugins are loaded during init().
|
// Controls which plugins are loaded during init().
|
||||||
// - undefined (default): guarded auto-detection of the first-party
|
// - undefined (default): guarded auto-detection of the first-party
|
||||||
|
|
|
||||||
|
|
@ -423,6 +423,46 @@ export class MetadataIndexManager implements MetadataIndexProvider {
|
||||||
prodLog.debug('✅ Type-aware cache warming completed')
|
prodLog.debug('✅ Type-aware cache warming completed')
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Full hydration — the {@link Brainy.warm} readiness seam for the metadata
|
||||||
|
* index. Unlike {@link warmCache} / {@link warmCacheForTopTypes} (which
|
||||||
|
* warm only a heuristic subset: common fields plus the top-N types' top
|
||||||
|
* fields), this loads EVERY field's sparse index the field registry knows
|
||||||
|
* about — a real read through {@link loadSparseIndex} into the unified
|
||||||
|
* cache for each field, not a stat/existence check. Idempotent: an
|
||||||
|
* already-cached field's `loadSparseIndex` call is a cheap cache hit.
|
||||||
|
*
|
||||||
|
* Re-reads the field registry first when `fieldIndexes` is empty (a warm()
|
||||||
|
* call issued before `init()` populated it would otherwise hydrate
|
||||||
|
* nothing), then loads every discovered field in parallel.
|
||||||
|
*/
|
||||||
|
async hydrateAll(): Promise<void> {
|
||||||
|
if (this.fieldIndexes.size === 0) {
|
||||||
|
await this.loadFieldRegistry()
|
||||||
|
}
|
||||||
|
|
||||||
|
const fields = Array.from(this.fieldIndexes.keys())
|
||||||
|
if (fields.length === 0) {
|
||||||
|
prodLog.debug('[MetadataIndex] hydrateAll: no persisted fields to hydrate')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
prodLog.debug(`[MetadataIndex] hydrateAll: loading ${fields.length} field(s) — ${fields.join(', ')}`)
|
||||||
|
|
||||||
|
await Promise.all(
|
||||||
|
fields.map(async field => {
|
||||||
|
try {
|
||||||
|
await this.loadSparseIndex(field)
|
||||||
|
} catch (error) {
|
||||||
|
// A single field's load failure doesn't abort the rest of the
|
||||||
|
// hydration — warm() is a best-effort readiness step, never a
|
||||||
|
// correctness gate (queries still demand-load on miss).
|
||||||
|
prodLog.debug(`[MetadataIndex] hydrateAll: field '${field}' failed to load:`, error)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Acquire an in-memory lock for coordinating concurrent metadata index writes
|
* Acquire an in-memory lock for coordinating concurrent metadata index writes
|
||||||
* Uses in-memory locks since MetadataIndexManager doesn't have direct file system access
|
* Uses in-memory locks since MetadataIndexManager doesn't have direct file system access
|
||||||
|
|
|
||||||
310
tests/unit/brainy/warm.test.ts
Normal file
310
tests/unit/brainy/warm.test.ts
Normal file
|
|
@ -0,0 +1,310 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/brainy/warm
|
||||||
|
* @description Coverage for `brain.warm()` / `warmOnOpen` / the provider
|
||||||
|
* `warm?()` contract (the cold-restart readiness fix): first operations after
|
||||||
|
* a cold restart should run at steady-state cost instead of paying
|
||||||
|
* demand-load latency on the critical path.
|
||||||
|
*
|
||||||
|
* Uses a real, in-process fake plugin provider implementing the actual
|
||||||
|
* `VectorIndexProvider` contract from `src/plugin.ts` — the real seam a
|
||||||
|
* native provider (e.g. a disk-native accelerator) plugs into. The real
|
||||||
|
* built-in JS metadata index and graph adjacency index run against real
|
||||||
|
* (filesystem or in-memory) storage with pre-existing data, so their
|
||||||
|
* hydration paths are exercised for real, not mocked.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect, afterEach } from 'vitest'
|
||||||
|
import * as fs from 'node:fs'
|
||||||
|
import * as os from 'node:os'
|
||||||
|
import * as path from 'node:path'
|
||||||
|
import { Brainy } from '../../../src/brainy.js'
|
||||||
|
import { NounType, VerbType } from '../../../src/types/graphTypes.js'
|
||||||
|
import type { VectorIndexProvider } from '../../../src/plugin.js'
|
||||||
|
import type { VectorDocument, Vector } from '../../../src/coreTypes.js'
|
||||||
|
import { MetadataIndexManager } from '../../../src/utils/metadataIndex.js'
|
||||||
|
import { GraphAdjacencyIndex } from '../../../src/graph/graphAdjacencyIndex.js'
|
||||||
|
|
||||||
|
const tmpDirs: string[] = []
|
||||||
|
function mkTmp(): string {
|
||||||
|
const d = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-warm-'))
|
||||||
|
tmpDirs.push(d)
|
||||||
|
return d
|
||||||
|
}
|
||||||
|
afterEach(() => {
|
||||||
|
for (const d of tmpDirs.splice(0)) fs.rmSync(d, { recursive: true, force: true })
|
||||||
|
})
|
||||||
|
|
||||||
|
// Brainy's ValidationConfig fixes vectors at exactly 384 dimensions
|
||||||
|
// (src/utils/paramValidation.ts) — match it so `add()` doesn't reject test data.
|
||||||
|
const DIM = 384
|
||||||
|
const V = (seed = 1): number[] => Array.from({ length: DIM }, (_, i) => Math.sin(seed + i))
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A real (not mocked) VectorIndexProvider implementation, backed by a plain
|
||||||
|
* Map, that optionally implements `warm()` — the exact seam
|
||||||
|
* `AddToVectorIndexOperation` / `brain.warm()` call through.
|
||||||
|
*/
|
||||||
|
class FakeVectorProvider implements VectorIndexProvider {
|
||||||
|
readonly name = 'fake-vector-provider'
|
||||||
|
readonly items = new Map<string, Vector>()
|
||||||
|
warmCalls = 0
|
||||||
|
searchCalls: Array<{ k?: number }> = []
|
||||||
|
// Only present on the instance when the constructor is told to — mirrors a
|
||||||
|
// real provider that may or may not implement the optional hook. Assigned
|
||||||
|
// in the constructor BODY (not as a field initializer): under native ES
|
||||||
|
// class fields, field initializers run before constructor-body statements
|
||||||
|
// — including the parameter-property assignment — so referencing a
|
||||||
|
// parameter property from a field initializer would see it as still
|
||||||
|
// `undefined`.
|
||||||
|
warm?: () => Promise<void>
|
||||||
|
|
||||||
|
constructor(hasWarm: boolean) {
|
||||||
|
if (hasWarm) {
|
||||||
|
this.warm = async (): Promise<void> => {
|
||||||
|
this.warmCalls++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async addItem(item: VectorDocument): Promise<string> {
|
||||||
|
this.items.set(item.id, item.vector)
|
||||||
|
return item.id
|
||||||
|
}
|
||||||
|
async removeItem(id: string): Promise<boolean> {
|
||||||
|
return this.items.delete(id)
|
||||||
|
}
|
||||||
|
async search(_queryVector: Vector, k?: number): Promise<Array<[string, number]>> {
|
||||||
|
this.searchCalls.push({ k })
|
||||||
|
return [...this.items.keys()].slice(0, k ?? 10).map((id) => [id, 0])
|
||||||
|
}
|
||||||
|
size(): number {
|
||||||
|
return this.items.size
|
||||||
|
}
|
||||||
|
clear(): void {
|
||||||
|
this.items.clear()
|
||||||
|
}
|
||||||
|
async rebuild(): Promise<void> {}
|
||||||
|
async flush(): Promise<number> {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
getPersistMode(): 'immediate' | 'deferred' {
|
||||||
|
return 'deferred'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Registers `provider` under the `'vector'` plugin key, before `init()`. */
|
||||||
|
function useFakeVectorProvider(brain: Brainy<any>, provider: FakeVectorProvider): void {
|
||||||
|
brain.use({
|
||||||
|
name: 'fake-vector-provider',
|
||||||
|
activate: async (ctx: any) => {
|
||||||
|
ctx.registerProvider('vector', () => provider)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('brain.warm()', () => {
|
||||||
|
it('(a) calls the vector provider\'s warm() when present and reports "warmed"', async () => {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
const provider = new FakeVectorProvider(true)
|
||||||
|
useFakeVectorProvider(brain, provider)
|
||||||
|
await brain.init()
|
||||||
|
await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
|
||||||
|
|
||||||
|
const report = await brain.warm()
|
||||||
|
|
||||||
|
expect(provider.warmCalls).toBe(1)
|
||||||
|
expect(provider.searchCalls.length).toBe(0) // warm() ran — no probe fallback
|
||||||
|
expect(report.vector.outcome).toBe('warmed')
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(b) falls back to a probe search() when warm() is absent and reports "probed", never "warmed"', async () => {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
const provider = new FakeVectorProvider(false)
|
||||||
|
useFakeVectorProvider(brain, provider)
|
||||||
|
await brain.init()
|
||||||
|
await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
|
||||||
|
await brain.add({ data: 'b', type: NounType.Thing, vector: V(2) })
|
||||||
|
|
||||||
|
const report = await brain.warm()
|
||||||
|
|
||||||
|
expect(provider.warmCalls).toBe(0) // no warm() on this provider
|
||||||
|
expect(provider.searchCalls.length).toBe(1) // the probe ran
|
||||||
|
expect(provider.searchCalls[0].k).toBe(Math.min(100, provider.size())) // k = min(100, size)
|
||||||
|
expect(report.vector.outcome).toBe('probed')
|
||||||
|
expect(report.vector.outcome).not.toBe('warmed') // never conflated
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('vector: reports "unavailable" when there is nothing to probe (empty index, unknown dimension)', async () => {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
// A disk-native provider MAY honestly report 0 resident entries while
|
||||||
|
// durable data exists on disk (the same posture documented on
|
||||||
|
// `VectorIndexProvider.isReady` — "an mmap/disk-native index may
|
||||||
|
// legitimately report 0 resident entries"). warm()'s probe fallback
|
||||||
|
// reads `size()`, so this is the real trigger for "nothing to probe":
|
||||||
|
// never a k=0 search, an honest skip.
|
||||||
|
const provider = new FakeVectorProvider(false)
|
||||||
|
provider.size = () => 0
|
||||||
|
useFakeVectorProvider(brain, provider)
|
||||||
|
await brain.init() // the VFS root bootstrap write sets `dimensions`, but size() still reports 0
|
||||||
|
|
||||||
|
const report = await brain.warm()
|
||||||
|
|
||||||
|
expect(provider.searchCalls.length).toBe(0) // nothing probed
|
||||||
|
expect(report.vector.outcome).toBe('unavailable')
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(c) metadata + graph hydration paths actually execute against filesystem storage with pre-existing data', async () => {
|
||||||
|
const dir = mkTmp()
|
||||||
|
|
||||||
|
// Build a brain with real data (including a field NOT in the metadata
|
||||||
|
// index's own common-fields warm subset — 'wave' — so hydrateAll()'s
|
||||||
|
// FULL hydration is distinguishable from init()'s partial warmCache()),
|
||||||
|
// and real graph edges, then close it (persisting everything).
|
||||||
|
const seed = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'filesystem', path: dir },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
await seed.init()
|
||||||
|
const ids: string[] = []
|
||||||
|
for (let i = 0; i < 6; i++) {
|
||||||
|
ids.push(
|
||||||
|
await seed.add({
|
||||||
|
data: `entity ${i}`,
|
||||||
|
type: NounType.Thing,
|
||||||
|
metadata: { wave: i % 3 },
|
||||||
|
vector: V(i + 1)
|
||||||
|
})
|
||||||
|
)
|
||||||
|
}
|
||||||
|
for (let i = 0; i + 1 < ids.length; i++) {
|
||||||
|
await seed.relate({ from: ids[i], to: ids[i + 1], type: VerbType.RelatedTo })
|
||||||
|
}
|
||||||
|
await seed.close()
|
||||||
|
|
||||||
|
// Cold-reopen a FRESH instance and spy on the real hydration seams before
|
||||||
|
// calling warm(), so we assert they actually ran (not just that the
|
||||||
|
// report claims they did).
|
||||||
|
const metaHydrateCalls: number[] = []
|
||||||
|
const loadedFields: string[] = []
|
||||||
|
const graphInitCalls: number[] = []
|
||||||
|
|
||||||
|
const origHydrateAll = MetadataIndexManager.prototype.hydrateAll
|
||||||
|
const origLoadSparseIndex = (MetadataIndexManager.prototype as any).loadSparseIndex
|
||||||
|
const origGraphInit = GraphAdjacencyIndex.prototype.init
|
||||||
|
|
||||||
|
MetadataIndexManager.prototype.hydrateAll = async function (...args: any[]) {
|
||||||
|
metaHydrateCalls.push(1)
|
||||||
|
return origHydrateAll.apply(this, args as any)
|
||||||
|
}
|
||||||
|
;(MetadataIndexManager.prototype as any).loadSparseIndex = async function (
|
||||||
|
field: string,
|
||||||
|
...args: any[]
|
||||||
|
) {
|
||||||
|
loadedFields.push(field)
|
||||||
|
return origLoadSparseIndex.apply(this, [field, ...args] as any)
|
||||||
|
}
|
||||||
|
GraphAdjacencyIndex.prototype.init = async function (...args: any[]) {
|
||||||
|
graphInitCalls.push(1)
|
||||||
|
return origGraphInit.apply(this, args as any)
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'filesystem', path: dir },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
|
const report = await brain.warm()
|
||||||
|
|
||||||
|
expect(metaHydrateCalls.length).toBe(1) // hydrateAll() actually ran
|
||||||
|
expect(loadedFields).toContain('wave') // a NON-common field was loaded — full hydration, not the heuristic subset
|
||||||
|
expect(report.metadata.outcome).toBe('warmed')
|
||||||
|
|
||||||
|
expect(graphInitCalls.length).toBeGreaterThanOrEqual(1) // graph's full-load seam ran (once at brain init, once via warm())
|
||||||
|
expect(report.graph.outcome).toBe('warmed')
|
||||||
|
|
||||||
|
// Correctness survives: the hydrated data still answers queries.
|
||||||
|
const byWhere = await brain.find({ type: NounType.Thing, where: { wave: 1 } })
|
||||||
|
expect(byWhere.length).toBe(2) // waves 1,4 of 0..5
|
||||||
|
|
||||||
|
await brain.close()
|
||||||
|
} finally {
|
||||||
|
MetadataIndexManager.prototype.hydrateAll = origHydrateAll
|
||||||
|
;(MetadataIndexManager.prototype as any).loadSparseIndex = origLoadSparseIndex
|
||||||
|
GraphAdjacencyIndex.prototype.init = origGraphInit
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(d) warmOnOpen: true runs warm() during init() — observable via the fake provider', async () => {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' },
|
||||||
|
warmOnOpen: true,
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
const provider = new FakeVectorProvider(true)
|
||||||
|
useFakeVectorProvider(brain, provider)
|
||||||
|
|
||||||
|
// No explicit brain.warm() call — warmOnOpen must have run it as part of init().
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
|
expect(provider.warmCalls).toBe(1)
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('warmOnOpen defaults to false — init() does NOT run warm() unless opted in', async () => {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
const provider = new FakeVectorProvider(true)
|
||||||
|
useFakeVectorProvider(brain, provider)
|
||||||
|
|
||||||
|
await brain.init()
|
||||||
|
|
||||||
|
expect(provider.warmCalls).toBe(0)
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(e) WarmReport shape: outcome literal + durationMs per surface, plus totalDurationMs', async () => {
|
||||||
|
const brain = new Brainy({
|
||||||
|
requireSubtype: false,
|
||||||
|
storage: { type: 'memory' },
|
||||||
|
silent: true
|
||||||
|
})
|
||||||
|
const provider = new FakeVectorProvider(true)
|
||||||
|
useFakeVectorProvider(brain, provider)
|
||||||
|
await brain.init()
|
||||||
|
await brain.add({ data: 'a', type: NounType.Thing, vector: V(1) })
|
||||||
|
|
||||||
|
const report = await brain.warm()
|
||||||
|
|
||||||
|
for (const surface of ['vector', 'metadata', 'graph'] as const) {
|
||||||
|
expect(['warmed', 'probed', 'unavailable']).toContain(report[surface].outcome)
|
||||||
|
expect(typeof report[surface].durationMs).toBe('number')
|
||||||
|
expect(report[surface].durationMs).toBeGreaterThanOrEqual(0)
|
||||||
|
}
|
||||||
|
expect(typeof report.totalDurationMs).toBe('number')
|
||||||
|
expect(report.totalDurationMs).toBeGreaterThanOrEqual(0)
|
||||||
|
await brain.close()
|
||||||
|
})
|
||||||
|
})
|
||||||
149
tests/unit/transaction/budget-commit-completed-work.test.ts
Normal file
149
tests/unit/transaction/budget-commit-completed-work.test.ts
Normal file
|
|
@ -0,0 +1,149 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/transaction/budget-commit-completed-work
|
||||||
|
* @description Regression coverage for the "commit-completed-work" transaction
|
||||||
|
* budget semantics: the budget gates STARTING the next operation; it never
|
||||||
|
* converts already-completed work into a rollback. Concretely:
|
||||||
|
*
|
||||||
|
* - A single-op transaction can never time out post-hoc — its one operation
|
||||||
|
* either runs (and the transaction commits, however long it took) or it
|
||||||
|
* never gets to start (which cannot happen: there is no operation before it
|
||||||
|
* to have overrun the budget).
|
||||||
|
* - A multi-op transaction whose op `i` overruns the budget stops op `i+1`
|
||||||
|
* from starting: everything applied so far rolls back atomically and a
|
||||||
|
* `TransactionTimeoutError` is thrown — the mid-batch zero-loss guarantee is
|
||||||
|
* unchanged.
|
||||||
|
* - `transactTimeoutBudget`'s scaling floor (`max(floorMs, opCount * 2000)`)
|
||||||
|
* is overridable per call, the seam `BrainyConfig.transactionBudgetFloorMs`
|
||||||
|
* feeds at the brainy.ts read sites.
|
||||||
|
*
|
||||||
|
* Uses a real class implementing the `Operation` interface (not an anonymous
|
||||||
|
* object literal, not a mock of Transaction internals) so the exercised path
|
||||||
|
* is exactly what a real caller's operation looks like.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect } from 'vitest'
|
||||||
|
import { Transaction, transactTimeoutBudget } from '../../../src/transaction/Transaction.js'
|
||||||
|
import type { Operation, RollbackAction } from '../../../src/transaction/types.js'
|
||||||
|
import { TransactionTimeoutError } from '../../../src/transaction/errors.js'
|
||||||
|
|
||||||
|
const sleep = (ms: number) => new Promise<void>((resolve) => setTimeout(resolve, ms))
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A real `Operation` implementation that sleeps for `delayMs` before applying
|
||||||
|
* a write to `log`, and returns a rollback action that removes it again. Used
|
||||||
|
* to deterministically make one operation "slow" relative to a transaction's
|
||||||
|
* budget without touching any Transaction internals.
|
||||||
|
*/
|
||||||
|
class SlowOperation implements Operation {
|
||||||
|
readonly name: string
|
||||||
|
executed = false
|
||||||
|
rolledBack = false
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
private readonly log: string[],
|
||||||
|
label: string,
|
||||||
|
private readonly delayMs: number
|
||||||
|
) {
|
||||||
|
this.name = label
|
||||||
|
}
|
||||||
|
|
||||||
|
async execute(): Promise<RollbackAction | undefined> {
|
||||||
|
this.executed = true
|
||||||
|
if (this.delayMs > 0) await sleep(this.delayMs)
|
||||||
|
this.log.push(this.name)
|
||||||
|
return async () => {
|
||||||
|
this.rolledBack = true
|
||||||
|
const idx = this.log.indexOf(this.name)
|
||||||
|
if (idx >= 0) this.log.splice(idx, 1)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('Transaction budget — commit-completed-work semantics', () => {
|
||||||
|
it('(a) single-op transaction whose op overruns the budget COMMITS — no rollback, no throw', async () => {
|
||||||
|
const log: string[] = []
|
||||||
|
const op = new SlowOperation(log, 'slow-single-op', 40)
|
||||||
|
|
||||||
|
// Budget (5ms) is far smaller than the op's 40ms — under the OLD
|
||||||
|
// (post-completion-check) semantics this would have thrown and rolled
|
||||||
|
// back after the op finished. Under the new semantics there is no
|
||||||
|
// operation after it to gate, so it commits.
|
||||||
|
const tx = new Transaction({ timeout: 5 })
|
||||||
|
tx.addOperation(op)
|
||||||
|
|
||||||
|
await expect(tx.execute()).resolves.toBeUndefined()
|
||||||
|
|
||||||
|
expect(tx.getState()).toBe('committed')
|
||||||
|
expect(op.executed).toBe(true)
|
||||||
|
expect(op.rolledBack).toBe(false)
|
||||||
|
expect(log).toEqual(['slow-single-op'])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(b) two-op transaction: op0 overruns the budget → op1 never starts, op0 rolls back, throws TransactionTimeoutError', async () => {
|
||||||
|
const log: string[] = []
|
||||||
|
const op0 = new SlowOperation(log, 'op0-overruns', 40)
|
||||||
|
const op1 = new SlowOperation(log, 'op1-never-starts', 0)
|
||||||
|
|
||||||
|
const tx = new Transaction({ timeout: 5 })
|
||||||
|
tx.addOperation(op0)
|
||||||
|
tx.addOperation(op1)
|
||||||
|
|
||||||
|
const error = await tx.execute().catch((e) => e)
|
||||||
|
|
||||||
|
expect(error).toBeInstanceOf(TransactionTimeoutError)
|
||||||
|
expect(op0.executed).toBe(true)
|
||||||
|
expect(op0.rolledBack).toBe(true)
|
||||||
|
expect(op1.executed).toBe(false)
|
||||||
|
expect(op1.rolledBack).toBe(false)
|
||||||
|
expect(log).toEqual([]) // op0's write was undone; op1 never wrote
|
||||||
|
expect(tx.getState()).toBe('rolled_back')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a single-op transaction never checks the budget before its only operation starts', async () => {
|
||||||
|
// Budget of 0ms: under the old "check before every op including 0" code
|
||||||
|
// this could theoretically trip before op 0 even started (if any
|
||||||
|
// measurable time elapsed between startTime capture and the check). The
|
||||||
|
// documented contract is stronger: operation 0 ALWAYS gets to start.
|
||||||
|
const log: string[] = []
|
||||||
|
const op = new SlowOperation(log, 'only-op', 5)
|
||||||
|
|
||||||
|
const tx = new Transaction({ timeout: 0 })
|
||||||
|
tx.addOperation(op)
|
||||||
|
|
||||||
|
await expect(tx.execute()).resolves.toBeUndefined()
|
||||||
|
expect(tx.getState()).toBe('committed')
|
||||||
|
expect(op.executed).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(c) transactTimeoutBudget: an explicit floorMs overrides the 30s default floor', () => {
|
||||||
|
// Below the per-op scaling term, a raised floor wins.
|
||||||
|
expect(transactTimeoutBudget(1, undefined, 5_000)).toBe(5_000) // 1 op × 2000 = 2000 < 5000 floor
|
||||||
|
expect(transactTimeoutBudget(1)).toBe(30_000) // unchanged default when floorMs omitted
|
||||||
|
|
||||||
|
// Above the floor, opCount * 2000 still wins over a smaller floor.
|
||||||
|
expect(transactTimeoutBudget(10, undefined, 5_000)).toBe(20_000) // 10 × 2000 = 20000 > 5000 floor
|
||||||
|
|
||||||
|
// A full `override` always wins, regardless of floorMs.
|
||||||
|
expect(transactTimeoutBudget(10, 999, 5_000)).toBe(999)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('(c) a configured floor changes real Transaction behavior end-to-end via TransactionManager-style options', async () => {
|
||||||
|
// Simulates how brainy.ts wires `this.config.transactionBudgetFloorMs`
|
||||||
|
// into `transactTimeoutBudget()`: opCount 0 isolates the floor term
|
||||||
|
// (0 × 2000 = 0), so a lowered floor makes a normally-generous budget
|
||||||
|
// fail-fast for a real 2-op Transaction.
|
||||||
|
const lowFloorBudget = transactTimeoutBudget(0, undefined, 10) // 10ms floor
|
||||||
|
expect(lowFloorBudget).toBe(10)
|
||||||
|
|
||||||
|
const log: string[] = []
|
||||||
|
const op0 = new SlowOperation(log, 'op0', 30)
|
||||||
|
const op1 = new SlowOperation(log, 'op1-gated-by-lowered-floor', 0)
|
||||||
|
|
||||||
|
const tx = new Transaction({ timeout: lowFloorBudget })
|
||||||
|
tx.addOperation(op0)
|
||||||
|
tx.addOperation(op1)
|
||||||
|
|
||||||
|
const error = await tx.execute().catch((e) => e)
|
||||||
|
expect(error).toBeInstanceOf(TransactionTimeoutError)
|
||||||
|
expect(op1.executed).toBe(false)
|
||||||
|
})
|
||||||
|
})
|
||||||
176
tests/unit/transaction/vectorIndexOperations-rename.test.ts
Normal file
176
tests/unit/transaction/vectorIndexOperations-rename.test.ts
Normal file
|
|
@ -0,0 +1,176 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/transaction/vectorIndexOperations-rename
|
||||||
|
* @description Coverage for the backend-neutral vector-index transaction
|
||||||
|
* operations (formerly `AddToHNSWOperation` / `RemoveFromHNSWOperation`).
|
||||||
|
* These op-name strings surface directly in consumer-visible transaction
|
||||||
|
* journals and timings — a fossil "HNSW" name misdirects an operator running
|
||||||
|
* a native (non-HNSW) vector provider. Verifies:
|
||||||
|
* 1. rollback wiring stayed byte-identical under the rename,
|
||||||
|
* 2. the emitted `name` stamps the active backend — `'hnsw-js'` for
|
||||||
|
* Brainy's own JS index, a provider's own required `name` when it
|
||||||
|
* self-identifies, and the tolerant-loud `'unknown-provider'` literal
|
||||||
|
* (plus one console.warn) when a runtime instance lacks `name`
|
||||||
|
* altogether (an older native provider compiled against the previous,
|
||||||
|
* optional `providerId` contract) — never a silently-wrong guess.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect, vi, afterEach } from 'vitest'
|
||||||
|
import {
|
||||||
|
AddToVectorIndexOperation,
|
||||||
|
RemoveFromVectorIndexOperation,
|
||||||
|
BatchAddToVectorIndexOperation
|
||||||
|
} from '../../../src/transaction/operations/IndexOperations.js'
|
||||||
|
import type { VectorIndexProvider } from '../../../src/plugin.js'
|
||||||
|
import type { VectorDocument, Vector } from '../../../src/coreTypes.js'
|
||||||
|
import { JsHnswVectorIndex } from '../../../src/hnsw/hnswIndex.js'
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A minimal, real (not mocked) VectorIndexProvider implementation backed by
|
||||||
|
* a plain Map — exercises the exact contract the operations call through,
|
||||||
|
* without any Brainy internals.
|
||||||
|
*/
|
||||||
|
class FakeVectorProvider implements VectorIndexProvider {
|
||||||
|
readonly items = new Map<string, Vector>()
|
||||||
|
constructor(readonly name: string) {}
|
||||||
|
|
||||||
|
async addItem(item: VectorDocument): Promise<string> {
|
||||||
|
this.items.set(item.id, item.vector)
|
||||||
|
return item.id
|
||||||
|
}
|
||||||
|
async removeItem(id: string): Promise<boolean> {
|
||||||
|
return this.items.delete(id)
|
||||||
|
}
|
||||||
|
async getItem(id: string): Promise<VectorDocument | undefined> {
|
||||||
|
const vector = this.items.get(id)
|
||||||
|
return vector ? { id, vector } : undefined
|
||||||
|
}
|
||||||
|
async search(): Promise<Array<[string, number]>> {
|
||||||
|
return []
|
||||||
|
}
|
||||||
|
size(): number {
|
||||||
|
return this.items.size
|
||||||
|
}
|
||||||
|
clear(): void {
|
||||||
|
this.items.clear()
|
||||||
|
}
|
||||||
|
async rebuild(): Promise<void> {}
|
||||||
|
async flush(): Promise<number> {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
getPersistMode(): 'immediate' | 'deferred' {
|
||||||
|
return 'immediate'
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('Vector index transaction operations — backend-neutral rename', () => {
|
||||||
|
describe('rollback wiring (byte-identical behavior under the rename)', () => {
|
||||||
|
it('AddToVectorIndexOperation rollback removes a newly-added item', async () => {
|
||||||
|
const provider = new FakeVectorProvider('fake-provider')
|
||||||
|
const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
|
||||||
|
|
||||||
|
const rollback = await op.execute()
|
||||||
|
expect(provider.items.has('id-1')).toBe(true)
|
||||||
|
|
||||||
|
await rollback()
|
||||||
|
expect(provider.items.has('id-1')).toBe(false)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('AddToVectorIndexOperation rollback is a no-op when the item pre-existed (update semantics)', async () => {
|
||||||
|
const provider = new FakeVectorProvider('fake-provider')
|
||||||
|
await provider.addItem({ id: 'id-1', vector: [9, 9, 9] })
|
||||||
|
|
||||||
|
const op = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
|
||||||
|
const rollback = await op.execute()
|
||||||
|
expect(provider.items.get('id-1')).toEqual([1, 2, 3])
|
||||||
|
|
||||||
|
await rollback()
|
||||||
|
// Pre-existing item is NOT removed by rollback — it stays (the update
|
||||||
|
// itself is not undone by this op; that's RemoveFromVectorIndexOperation's job).
|
||||||
|
expect(provider.items.has('id-1')).toBe(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('RemoveFromVectorIndexOperation rollback re-adds the item with its original vector', async () => {
|
||||||
|
const provider = new FakeVectorProvider('fake-provider')
|
||||||
|
await provider.addItem({ id: 'id-1', vector: [4, 5, 6] })
|
||||||
|
|
||||||
|
const op = new RemoveFromVectorIndexOperation(provider, 'id-1', [4, 5, 6])
|
||||||
|
const rollback = await op.execute()
|
||||||
|
expect(provider.items.has('id-1')).toBe(false)
|
||||||
|
|
||||||
|
await rollback()
|
||||||
|
expect(provider.items.get('id-1')).toEqual([4, 5, 6])
|
||||||
|
})
|
||||||
|
|
||||||
|
it('BatchAddToVectorIndexOperation rolls back every item in reverse order on undo', async () => {
|
||||||
|
const provider = new FakeVectorProvider('fake-provider')
|
||||||
|
const op = new BatchAddToVectorIndexOperation(provider, [
|
||||||
|
{ id: 'a', vector: [1] },
|
||||||
|
{ id: 'b', vector: [2] },
|
||||||
|
{ id: 'c', vector: [3] }
|
||||||
|
])
|
||||||
|
|
||||||
|
const rollback = await op.execute()
|
||||||
|
expect([...provider.items.keys()].sort()).toEqual(['a', 'b', 'c'])
|
||||||
|
|
||||||
|
await rollback()
|
||||||
|
expect(provider.items.size).toBe(0)
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('backend stamping in the emitted op name', () => {
|
||||||
|
afterEach(() => {
|
||||||
|
vi.restoreAllMocks()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('stamps the real JS HNSW index as hnsw-js (self-identifying name)', () => {
|
||||||
|
const index = new JsHnswVectorIndex()
|
||||||
|
const addOp = new AddToVectorIndexOperation(index, 'id-1', [1, 2, 3])
|
||||||
|
const removeOp = new RemoveFromVectorIndexOperation(index, 'id-1', [1, 2, 3])
|
||||||
|
|
||||||
|
expect(addOp.name).toBe('AddToVectorIndex(hnsw-js)')
|
||||||
|
expect(removeOp.name).toBe('RemoveFromVectorIndex(hnsw-js)')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('stamps a self-identifying provider\'s own name, never "HNSW"', () => {
|
||||||
|
const provider = new FakeVectorProvider('acme-diskann')
|
||||||
|
const addOp = new AddToVectorIndexOperation(provider, 'id-1', [1, 2, 3])
|
||||||
|
const removeOp = new RemoveFromVectorIndexOperation(provider, 'id-1', [1, 2, 3])
|
||||||
|
const batchOp = new BatchAddToVectorIndexOperation(provider, [{ id: 'a', vector: [1] }])
|
||||||
|
|
||||||
|
expect(addOp.name).toBe('AddToVectorIndex(acme-diskann)')
|
||||||
|
expect(removeOp.name).toBe('RemoveFromVectorIndex(acme-diskann)')
|
||||||
|
expect(batchOp.name).toBe('BatchAddToVectorIndex(acme-diskann)')
|
||||||
|
expect(addOp.name).not.toContain('HNSW')
|
||||||
|
expect(removeOp.name).not.toContain('HNSW')
|
||||||
|
})
|
||||||
|
|
||||||
|
it('tolerant-loud: stamps "unknown-provider" and warns exactly once when a runtime provider lacks the required `name` (an older native provider compiled against the previous optional `providerId` contract)', () => {
|
||||||
|
const warnSpy = vi.spyOn(console, 'warn').mockImplementation(() => {})
|
||||||
|
// Simulate a native provider compiled before `name` was required — the
|
||||||
|
// TypeScript contract requires `name`, but `as unknown as` mirrors what
|
||||||
|
// actually reaches this code at runtime from an un-rebuilt native addon.
|
||||||
|
const legacyProvider = {
|
||||||
|
addItem: async (item: VectorDocument) => item.id,
|
||||||
|
removeItem: async () => true,
|
||||||
|
search: async () => [],
|
||||||
|
size: () => 0,
|
||||||
|
clear: () => {},
|
||||||
|
rebuild: async () => {},
|
||||||
|
flush: async () => 0,
|
||||||
|
getPersistMode: () => 'immediate' as const
|
||||||
|
} as unknown as VectorIndexProvider
|
||||||
|
|
||||||
|
const addOp = new AddToVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3])
|
||||||
|
const removeOp = new RemoveFromVectorIndexOperation(legacyProvider, 'id-1', [1, 2, 3])
|
||||||
|
|
||||||
|
expect(addOp.name).toBe('AddToVectorIndex(unknown-provider)')
|
||||||
|
expect(removeOp.name).toBe('RemoveFromVectorIndex(unknown-provider)')
|
||||||
|
// Never silently falls back to the JS engine's name for a provider it
|
||||||
|
// knows nothing about — that's the exact fossil-naming bug being fixed.
|
||||||
|
expect(addOp.name).not.toContain('hnsw-js')
|
||||||
|
// Loud, not silent — but exactly once per provider instance, not once
|
||||||
|
// per op stamped against it.
|
||||||
|
expect(warnSpy).toHaveBeenCalledTimes(1)
|
||||||
|
expect(warnSpy.mock.calls[0][0]).toContain('name')
|
||||||
|
})
|
||||||
|
})
|
||||||
|
})
|
||||||
Loading…
Add table
Add a link
Reference in a new issue