feat: warm contract (warm/warmOnOpen/provider warm hook), configurable transact budget floor, backend-neutral vector index op names
Three pieces addressing the cold-restart-write incident where a production deployment's first writes after every restart (33-35s each on a cold page cache) blew the op-count-scaled transact budget mid-batch: every write is itself a multi-op transaction, so one cold operation consumed the whole budget, the gate before the next operation tripped, and the write rolled back atomically - refused, retried, and refused again until the page cache warmed passively. - The budget's start-gating contract is now explicit and pinned: it gates STARTING the next operation, never rolling back completed work for elapsed time (the shipped schedule since 8.7.0, now stated in contract JSDoc, guarded by code for operation 0, and enforced by regression tests). The 30s floor is configurable via transactionBudgetFloorMs for stores whose cold operations legitimately run long. - New brain.warm() eagerly loads the vector index, metadata index, and graph adjacency so first operations after a cold restart run at steady-state cost. Returns a WarmReport with an honest per-surface outcome (warmed / probed / unavailable) - never reports a probe as a warm. warmOnOpen: true runs it during init(). New optional provider hook warm() on the vector and graph plugin contracts. - Vector-index transaction op classes renamed from the backend-specific AddToHNSWOperation / RemoveFromHNSWOperation to backend-neutral AddToVectorIndexOperation / RemoveFromVectorIndexOperation, stamping the active backend into the emitted op-name string (AddToVectorIndex(js-hnsw) vs a native provider's own identity) so journals never misdirect an operator toward an index that isn't running.
This commit is contained in:
parent
d08679fc84
commit
55b867c998
13 changed files with 1099 additions and 57 deletions
|
|
@ -2,34 +2,58 @@
|
|||
* Index Operations with Rollback Support
|
||||
*
|
||||
* Provides transactional operations for all indexes:
|
||||
* - JsHnswVectorIndex (unified vector index)
|
||||
* - VectorIndexProvider (the JS HNSW fallback, or a native acceleration provider)
|
||||
* - MetadataIndexManager (roaring bitmap filtering)
|
||||
* - GraphAdjacencyIndex (LSM-tree graph storage)
|
||||
*
|
||||
* Each operation can be executed and rolled back atomically.
|
||||
*/
|
||||
|
||||
import type { JsHnswVectorIndex } from '../../hnsw/hnswIndex.js'
|
||||
import type { VectorIndexProvider, GraphIndexProvider } from '../../plugin.js'
|
||||
import type { MetadataIndexManager } from '../../utils/metadataIndex.js'
|
||||
import type { GraphIndexProvider } from '../../plugin.js'
|
||||
import type { GraphVerb } from '../../coreTypes.js'
|
||||
import type { Operation, RollbackAction } from '../types.js'
|
||||
|
||||
/**
|
||||
* Add to HNSW index with rollback support
|
||||
* Backend identity stamped into an operation's emitted `name` string (e.g.
|
||||
* `AddToVectorIndex(js-hnsw)`), resolved from the provider's own
|
||||
* {@link VectorIndexProvider.providerId} when it self-identifies.
|
||||
*
|
||||
* These operation classes are backend-neutral (the vector index they wrap may
|
||||
* be Brainy's own JS HNSW fallback OR a native acceleration provider), but
|
||||
* their names surface directly in consumer-visible transaction journals and
|
||||
* timings. A provider that hasn't set `providerId` resolves to
|
||||
* `'unknown-provider'` rather than guessing — silently defaulting to the JS
|
||||
* engine's name here is exactly the class of bug this stamping exists to
|
||||
* prevent (an operator diagnosing a backend that isn't the one that ran).
|
||||
*/
|
||||
function resolveVectorProviderId(index: VectorIndexProvider): string {
|
||||
return index.providerId ?? 'unknown-provider'
|
||||
}
|
||||
|
||||
/**
|
||||
* Add to the vector index with rollback support.
|
||||
*
|
||||
* Backend-neutral: `index` is whatever the `'vector'` provider factory
|
||||
* returns — Brainy's own JS HNSW fallback, or a native acceleration provider
|
||||
* (e.g. DiskANN). The emitted `name` stamps the active backend (see
|
||||
* {@link resolveVectorProviderId}) so operators reading a transaction journal
|
||||
* or timing trace see which engine actually ran, never a fossil name from
|
||||
* whichever engine happened to be active when this op class was written.
|
||||
*
|
||||
* Rollback strategy:
|
||||
* - Remove item from index
|
||||
|
||||
*/
|
||||
export class AddToHNSWOperation implements Operation {
|
||||
readonly name = 'AddToHNSW'
|
||||
export class AddToVectorIndexOperation implements Operation {
|
||||
readonly name: string
|
||||
|
||||
constructor(
|
||||
private readonly index: JsHnswVectorIndex,
|
||||
private readonly index: VectorIndexProvider,
|
||||
private readonly id: string,
|
||||
private readonly vector: number[]
|
||||
) {}
|
||||
) {
|
||||
this.name = `AddToVectorIndex(${resolveVectorProviderId(index)})`
|
||||
}
|
||||
|
||||
async execute(): Promise<RollbackAction> {
|
||||
// Check if item already exists (for rollback decision)
|
||||
|
|
@ -59,11 +83,12 @@ export class AddToHNSWOperation implements Operation {
|
|||
* pre-existence as "existed" made every rollback skip removeItem, leaving
|
||||
* phantom entries in the index after a failed transaction. The safe default
|
||||
* is to remove what this operation added — update flows pair this op with a
|
||||
* RemoveFromHNSWOperation whose own rollback restores the prior vector, so
|
||||
* reverse-order rollback reconstructs the original state either way.
|
||||
* RemoveFromVectorIndexOperation whose own rollback restores the prior
|
||||
* vector, so reverse-order rollback reconstructs the original state either
|
||||
* way.
|
||||
*/
|
||||
private async itemExists(id: string): Promise<boolean> {
|
||||
const index = this.index as JsHnswVectorIndex & {
|
||||
const index = this.index as VectorIndexProvider & {
|
||||
getItem?: (id: string) => Promise<unknown>
|
||||
}
|
||||
if (typeof index.getItem !== 'function') return false
|
||||
|
|
@ -77,21 +102,27 @@ export class AddToHNSWOperation implements Operation {
|
|||
}
|
||||
|
||||
/**
|
||||
* Remove from HNSW index with rollback support
|
||||
* Remove from the vector index with rollback support.
|
||||
*
|
||||
* Backend-neutral: see {@link AddToVectorIndexOperation} — `index` may be the
|
||||
* JS HNSW fallback or a native acceleration provider; the emitted `name`
|
||||
* stamps the active backend.
|
||||
*
|
||||
* Rollback strategy:
|
||||
* - Re-add item to index with original vector
|
||||
*
|
||||
* Note: Requires storing the vector for rollback
|
||||
*/
|
||||
export class RemoveFromHNSWOperation implements Operation {
|
||||
readonly name = 'RemoveFromHNSW'
|
||||
export class RemoveFromVectorIndexOperation implements Operation {
|
||||
readonly name: string
|
||||
|
||||
constructor(
|
||||
private readonly index: JsHnswVectorIndex,
|
||||
private readonly index: VectorIndexProvider,
|
||||
private readonly id: string,
|
||||
private readonly vector: number[] // Required for rollback
|
||||
) {}
|
||||
) {
|
||||
this.name = `RemoveFromVectorIndex(${resolveVectorProviderId(index)})`
|
||||
}
|
||||
|
||||
async execute(): Promise<RollbackAction> {
|
||||
// Remove from index
|
||||
|
|
@ -285,22 +316,24 @@ export class RemoveFromGraphIndexOperation implements Operation {
|
|||
}
|
||||
|
||||
/**
|
||||
* Batch operation: Add multiple items to HNSW index
|
||||
* Batch operation: Add multiple items to the vector index (backend-neutral —
|
||||
* see {@link AddToVectorIndexOperation}).
|
||||
*
|
||||
* Useful for bulk imports with transaction support.
|
||||
* Rolls back all items if any fail.
|
||||
*/
|
||||
export class BatchAddToHNSWOperation implements Operation {
|
||||
readonly name = 'BatchAddToHNSW'
|
||||
export class BatchAddToVectorIndexOperation implements Operation {
|
||||
readonly name: string
|
||||
|
||||
private operations: AddToHNSWOperation[]
|
||||
private operations: AddToVectorIndexOperation[]
|
||||
|
||||
constructor(
|
||||
index: JsHnswVectorIndex,
|
||||
index: VectorIndexProvider,
|
||||
items: Array<{ id: string; vector: number[] }>
|
||||
) {
|
||||
this.name = `BatchAddToVectorIndex(${resolveVectorProviderId(index)})`
|
||||
this.operations = items.map(
|
||||
item => new AddToHNSWOperation(index, item.id, item.vector)
|
||||
item => new AddToVectorIndexOperation(index, item.id, item.vector)
|
||||
)
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue