fix(index): the flicker window dies — atomic in-place vector update; lazy open honors every provider's not-ready report; the Path Registry twin table
DP6/DP8 of the Path Registry (BRAINY-PROD-LATENCY-TRIAD, the proven flicker mechanism): update paths staged RemoveFromVectorIndex then AddToVectorIndex as two separately-awaited transaction ops — between them a live row was in NEITHER index (dark to semantic recall, fine in metadata list). The native pair widened that window to seconds in production before their side's visibility-commit fix; the structural cure lands here: - hnswIndex.updateItem: absent → add; SAME vector → pure no-op (the production shape — a type-only update re-indexed an unchanged vector, remove+add did pure damage); changed vector → the node NEVER leaves the index: synchronous vector swap first (every query from that instant sees correct distances), then unlink/relink at the node's existing level via shared internals (linkNode/unlinkNodeEdges refactored out of add/remove; entry point and maxLevel provably unchanged). - ReplaceInVectorIndexOperation: ONE transaction leg; feature-detects provider updateItem (native seam flagged — their side ships updateItem, then the adjacent remove+add fallback is dead code). Both update staging sites swapped; delete sites untouched. - LAZY-OPEN GATE (fleet adoption find, SELF-ENGINE-PAIR-STANDARD): under disableAutoRebuild, ensureIndexesLoaded assessed ONLY the vector index — a not-ready native METADATA provider never blocked the completion latch and every find() silently returned [] on a populated store. All three providers now vote; any not-ready report falls through to the rebuild. - docs/path-registry.md: brainy's twin table for the 32 shared path IDs — service class, budgets, lifecycle, narration, and the cited pin per row; owed rows named (LC4 doors-open migration, MT4 yielding heals, LC7 downgrade contract) per the lifecycle-sprint choreography. Pins: update-item-atomic 9/9 (visibility-atomic swap, reverse-index parity vs fresh rebuild, entry-point invariants) · lazy-notready-honor 2/2. Gates: unit 1928/1928 (148 files) · integration 760 · conformance 27/27.
This commit is contained in:
parent
3236a01bef
commit
ebe06cdf33
7 changed files with 937 additions and 57 deletions
85
docs/path-registry.md
Normal file
85
docs/path-registry.md
Normal file
|
|
@ -0,0 +1,85 @@
|
||||||
|
# The Path Registry — brainy's twin table
|
||||||
|
|
||||||
|
The brainy half of the cross-engine Path Registry (the native accelerator
|
||||||
|
maintains the master list; IDs are shared and stable — `LC3`, `DP7`, … are
|
||||||
|
citable in commits, board rounds, release notes, and pins). Every row owes
|
||||||
|
five things: **service class** (INDEX-SERVED | BOUNDED-FALLBACK, announced |
|
||||||
|
TYPED REFUSAL), **latency budget** at 1k/10k/100k/1M (design bar: billions),
|
||||||
|
**lifecycle behavior**, **failure narration**, and a **test pin**. A path not
|
||||||
|
in this registry does not ship; an unregistered path is a red gate in the
|
||||||
|
scan audit.
|
||||||
|
|
||||||
|
**The availability bar governing every row: user-visible downtime is
|
||||||
|
seconds, at restart only.** Migration, heal, compaction, embedding, and
|
||||||
|
retention run behind the doors — yielding, budget-capped, narrated. No path
|
||||||
|
may hold the doors while it does housekeeping.
|
||||||
|
|
||||||
|
Status legend: ✅ contracted + pinned (test cited) · 🟡 partial (what holds
|
||||||
|
and what's missing, stated) · 🔴 owed (named, never silent).
|
||||||
|
|
||||||
|
## LC — Lifecycle
|
||||||
|
|
||||||
|
| ID | Brainy row | Status |
|
||||||
|
|----|-----------|--------|
|
||||||
|
| LC1 | Same-version reopen adopts everything: brain-format epoch match → zero rebuilds; aggregation state adopts by stamp; persisted indexes load. | ✅ `tests/unit/brainy/brain-format-handshake` + `migration-deference` (no-drift reopen never rebuilds) |
|
||||||
|
| LC2 | New empty brain: doors immediate. | ✅ exercised by every suite's setup |
|
||||||
|
| LC3 | Upgrade, same epoch: as LC1 — new code on unchanged formats owes nothing at open. | ✅ same pins as LC1 (epoch equality is the gate) |
|
||||||
|
| LC4 | Upgrade with epoch migration: TODAY brainy's epoch rebuild runs at open before doors. | 🔴 **owed — the sev's lockout row.** The doors-open-serving-old-structures design (yielding installments + atomic swap) lands measured-and-gated behind the service-class pair, per the lifecycle-sprint choreography. Acceptance case: the 9,184-row hours-lockout. |
|
||||||
|
| LC5 | Crash recovery: bounded, resumable, narrated. Aggregation leg ✅ (behind-stamp → incremental catch-up off the fact log + time-travel reconciliation, capped at 5,000 affected before an ANNOUNCED rescan). Vector/metadata legs ride epoch machinery (rebuild-from-canonical, narrated). | 🟡 aggregation pinned (`tests/integration/aggregation-lifecycle-catchup`); the rebuild legs are narrated but not yet installment-yielding (couples to LC4) |
|
||||||
|
| LC6 | Shutdown under load: close() drains the background flush flight, tears down cadence timers, runs ONE time-bounded compaction pass (~5s budget, resumable). | 🟡 pinned for flush/compaction (8.9.0 suites); SIGTERM drain budget not yet declared |
|
||||||
|
| LC7 | Rollback/downgrade: an N−1 build opening an N brain. | 🔴 owed — no declared read-compat window or typed refusal today (epoch mismatch triggers a rebuild, not a refusal; v2 nested-bag records read as a phantom user field on pre-law builds). Needs the declared-window contract. |
|
||||||
|
| LC8 | Relocatable brain directory: no absolute paths in artifacts; persist()/load() round-trips. | 🟡 persist/load pinned; byte-for-byte relocation depot cases are the pair gate's (shared corpora) |
|
||||||
|
| LC9 | Double-open: second writer gets a typed lock refusal (PID-liveness + heartbeat stale detection; `force` escape hatch logs loudly). | ✅ writer-lock suites (8.7.1) |
|
||||||
|
|
||||||
|
## DP — Data plane
|
||||||
|
|
||||||
|
| ID | Brainy row | Status |
|
||||||
|
|----|-----------|--------|
|
||||||
|
| DP1 | `get()` by id: direct storage read + hydrate. INDEX-SERVED (id-mapped). Milliseconds at every scale. | ✅ exercised everywhere; budget rides the pair speed table |
|
||||||
|
| DP2 | `find({query})`: embed + vector search. The embed dominates (native side owns the budget); JS HNSW serves the search leg. | 🟡 300ms-class p95 is the pair speed-table row; brainy-alone budget declared there |
|
||||||
|
| DP3 | Filtered/sorted list: column top-K when the field is columnized (INDEX-SERVED, zero canonical reads on the sorted page — value pairs come from ONE batched metadata-record pass); no-column fallback is BOUNDED-ANNOUNCED (one batch pass, announces once per field past 500 rows); unknown field → TYPED REFUSAL naming both candidate spellings. | ✅ `tests/unit/utils/metadataIndex-sort-callshape` (zero per-row reads, batch-only — latency-blind) + `metadataIndex-nested-orderby` (dotted keys serve-or-refuse) + `tests/integration/orderby-sort-bug` |
|
||||||
|
| DP4 | Aggregation/stats: ALWAYS answers. Write-time incremental; behind-stamp reconciles incrementally; genuine rebuilds go through the native parallel door or the paged JS walk; nothing ever latches off; before-image-less deletes flag a LOUD rescan, never a silent skip. | ✅ `tests/integration/aggregation-lifecycle-catchup` + `tests/unit/aggregation/aggregation-provider-rebuild` |
|
||||||
|
| DP5 | Graph traversal: `related()` paged via adjacency; whole-graph analytics carry declared cost. | 🟡 paged reads pinned; analytics cost-class declaration owed (rides VENUE-GRAPH-TRUST audit tool) |
|
||||||
|
| DP6 | Single write: ack at the canonical commit; visibility committed at ack (the atomic vector update kills the remove→add dark window); maintenance NEVER holds the ack (background flush cadence — THE ACK LAW pin: a hung flush cannot block a write). | 🟡 ack law pinned (`tests/unit/brainy/persistence-policy`); atomic-update pin lands with the flicker fix in this train |
|
||||||
|
| DP7 | Bulk ingest: sustained rate holds flat — per-write maintenance taxes must not grow with brain size (A4 removed caller-flush convoys; deferred embedding removes the per-write embed tax where opted). | 🟡 the decay-curve row is a pair speed-table RED GATE; brainy-alone sustained-rate run rides the same corpora |
|
||||||
|
| DP8 | Read under write pressure: no flicker window — a row that exists is never invisible to recall, even transiently (same-vector re-index is a no-op; changed-vector swaps in place, node never leaves the index). | 🟡 lands in this train (atomic `updateItem` + `ReplaceInVectorIndexOperation`); symmetry suite + sentinels are the B4 program |
|
||||||
|
| — | **The lazy-open gate honors EVERY provider's not-ready report** (a not-ready metadata provider can no longer latch the silent-empty state under `disableAutoRebuild`). | ✅ `tests/unit/brainy/lazy-notready-honor` |
|
||||||
|
|
||||||
|
## MT — Maintenance (never in the door path)
|
||||||
|
|
||||||
|
| ID | Brainy row | Status |
|
||||||
|
|----|-----------|--------|
|
||||||
|
| MT1 | Flush/checkpoint: ENGINE-OWNED cadence (write-count/interval/idle triggers, single-flight, background, loud on failure; callers never flush in hot paths; `flush()` stays as an awaitable barrier). | ✅ `tests/unit/brainy/persistence-policy` |
|
||||||
|
| MT2 | Compaction: never on flush (durability-only law, 8.9.0); close-time pass time-budgeted + resumable; explicit `compactHistory({timeBudgetMs})`. | ✅ 8.9.0 suites |
|
||||||
|
| MT3 | Index upkeep (mapper folds, delta promotion): native-side machinery; brainy's JS legs are small and synchronous-cheap. | 🟡 declared; yield audit rides the pair |
|
||||||
|
| MT4 | Heal/rebuild walks (`repairIndex`, backfill walks): paged; failure latches with cooldown; NOT yet yield-to-foreground installments. | 🔴 owed — the priority-isolation clause (couples to LC4; same choreography) |
|
||||||
|
| MT5 | Deferred embedding worker: ack at durability, durable pending markers, crash-recovered at open, single-flight batches. | 🔴 lands as A3 in this train (design frozen on the incident thread) |
|
||||||
|
| MT6 | Retention/archival walks: retention `'all'` does nothing by design; bounded-retention reclaim is close-time/explicit only. | 🟡 8.9.0 behavior pinned; archival profile is the co-frozen D1+D3 unit |
|
||||||
|
|
||||||
|
## FM — Failure modes
|
||||||
|
|
||||||
|
| ID | Brainy row | Status |
|
||||||
|
|----|-----------|--------|
|
||||||
|
| FM1 | Disk full / IO error mid-op: transaction rollback + typed error; failed rollback → StoreInconsistentError quarantines writes until repairIndex(). | 🟡 rollback paths pinned; explicit disk-full depot case owed |
|
||||||
|
| FM2 | Memory pressure: query limits + reserved-memory config; unified cache eviction. | 🟡 declared budgets; cascade pin owed |
|
||||||
|
| FM3 | Torn/corrupt file on open: malformed brain-format marker → safe rebuild (never trusting a bad epoch); corrupt records surface loudly. | 🟡 marker pin ✅ (`brain-format-handshake`); broader quarantine is native-side |
|
||||||
|
| FM4 | Native module unavailable: plugin load failure is LOUD (version-coupling law throws on range mismatch — never silently version-drifted); JS engine serves with its own declared budgets, named as the active backend in op names. | ✅ `tests/unit/plugin-version-coupling` + op-name stamping |
|
||||||
|
|
||||||
|
## FL — Fleet
|
||||||
|
|
||||||
|
| ID | Brainy row | Status |
|
||||||
|
|----|-----------|--------|
|
||||||
|
| FL1 | Cold open on demand: LC1's adopt-everything open; warm() available for eager paths. | 🟡 open cost pinned at LC1; millisecond budget rides the speed table |
|
||||||
|
| FL2–FL4 | Boot storm / upgrade wave / isolation: fleet-layer policies over LC1/LC4 — engine leg = budgeted opens + LC4's behind-doors migration. | 🔴 owed with LC4 |
|
||||||
|
| FL5 | Brain as product object: create instant (LC2) · erase = `clear()` explicit + complete · export = portable-graph, canon-complete mode available. | ✅ clear-persistence + portable-graph + canonical-enumeration suites |
|
||||||
|
|
||||||
|
## Status summary
|
||||||
|
|
||||||
|
Contracted + pinned this train: **DP3, DP4, MT1, LC5(aggregation), the
|
||||||
|
lazy-open not-ready gate, LC1/LC3/LC9, FM4, FL5** — each with the cited
|
||||||
|
test. Landing in this train: **DP6/DP8 (atomic vector update), MT5 (A3
|
||||||
|
deferred embedding)**. Owed, in production-risk order, all coupled to the
|
||||||
|
priority-isolation program the lifecycle sev opened: **LC4 (doors-open
|
||||||
|
migration), MT4 (yielding heals), LC7 (downgrade contract), LC6 (SIGTERM
|
||||||
|
budget), FL2–FL4, FM1/FM2 depot cases.** Rows move from owed to contracted
|
||||||
|
only with a cited test — none lands by prose.
|
||||||
|
|
@ -97,6 +97,7 @@ import {
|
||||||
SaveVerbOperation,
|
SaveVerbOperation,
|
||||||
AddToGraphIndexOperation,
|
AddToGraphIndexOperation,
|
||||||
RemoveFromVectorIndexOperation,
|
RemoveFromVectorIndexOperation,
|
||||||
|
ReplaceInVectorIndexOperation,
|
||||||
RemoveFromMetadataIndexOperation,
|
RemoveFromMetadataIndexOperation,
|
||||||
RemoveFromGraphIndexOperation,
|
RemoveFromGraphIndexOperation,
|
||||||
UpdateNounMetadataOperation,
|
UpdateNounMetadataOperation,
|
||||||
|
|
@ -2949,11 +2950,16 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
level: 0
|
level: 0
|
||||||
})
|
})
|
||||||
)
|
)
|
||||||
|
// ONE atomic vector-index leg: the historical Remove→Add pair was
|
||||||
|
// two separately-awaited operations — between them the row was in
|
||||||
|
// NEITHER index (dark to semantic recall, visible to metadata
|
||||||
|
// reads). ReplaceInVectorIndexOperation goes through the provider's
|
||||||
|
// in-place updateItem when available (row never absent; an
|
||||||
|
// element-wise UNCHANGED vector — the type-only-update shape that
|
||||||
|
// flickered in production — is a pure no-op), else remove+add
|
||||||
|
// adjacent within the single op.
|
||||||
tx.addOperation(
|
tx.addOperation(
|
||||||
new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector)
|
new ReplaceInVectorIndexOperation(this.index, params.id, existing.vector, vector)
|
||||||
)
|
|
||||||
tx.addOperation(
|
|
||||||
new AddToVectorIndexOperation(this.index, params.id, vector)
|
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -9364,8 +9370,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
connections: new Map(),
|
connections: new Map(),
|
||||||
level: 0
|
level: 0
|
||||||
}),
|
}),
|
||||||
new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector),
|
// ONE atomic vector-index leg — same law as update(): the row must
|
||||||
new AddToVectorIndexOperation(this.index, params.id, vector)
|
// never be absent from vector search during an update (see
|
||||||
|
// ReplaceInVectorIndexOperation).
|
||||||
|
new ReplaceInVectorIndexOperation(this.index, params.id, existing.vector, vector)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
plan.operations.push(
|
plan.operations.push(
|
||||||
|
|
@ -14958,14 +14966,30 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
||||||
}
|
}
|
||||||
|
|
||||||
// If indexes already populated AND honestly serving, mark complete and skip.
|
// If indexes already populated AND honestly serving, mark complete and skip.
|
||||||
// Honest gate: when the provider exposes isReady(), that REPLACES the size()>0
|
// Honest gate: when a provider exposes isReady(), that REPLACES the size()>0
|
||||||
// proxy (a native index can report a non-zero size while its serving structure
|
// proxy (a native index can report a non-zero size while its serving structure
|
||||||
// is not loaded — the silent-empty cold-load class). A not-ready provider falls
|
// is not loaded — the silent-empty cold-load class). A not-ready provider falls
|
||||||
// through so the rebuild path can load it; verifyVectorLive() is the query-time
|
// through so the rebuild path can load it; verifyVectorLive() is the query-time
|
||||||
// backstop either way. Providers without isReady() keep the size() heuristic
|
// backstop either way. Providers without isReady() keep the size() heuristic
|
||||||
// (the JS index's size()>0 genuinely means loaded).
|
// (the JS index's size()>0 genuinely means loaded).
|
||||||
|
//
|
||||||
|
// ALL THREE providers vote (fleet-adoption find, SELF-ENGINE-PAIR-STANDARD):
|
||||||
|
// this gate used to assess ONLY the vector index, so a not-ready native
|
||||||
|
// METADATA provider (its strand report) never blocked the completion latch
|
||||||
|
// — under disableAutoRebuild the promised lazy first-query rebuild never
|
||||||
|
// fired and every find() silently returned [] on a populated store. A
|
||||||
|
// not-ready report from ANY provider now falls through to the rebuild.
|
||||||
const vectorReadiness = assessIndexReadiness(this.index)
|
const vectorReadiness = assessIndexReadiness(this.index)
|
||||||
if (vectorReadiness === 'ready' || (vectorReadiness === 'unknown' && this.index.size() > 0)) {
|
const metadataReadiness = assessIndexReadiness(this.metadataIndex)
|
||||||
|
const graphReadiness = assessIndexReadiness(this.graphIndex)
|
||||||
|
const anyProviderNotReady =
|
||||||
|
vectorReadiness === 'not-ready' ||
|
||||||
|
metadataReadiness === 'not-ready' ||
|
||||||
|
graphReadiness === 'not-ready'
|
||||||
|
if (
|
||||||
|
!anyProviderNotReady &&
|
||||||
|
(vectorReadiness === 'ready' || (vectorReadiness === 'unknown' && this.index.size() > 0))
|
||||||
|
) {
|
||||||
this.lazyRebuildCompleted = true
|
this.lazyRebuildCompleted = true
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -486,6 +486,90 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
||||||
return id
|
return id
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Wire the node into the graph: greedy descent + per-level linking.
|
||||||
|
// Extracted to linkNode so updateItem's in-place relink runs the SAME
|
||||||
|
// insertion linking (one implementation, never a diverging copy).
|
||||||
|
await this.linkNode(noun, entryPoint)
|
||||||
|
|
||||||
|
// Update max level and entry point if needed
|
||||||
|
if (nounLevel > this.maxLevel) {
|
||||||
|
this.maxLevel = nounLevel
|
||||||
|
this.entryPointId = id
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add noun to the index
|
||||||
|
this.nouns.set(id, noun)
|
||||||
|
|
||||||
|
// Track high-level nodes for O(1) entry point selection
|
||||||
|
if (nounLevel >= 2 && nounLevel <= this.MAX_TRACKED_LEVELS) {
|
||||||
|
if (!this.highLevelNodes.has(nounLevel)) {
|
||||||
|
this.highLevelNodes.set(nounLevel, new Set())
|
||||||
|
}
|
||||||
|
this.highLevelNodes.get(nounLevel)!.add(id)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Lazy vector eviction (B2: graph-only memory after insert)
|
||||||
|
// After graph construction completes, evict the full vector from memory.
|
||||||
|
// Future searches will load vectors on-demand via getVectorSafe() + UnifiedCache.
|
||||||
|
if (this.vectorStorageMode === 'lazy' && this.storage) {
|
||||||
|
noun.vector = [] // Release float32 vector from memory
|
||||||
|
}
|
||||||
|
|
||||||
|
// Persist HNSW graph data to storage
|
||||||
|
// Respect persistMode setting
|
||||||
|
if (this.storage && this.persistMode === 'immediate') {
|
||||||
|
// IMMEDIATE MODE: Original behavior - persist new entity and system data.
|
||||||
|
// Goes through the per-node helper so the compressed-blob branch fires
|
||||||
|
// identically here vs. the deferred-flush + neighbor-update paths.
|
||||||
|
await this.persistNodeConnections(id, noun).catch((error) => {
|
||||||
|
console.error(`Failed to persist HNSW data for ${id}:`, error)
|
||||||
|
})
|
||||||
|
|
||||||
|
// Persist system data (entry point and max level)
|
||||||
|
await this.storage.saveHNSWSystem({
|
||||||
|
entryPointId: this.entryPointId,
|
||||||
|
maxLevel: this.maxLevel
|
||||||
|
}).catch((error) => {
|
||||||
|
console.error('Failed to persist HNSW system data:', error)
|
||||||
|
})
|
||||||
|
} else if (this.persistMode === 'deferred') {
|
||||||
|
// DEFERRED MODE: Track dirty nodes for later batch persistence
|
||||||
|
this.dirtyNodes.add(id)
|
||||||
|
this.dirtySystem = true
|
||||||
|
}
|
||||||
|
|
||||||
|
return id
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description The insertion LINKING phase shared by {@link addItem} and
|
||||||
|
* {@link updateItem}: greedy-descend from `entryPoint` through the levels
|
||||||
|
* above `noun.level`, then at each level from `min(noun.level, maxLevel)`
|
||||||
|
* down to 0 find `efConstruction` candidates, select the M nearest, and
|
||||||
|
* create bidirectional edges — maintaining the reverse-adjacency index via
|
||||||
|
* {@link addIncoming} and re-pruning any neighbor pushed over M.
|
||||||
|
*
|
||||||
|
* Persistence follows the caller's mode exactly as the historical inline
|
||||||
|
* addItem code did: `'immediate'` persists each touched neighbor's
|
||||||
|
* connections concurrently (batched by `maxConcurrentNeighborWrites`);
|
||||||
|
* `'deferred'` marks each touched neighbor dirty for the next flush.
|
||||||
|
*
|
||||||
|
* Does NOT touch index membership (`this.nouns`), the entry point, or
|
||||||
|
* `maxLevel` — the caller owns that bookkeeping: addItem inserts a NEW node
|
||||||
|
* afterwards and may raise maxLevel; updateItem relinks an EXISTING node in
|
||||||
|
* place whose level was already counted, so nothing may change. `noun.vector`
|
||||||
|
* must be the live in-memory vector at call time; both callers guarantee it
|
||||||
|
* (lazy-mode eviction happens only after linking completes).
|
||||||
|
*
|
||||||
|
* A `neighborId === noun.id` candidate is skipped defensively: during
|
||||||
|
* updateItem the node is already IN `this.nouns` (visibility-atomicity —
|
||||||
|
* unlike addItem, which links before inserting), and a self-edge must never
|
||||||
|
* be creatable no matter what the traversal surfaces.
|
||||||
|
*/
|
||||||
|
private async linkNode(noun: HNSWNoun, entryPoint: HNSWNoun): Promise<void> {
|
||||||
|
const { id, vector } = noun
|
||||||
|
const nounLevel = noun.level
|
||||||
|
|
||||||
let currObj = entryPoint
|
let currObj = entryPoint
|
||||||
|
|
||||||
// Calculate distance to entry point (handles lazy loading + sync fast path)
|
// Calculate distance to entry point (handles lazy loading + sync fast path)
|
||||||
|
|
@ -547,6 +631,10 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
||||||
}> = []
|
}> = []
|
||||||
|
|
||||||
for (const [neighborId, _] of neighbors) {
|
for (const [neighborId, _] of neighbors) {
|
||||||
|
if (neighborId === id) {
|
||||||
|
// Never self-link (see method JSDoc — reachable only via updateItem)
|
||||||
|
continue
|
||||||
|
}
|
||||||
const neighbor = this.nouns.get(neighborId)
|
const neighbor = this.nouns.get(neighborId)
|
||||||
if (!neighbor) {
|
if (!neighbor) {
|
||||||
// Skip neighbors that don't exist (expected during rapid additions/deletions)
|
// Skip neighbors that don't exist (expected during rapid additions/deletions)
|
||||||
|
|
@ -630,7 +718,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
||||||
const nearestNoun = this.nouns.get(nearestId)
|
const nearestNoun = this.nouns.get(nearestId)
|
||||||
if (!nearestNoun) {
|
if (!nearestNoun) {
|
||||||
console.error(
|
console.error(
|
||||||
`Nearest noun with ID ${nearestId} not found in addItem`
|
`Nearest noun with ID ${nearestId} not found in linkNode`
|
||||||
)
|
)
|
||||||
// Keep the current object as is
|
// Keep the current object as is
|
||||||
} else {
|
} else {
|
||||||
|
|
@ -639,55 +727,173 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Update max level and entry point if needed
|
|
||||||
if (nounLevel > this.maxLevel) {
|
|
||||||
this.maxLevel = nounLevel
|
|
||||||
this.entryPointId = id
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// Add noun to the index
|
/**
|
||||||
this.nouns.set(id, noun)
|
* @description Atomically replace an item's vector IN PLACE — the row is
|
||||||
|
* NEVER absent from the index during an update. The historical shape staged
|
||||||
// Track high-level nodes for O(1) entry point selection
|
* a remove followed by an add as two separately-awaited transaction
|
||||||
if (nounLevel >= 2 && nounLevel <= this.MAX_TRACKED_LEVELS) {
|
* operations; between them the row was in NEITHER index — dark to semantic
|
||||||
if (!this.highLevelNodes.has(nounLevel)) {
|
* recall while perfectly visible to metadata reads (observed as seconds-long
|
||||||
this.highLevelNodes.set(nounLevel, new Set())
|
* production flicker in a downstream deployment). Mandate: a row that
|
||||||
|
* exists must never be invisible to a read path, even transiently.
|
||||||
|
*
|
||||||
|
* Behavior:
|
||||||
|
* - id not in the index → delegates to {@link addItem} (plain insert).
|
||||||
|
* - SAME vector (element-wise equal) → pure no-op. This is the production
|
||||||
|
* flicker shape: a type-only update re-indexes an UNCHANGED vector, so the
|
||||||
|
* old remove+add did pure damage. (In lazy vector-storage mode the
|
||||||
|
* comparison baseline is whatever {@link getVectorSafe} serves — the
|
||||||
|
* cache, or the persisted record; if the caller already rewrote the
|
||||||
|
* record with the new vector before calling in, equality may report "no
|
||||||
|
* change" and skip the relink. Query correctness is unaffected either
|
||||||
|
* way — distances always use the live vector — the graph edges just keep
|
||||||
|
* their pre-update geometry, which HNSW tolerates by construction.)
|
||||||
|
* - DIFFERENT vector → the node never leaves `this.nouns`:
|
||||||
|
* 1. `node.vector` is swapped SYNCHRONOUSLY first (and the shared vector
|
||||||
|
* cache updated in the same tick), so from that point every query sees
|
||||||
|
* the node with correct distances;
|
||||||
|
* 2. its old edges are unlinked via the same reverse-adjacency walk
|
||||||
|
* removeItem uses ({@link unlinkNodeEdges}) — the node stays in the
|
||||||
|
* map and KEEPS its level;
|
||||||
|
* 3. the insertion linking re-runs at the node's EXISTING level
|
||||||
|
* ({@link linkNode}). Entry-point cases: if the node IS the entry
|
||||||
|
* point it REMAINS the entry point (still valid — same id, same
|
||||||
|
* level); the relink traversal then starts from another node via
|
||||||
|
* {@link resolveRelinkStart}, because the node's own edges were just
|
||||||
|
* cleared and a traversal starting AT it would find nothing and link
|
||||||
|
* nothing — stranding the whole graph behind an edgeless entry point.
|
||||||
|
* maxLevel never regresses: the node keeps its level and its
|
||||||
|
* membership, so the remove-side relevel bookkeeping never runs.
|
||||||
|
*
|
||||||
|
* Persistence mirrors {@link addItem}'s tail for the node itself plus the
|
||||||
|
* in-neighbors whose connection sets changed during the unlink:
|
||||||
|
* `'immediate'` persists their connections now; `'deferred'` marks them
|
||||||
|
* dirty for the next flush. The system record (entry point + maxLevel) is
|
||||||
|
* NOT rewritten — an in-place update changes neither.
|
||||||
|
*/
|
||||||
|
public async updateItem(item: VectorDocument): Promise<void> {
|
||||||
|
if (!item) {
|
||||||
|
throw new Error('Item is undefined or null')
|
||||||
}
|
}
|
||||||
this.highLevelNodes.get(nounLevel)!.add(id)
|
const { id, vector } = item
|
||||||
|
if (!vector) {
|
||||||
|
throw new Error('Vector is undefined or null')
|
||||||
}
|
}
|
||||||
|
|
||||||
// Lazy vector eviction (B2: graph-only memory after insert)
|
const node = this.nouns.get(id)
|
||||||
// After graph construction completes, evict the full vector from memory.
|
if (!node) {
|
||||||
// Future searches will load vectors on-demand via getVectorSafe() + UnifiedCache.
|
// Absent → plain insert.
|
||||||
if (this.vectorStorageMode === 'lazy' && this.storage) {
|
await this.addItem(item)
|
||||||
noun.vector = [] // Release float32 vector from memory
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
// Persist HNSW graph data to storage
|
if (this.dimension === null) {
|
||||||
// Respect persistMode setting
|
this.dimension = vector.length
|
||||||
|
} else if (vector.length !== this.dimension) {
|
||||||
|
throw new Error(
|
||||||
|
`Vector dimension mismatch: expected ${this.dimension}, got ${vector.length}`
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fast path: element-wise-equal vector → NOTHING to do (the production
|
||||||
|
// flicker shape — a type-only update re-indexing an unchanged vector).
|
||||||
|
// getVectorSafe handles the lazy-evicted case (loads from cache/storage).
|
||||||
|
const current = await this.getVectorSafe(node)
|
||||||
|
if (current.length === vector.length) {
|
||||||
|
let same = true
|
||||||
|
for (let i = 0; i < vector.length; i++) {
|
||||||
|
if (current[i] !== vector[i]) {
|
||||||
|
same = false
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (same) return
|
||||||
|
}
|
||||||
|
|
||||||
|
// (1) Visibility-atomic swap: from this synchronous assignment on, every
|
||||||
|
// query sees the node with correct distances. The shared vector cache is
|
||||||
|
// updated in the same tick so the lazy-mode read path can never serve the
|
||||||
|
// stale vector either.
|
||||||
|
node.vector = vector
|
||||||
|
this.unifiedCache.set(`hnsw:vector:${id}`, vector, 'vectors', vector.length * 4, 50)
|
||||||
|
|
||||||
|
// (2) Unlink the old edges — the node stays in the map, keeps its level.
|
||||||
|
const touchedReferrers = await this.unlinkNodeEdges(node)
|
||||||
|
node.connections = new Map()
|
||||||
|
for (let level = 0; level <= node.level; level++) {
|
||||||
|
node.connections.set(level, new Set<string>())
|
||||||
|
}
|
||||||
|
// The node's own reverse entry is rebuilt by the relink below.
|
||||||
|
this.incoming?.delete(id)
|
||||||
|
|
||||||
|
// (3) Relink at the node's EXISTING level (see JSDoc for the entry-point
|
||||||
|
// reasoning). A single-node index has nothing to link to — trivially done.
|
||||||
|
const start = this.resolveRelinkStart(id)
|
||||||
|
if (start) {
|
||||||
|
await this.linkNode(node, start)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Persistence — addItem's tail, minus the system record (entry point and
|
||||||
|
// maxLevel are untouched by an in-place update). Unlink-touched referrers
|
||||||
|
// are included so the persisted graph converges on the live one instead of
|
||||||
|
// keeping their pre-update edge sets forever.
|
||||||
if (this.storage && this.persistMode === 'immediate') {
|
if (this.storage && this.persistMode === 'immediate') {
|
||||||
// IMMEDIATE MODE: Original behavior - persist new entity and system data.
|
await this.persistNodeConnections(id, node).catch((error) => {
|
||||||
// Goes through the per-node helper so the compressed-blob branch fires
|
|
||||||
// identically here vs. the deferred-flush + neighbor-update paths.
|
|
||||||
await this.persistNodeConnections(id, noun).catch((error) => {
|
|
||||||
console.error(`Failed to persist HNSW data for ${id}:`, error)
|
console.error(`Failed to persist HNSW data for ${id}:`, error)
|
||||||
})
|
})
|
||||||
|
for (const refId of touchedReferrers) {
|
||||||
// Persist system data (entry point and max level)
|
const ref = this.nouns.get(refId)
|
||||||
await this.storage.saveHNSWSystem({
|
if (!ref) continue
|
||||||
entryPointId: this.entryPointId,
|
await this.persistNodeConnections(refId, ref).catch((error) => {
|
||||||
maxLevel: this.maxLevel
|
console.error(`Failed to persist HNSW data for ${refId}:`, error)
|
||||||
}).catch((error) => {
|
|
||||||
console.error('Failed to persist HNSW system data:', error)
|
|
||||||
})
|
})
|
||||||
|
}
|
||||||
} else if (this.persistMode === 'deferred') {
|
} else if (this.persistMode === 'deferred') {
|
||||||
// DEFERRED MODE: Track dirty nodes for later batch persistence
|
|
||||||
this.dirtyNodes.add(id)
|
this.dirtyNodes.add(id)
|
||||||
this.dirtySystem = true
|
for (const refId of touchedReferrers) {
|
||||||
|
this.dirtyNodes.add(refId)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
return id
|
// Lazy vector eviction — same contract as addItem: after graph work
|
||||||
|
// completes the float32 vector leaves memory; reads serve from the
|
||||||
|
// (just-updated) cache or the persisted record.
|
||||||
|
if (this.vectorStorageMode === 'lazy' && this.storage) {
|
||||||
|
node.vector = []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @description Pick the traversal start for an in-place relink
|
||||||
|
* ({@link updateItem} step 3): the current entry point — unless that IS the
|
||||||
|
* node being relinked. Its edges were just unlinked, so a traversal
|
||||||
|
* starting there would see an empty neighborhood and produce zero links,
|
||||||
|
* stranding the graph behind an edgeless entry point. In that case (or when
|
||||||
|
* the entry point is missing/stale) fall back to the best OTHER node:
|
||||||
|
* highest tracked level first (the same O(1) heuristic as
|
||||||
|
* {@link recoverEntryPointO1}), then any other node. Returns null when the
|
||||||
|
* node is the only one in the index — nothing to link to, trivially valid.
|
||||||
|
*/
|
||||||
|
private resolveRelinkStart(excludeId: string): HNSWNoun | null {
|
||||||
|
if (this.entryPointId && this.entryPointId !== excludeId) {
|
||||||
|
const entry = this.nouns.get(this.entryPointId)
|
||||||
|
if (entry) return entry
|
||||||
|
}
|
||||||
|
for (let level = this.MAX_TRACKED_LEVELS; level >= 2; level--) {
|
||||||
|
const nodesAtLevel = this.highLevelNodes.get(level)
|
||||||
|
if (!nodesAtLevel) continue
|
||||||
|
for (const nodeId of nodesAtLevel) {
|
||||||
|
if (nodeId !== excludeId) {
|
||||||
|
const candidate = this.nouns.get(nodeId)
|
||||||
|
if (candidate) return candidate
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
for (const [nodeId, candidate] of this.nouns) {
|
||||||
|
if (nodeId !== excludeId) return candidate
|
||||||
|
}
|
||||||
|
return null
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|
@ -948,20 +1154,34 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Remove an item from the index
|
* @description Unlink every graph edge touching `noun`, in BOTH directions,
|
||||||
|
* WITHOUT removing the node from `this.nouns` — the unlink walk shared by
|
||||||
|
* {@link removeItem} (which then drops the node) and {@link updateItem}
|
||||||
|
* (which relinks the node in place, so it must never leave the map and
|
||||||
|
* KEEPS its level).
|
||||||
|
*
|
||||||
|
* Reverse-adjacency lets us touch ONLY the nodes that actually reference
|
||||||
|
* `noun.id` (its in-neighbors) rather than scanning the whole corpus —
|
||||||
|
* turning a delete from O(N) into O(in-degree) and a bulk delete from O(N²)
|
||||||
|
* into O(N·degree). Each referrer set is snapshotted because
|
||||||
|
* pruneConnections mutates the index. Outgoing edges are unhooked from each
|
||||||
|
* target's reverse set so no stale referrer survives.
|
||||||
|
*
|
||||||
|
* `incoming[noun.id]` itself is intentionally NOT maintained edge-by-edge
|
||||||
|
* inside the walk — both callers dispose of it wholesale afterwards
|
||||||
|
* (removeItem deletes it with the node; updateItem clears it and lets the
|
||||||
|
* relink rebuild it).
|
||||||
|
*
|
||||||
|
* @returns The ids of in-neighbors whose connection sets were modified
|
||||||
|
* (they dropped their edge to `noun` and may have been re-pruned), so a
|
||||||
|
* caller that persists per-node connections (updateItem) can mark them
|
||||||
|
* dirty / persist them. removeItem ignores the return — its persistence
|
||||||
|
* story lives in the caller's delete path, unchanged.
|
||||||
*/
|
*/
|
||||||
public async removeItem(id: string): Promise<boolean> {
|
private async unlinkNodeEdges(noun: HNSWNoun): Promise<Set<string>> {
|
||||||
if (!this.nouns.has(id)) {
|
const id = noun.id
|
||||||
return false
|
const touchedReferrers = new Set<string>()
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
const noun = this.nouns.get(id)!
|
|
||||||
|
|
||||||
// Reverse-adjacency lets us touch ONLY the nodes that actually reference `id`
|
|
||||||
// (its in-neighbors) rather than scanning the whole corpus — turning a delete
|
|
||||||
// from O(N) into O(in-degree) and a bulk delete from O(N²) into O(N·degree).
|
|
||||||
// Snapshot each referrer set because pruneConnections mutates the index.
|
|
||||||
const incoming = this.ensureIncoming()
|
const incoming = this.ensureIncoming()
|
||||||
const referrers = incoming.get(id)
|
const referrers = incoming.get(id)
|
||||||
if (referrers) {
|
if (referrers) {
|
||||||
|
|
@ -969,11 +1189,11 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
||||||
for (const refId of Array.from(refSet)) {
|
for (const refId of Array.from(refSet)) {
|
||||||
const ref = this.nouns.get(refId)
|
const ref = this.nouns.get(refId)
|
||||||
if (ref && ref.connections.has(level)) {
|
if (ref && ref.connections.has(level)) {
|
||||||
// Drop the forward edge ref → id, then re-prune ref so the graph stays
|
// Drop the forward edge ref → id, then re-prune ref so the graph
|
||||||
// navigable. (id's own reverse entry is dropped wholesale below, so we
|
// stays navigable.
|
||||||
// intentionally do not maintain incoming[id] inside this loop.)
|
|
||||||
ref.connections.get(level)!.delete(id)
|
ref.connections.get(level)!.delete(id)
|
||||||
await this.pruneConnections(ref, level)
|
await this.pruneConnections(ref, level)
|
||||||
|
touchedReferrers.add(refId)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -987,6 +1207,26 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
return touchedReferrers
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Remove an item from the index
|
||||||
|
*/
|
||||||
|
public async removeItem(id: string): Promise<boolean> {
|
||||||
|
if (!this.nouns.has(id)) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
const noun = this.nouns.get(id)!
|
||||||
|
|
||||||
|
// Unlink every edge touching the node (shared with updateItem's in-place
|
||||||
|
// relink — see unlinkNodeEdges). The returned touched-referrer set is
|
||||||
|
// ignored here: removeItem's persistence story lives in the caller's
|
||||||
|
// delete path, unchanged.
|
||||||
|
await this.unlinkNodeEdges(noun)
|
||||||
|
|
||||||
// Remove the noun + its reverse-index entry.
|
// Remove the noun + its reverse-index entry.
|
||||||
this.nouns.delete(id)
|
this.nouns.delete(id)
|
||||||
this.incoming?.delete(id)
|
this.incoming?.delete(id)
|
||||||
|
|
|
||||||
|
|
@ -151,6 +151,95 @@ export class RemoveFromVectorIndexOperation implements Operation {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Replace an item's vector in the vector index as ONE atomic transaction leg —
|
||||||
|
* the row is never absent from vector search during an update.
|
||||||
|
*
|
||||||
|
* Backend-neutral: see {@link AddToVectorIndexOperation} — `index` may be the
|
||||||
|
* JS HNSW fallback or a native acceleration provider; the emitted `name`
|
||||||
|
* stamps the active backend.
|
||||||
|
*
|
||||||
|
* Why this op exists: update flows historically staged a
|
||||||
|
* {@link RemoveFromVectorIndexOperation} followed by an
|
||||||
|
* {@link AddToVectorIndexOperation} as two separately-awaited operations.
|
||||||
|
* Between them the row was in NEITHER index — dark to semantic recall while
|
||||||
|
* perfectly visible to metadata reads (a transient-invisibility window that
|
||||||
|
* stretched to seconds in a production deployment). The structural cure is a
|
||||||
|
* single leg that never removes without simultaneously re-inserting.
|
||||||
|
*
|
||||||
|
* Execution strategy (feature-detected, in preference order):
|
||||||
|
* 1. Provider exposes `updateItem` → ONE in-place call. The provider swaps
|
||||||
|
* the vector without the row ever leaving its index, and an element-wise
|
||||||
|
* UNCHANGED vector (the type-only-update production shape) is a pure
|
||||||
|
* no-op on its side.
|
||||||
|
* 2. Provider without `updateItem` (a native provider that has not shipped
|
||||||
|
* it yet) → `removeItem` + `addItem` executed ADJACENT within this single
|
||||||
|
* op. Still strictly better than the historical pair: no other transaction
|
||||||
|
* operation can interleave between the two calls. This is a temporary
|
||||||
|
* seam — the native side of the pair is expected to ship its own
|
||||||
|
* `updateItem` so path 1 applies everywhere; when it does, this fallback
|
||||||
|
* becomes dead code that costs nothing.
|
||||||
|
*
|
||||||
|
* Rollback strategy (mirrors the execute branch that ran):
|
||||||
|
* - `updateItem` path → `updateItem` back to `oldVector`.
|
||||||
|
* - Fallback path → `removeItem` + `addItem` back to `oldVector`.
|
||||||
|
*
|
||||||
|
* Rollback semantics when the item did not exist at execute time: this op's
|
||||||
|
* contract is that the caller read the entity and its CURRENT vector
|
||||||
|
* (`oldVector`) before staging — update flows only stage it for existing
|
||||||
|
* rows. If the item was somehow absent, execute() inserts it (`updateItem`
|
||||||
|
* delegates to add; the fallback's remove is a no-op before its add), and
|
||||||
|
* rollback restores `oldVector` rather than removing — the same posture as
|
||||||
|
* {@link RemoveFromVectorIndexOperation}'s unconditional re-add: by
|
||||||
|
* constructing the op with `oldVector` the caller DECLARED the before-state,
|
||||||
|
* and rollback reconstructs that declared state instead of silently deciding
|
||||||
|
* the row should vanish.
|
||||||
|
*/
|
||||||
|
export class ReplaceInVectorIndexOperation implements Operation {
|
||||||
|
readonly name: string
|
||||||
|
|
||||||
|
constructor(
|
||||||
|
private readonly index: VectorIndexProvider,
|
||||||
|
private readonly id: string,
|
||||||
|
private readonly oldVector: number[], // Required for rollback
|
||||||
|
private readonly newVector: number[]
|
||||||
|
) {
|
||||||
|
this.name = `ReplaceInVectorIndex(${resolveVectorProviderId(index)})`
|
||||||
|
}
|
||||||
|
|
||||||
|
async execute(): Promise<RollbackAction> {
|
||||||
|
// Feature-detect the in-place capability — optional on the provider
|
||||||
|
// contract, like `getItem`/`setPersistMode` (Brainy's JS HNSW index
|
||||||
|
// ships it; a native provider may not have yet).
|
||||||
|
const index = this.index as VectorIndexProvider & {
|
||||||
|
updateItem?: (item: { id: string; vector: number[] }) => Promise<void>
|
||||||
|
}
|
||||||
|
|
||||||
|
if (typeof index.updateItem === 'function') {
|
||||||
|
// Atomic path: one in-place call, the row never leaves the index.
|
||||||
|
await index.updateItem({ id: this.id, vector: this.newVector })
|
||||||
|
|
||||||
|
return async () => {
|
||||||
|
// Restore the declared before-state in place (see class JSDoc for
|
||||||
|
// the item-did-not-exist posture).
|
||||||
|
await index.updateItem!({ id: this.id, vector: this.oldVector })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fallback seam: remove+add ADJACENT within this single op — no other
|
||||||
|
// transaction operation can interleave between them (see class JSDoc).
|
||||||
|
await this.index.removeItem(this.id)
|
||||||
|
await this.index.addItem({ id: this.id, vector: this.newVector })
|
||||||
|
|
||||||
|
return async () => {
|
||||||
|
// updateItem-style restore via the same adjacent pair, back to the
|
||||||
|
// declared before-state.
|
||||||
|
await this.index.removeItem(this.id)
|
||||||
|
await this.index.addItem({ id: this.id, vector: this.oldVector })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Add to metadata index with rollback support
|
* Add to metadata index with rollback support
|
||||||
*
|
*
|
||||||
|
|
|
||||||
|
|
@ -23,6 +23,7 @@ export {
|
||||||
export {
|
export {
|
||||||
AddToVectorIndexOperation,
|
AddToVectorIndexOperation,
|
||||||
RemoveFromVectorIndexOperation,
|
RemoveFromVectorIndexOperation,
|
||||||
|
ReplaceInVectorIndexOperation,
|
||||||
AddToMetadataIndexOperation,
|
AddToMetadataIndexOperation,
|
||||||
RemoveFromMetadataIndexOperation,
|
RemoveFromMetadataIndexOperation,
|
||||||
AddToGraphIndexOperation,
|
AddToGraphIndexOperation,
|
||||||
|
|
|
||||||
75
tests/unit/brainy/lazy-notready-honor.test.ts
Normal file
75
tests/unit/brainy/lazy-notready-honor.test.ts
Normal file
|
|
@ -0,0 +1,75 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/brainy/lazy-notready-honor
|
||||||
|
* @description THE SILENT-EMPTY TRAP pin (found during a fleet adoption,
|
||||||
|
* SELF-ENGINE-PAIR-STANDARD): under `disableAutoRebuild: true`, the lazy
|
||||||
|
* first-query path (`ensureIndexesLoaded`) assessed ONLY the vector index's
|
||||||
|
* readiness — a native METADATA provider reporting not-ready (its strand
|
||||||
|
* report) never blocked the completion latch, so the promised lazy rebuild
|
||||||
|
* never fired and every `find()` silently returned `[]` on a populated
|
||||||
|
* store (measured: 52 entities durable-but-unqueryable, first query
|
||||||
|
* 0ms/0 rows). The law: a not-ready report from ANY provider falls through
|
||||||
|
* to the rebuild — never a silent empty.
|
||||||
|
*
|
||||||
|
* White-box provider-double pattern per tests/unit/brainy/migration-deference.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect, afterEach, vi } from 'vitest'
|
||||||
|
import { Brainy } from '../../../src/index.js'
|
||||||
|
import { NounType } from '../../../src/types/graphTypes.js'
|
||||||
|
import { createTestConfig } from '../../helpers/test-factory.js'
|
||||||
|
|
||||||
|
interface BrainInternals {
|
||||||
|
index: { size(): number }
|
||||||
|
metadataIndex: { isReady?: () => boolean }
|
||||||
|
lazyRebuildCompleted: boolean
|
||||||
|
ensureIndexesLoaded(): Promise<void>
|
||||||
|
rebuildIndexesIfNeeded(force?: boolean): Promise<void>
|
||||||
|
}
|
||||||
|
|
||||||
|
const brains: Brainy[] = []
|
||||||
|
|
||||||
|
afterEach(async () => {
|
||||||
|
for (const b of brains.splice(0)) await b.close().catch(() => {})
|
||||||
|
vi.restoreAllMocks()
|
||||||
|
})
|
||||||
|
|
||||||
|
async function warmLazyBrain(): Promise<{ brain: Brainy; internals: BrainInternals }> {
|
||||||
|
const brain = new Brainy(createTestConfig({ disableAutoRebuild: true }))
|
||||||
|
await brain.init()
|
||||||
|
brains.push(brain)
|
||||||
|
for (let i = 0; i < 3; i++) {
|
||||||
|
await brain.add({ data: `row ${i}`, type: NounType.Document, metadata: { i } })
|
||||||
|
}
|
||||||
|
const internals = brain as unknown as BrainInternals
|
||||||
|
internals.lazyRebuildCompleted = false // simulate the cold first query
|
||||||
|
return { brain, internals }
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('lazy path honors EVERY provider’s not-ready report', () => {
|
||||||
|
it('a not-ready METADATA provider blocks the completion latch and fires the rebuild', async () => {
|
||||||
|
const { internals } = await warmLazyBrain()
|
||||||
|
|
||||||
|
// The trap's shape: vector side looks fine (populated), metadata
|
||||||
|
// provider says NOT ready — the old gate latched complete here.
|
||||||
|
;(internals.metadataIndex as { isReady?: () => boolean }).isReady = () => false
|
||||||
|
const rebuildSpy = vi
|
||||||
|
.spyOn(internals, 'rebuildIndexesIfNeeded')
|
||||||
|
.mockResolvedValue(undefined)
|
||||||
|
|
||||||
|
await internals.ensureIndexesLoaded()
|
||||||
|
|
||||||
|
expect(rebuildSpy, 'not-ready metadata provider must fire the lazy rebuild').toHaveBeenCalledWith(true)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('control: all providers ready/unknown+populated → latch completes, no rebuild', async () => {
|
||||||
|
const { internals } = await warmLazyBrain()
|
||||||
|
;(internals.metadataIndex as { isReady?: () => boolean }).isReady = () => true
|
||||||
|
const rebuildSpy = vi
|
||||||
|
.spyOn(internals, 'rebuildIndexesIfNeeded')
|
||||||
|
.mockResolvedValue(undefined)
|
||||||
|
|
||||||
|
await internals.ensureIndexesLoaded()
|
||||||
|
|
||||||
|
expect(rebuildSpy).not.toHaveBeenCalled()
|
||||||
|
expect(internals.lazyRebuildCompleted).toBe(true)
|
||||||
|
})
|
||||||
|
})
|
||||||
366
tests/unit/hnsw/update-item-atomic.test.ts
Normal file
366
tests/unit/hnsw/update-item-atomic.test.ts
Normal file
|
|
@ -0,0 +1,366 @@
|
||||||
|
/**
|
||||||
|
* @module tests/unit/hnsw/update-item-atomic
|
||||||
|
* @description Guard for the atomic vector-index update: a row must NEVER be
|
||||||
|
* absent from vector search during an update. The historical update path
|
||||||
|
* staged a remove followed by an add as two separately-awaited transaction
|
||||||
|
* operations — between them the row was in NEITHER index (dark to semantic
|
||||||
|
* recall while perfectly visible to metadata reads; observed as seconds-long
|
||||||
|
* flicker in a production deployment). The structural cure verified here:
|
||||||
|
*
|
||||||
|
* 1. `JsHnswVectorIndex.updateItem` — same vector (element-wise) is a pure
|
||||||
|
* no-op (the production flicker shape: a type-only update re-indexing an
|
||||||
|
* UNCHANGED vector); a changed vector swaps in place, the node never
|
||||||
|
* leaving the map (white-box probe at the first internal step after the
|
||||||
|
* synchronous swap), including when the node IS the entry point.
|
||||||
|
* 2. `ReplaceInVectorIndexOperation` — one transaction leg that prefers the
|
||||||
|
* provider's in-place `updateItem`, with a remove+add-ADJACENT fallback
|
||||||
|
* for providers that have not shipped it; rollback restores the declared
|
||||||
|
* before-vector on both branches.
|
||||||
|
* 3. The brain's update path — with the JS index carrying `updateItem`,
|
||||||
|
* `removeItem` is never called during `brain.update()`, for the
|
||||||
|
* type-only shape AND for a genuine vector change.
|
||||||
|
*/
|
||||||
|
import { describe, it, expect, vi } from 'vitest'
|
||||||
|
import { JsHnswVectorIndex } from '../../../src/hnsw/hnswIndex.js'
|
||||||
|
import { ReplaceInVectorIndexOperation } from '../../../src/transaction/operations/IndexOperations.js'
|
||||||
|
import type { VectorIndexProvider } from '../../../src/plugin.js'
|
||||||
|
import type { Vector, VectorDocument } from '../../../src/coreTypes.js'
|
||||||
|
import { euclideanDistance } from '../../../src/utils/index.js'
|
||||||
|
import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js'
|
||||||
|
import { Brainy } from '../../../src/brainy'
|
||||||
|
import { createAddParams, createTestConfig } from '../../helpers/test-factory'
|
||||||
|
|
||||||
|
const DIM = 8
|
||||||
|
|
||||||
|
function seededRand(seed: number): () => number {
|
||||||
|
let s = seed >>> 0
|
||||||
|
return () => {
|
||||||
|
s = (s + 0x6d2b79f5) | 0
|
||||||
|
let t = Math.imul(s ^ (s >>> 15), 1 | s)
|
||||||
|
t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t
|
||||||
|
return ((t ^ (t >>> 14)) >>> 0) / 4294967296
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A deterministic vector pointing in a pseudo-random direction (well-connected graph). */
|
||||||
|
function vec(idx: number): number[] {
|
||||||
|
const rand = seededRand(idx + 1)
|
||||||
|
return Array.from({ length: DIM }, () => rand() * 2 - 1)
|
||||||
|
}
|
||||||
|
|
||||||
|
type Noun = { id: string; vector: number[]; connections: Map<number, Set<string>>; level: number }
|
||||||
|
|
||||||
|
function nounsOf(index: JsHnswVectorIndex): Map<string, Noun> {
|
||||||
|
return (index as unknown as { nouns: Map<string, Noun> }).nouns
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Flatten a reverse index to sorted `target|level|source` triples. */
|
||||||
|
function triplesFromIncoming(inc: Map<string, Map<number, Set<string>>>): string[] {
|
||||||
|
const out: string[] = []
|
||||||
|
for (const [target, byLevel] of inc) {
|
||||||
|
for (const [level, sources] of byLevel) {
|
||||||
|
for (const source of sources) out.push(`${target}|${level}|${source}`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out.sort()
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Derive the ground-truth reverse index directly from the live forward adjacency. */
|
||||||
|
function triplesFromAdjacency(nouns: Map<string, Noun>): string[] {
|
||||||
|
const out: string[] = []
|
||||||
|
for (const [nodeId, node] of nouns) {
|
||||||
|
for (const [level, targets] of node.connections) {
|
||||||
|
for (const target of targets) out.push(`${target}|${level}|${nodeId}`)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out.sort()
|
||||||
|
}
|
||||||
|
|
||||||
|
function assertReverseIndexConsistent(index: JsHnswVectorIndex): void {
|
||||||
|
const live = (
|
||||||
|
index as unknown as { ensureIncoming: () => Map<string, Map<number, Set<string>>> }
|
||||||
|
).ensureIncoming()
|
||||||
|
expect(triplesFromIncoming(live)).toEqual(triplesFromAdjacency(nounsOf(index)))
|
||||||
|
}
|
||||||
|
|
||||||
|
function assertNoSelfLoops(index: JsHnswVectorIndex, id: string): void {
|
||||||
|
const node = nounsOf(index).get(id)!
|
||||||
|
for (const [level, targets] of node.connections) {
|
||||||
|
expect(targets.has(id), `self-loop at level ${level}`).toBe(false)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function makeIndex(M = 16): JsHnswVectorIndex {
|
||||||
|
return new JsHnswVectorIndex(
|
||||||
|
{ M, efConstruction: 200, efSearch: 64, ml: 16 },
|
||||||
|
euclideanDistance,
|
||||||
|
{ useParallelization: false, storage: new MemoryStorage() }
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
async function fillIndex(index: JsHnswVectorIndex, count: number): Promise<void> {
|
||||||
|
for (let i = 0; i < count; i++) {
|
||||||
|
await index.addItem({ id: `n-${i}`, vector: vec(i) })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
describe('JsHnswVectorIndex.updateItem — atomic in-place vector update', () => {
|
||||||
|
it('same vector (element-wise equal, fresh array) is a pure no-op: no remove, no relink, still searchable', async () => {
|
||||||
|
const index = makeIndex()
|
||||||
|
await fillIndex(index, 30)
|
||||||
|
|
||||||
|
const target = 'n-7'
|
||||||
|
const sameVector = [...vec(7)] // fresh array, identical elements
|
||||||
|
|
||||||
|
const before = await index.search(vec(7), 1)
|
||||||
|
expect(before[0][0]).toBe(target)
|
||||||
|
|
||||||
|
const removeSpy = vi.spyOn(index, 'removeItem')
|
||||||
|
const nodeBefore = nounsOf(index).get(target)!
|
||||||
|
const connectionsBefore = nodeBefore.connections // reference — a relink replaces it
|
||||||
|
|
||||||
|
await index.updateItem({ id: target, vector: sameVector })
|
||||||
|
|
||||||
|
expect(removeSpy).not.toHaveBeenCalled()
|
||||||
|
expect(index.size()).toBe(30)
|
||||||
|
// No relink happened: the connections map is the SAME object, untouched.
|
||||||
|
expect(nounsOf(index).get(target)!.connections).toBe(connectionsBefore)
|
||||||
|
|
||||||
|
const after = await index.search(vec(7), 1)
|
||||||
|
expect(after[0][0]).toBe(target)
|
||||||
|
expect(after[0][1]).toBeCloseTo(0, 10)
|
||||||
|
|
||||||
|
removeSpy.mockRestore()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('changed vector: node never leaves the map (probe fires after the synchronous swap), removeItem never called, findable by the NEW vector', async () => {
|
||||||
|
const index = makeIndex()
|
||||||
|
await fillIndex(index, 40)
|
||||||
|
|
||||||
|
const target = 'n-5'
|
||||||
|
const newVector = vec(500)
|
||||||
|
|
||||||
|
// White-box probe: ensureIncoming is the FIRST internal step of the unlink
|
||||||
|
// walk, i.e. the first thing updateItem does after the synchronous vector
|
||||||
|
// swap. At that instant the node must (a) still be in the map and (b)
|
||||||
|
// already carry the NEW vector — the visibility-atomic ordering.
|
||||||
|
const inner = index as unknown as {
|
||||||
|
nouns: Map<string, Noun>
|
||||||
|
ensureIncoming: () => Map<string, Map<number, Set<string>>>
|
||||||
|
}
|
||||||
|
const origEnsure = inner.ensureIncoming.bind(index)
|
||||||
|
let probed = false
|
||||||
|
let presentDuring = false
|
||||||
|
let swappedFirst = false
|
||||||
|
;(index as any).ensureIncoming = function () {
|
||||||
|
if (!probed) {
|
||||||
|
probed = true
|
||||||
|
presentDuring = inner.nouns.has(target)
|
||||||
|
swappedFirst = inner.nouns.get(target)?.vector === newVector
|
||||||
|
}
|
||||||
|
return origEnsure()
|
||||||
|
}
|
||||||
|
|
||||||
|
const removeSpy = vi.spyOn(index, 'removeItem')
|
||||||
|
await index.updateItem({ id: target, vector: newVector })
|
||||||
|
delete (index as any).ensureIncoming // restore the prototype method
|
||||||
|
|
||||||
|
expect(probed).toBe(true)
|
||||||
|
expect(presentDuring).toBe(true)
|
||||||
|
expect(swappedFirst).toBe(true)
|
||||||
|
expect(removeSpy).not.toHaveBeenCalled()
|
||||||
|
expect(index.size()).toBe(40)
|
||||||
|
expect(nounsOf(index).has(target)).toBe(true)
|
||||||
|
|
||||||
|
// Findable by search with the NEW vector, at distance ~0.
|
||||||
|
const got = await index.search(newVector, 1)
|
||||||
|
expect(got[0][0]).toBe(target)
|
||||||
|
expect(got[0][1]).toBeCloseTo(0, 10)
|
||||||
|
|
||||||
|
// The relink left the graph bookkeeping exactly consistent.
|
||||||
|
assertNoSelfLoops(index, target)
|
||||||
|
assertReverseIndexConsistent(index)
|
||||||
|
|
||||||
|
removeSpy.mockRestore()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('keeps the node at its existing level (never releveled by an update)', async () => {
|
||||||
|
const index = makeIndex()
|
||||||
|
await fillIndex(index, 30)
|
||||||
|
|
||||||
|
const target = 'n-3'
|
||||||
|
const levelBefore = nounsOf(index).get(target)!.level
|
||||||
|
|
||||||
|
await index.updateItem({ id: target, vector: vec(600) })
|
||||||
|
|
||||||
|
expect(nounsOf(index).get(target)!.level).toBe(levelBefore)
|
||||||
|
expect(index.getMaxLevel()).toBeGreaterThanOrEqual(levelBefore)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('updating the ENTRY POINT in place keeps it valid — entry id and maxLevel unchanged, graph never stranded', async () => {
|
||||||
|
const index = makeIndex()
|
||||||
|
await fillIndex(index, 40)
|
||||||
|
|
||||||
|
const entryId = index.getEntryPointId()!
|
||||||
|
const maxLevelBefore = index.getMaxLevel()
|
||||||
|
const newVector = vec(700)
|
||||||
|
|
||||||
|
await index.updateItem({ id: entryId, vector: newVector })
|
||||||
|
|
||||||
|
// Entry-point bookkeeping must not regress.
|
||||||
|
expect(index.getEntryPointId()).toBe(entryId)
|
||||||
|
expect(index.getMaxLevel()).toBe(maxLevelBefore)
|
||||||
|
expect(index.size()).toBe(40)
|
||||||
|
|
||||||
|
// The entry point itself is findable by its new vector...
|
||||||
|
const gotEntry = await index.search(newVector, 1)
|
||||||
|
expect(gotEntry[0][0]).toBe(entryId)
|
||||||
|
|
||||||
|
// ...and the REST of the graph is still reachable through it (a stranded,
|
||||||
|
// edgeless entry point would make every other node invisible).
|
||||||
|
const otherId = [...nounsOf(index).keys()].find((id) => id !== entryId)!
|
||||||
|
const otherIdx = Number(otherId.slice(2))
|
||||||
|
const gotOther = await index.search(vec(otherIdx), 1)
|
||||||
|
expect(gotOther[0][0]).toBe(otherId)
|
||||||
|
|
||||||
|
assertNoSelfLoops(index, entryId)
|
||||||
|
assertReverseIndexConsistent(index)
|
||||||
|
})
|
||||||
|
|
||||||
|
it('absent id delegates to addItem (plain insert)', async () => {
|
||||||
|
const index = makeIndex()
|
||||||
|
await fillIndex(index, 10)
|
||||||
|
|
||||||
|
await index.updateItem({ id: 'fresh', vector: vec(900) })
|
||||||
|
|
||||||
|
expect(index.size()).toBe(11)
|
||||||
|
const got = await index.search(vec(900), 1)
|
||||||
|
expect(got[0][0]).toBe('fresh')
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('ReplaceInVectorIndexOperation — one atomic transaction leg', () => {
|
||||||
|
it('uses the provider updateItem path and rolls back to the old vector in place', async () => {
|
||||||
|
const index = makeIndex()
|
||||||
|
await fillIndex(index, 30)
|
||||||
|
|
||||||
|
const target = 'n-9'
|
||||||
|
const oldVector = vec(9)
|
||||||
|
const newVector = vec(800)
|
||||||
|
|
||||||
|
const removeSpy = vi.spyOn(index, 'removeItem')
|
||||||
|
const op = new ReplaceInVectorIndexOperation(index, target, oldVector, newVector)
|
||||||
|
expect(op.name).toBe('ReplaceInVectorIndex(hnsw-js)')
|
||||||
|
|
||||||
|
const rollback = await op.execute()
|
||||||
|
expect(removeSpy).not.toHaveBeenCalled()
|
||||||
|
expect((await index.search(newVector, 1))[0][0]).toBe(target)
|
||||||
|
|
||||||
|
await rollback()
|
||||||
|
expect(removeSpy).not.toHaveBeenCalled()
|
||||||
|
expect(index.size()).toBe(30)
|
||||||
|
|
||||||
|
// Old vector restored, element-wise, and searchable again.
|
||||||
|
const restored = nounsOf(index).get(target)!.vector
|
||||||
|
expect(restored.length).toBe(oldVector.length)
|
||||||
|
for (let i = 0; i < oldVector.length; i++) {
|
||||||
|
expect(restored[i]).toBe(oldVector[i])
|
||||||
|
}
|
||||||
|
const back = await index.search(oldVector, 1)
|
||||||
|
expect(back[0][0]).toBe(target)
|
||||||
|
expect(back[0][1]).toBeCloseTo(0, 10)
|
||||||
|
|
||||||
|
removeSpy.mockRestore()
|
||||||
|
})
|
||||||
|
|
||||||
|
it('falls back to remove+add ADJACENT within the single op for a provider without updateItem, and rolls back the same way', async () => {
|
||||||
|
// A provider that has not shipped updateItem — the temporary seam: the
|
||||||
|
// pair stays adjacent inside ONE op (no other transaction operation can
|
||||||
|
// interleave), until the provider ships its own in-place updateItem.
|
||||||
|
const calls: string[] = []
|
||||||
|
const store = new Map<string, Vector>()
|
||||||
|
const legacyProvider = {
|
||||||
|
name: 'legacy-native',
|
||||||
|
addItem: async (item: VectorDocument) => {
|
||||||
|
calls.push(`add:${item.id}`)
|
||||||
|
store.set(item.id, item.vector)
|
||||||
|
return item.id
|
||||||
|
},
|
||||||
|
removeItem: async (id: string) => {
|
||||||
|
calls.push(`remove:${id}`)
|
||||||
|
return store.delete(id)
|
||||||
|
},
|
||||||
|
search: async () => [],
|
||||||
|
size: () => store.size,
|
||||||
|
clear: () => store.clear(),
|
||||||
|
rebuild: async () => {},
|
||||||
|
flush: async () => 0,
|
||||||
|
getPersistMode: () => 'immediate' as const
|
||||||
|
} as unknown as VectorIndexProvider
|
||||||
|
|
||||||
|
store.set('x', [1, 0])
|
||||||
|
const op = new ReplaceInVectorIndexOperation(legacyProvider, 'x', [1, 0], [0, 1])
|
||||||
|
|
||||||
|
const rollback = await op.execute()
|
||||||
|
expect(calls).toEqual(['remove:x', 'add:x'])
|
||||||
|
expect(store.get('x')).toEqual([0, 1])
|
||||||
|
|
||||||
|
await rollback()
|
||||||
|
expect(calls).toEqual(['remove:x', 'add:x', 'remove:x', 'add:x'])
|
||||||
|
expect(store.get('x')).toEqual([1, 0])
|
||||||
|
})
|
||||||
|
})
|
||||||
|
|
||||||
|
describe('brain.update() — the update path stages ONE atomic vector-index leg', () => {
|
||||||
|
it('a type-only update (unchanged vector — the production flicker shape) never calls removeItem on the vector index', async () => {
|
||||||
|
const brain = new Brainy(createTestConfig())
|
||||||
|
await brain.init()
|
||||||
|
try {
|
||||||
|
const id = await brain.add(
|
||||||
|
createAddParams({ data: 'atomic flicker guard entity', type: 'thing' })
|
||||||
|
)
|
||||||
|
|
||||||
|
const index = (brain as unknown as { index: JsHnswVectorIndex }).index
|
||||||
|
const removeSpy = vi.spyOn(index, 'removeItem')
|
||||||
|
const sizeBefore = index.size()
|
||||||
|
|
||||||
|
await brain.update({ id, type: 'document' })
|
||||||
|
|
||||||
|
expect(removeSpy).not.toHaveBeenCalled()
|
||||||
|
expect(index.size()).toBe(sizeBefore)
|
||||||
|
|
||||||
|
const updated = await brain.get(id)
|
||||||
|
expect(updated).not.toBeNull()
|
||||||
|
expect(updated!.type).toBe('document')
|
||||||
|
|
||||||
|
removeSpy.mockRestore()
|
||||||
|
} finally {
|
||||||
|
await brain.close()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
|
||||||
|
it('a genuine vector change on update also never calls removeItem (in-place replace)', async () => {
|
||||||
|
const brain = new Brainy(createTestConfig())
|
||||||
|
await brain.init()
|
||||||
|
try {
|
||||||
|
const id = await brain.add(
|
||||||
|
createAddParams({ data: 'vector change stays visible', type: 'thing' })
|
||||||
|
)
|
||||||
|
const existing = await brain.get(id, { includeVectors: true })
|
||||||
|
// Same dimensionality, guaranteed-different content.
|
||||||
|
const changed = existing!.vector.map((x: number, i: number) => (i === 0 ? x + 0.25 : x))
|
||||||
|
|
||||||
|
const index = (brain as unknown as { index: JsHnswVectorIndex }).index
|
||||||
|
const removeSpy = vi.spyOn(index, 'removeItem')
|
||||||
|
|
||||||
|
await brain.update({ id, vector: changed })
|
||||||
|
|
||||||
|
expect(removeSpy).not.toHaveBeenCalled()
|
||||||
|
expect(nounsOf(index).has(id)).toBe(true)
|
||||||
|
|
||||||
|
removeSpy.mockRestore()
|
||||||
|
} finally {
|
||||||
|
await brain.close()
|
||||||
|
}
|
||||||
|
})
|
||||||
|
})
|
||||||
Loading…
Add table
Add a link
Reference in a new issue