From ebe06cdf33d1078a26f8f41f05f09a8b2659d8c4 Mon Sep 17 00:00:00 2001 From: David Snelling Date: Wed, 5 Aug 2026 16:11:23 -0700 Subject: [PATCH] =?UTF-8?q?fix(index):=20the=20flicker=20window=20dies=20?= =?UTF-8?q?=E2=80=94=20atomic=20in-place=20vector=20update;=20lazy=20open?= =?UTF-8?q?=20honors=20every=20provider's=20not-ready=20report;=20the=20Pa?= =?UTF-8?q?th=20Registry=20twin=20table?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DP6/DP8 of the Path Registry (BRAINY-PROD-LATENCY-TRIAD, the proven flicker mechanism): update paths staged RemoveFromVectorIndex then AddToVectorIndex as two separately-awaited transaction ops — between them a live row was in NEITHER index (dark to semantic recall, fine in metadata list). The native pair widened that window to seconds in production before their side's visibility-commit fix; the structural cure lands here: - hnswIndex.updateItem: absent → add; SAME vector → pure no-op (the production shape — a type-only update re-indexed an unchanged vector, remove+add did pure damage); changed vector → the node NEVER leaves the index: synchronous vector swap first (every query from that instant sees correct distances), then unlink/relink at the node's existing level via shared internals (linkNode/unlinkNodeEdges refactored out of add/remove; entry point and maxLevel provably unchanged). - ReplaceInVectorIndexOperation: ONE transaction leg; feature-detects provider updateItem (native seam flagged — their side ships updateItem, then the adjacent remove+add fallback is dead code). Both update staging sites swapped; delete sites untouched. - LAZY-OPEN GATE (fleet adoption find, SELF-ENGINE-PAIR-STANDARD): under disableAutoRebuild, ensureIndexesLoaded assessed ONLY the vector index — a not-ready native METADATA provider never blocked the completion latch and every find() silently returned [] on a populated store. All three providers now vote; any not-ready report falls through to the rebuild. - docs/path-registry.md: brainy's twin table for the 32 shared path IDs — service class, budgets, lifecycle, narration, and the cited pin per row; owed rows named (LC4 doors-open migration, MT4 yielding heals, LC7 downgrade contract) per the lifecycle-sprint choreography. Pins: update-item-atomic 9/9 (visibility-atomic swap, reverse-index parity vs fresh rebuild, entry-point invariants) · lazy-notready-honor 2/2. Gates: unit 1928/1928 (148 files) · integration 760 · conformance 27/27. --- docs/path-registry.md | 85 ++++ src/brainy.ts | 40 +- src/hnsw/hnswIndex.ts | 338 +++++++++++++--- src/transaction/operations/IndexOperations.ts | 89 +++++ src/transaction/operations/index.ts | 1 + tests/unit/brainy/lazy-notready-honor.test.ts | 75 ++++ tests/unit/hnsw/update-item-atomic.test.ts | 366 ++++++++++++++++++ 7 files changed, 937 insertions(+), 57 deletions(-) create mode 100644 docs/path-registry.md create mode 100644 tests/unit/brainy/lazy-notready-honor.test.ts create mode 100644 tests/unit/hnsw/update-item-atomic.test.ts diff --git a/docs/path-registry.md b/docs/path-registry.md new file mode 100644 index 00000000..a8c694ac --- /dev/null +++ b/docs/path-registry.md @@ -0,0 +1,85 @@ +# The Path Registry — brainy's twin table + +The brainy half of the cross-engine Path Registry (the native accelerator +maintains the master list; IDs are shared and stable — `LC3`, `DP7`, … are +citable in commits, board rounds, release notes, and pins). Every row owes +five things: **service class** (INDEX-SERVED | BOUNDED-FALLBACK, announced | +TYPED REFUSAL), **latency budget** at 1k/10k/100k/1M (design bar: billions), +**lifecycle behavior**, **failure narration**, and a **test pin**. A path not +in this registry does not ship; an unregistered path is a red gate in the +scan audit. + +**The availability bar governing every row: user-visible downtime is +seconds, at restart only.** Migration, heal, compaction, embedding, and +retention run behind the doors — yielding, budget-capped, narrated. No path +may hold the doors while it does housekeeping. + +Status legend: ✅ contracted + pinned (test cited) · 🟡 partial (what holds +and what's missing, stated) · 🔴 owed (named, never silent). + +## LC — Lifecycle + +| ID | Brainy row | Status | +|----|-----------|--------| +| LC1 | Same-version reopen adopts everything: brain-format epoch match → zero rebuilds; aggregation state adopts by stamp; persisted indexes load. | ✅ `tests/unit/brainy/brain-format-handshake` + `migration-deference` (no-drift reopen never rebuilds) | +| LC2 | New empty brain: doors immediate. | ✅ exercised by every suite's setup | +| LC3 | Upgrade, same epoch: as LC1 — new code on unchanged formats owes nothing at open. | ✅ same pins as LC1 (epoch equality is the gate) | +| LC4 | Upgrade with epoch migration: TODAY brainy's epoch rebuild runs at open before doors. | 🔴 **owed — the sev's lockout row.** The doors-open-serving-old-structures design (yielding installments + atomic swap) lands measured-and-gated behind the service-class pair, per the lifecycle-sprint choreography. Acceptance case: the 9,184-row hours-lockout. | +| LC5 | Crash recovery: bounded, resumable, narrated. Aggregation leg ✅ (behind-stamp → incremental catch-up off the fact log + time-travel reconciliation, capped at 5,000 affected before an ANNOUNCED rescan). Vector/metadata legs ride epoch machinery (rebuild-from-canonical, narrated). | 🟡 aggregation pinned (`tests/integration/aggregation-lifecycle-catchup`); the rebuild legs are narrated but not yet installment-yielding (couples to LC4) | +| LC6 | Shutdown under load: close() drains the background flush flight, tears down cadence timers, runs ONE time-bounded compaction pass (~5s budget, resumable). | 🟡 pinned for flush/compaction (8.9.0 suites); SIGTERM drain budget not yet declared | +| LC7 | Rollback/downgrade: an N−1 build opening an N brain. | 🔴 owed — no declared read-compat window or typed refusal today (epoch mismatch triggers a rebuild, not a refusal; v2 nested-bag records read as a phantom user field on pre-law builds). Needs the declared-window contract. | +| LC8 | Relocatable brain directory: no absolute paths in artifacts; persist()/load() round-trips. | 🟡 persist/load pinned; byte-for-byte relocation depot cases are the pair gate's (shared corpora) | +| LC9 | Double-open: second writer gets a typed lock refusal (PID-liveness + heartbeat stale detection; `force` escape hatch logs loudly). | ✅ writer-lock suites (8.7.1) | + +## DP — Data plane + +| ID | Brainy row | Status | +|----|-----------|--------| +| DP1 | `get()` by id: direct storage read + hydrate. INDEX-SERVED (id-mapped). Milliseconds at every scale. | ✅ exercised everywhere; budget rides the pair speed table | +| DP2 | `find({query})`: embed + vector search. The embed dominates (native side owns the budget); JS HNSW serves the search leg. | 🟡 300ms-class p95 is the pair speed-table row; brainy-alone budget declared there | +| DP3 | Filtered/sorted list: column top-K when the field is columnized (INDEX-SERVED, zero canonical reads on the sorted page — value pairs come from ONE batched metadata-record pass); no-column fallback is BOUNDED-ANNOUNCED (one batch pass, announces once per field past 500 rows); unknown field → TYPED REFUSAL naming both candidate spellings. | ✅ `tests/unit/utils/metadataIndex-sort-callshape` (zero per-row reads, batch-only — latency-blind) + `metadataIndex-nested-orderby` (dotted keys serve-or-refuse) + `tests/integration/orderby-sort-bug` | +| DP4 | Aggregation/stats: ALWAYS answers. Write-time incremental; behind-stamp reconciles incrementally; genuine rebuilds go through the native parallel door or the paged JS walk; nothing ever latches off; before-image-less deletes flag a LOUD rescan, never a silent skip. | ✅ `tests/integration/aggregation-lifecycle-catchup` + `tests/unit/aggregation/aggregation-provider-rebuild` | +| DP5 | Graph traversal: `related()` paged via adjacency; whole-graph analytics carry declared cost. | 🟡 paged reads pinned; analytics cost-class declaration owed (rides VENUE-GRAPH-TRUST audit tool) | +| DP6 | Single write: ack at the canonical commit; visibility committed at ack (the atomic vector update kills the remove→add dark window); maintenance NEVER holds the ack (background flush cadence — THE ACK LAW pin: a hung flush cannot block a write). | 🟡 ack law pinned (`tests/unit/brainy/persistence-policy`); atomic-update pin lands with the flicker fix in this train | +| DP7 | Bulk ingest: sustained rate holds flat — per-write maintenance taxes must not grow with brain size (A4 removed caller-flush convoys; deferred embedding removes the per-write embed tax where opted). | 🟡 the decay-curve row is a pair speed-table RED GATE; brainy-alone sustained-rate run rides the same corpora | +| DP8 | Read under write pressure: no flicker window — a row that exists is never invisible to recall, even transiently (same-vector re-index is a no-op; changed-vector swaps in place, node never leaves the index). | 🟡 lands in this train (atomic `updateItem` + `ReplaceInVectorIndexOperation`); symmetry suite + sentinels are the B4 program | +| — | **The lazy-open gate honors EVERY provider's not-ready report** (a not-ready metadata provider can no longer latch the silent-empty state under `disableAutoRebuild`). | ✅ `tests/unit/brainy/lazy-notready-honor` | + +## MT — Maintenance (never in the door path) + +| ID | Brainy row | Status | +|----|-----------|--------| +| MT1 | Flush/checkpoint: ENGINE-OWNED cadence (write-count/interval/idle triggers, single-flight, background, loud on failure; callers never flush in hot paths; `flush()` stays as an awaitable barrier). | ✅ `tests/unit/brainy/persistence-policy` | +| MT2 | Compaction: never on flush (durability-only law, 8.9.0); close-time pass time-budgeted + resumable; explicit `compactHistory({timeBudgetMs})`. | ✅ 8.9.0 suites | +| MT3 | Index upkeep (mapper folds, delta promotion): native-side machinery; brainy's JS legs are small and synchronous-cheap. | 🟡 declared; yield audit rides the pair | +| MT4 | Heal/rebuild walks (`repairIndex`, backfill walks): paged; failure latches with cooldown; NOT yet yield-to-foreground installments. | 🔴 owed — the priority-isolation clause (couples to LC4; same choreography) | +| MT5 | Deferred embedding worker: ack at durability, durable pending markers, crash-recovered at open, single-flight batches. | 🔴 lands as A3 in this train (design frozen on the incident thread) | +| MT6 | Retention/archival walks: retention `'all'` does nothing by design; bounded-retention reclaim is close-time/explicit only. | 🟡 8.9.0 behavior pinned; archival profile is the co-frozen D1+D3 unit | + +## FM — Failure modes + +| ID | Brainy row | Status | +|----|-----------|--------| +| FM1 | Disk full / IO error mid-op: transaction rollback + typed error; failed rollback → StoreInconsistentError quarantines writes until repairIndex(). | 🟡 rollback paths pinned; explicit disk-full depot case owed | +| FM2 | Memory pressure: query limits + reserved-memory config; unified cache eviction. | 🟡 declared budgets; cascade pin owed | +| FM3 | Torn/corrupt file on open: malformed brain-format marker → safe rebuild (never trusting a bad epoch); corrupt records surface loudly. | 🟡 marker pin ✅ (`brain-format-handshake`); broader quarantine is native-side | +| FM4 | Native module unavailable: plugin load failure is LOUD (version-coupling law throws on range mismatch — never silently version-drifted); JS engine serves with its own declared budgets, named as the active backend in op names. | ✅ `tests/unit/plugin-version-coupling` + op-name stamping | + +## FL — Fleet + +| ID | Brainy row | Status | +|----|-----------|--------| +| FL1 | Cold open on demand: LC1's adopt-everything open; warm() available for eager paths. | 🟡 open cost pinned at LC1; millisecond budget rides the speed table | +| FL2–FL4 | Boot storm / upgrade wave / isolation: fleet-layer policies over LC1/LC4 — engine leg = budgeted opens + LC4's behind-doors migration. | 🔴 owed with LC4 | +| FL5 | Brain as product object: create instant (LC2) · erase = `clear()` explicit + complete · export = portable-graph, canon-complete mode available. | ✅ clear-persistence + portable-graph + canonical-enumeration suites | + +## Status summary + +Contracted + pinned this train: **DP3, DP4, MT1, LC5(aggregation), the +lazy-open not-ready gate, LC1/LC3/LC9, FM4, FL5** — each with the cited +test. Landing in this train: **DP6/DP8 (atomic vector update), MT5 (A3 +deferred embedding)**. Owed, in production-risk order, all coupled to the +priority-isolation program the lifecycle sev opened: **LC4 (doors-open +migration), MT4 (yielding heals), LC7 (downgrade contract), LC6 (SIGTERM +budget), FL2–FL4, FM1/FM2 depot cases.** Rows move from owed to contracted +only with a cited test — none lands by prose. diff --git a/src/brainy.ts b/src/brainy.ts index 6d3a7927..3ad8ba31 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -97,6 +97,7 @@ import { SaveVerbOperation, AddToGraphIndexOperation, RemoveFromVectorIndexOperation, + ReplaceInVectorIndexOperation, RemoveFromMetadataIndexOperation, RemoveFromGraphIndexOperation, UpdateNounMetadataOperation, @@ -2949,11 +2950,16 @@ export class Brainy implements BrainyInterface { level: 0 }) ) + // ONE atomic vector-index leg: the historical Remove→Add pair was + // two separately-awaited operations — between them the row was in + // NEITHER index (dark to semantic recall, visible to metadata + // reads). ReplaceInVectorIndexOperation goes through the provider's + // in-place updateItem when available (row never absent; an + // element-wise UNCHANGED vector — the type-only-update shape that + // flickered in production — is a pure no-op), else remove+add + // adjacent within the single op. tx.addOperation( - new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector) - ) - tx.addOperation( - new AddToVectorIndexOperation(this.index, params.id, vector) + new ReplaceInVectorIndexOperation(this.index, params.id, existing.vector, vector) ) } @@ -9364,8 +9370,10 @@ export class Brainy implements BrainyInterface { connections: new Map(), level: 0 }), - new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector), - new AddToVectorIndexOperation(this.index, params.id, vector) + // ONE atomic vector-index leg — same law as update(): the row must + // never be absent from vector search during an update (see + // ReplaceInVectorIndexOperation). + new ReplaceInVectorIndexOperation(this.index, params.id, existing.vector, vector) ) } plan.operations.push( @@ -14958,14 +14966,30 @@ export class Brainy implements BrainyInterface { } // If indexes already populated AND honestly serving, mark complete and skip. - // Honest gate: when the provider exposes isReady(), that REPLACES the size()>0 + // Honest gate: when a provider exposes isReady(), that REPLACES the size()>0 // proxy (a native index can report a non-zero size while its serving structure // is not loaded — the silent-empty cold-load class). A not-ready provider falls // through so the rebuild path can load it; verifyVectorLive() is the query-time // backstop either way. Providers without isReady() keep the size() heuristic // (the JS index's size()>0 genuinely means loaded). + // + // ALL THREE providers vote (fleet-adoption find, SELF-ENGINE-PAIR-STANDARD): + // this gate used to assess ONLY the vector index, so a not-ready native + // METADATA provider (its strand report) never blocked the completion latch + // — under disableAutoRebuild the promised lazy first-query rebuild never + // fired and every find() silently returned [] on a populated store. A + // not-ready report from ANY provider now falls through to the rebuild. const vectorReadiness = assessIndexReadiness(this.index) - if (vectorReadiness === 'ready' || (vectorReadiness === 'unknown' && this.index.size() > 0)) { + const metadataReadiness = assessIndexReadiness(this.metadataIndex) + const graphReadiness = assessIndexReadiness(this.graphIndex) + const anyProviderNotReady = + vectorReadiness === 'not-ready' || + metadataReadiness === 'not-ready' || + graphReadiness === 'not-ready' + if ( + !anyProviderNotReady && + (vectorReadiness === 'ready' || (vectorReadiness === 'unknown' && this.index.size() > 0)) + ) { this.lazyRebuildCompleted = true return } diff --git a/src/hnsw/hnswIndex.ts b/src/hnsw/hnswIndex.ts index eb2acd71..a5b8e834 100644 --- a/src/hnsw/hnswIndex.ts +++ b/src/hnsw/hnswIndex.ts @@ -486,6 +486,90 @@ export class JsHnswVectorIndex implements VectorIndexProvider { return id } + // Wire the node into the graph: greedy descent + per-level linking. + // Extracted to linkNode so updateItem's in-place relink runs the SAME + // insertion linking (one implementation, never a diverging copy). + await this.linkNode(noun, entryPoint) + + // Update max level and entry point if needed + if (nounLevel > this.maxLevel) { + this.maxLevel = nounLevel + this.entryPointId = id + } + + // Add noun to the index + this.nouns.set(id, noun) + + // Track high-level nodes for O(1) entry point selection + if (nounLevel >= 2 && nounLevel <= this.MAX_TRACKED_LEVELS) { + if (!this.highLevelNodes.has(nounLevel)) { + this.highLevelNodes.set(nounLevel, new Set()) + } + this.highLevelNodes.get(nounLevel)!.add(id) + } + + // Lazy vector eviction (B2: graph-only memory after insert) + // After graph construction completes, evict the full vector from memory. + // Future searches will load vectors on-demand via getVectorSafe() + UnifiedCache. + if (this.vectorStorageMode === 'lazy' && this.storage) { + noun.vector = [] // Release float32 vector from memory + } + + // Persist HNSW graph data to storage + // Respect persistMode setting + if (this.storage && this.persistMode === 'immediate') { + // IMMEDIATE MODE: Original behavior - persist new entity and system data. + // Goes through the per-node helper so the compressed-blob branch fires + // identically here vs. the deferred-flush + neighbor-update paths. + await this.persistNodeConnections(id, noun).catch((error) => { + console.error(`Failed to persist HNSW data for ${id}:`, error) + }) + + // Persist system data (entry point and max level) + await this.storage.saveHNSWSystem({ + entryPointId: this.entryPointId, + maxLevel: this.maxLevel + }).catch((error) => { + console.error('Failed to persist HNSW system data:', error) + }) + } else if (this.persistMode === 'deferred') { + // DEFERRED MODE: Track dirty nodes for later batch persistence + this.dirtyNodes.add(id) + this.dirtySystem = true + } + + return id + } + + /** + * @description The insertion LINKING phase shared by {@link addItem} and + * {@link updateItem}: greedy-descend from `entryPoint` through the levels + * above `noun.level`, then at each level from `min(noun.level, maxLevel)` + * down to 0 find `efConstruction` candidates, select the M nearest, and + * create bidirectional edges — maintaining the reverse-adjacency index via + * {@link addIncoming} and re-pruning any neighbor pushed over M. + * + * Persistence follows the caller's mode exactly as the historical inline + * addItem code did: `'immediate'` persists each touched neighbor's + * connections concurrently (batched by `maxConcurrentNeighborWrites`); + * `'deferred'` marks each touched neighbor dirty for the next flush. + * + * Does NOT touch index membership (`this.nouns`), the entry point, or + * `maxLevel` — the caller owns that bookkeeping: addItem inserts a NEW node + * afterwards and may raise maxLevel; updateItem relinks an EXISTING node in + * place whose level was already counted, so nothing may change. `noun.vector` + * must be the live in-memory vector at call time; both callers guarantee it + * (lazy-mode eviction happens only after linking completes). + * + * A `neighborId === noun.id` candidate is skipped defensively: during + * updateItem the node is already IN `this.nouns` (visibility-atomicity — + * unlike addItem, which links before inserting), and a self-edge must never + * be creatable no matter what the traversal surfaces. + */ + private async linkNode(noun: HNSWNoun, entryPoint: HNSWNoun): Promise { + const { id, vector } = noun + const nounLevel = noun.level + let currObj = entryPoint // Calculate distance to entry point (handles lazy loading + sync fast path) @@ -547,6 +631,10 @@ export class JsHnswVectorIndex implements VectorIndexProvider { }> = [] for (const [neighborId, _] of neighbors) { + if (neighborId === id) { + // Never self-link (see method JSDoc — reachable only via updateItem) + continue + } const neighbor = this.nouns.get(neighborId) if (!neighbor) { // Skip neighbors that don't exist (expected during rapid additions/deletions) @@ -630,7 +718,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider { const nearestNoun = this.nouns.get(nearestId) if (!nearestNoun) { console.error( - `Nearest noun with ID ${nearestId} not found in addItem` + `Nearest noun with ID ${nearestId} not found in linkNode` ) // Keep the current object as is } else { @@ -639,55 +727,173 @@ export class JsHnswVectorIndex implements VectorIndexProvider { } } } + } - // Update max level and entry point if needed - if (nounLevel > this.maxLevel) { - this.maxLevel = nounLevel - this.entryPointId = id + /** + * @description Atomically replace an item's vector IN PLACE — the row is + * NEVER absent from the index during an update. The historical shape staged + * a remove followed by an add as two separately-awaited transaction + * operations; between them the row was in NEITHER index — dark to semantic + * recall while perfectly visible to metadata reads (observed as seconds-long + * production flicker in a downstream deployment). Mandate: a row that + * exists must never be invisible to a read path, even transiently. + * + * Behavior: + * - id not in the index → delegates to {@link addItem} (plain insert). + * - SAME vector (element-wise equal) → pure no-op. This is the production + * flicker shape: a type-only update re-indexes an UNCHANGED vector, so the + * old remove+add did pure damage. (In lazy vector-storage mode the + * comparison baseline is whatever {@link getVectorSafe} serves — the + * cache, or the persisted record; if the caller already rewrote the + * record with the new vector before calling in, equality may report "no + * change" and skip the relink. Query correctness is unaffected either + * way — distances always use the live vector — the graph edges just keep + * their pre-update geometry, which HNSW tolerates by construction.) + * - DIFFERENT vector → the node never leaves `this.nouns`: + * 1. `node.vector` is swapped SYNCHRONOUSLY first (and the shared vector + * cache updated in the same tick), so from that point every query sees + * the node with correct distances; + * 2. its old edges are unlinked via the same reverse-adjacency walk + * removeItem uses ({@link unlinkNodeEdges}) — the node stays in the + * map and KEEPS its level; + * 3. the insertion linking re-runs at the node's EXISTING level + * ({@link linkNode}). Entry-point cases: if the node IS the entry + * point it REMAINS the entry point (still valid — same id, same + * level); the relink traversal then starts from another node via + * {@link resolveRelinkStart}, because the node's own edges were just + * cleared and a traversal starting AT it would find nothing and link + * nothing — stranding the whole graph behind an edgeless entry point. + * maxLevel never regresses: the node keeps its level and its + * membership, so the remove-side relevel bookkeeping never runs. + * + * Persistence mirrors {@link addItem}'s tail for the node itself plus the + * in-neighbors whose connection sets changed during the unlink: + * `'immediate'` persists their connections now; `'deferred'` marks them + * dirty for the next flush. The system record (entry point + maxLevel) is + * NOT rewritten — an in-place update changes neither. + */ + public async updateItem(item: VectorDocument): Promise { + if (!item) { + throw new Error('Item is undefined or null') + } + const { id, vector } = item + if (!vector) { + throw new Error('Vector is undefined or null') } - // Add noun to the index - this.nouns.set(id, noun) + const node = this.nouns.get(id) + if (!node) { + // Absent → plain insert. + await this.addItem(item) + return + } - // Track high-level nodes for O(1) entry point selection - if (nounLevel >= 2 && nounLevel <= this.MAX_TRACKED_LEVELS) { - if (!this.highLevelNodes.has(nounLevel)) { - this.highLevelNodes.set(nounLevel, new Set()) + if (this.dimension === null) { + this.dimension = vector.length + } else if (vector.length !== this.dimension) { + throw new Error( + `Vector dimension mismatch: expected ${this.dimension}, got ${vector.length}` + ) + } + + // Fast path: element-wise-equal vector → NOTHING to do (the production + // flicker shape — a type-only update re-indexing an unchanged vector). + // getVectorSafe handles the lazy-evicted case (loads from cache/storage). + const current = await this.getVectorSafe(node) + if (current.length === vector.length) { + let same = true + for (let i = 0; i < vector.length; i++) { + if (current[i] !== vector[i]) { + same = false + break + } } - this.highLevelNodes.get(nounLevel)!.add(id) + if (same) return } - // Lazy vector eviction (B2: graph-only memory after insert) - // After graph construction completes, evict the full vector from memory. - // Future searches will load vectors on-demand via getVectorSafe() + UnifiedCache. - if (this.vectorStorageMode === 'lazy' && this.storage) { - noun.vector = [] // Release float32 vector from memory + // (1) Visibility-atomic swap: from this synchronous assignment on, every + // query sees the node with correct distances. The shared vector cache is + // updated in the same tick so the lazy-mode read path can never serve the + // stale vector either. + node.vector = vector + this.unifiedCache.set(`hnsw:vector:${id}`, vector, 'vectors', vector.length * 4, 50) + + // (2) Unlink the old edges — the node stays in the map, keeps its level. + const touchedReferrers = await this.unlinkNodeEdges(node) + node.connections = new Map() + for (let level = 0; level <= node.level; level++) { + node.connections.set(level, new Set()) + } + // The node's own reverse entry is rebuilt by the relink below. + this.incoming?.delete(id) + + // (3) Relink at the node's EXISTING level (see JSDoc for the entry-point + // reasoning). A single-node index has nothing to link to — trivially done. + const start = this.resolveRelinkStart(id) + if (start) { + await this.linkNode(node, start) } - // Persist HNSW graph data to storage - // Respect persistMode setting + // Persistence — addItem's tail, minus the system record (entry point and + // maxLevel are untouched by an in-place update). Unlink-touched referrers + // are included so the persisted graph converges on the live one instead of + // keeping their pre-update edge sets forever. if (this.storage && this.persistMode === 'immediate') { - // IMMEDIATE MODE: Original behavior - persist new entity and system data. - // Goes through the per-node helper so the compressed-blob branch fires - // identically here vs. the deferred-flush + neighbor-update paths. - await this.persistNodeConnections(id, noun).catch((error) => { + await this.persistNodeConnections(id, node).catch((error) => { console.error(`Failed to persist HNSW data for ${id}:`, error) }) - - // Persist system data (entry point and max level) - await this.storage.saveHNSWSystem({ - entryPointId: this.entryPointId, - maxLevel: this.maxLevel - }).catch((error) => { - console.error('Failed to persist HNSW system data:', error) - }) + for (const refId of touchedReferrers) { + const ref = this.nouns.get(refId) + if (!ref) continue + await this.persistNodeConnections(refId, ref).catch((error) => { + console.error(`Failed to persist HNSW data for ${refId}:`, error) + }) + } } else if (this.persistMode === 'deferred') { - // DEFERRED MODE: Track dirty nodes for later batch persistence this.dirtyNodes.add(id) - this.dirtySystem = true + for (const refId of touchedReferrers) { + this.dirtyNodes.add(refId) + } } - return id + // Lazy vector eviction — same contract as addItem: after graph work + // completes the float32 vector leaves memory; reads serve from the + // (just-updated) cache or the persisted record. + if (this.vectorStorageMode === 'lazy' && this.storage) { + node.vector = [] + } + } + + /** + * @description Pick the traversal start for an in-place relink + * ({@link updateItem} step 3): the current entry point — unless that IS the + * node being relinked. Its edges were just unlinked, so a traversal + * starting there would see an empty neighborhood and produce zero links, + * stranding the graph behind an edgeless entry point. In that case (or when + * the entry point is missing/stale) fall back to the best OTHER node: + * highest tracked level first (the same O(1) heuristic as + * {@link recoverEntryPointO1}), then any other node. Returns null when the + * node is the only one in the index — nothing to link to, trivially valid. + */ + private resolveRelinkStart(excludeId: string): HNSWNoun | null { + if (this.entryPointId && this.entryPointId !== excludeId) { + const entry = this.nouns.get(this.entryPointId) + if (entry) return entry + } + for (let level = this.MAX_TRACKED_LEVELS; level >= 2; level--) { + const nodesAtLevel = this.highLevelNodes.get(level) + if (!nodesAtLevel) continue + for (const nodeId of nodesAtLevel) { + if (nodeId !== excludeId) { + const candidate = this.nouns.get(nodeId) + if (candidate) return candidate + } + } + } + for (const [nodeId, candidate] of this.nouns) { + if (nodeId !== excludeId) return candidate + } + return null } /** @@ -948,20 +1154,34 @@ export class JsHnswVectorIndex implements VectorIndexProvider { } /** - * Remove an item from the index + * @description Unlink every graph edge touching `noun`, in BOTH directions, + * WITHOUT removing the node from `this.nouns` — the unlink walk shared by + * {@link removeItem} (which then drops the node) and {@link updateItem} + * (which relinks the node in place, so it must never leave the map and + * KEEPS its level). + * + * Reverse-adjacency lets us touch ONLY the nodes that actually reference + * `noun.id` (its in-neighbors) rather than scanning the whole corpus — + * turning a delete from O(N) into O(in-degree) and a bulk delete from O(N²) + * into O(N·degree). Each referrer set is snapshotted because + * pruneConnections mutates the index. Outgoing edges are unhooked from each + * target's reverse set so no stale referrer survives. + * + * `incoming[noun.id]` itself is intentionally NOT maintained edge-by-edge + * inside the walk — both callers dispose of it wholesale afterwards + * (removeItem deletes it with the node; updateItem clears it and lets the + * relink rebuild it). + * + * @returns The ids of in-neighbors whose connection sets were modified + * (they dropped their edge to `noun` and may have been re-pruned), so a + * caller that persists per-node connections (updateItem) can mark them + * dirty / persist them. removeItem ignores the return — its persistence + * story lives in the caller's delete path, unchanged. */ - public async removeItem(id: string): Promise { - if (!this.nouns.has(id)) { - return false - } + private async unlinkNodeEdges(noun: HNSWNoun): Promise> { + const id = noun.id + const touchedReferrers = new Set() - - const noun = this.nouns.get(id)! - - // Reverse-adjacency lets us touch ONLY the nodes that actually reference `id` - // (its in-neighbors) rather than scanning the whole corpus — turning a delete - // from O(N) into O(in-degree) and a bulk delete from O(N²) into O(N·degree). - // Snapshot each referrer set because pruneConnections mutates the index. const incoming = this.ensureIncoming() const referrers = incoming.get(id) if (referrers) { @@ -969,11 +1189,11 @@ export class JsHnswVectorIndex implements VectorIndexProvider { for (const refId of Array.from(refSet)) { const ref = this.nouns.get(refId) if (ref && ref.connections.has(level)) { - // Drop the forward edge ref → id, then re-prune ref so the graph stays - // navigable. (id's own reverse entry is dropped wholesale below, so we - // intentionally do not maintain incoming[id] inside this loop.) + // Drop the forward edge ref → id, then re-prune ref so the graph + // stays navigable. ref.connections.get(level)!.delete(id) await this.pruneConnections(ref, level) + touchedReferrers.add(refId) } } } @@ -987,6 +1207,26 @@ export class JsHnswVectorIndex implements VectorIndexProvider { } } + return touchedReferrers + } + + /** + * Remove an item from the index + */ + public async removeItem(id: string): Promise { + if (!this.nouns.has(id)) { + return false + } + + + const noun = this.nouns.get(id)! + + // Unlink every edge touching the node (shared with updateItem's in-place + // relink — see unlinkNodeEdges). The returned touched-referrer set is + // ignored here: removeItem's persistence story lives in the caller's + // delete path, unchanged. + await this.unlinkNodeEdges(noun) + // Remove the noun + its reverse-index entry. this.nouns.delete(id) this.incoming?.delete(id) diff --git a/src/transaction/operations/IndexOperations.ts b/src/transaction/operations/IndexOperations.ts index d130bb3f..679a6d4d 100644 --- a/src/transaction/operations/IndexOperations.ts +++ b/src/transaction/operations/IndexOperations.ts @@ -151,6 +151,95 @@ export class RemoveFromVectorIndexOperation implements Operation { } } +/** + * Replace an item's vector in the vector index as ONE atomic transaction leg — + * the row is never absent from vector search during an update. + * + * Backend-neutral: see {@link AddToVectorIndexOperation} — `index` may be the + * JS HNSW fallback or a native acceleration provider; the emitted `name` + * stamps the active backend. + * + * Why this op exists: update flows historically staged a + * {@link RemoveFromVectorIndexOperation} followed by an + * {@link AddToVectorIndexOperation} as two separately-awaited operations. + * Between them the row was in NEITHER index — dark to semantic recall while + * perfectly visible to metadata reads (a transient-invisibility window that + * stretched to seconds in a production deployment). The structural cure is a + * single leg that never removes without simultaneously re-inserting. + * + * Execution strategy (feature-detected, in preference order): + * 1. Provider exposes `updateItem` → ONE in-place call. The provider swaps + * the vector without the row ever leaving its index, and an element-wise + * UNCHANGED vector (the type-only-update production shape) is a pure + * no-op on its side. + * 2. Provider without `updateItem` (a native provider that has not shipped + * it yet) → `removeItem` + `addItem` executed ADJACENT within this single + * op. Still strictly better than the historical pair: no other transaction + * operation can interleave between the two calls. This is a temporary + * seam — the native side of the pair is expected to ship its own + * `updateItem` so path 1 applies everywhere; when it does, this fallback + * becomes dead code that costs nothing. + * + * Rollback strategy (mirrors the execute branch that ran): + * - `updateItem` path → `updateItem` back to `oldVector`. + * - Fallback path → `removeItem` + `addItem` back to `oldVector`. + * + * Rollback semantics when the item did not exist at execute time: this op's + * contract is that the caller read the entity and its CURRENT vector + * (`oldVector`) before staging — update flows only stage it for existing + * rows. If the item was somehow absent, execute() inserts it (`updateItem` + * delegates to add; the fallback's remove is a no-op before its add), and + * rollback restores `oldVector` rather than removing — the same posture as + * {@link RemoveFromVectorIndexOperation}'s unconditional re-add: by + * constructing the op with `oldVector` the caller DECLARED the before-state, + * and rollback reconstructs that declared state instead of silently deciding + * the row should vanish. + */ +export class ReplaceInVectorIndexOperation implements Operation { + readonly name: string + + constructor( + private readonly index: VectorIndexProvider, + private readonly id: string, + private readonly oldVector: number[], // Required for rollback + private readonly newVector: number[] + ) { + this.name = `ReplaceInVectorIndex(${resolveVectorProviderId(index)})` + } + + async execute(): Promise { + // Feature-detect the in-place capability — optional on the provider + // contract, like `getItem`/`setPersistMode` (Brainy's JS HNSW index + // ships it; a native provider may not have yet). + const index = this.index as VectorIndexProvider & { + updateItem?: (item: { id: string; vector: number[] }) => Promise + } + + if (typeof index.updateItem === 'function') { + // Atomic path: one in-place call, the row never leaves the index. + await index.updateItem({ id: this.id, vector: this.newVector }) + + return async () => { + // Restore the declared before-state in place (see class JSDoc for + // the item-did-not-exist posture). + await index.updateItem!({ id: this.id, vector: this.oldVector }) + } + } + + // Fallback seam: remove+add ADJACENT within this single op — no other + // transaction operation can interleave between them (see class JSDoc). + await this.index.removeItem(this.id) + await this.index.addItem({ id: this.id, vector: this.newVector }) + + return async () => { + // updateItem-style restore via the same adjacent pair, back to the + // declared before-state. + await this.index.removeItem(this.id) + await this.index.addItem({ id: this.id, vector: this.oldVector }) + } + } +} + /** * Add to metadata index with rollback support * diff --git a/src/transaction/operations/index.ts b/src/transaction/operations/index.ts index c5548e70..32a69a21 100644 --- a/src/transaction/operations/index.ts +++ b/src/transaction/operations/index.ts @@ -23,6 +23,7 @@ export { export { AddToVectorIndexOperation, RemoveFromVectorIndexOperation, + ReplaceInVectorIndexOperation, AddToMetadataIndexOperation, RemoveFromMetadataIndexOperation, AddToGraphIndexOperation, diff --git a/tests/unit/brainy/lazy-notready-honor.test.ts b/tests/unit/brainy/lazy-notready-honor.test.ts new file mode 100644 index 00000000..4cfc6857 --- /dev/null +++ b/tests/unit/brainy/lazy-notready-honor.test.ts @@ -0,0 +1,75 @@ +/** + * @module tests/unit/brainy/lazy-notready-honor + * @description THE SILENT-EMPTY TRAP pin (found during a fleet adoption, + * SELF-ENGINE-PAIR-STANDARD): under `disableAutoRebuild: true`, the lazy + * first-query path (`ensureIndexesLoaded`) assessed ONLY the vector index's + * readiness — a native METADATA provider reporting not-ready (its strand + * report) never blocked the completion latch, so the promised lazy rebuild + * never fired and every `find()` silently returned `[]` on a populated + * store (measured: 52 entities durable-but-unqueryable, first query + * 0ms/0 rows). The law: a not-ready report from ANY provider falls through + * to the rebuild — never a silent empty. + * + * White-box provider-double pattern per tests/unit/brainy/migration-deference. + */ +import { describe, it, expect, afterEach, vi } from 'vitest' +import { Brainy } from '../../../src/index.js' +import { NounType } from '../../../src/types/graphTypes.js' +import { createTestConfig } from '../../helpers/test-factory.js' + +interface BrainInternals { + index: { size(): number } + metadataIndex: { isReady?: () => boolean } + lazyRebuildCompleted: boolean + ensureIndexesLoaded(): Promise + rebuildIndexesIfNeeded(force?: boolean): Promise +} + +const brains: Brainy[] = [] + +afterEach(async () => { + for (const b of brains.splice(0)) await b.close().catch(() => {}) + vi.restoreAllMocks() +}) + +async function warmLazyBrain(): Promise<{ brain: Brainy; internals: BrainInternals }> { + const brain = new Brainy(createTestConfig({ disableAutoRebuild: true })) + await brain.init() + brains.push(brain) + for (let i = 0; i < 3; i++) { + await brain.add({ data: `row ${i}`, type: NounType.Document, metadata: { i } }) + } + const internals = brain as unknown as BrainInternals + internals.lazyRebuildCompleted = false // simulate the cold first query + return { brain, internals } +} + +describe('lazy path honors EVERY provider’s not-ready report', () => { + it('a not-ready METADATA provider blocks the completion latch and fires the rebuild', async () => { + const { internals } = await warmLazyBrain() + + // The trap's shape: vector side looks fine (populated), metadata + // provider says NOT ready — the old gate latched complete here. + ;(internals.metadataIndex as { isReady?: () => boolean }).isReady = () => false + const rebuildSpy = vi + .spyOn(internals, 'rebuildIndexesIfNeeded') + .mockResolvedValue(undefined) + + await internals.ensureIndexesLoaded() + + expect(rebuildSpy, 'not-ready metadata provider must fire the lazy rebuild').toHaveBeenCalledWith(true) + }) + + it('control: all providers ready/unknown+populated → latch completes, no rebuild', async () => { + const { internals } = await warmLazyBrain() + ;(internals.metadataIndex as { isReady?: () => boolean }).isReady = () => true + const rebuildSpy = vi + .spyOn(internals, 'rebuildIndexesIfNeeded') + .mockResolvedValue(undefined) + + await internals.ensureIndexesLoaded() + + expect(rebuildSpy).not.toHaveBeenCalled() + expect(internals.lazyRebuildCompleted).toBe(true) + }) +}) diff --git a/tests/unit/hnsw/update-item-atomic.test.ts b/tests/unit/hnsw/update-item-atomic.test.ts new file mode 100644 index 00000000..f8798949 --- /dev/null +++ b/tests/unit/hnsw/update-item-atomic.test.ts @@ -0,0 +1,366 @@ +/** + * @module tests/unit/hnsw/update-item-atomic + * @description Guard for the atomic vector-index update: a row must NEVER be + * absent from vector search during an update. The historical update path + * staged a remove followed by an add as two separately-awaited transaction + * operations — between them the row was in NEITHER index (dark to semantic + * recall while perfectly visible to metadata reads; observed as seconds-long + * flicker in a production deployment). The structural cure verified here: + * + * 1. `JsHnswVectorIndex.updateItem` — same vector (element-wise) is a pure + * no-op (the production flicker shape: a type-only update re-indexing an + * UNCHANGED vector); a changed vector swaps in place, the node never + * leaving the map (white-box probe at the first internal step after the + * synchronous swap), including when the node IS the entry point. + * 2. `ReplaceInVectorIndexOperation` — one transaction leg that prefers the + * provider's in-place `updateItem`, with a remove+add-ADJACENT fallback + * for providers that have not shipped it; rollback restores the declared + * before-vector on both branches. + * 3. The brain's update path — with the JS index carrying `updateItem`, + * `removeItem` is never called during `brain.update()`, for the + * type-only shape AND for a genuine vector change. + */ +import { describe, it, expect, vi } from 'vitest' +import { JsHnswVectorIndex } from '../../../src/hnsw/hnswIndex.js' +import { ReplaceInVectorIndexOperation } from '../../../src/transaction/operations/IndexOperations.js' +import type { VectorIndexProvider } from '../../../src/plugin.js' +import type { Vector, VectorDocument } from '../../../src/coreTypes.js' +import { euclideanDistance } from '../../../src/utils/index.js' +import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js' +import { Brainy } from '../../../src/brainy' +import { createAddParams, createTestConfig } from '../../helpers/test-factory' + +const DIM = 8 + +function seededRand(seed: number): () => number { + let s = seed >>> 0 + return () => { + s = (s + 0x6d2b79f5) | 0 + let t = Math.imul(s ^ (s >>> 15), 1 | s) + t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t + return ((t ^ (t >>> 14)) >>> 0) / 4294967296 + } +} + +/** A deterministic vector pointing in a pseudo-random direction (well-connected graph). */ +function vec(idx: number): number[] { + const rand = seededRand(idx + 1) + return Array.from({ length: DIM }, () => rand() * 2 - 1) +} + +type Noun = { id: string; vector: number[]; connections: Map>; level: number } + +function nounsOf(index: JsHnswVectorIndex): Map { + return (index as unknown as { nouns: Map }).nouns +} + +/** Flatten a reverse index to sorted `target|level|source` triples. */ +function triplesFromIncoming(inc: Map>>): string[] { + const out: string[] = [] + for (const [target, byLevel] of inc) { + for (const [level, sources] of byLevel) { + for (const source of sources) out.push(`${target}|${level}|${source}`) + } + } + return out.sort() +} + +/** Derive the ground-truth reverse index directly from the live forward adjacency. */ +function triplesFromAdjacency(nouns: Map): string[] { + const out: string[] = [] + for (const [nodeId, node] of nouns) { + for (const [level, targets] of node.connections) { + for (const target of targets) out.push(`${target}|${level}|${nodeId}`) + } + } + return out.sort() +} + +function assertReverseIndexConsistent(index: JsHnswVectorIndex): void { + const live = ( + index as unknown as { ensureIncoming: () => Map>> } + ).ensureIncoming() + expect(triplesFromIncoming(live)).toEqual(triplesFromAdjacency(nounsOf(index))) +} + +function assertNoSelfLoops(index: JsHnswVectorIndex, id: string): void { + const node = nounsOf(index).get(id)! + for (const [level, targets] of node.connections) { + expect(targets.has(id), `self-loop at level ${level}`).toBe(false) + } +} + +function makeIndex(M = 16): JsHnswVectorIndex { + return new JsHnswVectorIndex( + { M, efConstruction: 200, efSearch: 64, ml: 16 }, + euclideanDistance, + { useParallelization: false, storage: new MemoryStorage() } + ) +} + +async function fillIndex(index: JsHnswVectorIndex, count: number): Promise { + for (let i = 0; i < count; i++) { + await index.addItem({ id: `n-${i}`, vector: vec(i) }) + } +} + +describe('JsHnswVectorIndex.updateItem — atomic in-place vector update', () => { + it('same vector (element-wise equal, fresh array) is a pure no-op: no remove, no relink, still searchable', async () => { + const index = makeIndex() + await fillIndex(index, 30) + + const target = 'n-7' + const sameVector = [...vec(7)] // fresh array, identical elements + + const before = await index.search(vec(7), 1) + expect(before[0][0]).toBe(target) + + const removeSpy = vi.spyOn(index, 'removeItem') + const nodeBefore = nounsOf(index).get(target)! + const connectionsBefore = nodeBefore.connections // reference — a relink replaces it + + await index.updateItem({ id: target, vector: sameVector }) + + expect(removeSpy).not.toHaveBeenCalled() + expect(index.size()).toBe(30) + // No relink happened: the connections map is the SAME object, untouched. + expect(nounsOf(index).get(target)!.connections).toBe(connectionsBefore) + + const after = await index.search(vec(7), 1) + expect(after[0][0]).toBe(target) + expect(after[0][1]).toBeCloseTo(0, 10) + + removeSpy.mockRestore() + }) + + it('changed vector: node never leaves the map (probe fires after the synchronous swap), removeItem never called, findable by the NEW vector', async () => { + const index = makeIndex() + await fillIndex(index, 40) + + const target = 'n-5' + const newVector = vec(500) + + // White-box probe: ensureIncoming is the FIRST internal step of the unlink + // walk, i.e. the first thing updateItem does after the synchronous vector + // swap. At that instant the node must (a) still be in the map and (b) + // already carry the NEW vector — the visibility-atomic ordering. + const inner = index as unknown as { + nouns: Map + ensureIncoming: () => Map>> + } + const origEnsure = inner.ensureIncoming.bind(index) + let probed = false + let presentDuring = false + let swappedFirst = false + ;(index as any).ensureIncoming = function () { + if (!probed) { + probed = true + presentDuring = inner.nouns.has(target) + swappedFirst = inner.nouns.get(target)?.vector === newVector + } + return origEnsure() + } + + const removeSpy = vi.spyOn(index, 'removeItem') + await index.updateItem({ id: target, vector: newVector }) + delete (index as any).ensureIncoming // restore the prototype method + + expect(probed).toBe(true) + expect(presentDuring).toBe(true) + expect(swappedFirst).toBe(true) + expect(removeSpy).not.toHaveBeenCalled() + expect(index.size()).toBe(40) + expect(nounsOf(index).has(target)).toBe(true) + + // Findable by search with the NEW vector, at distance ~0. + const got = await index.search(newVector, 1) + expect(got[0][0]).toBe(target) + expect(got[0][1]).toBeCloseTo(0, 10) + + // The relink left the graph bookkeeping exactly consistent. + assertNoSelfLoops(index, target) + assertReverseIndexConsistent(index) + + removeSpy.mockRestore() + }) + + it('keeps the node at its existing level (never releveled by an update)', async () => { + const index = makeIndex() + await fillIndex(index, 30) + + const target = 'n-3' + const levelBefore = nounsOf(index).get(target)!.level + + await index.updateItem({ id: target, vector: vec(600) }) + + expect(nounsOf(index).get(target)!.level).toBe(levelBefore) + expect(index.getMaxLevel()).toBeGreaterThanOrEqual(levelBefore) + }) + + it('updating the ENTRY POINT in place keeps it valid — entry id and maxLevel unchanged, graph never stranded', async () => { + const index = makeIndex() + await fillIndex(index, 40) + + const entryId = index.getEntryPointId()! + const maxLevelBefore = index.getMaxLevel() + const newVector = vec(700) + + await index.updateItem({ id: entryId, vector: newVector }) + + // Entry-point bookkeeping must not regress. + expect(index.getEntryPointId()).toBe(entryId) + expect(index.getMaxLevel()).toBe(maxLevelBefore) + expect(index.size()).toBe(40) + + // The entry point itself is findable by its new vector... + const gotEntry = await index.search(newVector, 1) + expect(gotEntry[0][0]).toBe(entryId) + + // ...and the REST of the graph is still reachable through it (a stranded, + // edgeless entry point would make every other node invisible). + const otherId = [...nounsOf(index).keys()].find((id) => id !== entryId)! + const otherIdx = Number(otherId.slice(2)) + const gotOther = await index.search(vec(otherIdx), 1) + expect(gotOther[0][0]).toBe(otherId) + + assertNoSelfLoops(index, entryId) + assertReverseIndexConsistent(index) + }) + + it('absent id delegates to addItem (plain insert)', async () => { + const index = makeIndex() + await fillIndex(index, 10) + + await index.updateItem({ id: 'fresh', vector: vec(900) }) + + expect(index.size()).toBe(11) + const got = await index.search(vec(900), 1) + expect(got[0][0]).toBe('fresh') + }) +}) + +describe('ReplaceInVectorIndexOperation — one atomic transaction leg', () => { + it('uses the provider updateItem path and rolls back to the old vector in place', async () => { + const index = makeIndex() + await fillIndex(index, 30) + + const target = 'n-9' + const oldVector = vec(9) + const newVector = vec(800) + + const removeSpy = vi.spyOn(index, 'removeItem') + const op = new ReplaceInVectorIndexOperation(index, target, oldVector, newVector) + expect(op.name).toBe('ReplaceInVectorIndex(hnsw-js)') + + const rollback = await op.execute() + expect(removeSpy).not.toHaveBeenCalled() + expect((await index.search(newVector, 1))[0][0]).toBe(target) + + await rollback() + expect(removeSpy).not.toHaveBeenCalled() + expect(index.size()).toBe(30) + + // Old vector restored, element-wise, and searchable again. + const restored = nounsOf(index).get(target)!.vector + expect(restored.length).toBe(oldVector.length) + for (let i = 0; i < oldVector.length; i++) { + expect(restored[i]).toBe(oldVector[i]) + } + const back = await index.search(oldVector, 1) + expect(back[0][0]).toBe(target) + expect(back[0][1]).toBeCloseTo(0, 10) + + removeSpy.mockRestore() + }) + + it('falls back to remove+add ADJACENT within the single op for a provider without updateItem, and rolls back the same way', async () => { + // A provider that has not shipped updateItem — the temporary seam: the + // pair stays adjacent inside ONE op (no other transaction operation can + // interleave), until the provider ships its own in-place updateItem. + const calls: string[] = [] + const store = new Map() + const legacyProvider = { + name: 'legacy-native', + addItem: async (item: VectorDocument) => { + calls.push(`add:${item.id}`) + store.set(item.id, item.vector) + return item.id + }, + removeItem: async (id: string) => { + calls.push(`remove:${id}`) + return store.delete(id) + }, + search: async () => [], + size: () => store.size, + clear: () => store.clear(), + rebuild: async () => {}, + flush: async () => 0, + getPersistMode: () => 'immediate' as const + } as unknown as VectorIndexProvider + + store.set('x', [1, 0]) + const op = new ReplaceInVectorIndexOperation(legacyProvider, 'x', [1, 0], [0, 1]) + + const rollback = await op.execute() + expect(calls).toEqual(['remove:x', 'add:x']) + expect(store.get('x')).toEqual([0, 1]) + + await rollback() + expect(calls).toEqual(['remove:x', 'add:x', 'remove:x', 'add:x']) + expect(store.get('x')).toEqual([1, 0]) + }) +}) + +describe('brain.update() — the update path stages ONE atomic vector-index leg', () => { + it('a type-only update (unchanged vector — the production flicker shape) never calls removeItem on the vector index', async () => { + const brain = new Brainy(createTestConfig()) + await brain.init() + try { + const id = await brain.add( + createAddParams({ data: 'atomic flicker guard entity', type: 'thing' }) + ) + + const index = (brain as unknown as { index: JsHnswVectorIndex }).index + const removeSpy = vi.spyOn(index, 'removeItem') + const sizeBefore = index.size() + + await brain.update({ id, type: 'document' }) + + expect(removeSpy).not.toHaveBeenCalled() + expect(index.size()).toBe(sizeBefore) + + const updated = await brain.get(id) + expect(updated).not.toBeNull() + expect(updated!.type).toBe('document') + + removeSpy.mockRestore() + } finally { + await brain.close() + } + }) + + it('a genuine vector change on update also never calls removeItem (in-place replace)', async () => { + const brain = new Brainy(createTestConfig()) + await brain.init() + try { + const id = await brain.add( + createAddParams({ data: 'vector change stays visible', type: 'thing' }) + ) + const existing = await brain.get(id, { includeVectors: true }) + // Same dimensionality, guaranteed-different content. + const changed = existing!.vector.map((x: number, i: number) => (i === 0 ? x + 0.25 : x)) + + const index = (brain as unknown as { index: JsHnswVectorIndex }).index + const removeSpy = vi.spyOn(index, 'removeItem') + + await brain.update({ id, vector: changed }) + + expect(removeSpy).not.toHaveBeenCalled() + expect(nounsOf(index).has(id)).toBe(true) + + removeSpy.mockRestore() + } finally { + await brain.close() + } + }) +})