fix(index): the flicker window dies — atomic in-place vector update; lazy open honors every provider's not-ready report; the Path Registry twin table
DP6/DP8 of the Path Registry (BRAINY-PROD-LATENCY-TRIAD, the proven flicker mechanism): update paths staged RemoveFromVectorIndex then AddToVectorIndex as two separately-awaited transaction ops — between them a live row was in NEITHER index (dark to semantic recall, fine in metadata list). The native pair widened that window to seconds in production before their side's visibility-commit fix; the structural cure lands here: - hnswIndex.updateItem: absent → add; SAME vector → pure no-op (the production shape — a type-only update re-indexed an unchanged vector, remove+add did pure damage); changed vector → the node NEVER leaves the index: synchronous vector swap first (every query from that instant sees correct distances), then unlink/relink at the node's existing level via shared internals (linkNode/unlinkNodeEdges refactored out of add/remove; entry point and maxLevel provably unchanged). - ReplaceInVectorIndexOperation: ONE transaction leg; feature-detects provider updateItem (native seam flagged — their side ships updateItem, then the adjacent remove+add fallback is dead code). Both update staging sites swapped; delete sites untouched. - LAZY-OPEN GATE (fleet adoption find, SELF-ENGINE-PAIR-STANDARD): under disableAutoRebuild, ensureIndexesLoaded assessed ONLY the vector index — a not-ready native METADATA provider never blocked the completion latch and every find() silently returned [] on a populated store. All three providers now vote; any not-ready report falls through to the rebuild. - docs/path-registry.md: brainy's twin table for the 32 shared path IDs — service class, budgets, lifecycle, narration, and the cited pin per row; owed rows named (LC4 doors-open migration, MT4 yielding heals, LC7 downgrade contract) per the lifecycle-sprint choreography. Pins: update-item-atomic 9/9 (visibility-atomic swap, reverse-index parity vs fresh rebuild, entry-point invariants) · lazy-notready-honor 2/2. Gates: unit 1928/1928 (148 files) · integration 760 · conformance 27/27.
This commit is contained in:
parent
3236a01bef
commit
ebe06cdf33
7 changed files with 937 additions and 57 deletions
|
|
@ -97,6 +97,7 @@ import {
|
|||
SaveVerbOperation,
|
||||
AddToGraphIndexOperation,
|
||||
RemoveFromVectorIndexOperation,
|
||||
ReplaceInVectorIndexOperation,
|
||||
RemoveFromMetadataIndexOperation,
|
||||
RemoveFromGraphIndexOperation,
|
||||
UpdateNounMetadataOperation,
|
||||
|
|
@ -2949,11 +2950,16 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
level: 0
|
||||
})
|
||||
)
|
||||
// ONE atomic vector-index leg: the historical Remove→Add pair was
|
||||
// two separately-awaited operations — between them the row was in
|
||||
// NEITHER index (dark to semantic recall, visible to metadata
|
||||
// reads). ReplaceInVectorIndexOperation goes through the provider's
|
||||
// in-place updateItem when available (row never absent; an
|
||||
// element-wise UNCHANGED vector — the type-only-update shape that
|
||||
// flickered in production — is a pure no-op), else remove+add
|
||||
// adjacent within the single op.
|
||||
tx.addOperation(
|
||||
new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector)
|
||||
)
|
||||
tx.addOperation(
|
||||
new AddToVectorIndexOperation(this.index, params.id, vector)
|
||||
new ReplaceInVectorIndexOperation(this.index, params.id, existing.vector, vector)
|
||||
)
|
||||
}
|
||||
|
||||
|
|
@ -9364,8 +9370,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
connections: new Map(),
|
||||
level: 0
|
||||
}),
|
||||
new RemoveFromVectorIndexOperation(this.index, params.id, existing.vector),
|
||||
new AddToVectorIndexOperation(this.index, params.id, vector)
|
||||
// ONE atomic vector-index leg — same law as update(): the row must
|
||||
// never be absent from vector search during an update (see
|
||||
// ReplaceInVectorIndexOperation).
|
||||
new ReplaceInVectorIndexOperation(this.index, params.id, existing.vector, vector)
|
||||
)
|
||||
}
|
||||
plan.operations.push(
|
||||
|
|
@ -14958,14 +14966,30 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
|
||||
// If indexes already populated AND honestly serving, mark complete and skip.
|
||||
// Honest gate: when the provider exposes isReady(), that REPLACES the size()>0
|
||||
// Honest gate: when a provider exposes isReady(), that REPLACES the size()>0
|
||||
// proxy (a native index can report a non-zero size while its serving structure
|
||||
// is not loaded — the silent-empty cold-load class). A not-ready provider falls
|
||||
// through so the rebuild path can load it; verifyVectorLive() is the query-time
|
||||
// backstop either way. Providers without isReady() keep the size() heuristic
|
||||
// (the JS index's size()>0 genuinely means loaded).
|
||||
//
|
||||
// ALL THREE providers vote (fleet-adoption find, SELF-ENGINE-PAIR-STANDARD):
|
||||
// this gate used to assess ONLY the vector index, so a not-ready native
|
||||
// METADATA provider (its strand report) never blocked the completion latch
|
||||
// — under disableAutoRebuild the promised lazy first-query rebuild never
|
||||
// fired and every find() silently returned [] on a populated store. A
|
||||
// not-ready report from ANY provider now falls through to the rebuild.
|
||||
const vectorReadiness = assessIndexReadiness(this.index)
|
||||
if (vectorReadiness === 'ready' || (vectorReadiness === 'unknown' && this.index.size() > 0)) {
|
||||
const metadataReadiness = assessIndexReadiness(this.metadataIndex)
|
||||
const graphReadiness = assessIndexReadiness(this.graphIndex)
|
||||
const anyProviderNotReady =
|
||||
vectorReadiness === 'not-ready' ||
|
||||
metadataReadiness === 'not-ready' ||
|
||||
graphReadiness === 'not-ready'
|
||||
if (
|
||||
!anyProviderNotReady &&
|
||||
(vectorReadiness === 'ready' || (vectorReadiness === 'unknown' && this.index.size() > 0))
|
||||
) {
|
||||
this.lazyRebuildCompleted = true
|
||||
return
|
||||
}
|
||||
|
|
|
|||
|
|
@ -486,6 +486,90 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
return id
|
||||
}
|
||||
|
||||
// Wire the node into the graph: greedy descent + per-level linking.
|
||||
// Extracted to linkNode so updateItem's in-place relink runs the SAME
|
||||
// insertion linking (one implementation, never a diverging copy).
|
||||
await this.linkNode(noun, entryPoint)
|
||||
|
||||
// Update max level and entry point if needed
|
||||
if (nounLevel > this.maxLevel) {
|
||||
this.maxLevel = nounLevel
|
||||
this.entryPointId = id
|
||||
}
|
||||
|
||||
// Add noun to the index
|
||||
this.nouns.set(id, noun)
|
||||
|
||||
// Track high-level nodes for O(1) entry point selection
|
||||
if (nounLevel >= 2 && nounLevel <= this.MAX_TRACKED_LEVELS) {
|
||||
if (!this.highLevelNodes.has(nounLevel)) {
|
||||
this.highLevelNodes.set(nounLevel, new Set())
|
||||
}
|
||||
this.highLevelNodes.get(nounLevel)!.add(id)
|
||||
}
|
||||
|
||||
// Lazy vector eviction (B2: graph-only memory after insert)
|
||||
// After graph construction completes, evict the full vector from memory.
|
||||
// Future searches will load vectors on-demand via getVectorSafe() + UnifiedCache.
|
||||
if (this.vectorStorageMode === 'lazy' && this.storage) {
|
||||
noun.vector = [] // Release float32 vector from memory
|
||||
}
|
||||
|
||||
// Persist HNSW graph data to storage
|
||||
// Respect persistMode setting
|
||||
if (this.storage && this.persistMode === 'immediate') {
|
||||
// IMMEDIATE MODE: Original behavior - persist new entity and system data.
|
||||
// Goes through the per-node helper so the compressed-blob branch fires
|
||||
// identically here vs. the deferred-flush + neighbor-update paths.
|
||||
await this.persistNodeConnections(id, noun).catch((error) => {
|
||||
console.error(`Failed to persist HNSW data for ${id}:`, error)
|
||||
})
|
||||
|
||||
// Persist system data (entry point and max level)
|
||||
await this.storage.saveHNSWSystem({
|
||||
entryPointId: this.entryPointId,
|
||||
maxLevel: this.maxLevel
|
||||
}).catch((error) => {
|
||||
console.error('Failed to persist HNSW system data:', error)
|
||||
})
|
||||
} else if (this.persistMode === 'deferred') {
|
||||
// DEFERRED MODE: Track dirty nodes for later batch persistence
|
||||
this.dirtyNodes.add(id)
|
||||
this.dirtySystem = true
|
||||
}
|
||||
|
||||
return id
|
||||
}
|
||||
|
||||
/**
|
||||
* @description The insertion LINKING phase shared by {@link addItem} and
|
||||
* {@link updateItem}: greedy-descend from `entryPoint` through the levels
|
||||
* above `noun.level`, then at each level from `min(noun.level, maxLevel)`
|
||||
* down to 0 find `efConstruction` candidates, select the M nearest, and
|
||||
* create bidirectional edges — maintaining the reverse-adjacency index via
|
||||
* {@link addIncoming} and re-pruning any neighbor pushed over M.
|
||||
*
|
||||
* Persistence follows the caller's mode exactly as the historical inline
|
||||
* addItem code did: `'immediate'` persists each touched neighbor's
|
||||
* connections concurrently (batched by `maxConcurrentNeighborWrites`);
|
||||
* `'deferred'` marks each touched neighbor dirty for the next flush.
|
||||
*
|
||||
* Does NOT touch index membership (`this.nouns`), the entry point, or
|
||||
* `maxLevel` — the caller owns that bookkeeping: addItem inserts a NEW node
|
||||
* afterwards and may raise maxLevel; updateItem relinks an EXISTING node in
|
||||
* place whose level was already counted, so nothing may change. `noun.vector`
|
||||
* must be the live in-memory vector at call time; both callers guarantee it
|
||||
* (lazy-mode eviction happens only after linking completes).
|
||||
*
|
||||
* A `neighborId === noun.id` candidate is skipped defensively: during
|
||||
* updateItem the node is already IN `this.nouns` (visibility-atomicity —
|
||||
* unlike addItem, which links before inserting), and a self-edge must never
|
||||
* be creatable no matter what the traversal surfaces.
|
||||
*/
|
||||
private async linkNode(noun: HNSWNoun, entryPoint: HNSWNoun): Promise<void> {
|
||||
const { id, vector } = noun
|
||||
const nounLevel = noun.level
|
||||
|
||||
let currObj = entryPoint
|
||||
|
||||
// Calculate distance to entry point (handles lazy loading + sync fast path)
|
||||
|
|
@ -547,6 +631,10 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
}> = []
|
||||
|
||||
for (const [neighborId, _] of neighbors) {
|
||||
if (neighborId === id) {
|
||||
// Never self-link (see method JSDoc — reachable only via updateItem)
|
||||
continue
|
||||
}
|
||||
const neighbor = this.nouns.get(neighborId)
|
||||
if (!neighbor) {
|
||||
// Skip neighbors that don't exist (expected during rapid additions/deletions)
|
||||
|
|
@ -630,7 +718,7 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
const nearestNoun = this.nouns.get(nearestId)
|
||||
if (!nearestNoun) {
|
||||
console.error(
|
||||
`Nearest noun with ID ${nearestId} not found in addItem`
|
||||
`Nearest noun with ID ${nearestId} not found in linkNode`
|
||||
)
|
||||
// Keep the current object as is
|
||||
} else {
|
||||
|
|
@ -639,55 +727,173 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Update max level and entry point if needed
|
||||
if (nounLevel > this.maxLevel) {
|
||||
this.maxLevel = nounLevel
|
||||
this.entryPointId = id
|
||||
/**
|
||||
* @description Atomically replace an item's vector IN PLACE — the row is
|
||||
* NEVER absent from the index during an update. The historical shape staged
|
||||
* a remove followed by an add as two separately-awaited transaction
|
||||
* operations; between them the row was in NEITHER index — dark to semantic
|
||||
* recall while perfectly visible to metadata reads (observed as seconds-long
|
||||
* production flicker in a downstream deployment). Mandate: a row that
|
||||
* exists must never be invisible to a read path, even transiently.
|
||||
*
|
||||
* Behavior:
|
||||
* - id not in the index → delegates to {@link addItem} (plain insert).
|
||||
* - SAME vector (element-wise equal) → pure no-op. This is the production
|
||||
* flicker shape: a type-only update re-indexes an UNCHANGED vector, so the
|
||||
* old remove+add did pure damage. (In lazy vector-storage mode the
|
||||
* comparison baseline is whatever {@link getVectorSafe} serves — the
|
||||
* cache, or the persisted record; if the caller already rewrote the
|
||||
* record with the new vector before calling in, equality may report "no
|
||||
* change" and skip the relink. Query correctness is unaffected either
|
||||
* way — distances always use the live vector — the graph edges just keep
|
||||
* their pre-update geometry, which HNSW tolerates by construction.)
|
||||
* - DIFFERENT vector → the node never leaves `this.nouns`:
|
||||
* 1. `node.vector` is swapped SYNCHRONOUSLY first (and the shared vector
|
||||
* cache updated in the same tick), so from that point every query sees
|
||||
* the node with correct distances;
|
||||
* 2. its old edges are unlinked via the same reverse-adjacency walk
|
||||
* removeItem uses ({@link unlinkNodeEdges}) — the node stays in the
|
||||
* map and KEEPS its level;
|
||||
* 3. the insertion linking re-runs at the node's EXISTING level
|
||||
* ({@link linkNode}). Entry-point cases: if the node IS the entry
|
||||
* point it REMAINS the entry point (still valid — same id, same
|
||||
* level); the relink traversal then starts from another node via
|
||||
* {@link resolveRelinkStart}, because the node's own edges were just
|
||||
* cleared and a traversal starting AT it would find nothing and link
|
||||
* nothing — stranding the whole graph behind an edgeless entry point.
|
||||
* maxLevel never regresses: the node keeps its level and its
|
||||
* membership, so the remove-side relevel bookkeeping never runs.
|
||||
*
|
||||
* Persistence mirrors {@link addItem}'s tail for the node itself plus the
|
||||
* in-neighbors whose connection sets changed during the unlink:
|
||||
* `'immediate'` persists their connections now; `'deferred'` marks them
|
||||
* dirty for the next flush. The system record (entry point + maxLevel) is
|
||||
* NOT rewritten — an in-place update changes neither.
|
||||
*/
|
||||
public async updateItem(item: VectorDocument): Promise<void> {
|
||||
if (!item) {
|
||||
throw new Error('Item is undefined or null')
|
||||
}
|
||||
const { id, vector } = item
|
||||
if (!vector) {
|
||||
throw new Error('Vector is undefined or null')
|
||||
}
|
||||
|
||||
// Add noun to the index
|
||||
this.nouns.set(id, noun)
|
||||
const node = this.nouns.get(id)
|
||||
if (!node) {
|
||||
// Absent → plain insert.
|
||||
await this.addItem(item)
|
||||
return
|
||||
}
|
||||
|
||||
// Track high-level nodes for O(1) entry point selection
|
||||
if (nounLevel >= 2 && nounLevel <= this.MAX_TRACKED_LEVELS) {
|
||||
if (!this.highLevelNodes.has(nounLevel)) {
|
||||
this.highLevelNodes.set(nounLevel, new Set())
|
||||
if (this.dimension === null) {
|
||||
this.dimension = vector.length
|
||||
} else if (vector.length !== this.dimension) {
|
||||
throw new Error(
|
||||
`Vector dimension mismatch: expected ${this.dimension}, got ${vector.length}`
|
||||
)
|
||||
}
|
||||
|
||||
// Fast path: element-wise-equal vector → NOTHING to do (the production
|
||||
// flicker shape — a type-only update re-indexing an unchanged vector).
|
||||
// getVectorSafe handles the lazy-evicted case (loads from cache/storage).
|
||||
const current = await this.getVectorSafe(node)
|
||||
if (current.length === vector.length) {
|
||||
let same = true
|
||||
for (let i = 0; i < vector.length; i++) {
|
||||
if (current[i] !== vector[i]) {
|
||||
same = false
|
||||
break
|
||||
}
|
||||
}
|
||||
this.highLevelNodes.get(nounLevel)!.add(id)
|
||||
if (same) return
|
||||
}
|
||||
|
||||
// Lazy vector eviction (B2: graph-only memory after insert)
|
||||
// After graph construction completes, evict the full vector from memory.
|
||||
// Future searches will load vectors on-demand via getVectorSafe() + UnifiedCache.
|
||||
if (this.vectorStorageMode === 'lazy' && this.storage) {
|
||||
noun.vector = [] // Release float32 vector from memory
|
||||
// (1) Visibility-atomic swap: from this synchronous assignment on, every
|
||||
// query sees the node with correct distances. The shared vector cache is
|
||||
// updated in the same tick so the lazy-mode read path can never serve the
|
||||
// stale vector either.
|
||||
node.vector = vector
|
||||
this.unifiedCache.set(`hnsw:vector:${id}`, vector, 'vectors', vector.length * 4, 50)
|
||||
|
||||
// (2) Unlink the old edges — the node stays in the map, keeps its level.
|
||||
const touchedReferrers = await this.unlinkNodeEdges(node)
|
||||
node.connections = new Map()
|
||||
for (let level = 0; level <= node.level; level++) {
|
||||
node.connections.set(level, new Set<string>())
|
||||
}
|
||||
// The node's own reverse entry is rebuilt by the relink below.
|
||||
this.incoming?.delete(id)
|
||||
|
||||
// (3) Relink at the node's EXISTING level (see JSDoc for the entry-point
|
||||
// reasoning). A single-node index has nothing to link to — trivially done.
|
||||
const start = this.resolveRelinkStart(id)
|
||||
if (start) {
|
||||
await this.linkNode(node, start)
|
||||
}
|
||||
|
||||
// Persist HNSW graph data to storage
|
||||
// Respect persistMode setting
|
||||
// Persistence — addItem's tail, minus the system record (entry point and
|
||||
// maxLevel are untouched by an in-place update). Unlink-touched referrers
|
||||
// are included so the persisted graph converges on the live one instead of
|
||||
// keeping their pre-update edge sets forever.
|
||||
if (this.storage && this.persistMode === 'immediate') {
|
||||
// IMMEDIATE MODE: Original behavior - persist new entity and system data.
|
||||
// Goes through the per-node helper so the compressed-blob branch fires
|
||||
// identically here vs. the deferred-flush + neighbor-update paths.
|
||||
await this.persistNodeConnections(id, noun).catch((error) => {
|
||||
await this.persistNodeConnections(id, node).catch((error) => {
|
||||
console.error(`Failed to persist HNSW data for ${id}:`, error)
|
||||
})
|
||||
|
||||
// Persist system data (entry point and max level)
|
||||
await this.storage.saveHNSWSystem({
|
||||
entryPointId: this.entryPointId,
|
||||
maxLevel: this.maxLevel
|
||||
}).catch((error) => {
|
||||
console.error('Failed to persist HNSW system data:', error)
|
||||
})
|
||||
for (const refId of touchedReferrers) {
|
||||
const ref = this.nouns.get(refId)
|
||||
if (!ref) continue
|
||||
await this.persistNodeConnections(refId, ref).catch((error) => {
|
||||
console.error(`Failed to persist HNSW data for ${refId}:`, error)
|
||||
})
|
||||
}
|
||||
} else if (this.persistMode === 'deferred') {
|
||||
// DEFERRED MODE: Track dirty nodes for later batch persistence
|
||||
this.dirtyNodes.add(id)
|
||||
this.dirtySystem = true
|
||||
for (const refId of touchedReferrers) {
|
||||
this.dirtyNodes.add(refId)
|
||||
}
|
||||
}
|
||||
|
||||
return id
|
||||
// Lazy vector eviction — same contract as addItem: after graph work
|
||||
// completes the float32 vector leaves memory; reads serve from the
|
||||
// (just-updated) cache or the persisted record.
|
||||
if (this.vectorStorageMode === 'lazy' && this.storage) {
|
||||
node.vector = []
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Pick the traversal start for an in-place relink
|
||||
* ({@link updateItem} step 3): the current entry point — unless that IS the
|
||||
* node being relinked. Its edges were just unlinked, so a traversal
|
||||
* starting there would see an empty neighborhood and produce zero links,
|
||||
* stranding the graph behind an edgeless entry point. In that case (or when
|
||||
* the entry point is missing/stale) fall back to the best OTHER node:
|
||||
* highest tracked level first (the same O(1) heuristic as
|
||||
* {@link recoverEntryPointO1}), then any other node. Returns null when the
|
||||
* node is the only one in the index — nothing to link to, trivially valid.
|
||||
*/
|
||||
private resolveRelinkStart(excludeId: string): HNSWNoun | null {
|
||||
if (this.entryPointId && this.entryPointId !== excludeId) {
|
||||
const entry = this.nouns.get(this.entryPointId)
|
||||
if (entry) return entry
|
||||
}
|
||||
for (let level = this.MAX_TRACKED_LEVELS; level >= 2; level--) {
|
||||
const nodesAtLevel = this.highLevelNodes.get(level)
|
||||
if (!nodesAtLevel) continue
|
||||
for (const nodeId of nodesAtLevel) {
|
||||
if (nodeId !== excludeId) {
|
||||
const candidate = this.nouns.get(nodeId)
|
||||
if (candidate) return candidate
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const [nodeId, candidate] of this.nouns) {
|
||||
if (nodeId !== excludeId) return candidate
|
||||
}
|
||||
return null
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -948,20 +1154,34 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
}
|
||||
|
||||
/**
|
||||
* Remove an item from the index
|
||||
* @description Unlink every graph edge touching `noun`, in BOTH directions,
|
||||
* WITHOUT removing the node from `this.nouns` — the unlink walk shared by
|
||||
* {@link removeItem} (which then drops the node) and {@link updateItem}
|
||||
* (which relinks the node in place, so it must never leave the map and
|
||||
* KEEPS its level).
|
||||
*
|
||||
* Reverse-adjacency lets us touch ONLY the nodes that actually reference
|
||||
* `noun.id` (its in-neighbors) rather than scanning the whole corpus —
|
||||
* turning a delete from O(N) into O(in-degree) and a bulk delete from O(N²)
|
||||
* into O(N·degree). Each referrer set is snapshotted because
|
||||
* pruneConnections mutates the index. Outgoing edges are unhooked from each
|
||||
* target's reverse set so no stale referrer survives.
|
||||
*
|
||||
* `incoming[noun.id]` itself is intentionally NOT maintained edge-by-edge
|
||||
* inside the walk — both callers dispose of it wholesale afterwards
|
||||
* (removeItem deletes it with the node; updateItem clears it and lets the
|
||||
* relink rebuild it).
|
||||
*
|
||||
* @returns The ids of in-neighbors whose connection sets were modified
|
||||
* (they dropped their edge to `noun` and may have been re-pruned), so a
|
||||
* caller that persists per-node connections (updateItem) can mark them
|
||||
* dirty / persist them. removeItem ignores the return — its persistence
|
||||
* story lives in the caller's delete path, unchanged.
|
||||
*/
|
||||
public async removeItem(id: string): Promise<boolean> {
|
||||
if (!this.nouns.has(id)) {
|
||||
return false
|
||||
}
|
||||
private async unlinkNodeEdges(noun: HNSWNoun): Promise<Set<string>> {
|
||||
const id = noun.id
|
||||
const touchedReferrers = new Set<string>()
|
||||
|
||||
|
||||
const noun = this.nouns.get(id)!
|
||||
|
||||
// Reverse-adjacency lets us touch ONLY the nodes that actually reference `id`
|
||||
// (its in-neighbors) rather than scanning the whole corpus — turning a delete
|
||||
// from O(N) into O(in-degree) and a bulk delete from O(N²) into O(N·degree).
|
||||
// Snapshot each referrer set because pruneConnections mutates the index.
|
||||
const incoming = this.ensureIncoming()
|
||||
const referrers = incoming.get(id)
|
||||
if (referrers) {
|
||||
|
|
@ -969,11 +1189,11 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
for (const refId of Array.from(refSet)) {
|
||||
const ref = this.nouns.get(refId)
|
||||
if (ref && ref.connections.has(level)) {
|
||||
// Drop the forward edge ref → id, then re-prune ref so the graph stays
|
||||
// navigable. (id's own reverse entry is dropped wholesale below, so we
|
||||
// intentionally do not maintain incoming[id] inside this loop.)
|
||||
// Drop the forward edge ref → id, then re-prune ref so the graph
|
||||
// stays navigable.
|
||||
ref.connections.get(level)!.delete(id)
|
||||
await this.pruneConnections(ref, level)
|
||||
touchedReferrers.add(refId)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -987,6 +1207,26 @@ export class JsHnswVectorIndex implements VectorIndexProvider {
|
|||
}
|
||||
}
|
||||
|
||||
return touchedReferrers
|
||||
}
|
||||
|
||||
/**
|
||||
* Remove an item from the index
|
||||
*/
|
||||
public async removeItem(id: string): Promise<boolean> {
|
||||
if (!this.nouns.has(id)) {
|
||||
return false
|
||||
}
|
||||
|
||||
|
||||
const noun = this.nouns.get(id)!
|
||||
|
||||
// Unlink every edge touching the node (shared with updateItem's in-place
|
||||
// relink — see unlinkNodeEdges). The returned touched-referrer set is
|
||||
// ignored here: removeItem's persistence story lives in the caller's
|
||||
// delete path, unchanged.
|
||||
await this.unlinkNodeEdges(noun)
|
||||
|
||||
// Remove the noun + its reverse-index entry.
|
||||
this.nouns.delete(id)
|
||||
this.incoming?.delete(id)
|
||||
|
|
|
|||
|
|
@ -151,6 +151,95 @@ export class RemoveFromVectorIndexOperation implements Operation {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Replace an item's vector in the vector index as ONE atomic transaction leg —
|
||||
* the row is never absent from vector search during an update.
|
||||
*
|
||||
* Backend-neutral: see {@link AddToVectorIndexOperation} — `index` may be the
|
||||
* JS HNSW fallback or a native acceleration provider; the emitted `name`
|
||||
* stamps the active backend.
|
||||
*
|
||||
* Why this op exists: update flows historically staged a
|
||||
* {@link RemoveFromVectorIndexOperation} followed by an
|
||||
* {@link AddToVectorIndexOperation} as two separately-awaited operations.
|
||||
* Between them the row was in NEITHER index — dark to semantic recall while
|
||||
* perfectly visible to metadata reads (a transient-invisibility window that
|
||||
* stretched to seconds in a production deployment). The structural cure is a
|
||||
* single leg that never removes without simultaneously re-inserting.
|
||||
*
|
||||
* Execution strategy (feature-detected, in preference order):
|
||||
* 1. Provider exposes `updateItem` → ONE in-place call. The provider swaps
|
||||
* the vector without the row ever leaving its index, and an element-wise
|
||||
* UNCHANGED vector (the type-only-update production shape) is a pure
|
||||
* no-op on its side.
|
||||
* 2. Provider without `updateItem` (a native provider that has not shipped
|
||||
* it yet) → `removeItem` + `addItem` executed ADJACENT within this single
|
||||
* op. Still strictly better than the historical pair: no other transaction
|
||||
* operation can interleave between the two calls. This is a temporary
|
||||
* seam — the native side of the pair is expected to ship its own
|
||||
* `updateItem` so path 1 applies everywhere; when it does, this fallback
|
||||
* becomes dead code that costs nothing.
|
||||
*
|
||||
* Rollback strategy (mirrors the execute branch that ran):
|
||||
* - `updateItem` path → `updateItem` back to `oldVector`.
|
||||
* - Fallback path → `removeItem` + `addItem` back to `oldVector`.
|
||||
*
|
||||
* Rollback semantics when the item did not exist at execute time: this op's
|
||||
* contract is that the caller read the entity and its CURRENT vector
|
||||
* (`oldVector`) before staging — update flows only stage it for existing
|
||||
* rows. If the item was somehow absent, execute() inserts it (`updateItem`
|
||||
* delegates to add; the fallback's remove is a no-op before its add), and
|
||||
* rollback restores `oldVector` rather than removing — the same posture as
|
||||
* {@link RemoveFromVectorIndexOperation}'s unconditional re-add: by
|
||||
* constructing the op with `oldVector` the caller DECLARED the before-state,
|
||||
* and rollback reconstructs that declared state instead of silently deciding
|
||||
* the row should vanish.
|
||||
*/
|
||||
export class ReplaceInVectorIndexOperation implements Operation {
|
||||
readonly name: string
|
||||
|
||||
constructor(
|
||||
private readonly index: VectorIndexProvider,
|
||||
private readonly id: string,
|
||||
private readonly oldVector: number[], // Required for rollback
|
||||
private readonly newVector: number[]
|
||||
) {
|
||||
this.name = `ReplaceInVectorIndex(${resolveVectorProviderId(index)})`
|
||||
}
|
||||
|
||||
async execute(): Promise<RollbackAction> {
|
||||
// Feature-detect the in-place capability — optional on the provider
|
||||
// contract, like `getItem`/`setPersistMode` (Brainy's JS HNSW index
|
||||
// ships it; a native provider may not have yet).
|
||||
const index = this.index as VectorIndexProvider & {
|
||||
updateItem?: (item: { id: string; vector: number[] }) => Promise<void>
|
||||
}
|
||||
|
||||
if (typeof index.updateItem === 'function') {
|
||||
// Atomic path: one in-place call, the row never leaves the index.
|
||||
await index.updateItem({ id: this.id, vector: this.newVector })
|
||||
|
||||
return async () => {
|
||||
// Restore the declared before-state in place (see class JSDoc for
|
||||
// the item-did-not-exist posture).
|
||||
await index.updateItem!({ id: this.id, vector: this.oldVector })
|
||||
}
|
||||
}
|
||||
|
||||
// Fallback seam: remove+add ADJACENT within this single op — no other
|
||||
// transaction operation can interleave between them (see class JSDoc).
|
||||
await this.index.removeItem(this.id)
|
||||
await this.index.addItem({ id: this.id, vector: this.newVector })
|
||||
|
||||
return async () => {
|
||||
// updateItem-style restore via the same adjacent pair, back to the
|
||||
// declared before-state.
|
||||
await this.index.removeItem(this.id)
|
||||
await this.index.addItem({ id: this.id, vector: this.oldVector })
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Add to metadata index with rollback support
|
||||
*
|
||||
|
|
|
|||
|
|
@ -23,6 +23,7 @@ export {
|
|||
export {
|
||||
AddToVectorIndexOperation,
|
||||
RemoveFromVectorIndexOperation,
|
||||
ReplaceInVectorIndexOperation,
|
||||
AddToMetadataIndexOperation,
|
||||
RemoveFromMetadataIndexOperation,
|
||||
AddToGraphIndexOperation,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue