fix(vfs): the VFS root never persists a zero-norm vector
A zero-norm vector is lawful inside brainy (cosine distance scores it at maximum, never a false top hit) but a false attractor for a downstream engine serving squared-euclidean distance, which cannot tell a real all-zero vector apart from a legitimate origin point. - The VFS root now persists with vector [] (the existing "unvectored" shape) instead of a real all-zero 384-dim placeholder, and is never routed into the deferred-embed pipeline. - A one-time migration in the root-init path detects a pre-fix store's all-zero placeholder root (by norm, not length) and rewrites it to [] through a new sanctioned Brainy method that keeps the canonical vectored-noun ledger honest and removes the row from the vector index. - The vector-index write seam (AddToVectorIndexOperation, ReplaceInVectorIndexOperation, and the generation materializer's direct insert) now refuses any real all-zero vector before it reaches a provider, loudly naming the entity, while the canonical write still lands. - add()'s dimension-pinning and HNSW-insert gates, and the add-params validator, now treat any empty vector as carrying no dimension information, closing a latent trap where an explicit `vector: []` would have pinned dimensions to 0.
This commit is contained in:
parent
aad9e2eeb1
commit
c6cc0de955
10 changed files with 516 additions and 64 deletions
|
|
@ -13,6 +13,8 @@ import type { VectorIndexProvider, GraphIndexProvider } from '../../plugin.js'
|
|||
import type { MetadataIndexManager } from '../../utils/metadataIndex.js'
|
||||
import type { GraphVerb } from '../../coreTypes.js'
|
||||
import type { Operation, RollbackAction } from '../types.js'
|
||||
import { isZeroNormVector } from '../../utils/distance.js'
|
||||
import { prodLog } from '../../utils/logger.js'
|
||||
|
||||
/**
|
||||
* Backend identity stamped into an operation's emitted `name` string (e.g.
|
||||
|
|
@ -88,6 +90,30 @@ export class AddToVectorIndexOperation implements Operation {
|
|||
}
|
||||
|
||||
async execute(): Promise<RollbackAction> {
|
||||
// THE ZERO-NORM LAW (the live provider-write seam's belt): a zero-norm
|
||||
// vector is not a vector — it never crosses an engine boundary. This
|
||||
// engine's own cosine distance treats an all-zero vector safely (a
|
||||
// zero-norm operand always scores MAXIMUM distance, see
|
||||
// {@link isZeroNormVector}'s JSDoc), but a downstream engine serving
|
||||
// squared-euclidean distance cannot tell it apart from a legitimate
|
||||
// origin point — a false attractor that silently darkens real results.
|
||||
// The canonical write already landed (SaveNoun/SaveNounMetadata
|
||||
// operations are staged ahead of this one in every caller) — only the
|
||||
// INDEX INSERT is refused here, loudly, never a throw. A length-0
|
||||
// vector is the unrelated "unvectored" shape and is skipped silently
|
||||
// (the same contract callers already rely on for deferred embeds).
|
||||
if (this.vector.length === 0) {
|
||||
return async () => {}
|
||||
}
|
||||
if (isZeroNormVector(this.vector)) {
|
||||
prodLog.warn(
|
||||
`[vector-index] refusing to index a zero-norm vector for entity ${this.id} — ` +
|
||||
`a zero-norm vector is not a vector and never crosses an engine boundary ` +
|
||||
`(the canonical write is unaffected; only the vector-index insert is skipped)`
|
||||
)
|
||||
return async () => {}
|
||||
}
|
||||
|
||||
// Check if item already exists (for rollback decision)
|
||||
const existed = await this.itemExists(this.id)
|
||||
|
||||
|
|
@ -263,6 +289,35 @@ export class ReplaceInVectorIndexOperation implements Operation {
|
|||
// One commit generation for the whole replace (both branches + rollback).
|
||||
const generation = this.generationFn?.()
|
||||
|
||||
// THE ZERO-NORM LAW (see AddToVectorIndexOperation's matching JSDoc): a
|
||||
// real all-zero replacement vector must never land in the index — refuse
|
||||
// loudly, canonical write unaffected. The row must not be left stale
|
||||
// either: if it was genuinely indexed under `oldVector`, remove it
|
||||
// rather than pretend the old vector still describes the row. A
|
||||
// length-0 `newVector` (the unrelated "unvectored" shape) is handled the
|
||||
// same way, silently — no caller today reaches this with an empty
|
||||
// replacement (update() rejects a dimension-mismatched empty vector),
|
||||
// but the seam stays consistent in case one ever legitimately does.
|
||||
if (isZeroNormVector(this.newVector) || this.newVector.length === 0) {
|
||||
const wasIndexed = this.oldVector.length > 0 && !isZeroNormVector(this.oldVector)
|
||||
if (isZeroNormVector(this.newVector)) {
|
||||
prodLog.warn(
|
||||
`[vector-index] refusing to replace with a zero-norm vector for entity ${this.id} — ` +
|
||||
`a zero-norm vector is not a vector and never crosses an engine boundary ` +
|
||||
`(the canonical write is unaffected; the row is removed from the vector index instead)`
|
||||
)
|
||||
}
|
||||
if (wasIndexed) {
|
||||
await this.index.removeItem(this.id, generation)
|
||||
}
|
||||
return async () => {
|
||||
// Restore the declared before-state.
|
||||
if (wasIndexed) {
|
||||
await this.index.addItem({ id: this.id, vector: this.oldVector }, generation)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (typeof index.updateItem === 'function') {
|
||||
// Atomic path: one in-place call, the row never leaves the index.
|
||||
await index.updateItem({ id: this.id, vector: this.newVector }, generation)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue