fix(vfs): the VFS root never persists a zero-norm vector
Some checks failed
CI / Node 22 (push) Successful in 12m23s
CI / Node 24 (push) Successful in 12m12s
CI / Integration + conformance (Node 22) (push) Failing after 14m47s
CI / Bun (latest) (push) Successful in 12m38s

A zero-norm vector is lawful inside brainy (cosine distance scores it at
maximum, never a false top hit) but a false attractor for a downstream
engine serving squared-euclidean distance, which cannot tell a real
all-zero vector apart from a legitimate origin point.

- The VFS root now persists with vector [] (the existing "unvectored"
  shape) instead of a real all-zero 384-dim placeholder, and is never
  routed into the deferred-embed pipeline.
- A one-time migration in the root-init path detects a pre-fix store's
  all-zero placeholder root (by norm, not length) and rewrites it to []
  through a new sanctioned Brainy method that keeps the canonical
  vectored-noun ledger honest and removes the row from the vector index.
- The vector-index write seam (AddToVectorIndexOperation,
  ReplaceInVectorIndexOperation, and the generation materializer's direct
  insert) now refuses any real all-zero vector before it reaches a
  provider, loudly naming the entity, while the canonical write still
  lands.
- add()'s dimension-pinning and HNSW-insert gates, and the add-params
  validator, now treat any empty vector as carrying no dimension
  information, closing a latent trap where an explicit `vector: []`
  would have pinned dimensions to 0.
This commit is contained in:
David Snelling 2026-08-27 09:28:44 -07:00
parent aad9e2eeb1
commit c6cc0de955
10 changed files with 516 additions and 64 deletions

View file

@ -13,6 +13,8 @@ import type { VectorIndexProvider, GraphIndexProvider } from '../../plugin.js'
import type { MetadataIndexManager } from '../../utils/metadataIndex.js'
import type { GraphVerb } from '../../coreTypes.js'
import type { Operation, RollbackAction } from '../types.js'
import { isZeroNormVector } from '../../utils/distance.js'
import { prodLog } from '../../utils/logger.js'
/**
* Backend identity stamped into an operation's emitted `name` string (e.g.
@ -88,6 +90,30 @@ export class AddToVectorIndexOperation implements Operation {
}
async execute(): Promise<RollbackAction> {
// THE ZERO-NORM LAW (the live provider-write seam's belt): a zero-norm
// vector is not a vector — it never crosses an engine boundary. This
// engine's own cosine distance treats an all-zero vector safely (a
// zero-norm operand always scores MAXIMUM distance, see
// {@link isZeroNormVector}'s JSDoc), but a downstream engine serving
// squared-euclidean distance cannot tell it apart from a legitimate
// origin point — a false attractor that silently darkens real results.
// The canonical write already landed (SaveNoun/SaveNounMetadata
// operations are staged ahead of this one in every caller) — only the
// INDEX INSERT is refused here, loudly, never a throw. A length-0
// vector is the unrelated "unvectored" shape and is skipped silently
// (the same contract callers already rely on for deferred embeds).
if (this.vector.length === 0) {
return async () => {}
}
if (isZeroNormVector(this.vector)) {
prodLog.warn(
`[vector-index] refusing to index a zero-norm vector for entity ${this.id}` +
`a zero-norm vector is not a vector and never crosses an engine boundary ` +
`(the canonical write is unaffected; only the vector-index insert is skipped)`
)
return async () => {}
}
// Check if item already exists (for rollback decision)
const existed = await this.itemExists(this.id)
@ -263,6 +289,35 @@ export class ReplaceInVectorIndexOperation implements Operation {
// One commit generation for the whole replace (both branches + rollback).
const generation = this.generationFn?.()
// THE ZERO-NORM LAW (see AddToVectorIndexOperation's matching JSDoc): a
// real all-zero replacement vector must never land in the index — refuse
// loudly, canonical write unaffected. The row must not be left stale
// either: if it was genuinely indexed under `oldVector`, remove it
// rather than pretend the old vector still describes the row. A
// length-0 `newVector` (the unrelated "unvectored" shape) is handled the
// same way, silently — no caller today reaches this with an empty
// replacement (update() rejects a dimension-mismatched empty vector),
// but the seam stays consistent in case one ever legitimately does.
if (isZeroNormVector(this.newVector) || this.newVector.length === 0) {
const wasIndexed = this.oldVector.length > 0 && !isZeroNormVector(this.oldVector)
if (isZeroNormVector(this.newVector)) {
prodLog.warn(
`[vector-index] refusing to replace with a zero-norm vector for entity ${this.id}` +
`a zero-norm vector is not a vector and never crosses an engine boundary ` +
`(the canonical write is unaffected; the row is removed from the vector index instead)`
)
}
if (wasIndexed) {
await this.index.removeItem(this.id, generation)
}
return async () => {
// Restore the declared before-state.
if (wasIndexed) {
await this.index.addItem({ id: this.id, vector: this.oldVector }, generation)
}
}
}
if (typeof index.updateItem === 'function') {
// Atomic path: one in-place call, the row never leaves the index.
await index.updateItem({ id: this.id, vector: this.newVector }, generation)