fix(vfs): the VFS root never persists a zero-norm vector
A zero-norm vector is lawful inside brainy (cosine distance scores it at maximum, never a false top hit) but a false attractor for a downstream engine serving squared-euclidean distance, which cannot tell a real all-zero vector apart from a legitimate origin point. - The VFS root now persists with vector [] (the existing "unvectored" shape) instead of a real all-zero 384-dim placeholder, and is never routed into the deferred-embed pipeline. - A one-time migration in the root-init path detects a pre-fix store's all-zero placeholder root (by norm, not length) and rewrites it to [] through a new sanctioned Brainy method that keeps the canonical vectored-noun ledger honest and removes the row from the vector index. - The vector-index write seam (AddToVectorIndexOperation, ReplaceInVectorIndexOperation, and the generation materializer's direct insert) now refuses any real all-zero vector before it reaches a provider, loudly naming the entity, while the canonical write still lands. - add()'s dimension-pinning and HNSW-insert gates, and the add-params validator, now treat any empty vector as carrying no dimension information, closing a latent trap where an explicit `vector: []` would have pinned dimensions to 0.
This commit is contained in:
parent
aad9e2eeb1
commit
c6cc0de955
10 changed files with 516 additions and 64 deletions
|
|
@ -11,6 +11,7 @@ import { v4 as uuidv4 } from '../universal/uuid.js'
|
|||
import { Brainy } from '../brainy.js'
|
||||
import { Entity, AddParams, RelateParams, FindParams, Relation } from '../types/brainy.types.js'
|
||||
import { NounType, VerbType } from '../types/graphTypes.js'
|
||||
import { isZeroNormVector } from '../utils/distance.js'
|
||||
import { PathResolver } from './PathResolver.js'
|
||||
import { mimeDetector } from './MimeTypeDetector.js'
|
||||
import {
|
||||
|
|
@ -89,16 +90,6 @@ export class VirtualFileSystem implements IVirtualFileSystem {
|
|||
// Uses deterministic UUID format for storage compatibility
|
||||
private static readonly VFS_ROOT_ID = '00000000-0000-0000-0000-000000000000'
|
||||
|
||||
// OPEN-PATH FIX: the dimension of the placeholder vector given to the VFS
|
||||
// root when it is first created (see `doInitializeRoot`). Mirrors the
|
||||
// built-in WASM embedding engine's fixed, hardcoded output size
|
||||
// (all-MiniLM-L6-v2 — see src/embeddings/candle-wasm/src/lib.rs
|
||||
// HIDDEN_SIZE and src/embeddings/EmbeddingManager.ts). Deliberately NOT
|
||||
// derived from `brain.dimensions` — this constant only applies to the
|
||||
// default-embedder branch, where the true output dimension is this fixed
|
||||
// value by construction, never a moving target.
|
||||
private static readonly VFS_ROOT_VECTOR_DIMENSIONS = 384
|
||||
|
||||
/**
|
||||
* Construct a VFS bound to a Brainy instance.
|
||||
*
|
||||
|
|
@ -241,9 +232,11 @@ export class VirtualFileSystem implements IVirtualFileSystem {
|
|||
private async doInitializeRoot(): Promise<string> {
|
||||
const rootId = VirtualFileSystem.VFS_ROOT_ID
|
||||
|
||||
// Try to get existing root by fixed ID (O(1) lookup, not query)
|
||||
// Try to get existing root by fixed ID (O(1) lookup, not query).
|
||||
// includeVectors: true — the zero-norm migration below (leg 2) needs to
|
||||
// inspect the persisted vector to detect the legacy placeholder shape.
|
||||
try {
|
||||
const existingRoot = await this.brain.get(rootId)
|
||||
const existingRoot = await this.brain.get(rootId, { includeVectors: true })
|
||||
|
||||
if (existingRoot) {
|
||||
// Root exists - verify metadata is correct
|
||||
|
|
@ -260,6 +253,34 @@ export class VirtualFileSystem implements IVirtualFileSystem {
|
|||
})
|
||||
}
|
||||
|
||||
// ZERO-NORM ROOT MIGRATION (one-time): a pre-fix store persisted the
|
||||
// root with a REAL all-zero placeholder vector — lawful inside
|
||||
// brainy (cosineDistance treats a zero-norm operand as MAXIMUM
|
||||
// distance, see src/utils/distance.ts) but a "false attractor" for a
|
||||
// downstream engine serving squared-euclidean distance, which cannot
|
||||
// tell an all-zero vector apart from a legitimate origin point (a
|
||||
// production incident silently darkened 150+ rows in a partner
|
||||
// engine's index this way). THE LAW: a zero-norm vector is not a
|
||||
// vector — it never crosses an engine boundary. Detect the legacy
|
||||
// shape via NORM, not length or dimension (any real all-zero vector
|
||||
// qualifies, not just the historical 384-dim one), and rewrite it to
|
||||
// the "unvectored" `[]` shape through the sanctioned migration path
|
||||
// (Brainy.unvectorNounForRootMigration — see its JSDoc), which keeps
|
||||
// `getCanonicalCounts().vectors.all` honest and removes the row from
|
||||
// the vector index. Idempotent: a store already on the new shape
|
||||
// (vector.length === 0) takes the false branch below on every
|
||||
// subsequent init() — a permanent no-op, not a one-time flag.
|
||||
const existingVector = existingRoot.vector ?? []
|
||||
if (existingVector.length > 0 && isZeroNormVector(existingVector)) {
|
||||
const migrated = await this.brain.unvectorNounForRootMigration(rootId)
|
||||
if (migrated) {
|
||||
console.log(
|
||||
'VFS: migrated root vector from the legacy all-zero placeholder to the ' +
|
||||
'unvectored shape (zero-norm vectors never cross an engine boundary)'
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
return rootId
|
||||
}
|
||||
} catch (error) {
|
||||
|
|
@ -277,31 +298,42 @@ export class VirtualFileSystem implements IVirtualFileSystem {
|
|||
// meant every writer's FIRST-EVER open forced the process-global WASM
|
||||
// engine to cold-compile its model (measured 90-140s on throttled
|
||||
// CPUs) before the brain could even finish init(). This branch only
|
||||
// runs once per store (the root already exists — with a real vector —
|
||||
// in every previously-opened production store; reopening never re-adds
|
||||
// it), so the fix applies only to brand-new stores.
|
||||
// runs once per store (a previously-opened store already has a root —
|
||||
// see the migration above for the pre-fix shape — reopening never
|
||||
// re-adds it), so the fix applies only to brand-new stores.
|
||||
//
|
||||
// Chosen fix: an explicit all-zero vector, not `deferEmbedding: true`.
|
||||
// `deferEmbedding` looked attractive (ack now, embed later) but its
|
||||
// landing path (`kickEmbedWorker()`, called synchronously right after
|
||||
// commit — see brainy.ts add()/update()) would still force the WASM
|
||||
// engine to cold-compile within milliseconds of open, just off the
|
||||
// awaited path instead of never paying it at all — worse than the
|
||||
// explicit-vector path, which never touches the engine for this row.
|
||||
// An all-zero vector is safe: `cosineDistance` (src/utils/distance.ts)
|
||||
// explicitly returns the MAXIMUM distance whenever either operand's
|
||||
// norm is zero, so the root can never rank ahead of real content in a
|
||||
// similarity search, and HNSW indexes it like any other vector.
|
||||
// ZERO-NORM LAW (current shape, superseding the historical all-zero
|
||||
// placeholder): the root's vector is `[]` — the SAME "unvectored"
|
||||
// empty-array shape used for a deferred embed's stub and any other
|
||||
// not-yet-embedded row — never a real all-zero vector. A zero-norm
|
||||
// vector is lawful inside brainy (`cosineDistance`, see
|
||||
// src/utils/distance.ts, returns the MAXIMUM distance whenever either
|
||||
// operand's norm is zero) but is a "false attractor" for a downstream
|
||||
// engine serving squared-euclidean distance, which cannot tell a real
|
||||
// all-zero vector apart from a legitimate origin point — it silently
|
||||
// darkened 150+ rows in a partner engine's index in production. THE
|
||||
// LAW: a zero-norm vector is not a vector — it never crosses an engine
|
||||
// boundary. `vector: []` achieves the SAME cold-compile avoidance the
|
||||
// original placeholder did (`add()`'s dimension-pin and HNSW-insert
|
||||
// gates both key off `vector.length > 0`, so an empty vector never
|
||||
// calls embed(), never pins `brain.dimensions`, and never reaches the
|
||||
// vector index — see brainy.ts add()'s matching comments) while never
|
||||
// persisting a searchable zero vector for a downstream engine to trip
|
||||
// over. Deliberately NOT `deferEmbedding: true`: that flag's landing
|
||||
// path (`kickEmbedWorker()`, called synchronously right after commit —
|
||||
// see brainy.ts add()/update()) would still force the WASM engine to
|
||||
// cold-compile within milliseconds of open (just off the awaited path
|
||||
// instead of never paying it at all) AND would eventually embed the
|
||||
// root's data for real, which this fix forbids — the root must NEVER
|
||||
// be embedded, not merely "not yet".
|
||||
//
|
||||
// Only the default WASM engine gets this treatment — its output
|
||||
// dimension (384) is fixed and hardcoded, so the placeholder can never
|
||||
// mis-pin `brain.dimensions` for it. A plugin-registered native
|
||||
// 'embeddings' provider has no cold-compile cost AND may use a
|
||||
// different dimension, so it keeps embedding the root for real (same
|
||||
// as before this fix) rather than risk pinning the wrong dimension
|
||||
// ahead of the caller's own first real embed.
|
||||
// Only the default WASM engine gets this treatment — a plugin-
|
||||
// registered native 'embeddings' provider has no cold-compile cost and
|
||||
// may use a different dimension, so it keeps embedding the root for
|
||||
// real (same as before this fix) rather than leave Brainy's own
|
||||
// plumbing permanently unvectored on a store where embedding is cheap.
|
||||
const rootVector = this.brain.usesDefaultWasmEmbedder()
|
||||
? new Array(VirtualFileSystem.VFS_ROOT_VECTOR_DIMENSIONS).fill(0)
|
||||
? ([] as number[])
|
||||
: undefined
|
||||
|
||||
await this.brain.add({
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue