fix(vfs): the VFS root never persists a zero-norm vector

A zero-norm vector is lawful inside brainy (cosine distance scores it at
maximum, never a false top hit) but a false attractor for a downstream
engine serving squared-euclidean distance, which cannot tell a real
all-zero vector apart from a legitimate origin point.

- The VFS root now persists with vector [] (the existing "unvectored"
  shape) instead of a real all-zero 384-dim placeholder, and is never
  routed into the deferred-embed pipeline.
- A one-time migration in the root-init path detects a pre-fix store's
  all-zero placeholder root (by norm, not length) and rewrites it to []
  through a new sanctioned Brainy method that keeps the canonical
  vectored-noun ledger honest and removes the row from the vector index.
- The vector-index write seam (AddToVectorIndexOperation,
  ReplaceInVectorIndexOperation, and the generation materializer's direct
  insert) now refuses any real all-zero vector before it reaches a
  provider, loudly naming the entity, while the canonical write still
  lands.
- add()'s dimension-pinning and HNSW-insert gates, and the add-params
  validator, now treat any empty vector as carrying no dimension
  information, closing a latent trap where an explicit `vector: []`
  would have pinned dimensions to 0.

tests/integration/canonical-count-ledger.test.ts

@@ -225,9 +225,12 @@ describe('canonical count ledger — the vectored-noun scalar (the vector leg\'s
   }
 
   /** Baseline vectored count right after a fresh open() — init() creates a
-   *  hidden system VFS-root noun that itself carries a real vector, so a
-   *  brand-new store's `vectors.all` is 1, not 0. Tests assert DELTAS off
-   *  this baseline rather than hardcoding it away. */
+   *  hidden system VFS-root noun, but (the zero-norm root cure) it is
+   *  deliberately UNVECTORED (`vector: []`, never a real all-zero
+   *  placeholder — a zero-norm vector never crosses an engine boundary), so
+   *  a brand-new store's `vectors.all` is 0. Tests still assert DELTAS off
+   *  this baseline rather than hardcoding it away, in case that ever
+   *  changes again. */
   let baseline: number
 
   beforeEach(async () => {
@@ -235,6 +238,7 @@ describe('canonical count ledger — the vectored-noun scalar (the vector leg\'s
     dir = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-vectored-ledger-'))
     brain = await open()
     baseline = (await brain.storage.getCanonicalCounts()).vectors.all
+    expect(baseline).toBe(0) // the unvectored VFS root contributes nothing
   })
   afterEach(async () => {
     vi.restoreAllMocks()
@@ -348,7 +352,7 @@ describe('canonical count ledger — the vectored-noun scalar (the vector leg\'s
 
     brain = await open()
     const ledger = await brain.storage.getCanonicalCounts()
-    expect(ledger.vectors.all).toBe(baseline + 1) // the root + the one non-deferred noun
+    expect(ledger.vectors.all).toBe(baseline + 1) // just the one non-deferred noun — the root is unvectored
     expect(ledger.vectors.all).toBe(countVectoredNouns(dir))
     const persisted = JSON.parse(fs.readFileSync(countsPath, 'utf-8'))
     expect(persisted.totalVectoredNounCount).toBe(baseline + 1)

tests/integration/vector-leg-open-build.test.ts

@@ -176,21 +176,22 @@ describe('vector-leg open-build (two-engine gate, last red)', () => {
   })
 
   it('the inverse: only deferred (never-landed) user nouns — the ledger is never inflated by them, and search over them honestly returns []', async () => {
-    // ARCHITECTURAL NOTE (found while building this pin): every brainy store
-    // carries ONE permanent, always-vectored noun beyond user data — the VFS
-    // root (`entities/nouns/.../00000000-0000-0000-0000-000000000000`,
-    // src/vfs/VirtualFileSystem.ts). It is inserted with an explicit all-zero
-    // (but non-empty, length-384) vector on EVERY store's first open — never
-    // deferred (a deliberate WASM-cold-compile-avoidance fix, see that
-    // file's comment) — and VFS init unconditionally re-creates it if
-    // missing, before the rebuild gate ever runs. A literal "0 vectored
-    // nouns" store is therefore unreachable through the public API; a
-    // brand-new store's `vectors.all` floor is 1, not 0. This pin verifies
-    // the law the task names in the ACHIEVABLE form: nouns whose embed is
-    // still deferred/unlanded contribute NOTHING to the vectored-noun ledger
-    // — the coverage-gap comparison sees exactly the root (1), never
-    // root+deferred — and semantic search over deferred-only user content
-    // honestly returns `[]` (no error, no false "coverage restored" claim).
+    // ARCHITECTURAL NOTE (updated by the zero-norm root cure): every brainy
+    // store carries ONE permanent VFS root noun beyond user data
+    // (`entities/nouns/.../00000000-0000-0000-0000-000000000000`,
+    // src/vfs/VirtualFileSystem.ts), created (or, on a pre-fix store,
+    // migrated) on every open — but it is deliberately UNVECTORED (vector
+    // `[]`), never a real all-zero placeholder: a zero-norm vector is not a
+    // vector and never crosses an engine boundary (see that file's
+    // doInitializeRoot() comment). It therefore contributes NOTHING to the
+    // vectored-noun ledger — a brand-new store's `vectors.all` floor is 0,
+    // not 1. This pin verifies the law the task names in the ACHIEVABLE
+    // form: nouns whose embed is still deferred/unlanded contribute NOTHING
+    // to the vectored-noun ledger either — the coverage-gap comparison sees
+    // exactly the baseline (the root, contributing 0), never
+    // baseline+deferred — and semantic search over deferred-only user
+    // content honestly returns `[]` (no error, no false "coverage restored"
+    // claim).
     const dir = mkTmp()
 
     const build: any = new Brainy({
@@ -202,6 +203,8 @@ describe('vector-leg open-build (two-engine gate, last red)', () => {
     })
     await build.init()
     const rootOnlyLedger = await build.storage.getCanonicalCounts()
+    // THE NEW LAW: the root is unvectored — a brand-new store's floor is 0.
+    expect(rootOnlyLedger.vectors.all).toBe(0)
     // Block the embedder permanently so every add below stays deferred and
     // unlanded for the rest of this test (a fast deterministic embedder
     // could otherwise land it before we ever observe the "still 0 extra"

tests/integration/vfs-root-zero-norm.test.ts

@@ -0,0 +1,203 @@
+/**
+ * @module tests/integration/vfs-root-zero-norm
+ * @description THE ZERO-NORM ROOT CURE — a production incident traced 150+
+ * darkened rows in a downstream engine's index to the VFS root's persisted
+ * ALL-ZERO placeholder vector: lawful inside brainy (`cosineDistance`
+ * treats a zero-norm operand as MAXIMUM distance, src/utils/distance.ts)
+ * but a "false attractor" for an engine serving squared-euclidean distance,
+ * which cannot tell a real all-zero vector apart from a legitimate origin
+ * point. THE LAW: a zero-norm vector is not a vector — it never crosses an
+ * engine boundary.
+ *
+ * Three legs pinned here:
+ * (a) the root persists NO zeros — a brand-new store creates it with
+ *     vector `[]` (the "unvectored" shape), absent from the HNSW index, and
+ *     the canonical vectored-noun ledger does not count it.
+ * (b) a ONE-TIME migration heals an existing (pre-fix) store: an old-shape
+ *     root (a REAL all-zero vector, genuinely indexed and ledgered — the
+ *     harness reproduces exactly what a pre-fix store looked like on disk)
+ *     is rewritten to `[]` on the next `init()`, the ledger is decremented
+ *     through the sanctioned path, and a second `init()` is a no-op.
+ * (c) THE BELT at the live provider-write seam: an entity added with an
+ *     EXPLICIT all-zero vector (any dimension) still lands its canonical
+ *     write, but the vector-index insert is refused loudly.
+ * (d) the migrated root never surfaces in `find()` results (it was already
+ *     hidden behind `visibility: 'system'` — this pin holds regardless).
+ */
+import { describe, it, expect, afterEach, vi } from 'vitest'
+import * as fs from 'node:fs'
+import * as os from 'node:os'
+import * as path from 'node:path'
+import { Brainy } from '../../src/index.js'
+import { NounType } from '../../src/types/graphTypes.js'
+import { prodLog } from '../../src/utils/logger.js'
+
+const ROOT_ID = '00000000-0000-0000-0000-000000000000'
+
+process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
+
+const tmpDirs: string[] = []
+function mkTmp(): string {
+  const d = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-vfs-root-zero-norm-'))
+  tmpDirs.push(d)
+  return d
+}
+afterEach(() => {
+  vi.restoreAllMocks()
+  for (const d of tmpDirs.splice(0)) fs.rmSync(d, { recursive: true, force: true })
+})
+
+function openBrain(dir: string): any {
+  return new Brainy({
+    requireSubtype: false,
+    storage: { type: 'filesystem', path: dir },
+    silent: true,
+    dimensions: 384
+  })
+}
+
+describe('VFS root zero-norm cure', () => {
+  it('(a) a brand-new store persists the root with vector [], absent from the HNSW index, and the canonical ledger counts it unvectored', async () => {
+    const dir = mkTmp()
+    const brain = openBrain(dir)
+    await brain.init()
+
+    const root = await brain.get(ROOT_ID, { includeVectors: true })
+    expect(root).not.toBeNull()
+    expect(root.vector).toEqual([])
+
+    const status = await brain.getIndexStatus()
+    expect(status.hnswIndex.size).toBe(0)
+
+    const ledger = await brain.storage.getCanonicalCounts()
+    expect(ledger.vectors.all).toBe(0)
+
+    await brain.close()
+  })
+
+  it('(b) an old-shape store (a real all-zero placeholder root) migrates to [] exactly once on init; the ledger is decremented through the sanctioned path; a second init is a no-op', async () => {
+    const dir = mkTmp()
+
+    // SESSION 1 — build the store, then hand-rewrite the root to the LEGACY
+    // shape: a REAL all-zero 384-dim vector, genuinely inserted into the
+    // vector index and genuinely counted by the vectored-noun ledger —
+    // reproducing exactly what a pre-fix store's root looked like on disk
+    // (the pre-fix add() always indexed + counted it). `index.addItem` is
+    // called directly (bypassing AddToVectorIndexOperation's own zero-norm
+    // belt, added by this same fix) precisely because the pre-fix code path
+    // had no such belt — this harness must match history, not the cure.
+    let brain = openBrain(dir)
+    await brain.init()
+    const oldVector = new Array(384).fill(0)
+    await brain.storage.saveNoun({ id: ROOT_ID, vector: oldVector, connections: new Map(), level: 0 })
+    await brain.index.addItem({ id: ROOT_ID, vector: oldVector })
+    await brain.storage.noteVectorLanded(ROOT_ID)
+    await brain.storage.persistCounts()
+    await brain.flush()
+
+    const ledgerBeforeMigration = await brain.storage.getCanonicalCounts()
+    expect(ledgerBeforeMigration.vectors.all).toBe(1)
+    await brain.close()
+
+    // SESSION 2 — reopen: VFS init must detect the legacy shape and migrate.
+    // Spy on the sanctioned migration method itself (not console output —
+    // `silent: true` monkey-patches `console.log` to a no-op INSIDE init(),
+    // which would silently swallow any pre-installed console spy too).
+    brain = openBrain(dir)
+    const migrateSpy = vi.spyOn(brain, 'unvectorNounForRootMigration')
+    await brain.init()
+
+    expect(migrateSpy).toHaveBeenCalledTimes(1)
+    expect(migrateSpy).toHaveBeenCalledWith(ROOT_ID)
+    await expect(migrateSpy.mock.results[0].value).resolves.toBe(true)
+
+    const migratedRoot = await brain.get(ROOT_ID, { includeVectors: true })
+    expect(migratedRoot.vector).toEqual([])
+
+    const ledgerAfterMigration = await brain.storage.getCanonicalCounts()
+    expect(ledgerAfterMigration.vectors.all).toBe(0)
+
+    const statusAfterMigration = await brain.getIndexStatus()
+    expect(statusAfterMigration.hnswIndex.size).toBe(0)
+
+    await brain.flush()
+    await brain.close()
+
+    // SESSION 3 — reopen again: the migration is a permanent no-op, not a
+    // one-time flag that silently re-drifts or re-fires. The zero-norm
+    // detection at the VFS init site never even calls the migration method
+    // again — the root's vector is already `[]`.
+    brain = openBrain(dir)
+    const migrateSpy2 = vi.spyOn(brain, 'unvectorNounForRootMigration')
+    await brain.init()
+
+    expect(migrateSpy2).not.toHaveBeenCalled()
+
+    const rootAfterSecondInit = await brain.get(ROOT_ID, { includeVectors: true })
+    expect(rootAfterSecondInit.vector).toEqual([])
+
+    const ledgerAfterSecondInit = await brain.storage.getCanonicalCounts()
+    expect(ledgerAfterSecondInit.vectors.all).toBe(0)
+
+    await brain.close()
+  })
+
+  it('(c) the live-write belt: an entity added with an explicit all-zero vector lands its canonical write, but the vector-index insert is refused loudly', async () => {
+    const dir = mkTmp()
+    const brain = openBrain(dir)
+    await brain.init()
+
+    const warnSpy = vi.spyOn(prodLog, 'warn')
+
+    const sizeBefore = (await brain.getIndexStatus()).hnswIndex.size
+    const zeroVector = new Array(384).fill(0)
+    const id = await brain.add({ data: 'poisoned entity', type: NounType.Document, vector: zeroVector })
+
+    // The canonical write succeeded — the entity is fully readable with its
+    // (real, all-zero) vector intact.
+    const entity = await brain.get(id, { includeVectors: true })
+    expect(entity).not.toBeNull()
+    expect(entity.vector).toEqual(zeroVector)
+
+    // The vector-index insert was skipped — the index size never moved.
+    const sizeAfter = (await brain.getIndexStatus()).hnswIndex.size
+    expect(sizeAfter).toBe(sizeBefore)
+
+    // The refusal was LOUD and named the entity.
+    const loudCall = warnSpy.mock.calls.find(
+      (call) => typeof call[0] === 'string' && call[0].includes(id) && call[0].toLowerCase().includes('zero-norm')
+    )
+    expect(loudCall).toBeDefined()
+
+    await brain.close()
+  })
+
+  it('(d) find() over a store whose root has been migrated never returns the root (already hidden behind visibility: system — pinned anyway)', async () => {
+    const dir = mkTmp()
+
+    // Build an old-shape store (same harness as pin (b)) and let it migrate.
+    let brain = openBrain(dir)
+    await brain.init()
+    const oldVector = new Array(384).fill(0)
+    await brain.storage.saveNoun({ id: ROOT_ID, vector: oldVector, connections: new Map(), level: 0 })
+    await brain.index.addItem({ id: ROOT_ID, vector: oldVector })
+    await brain.storage.noteVectorLanded(ROOT_ID)
+    await brain.storage.persistCounts()
+    await brain.add({ data: 'a document about technology', type: NounType.Document })
+    await brain.flush()
+    await brain.close()
+
+    brain = openBrain(dir) // migrates on init()
+    await brain.init()
+
+    const results = await brain.find({ query: 'technology', limit: 10 })
+    expect(results.some((r: any) => r.id === ROOT_ID)).toBe(false)
+
+    // Even asking explicitly for system-tier entities must never surface the
+    // root as a semantic-search HIT (it carries no vector to match against).
+    const resultsIncludingSystem = await brain.find({ query: 'technology', limit: 10, includeSystem: true })
+    expect(resultsIncludingSystem.some((r: any) => r.id === ROOT_ID)).toBe(false)
+
+    await brain.close()
+  })
+})