fix(vectors): a zero-norm vector is not a vector, canonical side included, plus the sanctioned unvector door
Some checks failed
CI / Node 22 (push) Successful in 12m22s
CI / Node 24 (push) Successful in 12m15s
CI / Integration + conformance (Node 22) (push) Failing after 14m49s
CI / Bun (latest) (push) Successful in 12m28s

The engine-pair seam law: a zero-norm vector never crosses an engine
boundary. The index belt already refused to insert one, but the canonical
write and the vectored-noun ledger still counted it, so a near-empty store
whose only vectored row was zero-norm read "1 canonical vectored vs 0
indexed" and threw a not-ready error at open, and a store's own legacy
zero-norm VFS root could trip the same gate before its VFS-init-time cure
ever ran.

- add()/update() (single and transact()) now normalize an explicit
  real all-zero vector to the unvectored [] shape before the dimension
  pin, the ledger flag, and the index ops ever see it (loud, one warn per
  write, canonical write still succeeds).
- The legacy counts.json derivation walk (scanVectoredNounCount) excludes
  a persisted zero-norm row, matching the live ledger's definition.
- A legacy zero-norm VFS root now migrates at open, before the vector-leg
  gate evaluates, via one O(1) fixed-path read (torn-tolerant — skips
  rather than aborting init on a torn root, letting the recovery walk
  heal it) — independent of whether a VirtualFileSystem is ever
  constructed this session.
- update({ id, vector: [] }) (and the same op inside transact()) is now
  the sanctioned, idempotent unvector door: index removal, exactly-once
  ledger decrement, no re-embed, and it clears a pending deferred-embed
  marker rather than leaving it to re-vectorize the row later. The
  combination with deferEmbedding is a typed refusal.
- JsHnswVectorIndex.rebuild() now skips a zero-norm/empty persisted
  vector when repopulating from canonical (the same belt the live
  add/replace paths already had), and health()'s index-parity check now
  compares HNSW size against the vectored-noun ledger rather than the
  raw metadata-entry count, since a store's VFS root is permanently
  unvectored by design.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
David Snelling 2026-08-27 13:53:09 -07:00
parent fd6b4ce4ff
commit 0de7665930
6 changed files with 810 additions and 53 deletions

View file

@ -19,6 +19,7 @@ import {
import { getBrainyVersion } from '../../utils/index.js'
import { isAbsentError } from '../../utils/errorClassification.js'
import { prodLog } from '../../utils/logger.js'
import { isZeroNormVector } from '../../utils/distance.js'
import {
TornRecordError,
isUnparseablePayloadError,
@ -2841,14 +2842,20 @@ export class FileSystemStorage extends BaseStorage {
}
/**
* Count canonical nouns holding a REAL (non-empty) vector the vectored-
* noun ledger scalar. UNLIKE {@link scanCanonicalEntities}, presence
* cannot be decided from the id-directory listing alone: a deferred-embed
* noun's `vectors.json` EXISTS (written at `add()` time with `vector: []`)
* until its embed LANDS, so this walk reads every noun's `vectors.json`
* CONTENT O(nouns) reads, not O(ids) listing. Used ONLY for a one-time
* legacy-counts.json derivation or a lost/corrupted counts.json recovery;
* the result is persisted so this scan never repeats.
* Count canonical nouns holding a REAL (non-empty, non-zero-norm) vector
* the vectored-noun ledger scalar. UNLIKE {@link scanCanonicalEntities},
* presence cannot be decided from the id-directory listing alone: a
* deferred-embed noun's `vectors.json` EXISTS (written at `add()` time
* with `vector: []`) until its embed LANDS, so this walk reads every
* noun's `vectors.json` CONTENT O(nouns) reads, not O(ids) listing.
* ZERO-NORM LAW: a real all-zero vector is not a vector it never counts
* here either (Brainy's write paths normalize an explicit zero-norm
* vector to `[]` at write time, but a store created before that fix may
* still carry legacy all-zero rows on disk; this derivation must agree
* with the live ledger's definition of "vectored" regardless of when the
* row was written). Used ONLY for a one-time legacy-counts.json derivation
* or a lost/corrupted counts.json recovery; the result is persisted so
* this scan never repeats.
*/
private async scanVectoredNounCount(): Promise<number> {
const base = path.join(this.rootDir, 'entities', 'nouns')
@ -2862,7 +2869,12 @@ export class FileSystemStorage extends BaseStorage {
for (const entry of ids) {
if (!entry.isDirectory()) continue
const record = await this.readEntityVectorRaw(path.join(shardPath, entry.name))
if (record && Array.isArray(record.vector) && record.vector.length > 0) {
if (
record &&
Array.isArray(record.vector) &&
record.vector.length > 0 &&
!isZeroNormVector(record.vector)
) {
vectored++
}
}