fix(vectors): a zero-norm vector is not a vector, canonical side included, plus the sanctioned unvector door
Some checks failed
CI / Node 22 (push) Successful in 12m22s
CI / Node 24 (push) Successful in 12m15s
CI / Integration + conformance (Node 22) (push) Failing after 14m49s
CI / Bun (latest) (push) Successful in 12m28s

The engine-pair seam law: a zero-norm vector never crosses an engine
boundary. The index belt already refused to insert one, but the canonical
write and the vectored-noun ledger still counted it, so a near-empty store
whose only vectored row was zero-norm read "1 canonical vectored vs 0
indexed" and threw a not-ready error at open, and a store's own legacy
zero-norm VFS root could trip the same gate before its VFS-init-time cure
ever ran.

- add()/update() (single and transact()) now normalize an explicit
  real all-zero vector to the unvectored [] shape before the dimension
  pin, the ledger flag, and the index ops ever see it (loud, one warn per
  write, canonical write still succeeds).
- The legacy counts.json derivation walk (scanVectoredNounCount) excludes
  a persisted zero-norm row, matching the live ledger's definition.
- A legacy zero-norm VFS root now migrates at open, before the vector-leg
  gate evaluates, via one O(1) fixed-path read (torn-tolerant — skips
  rather than aborting init on a torn root, letting the recovery walk
  heal it) — independent of whether a VirtualFileSystem is ever
  constructed this session.
- update({ id, vector: [] }) (and the same op inside transact()) is now
  the sanctioned, idempotent unvector door: index removal, exactly-once
  ledger decrement, no re-embed, and it clears a pending deferred-embed
  marker rather than leaving it to re-vectorize the row later. The
  combination with deferEmbedding is a typed refusal.
- JsHnswVectorIndex.rebuild() now skips a zero-norm/empty persisted
  vector when repopulating from canonical (the same belt the live
  add/replace paths already had), and health()'s index-parity check now
  compares HNSW size against the vectored-noun ledger rather than the
  raw metadata-entry count, since a store's VFS root is permanently
  unvectored by design.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
David Snelling 2026-08-27 13:53:09 -07:00
parent fd6b4ce4ff
commit 0de7665930
6 changed files with 810 additions and 53 deletions

View file

@ -18,9 +18,16 @@
* harness reproduces exactly what a pre-fix store looked like on disk)
* is rewritten to `[]` on the next `init()`, the ledger is decremented
* through the sanctioned path, and a second `init()` is a no-op.
* (c) THE BELT at the live provider-write seam: an entity added with an
* EXPLICIT all-zero vector (any dimension) still lands its canonical
* write, but the vector-index insert is refused loudly.
* (c) THE CANONICAL-WRITE NORMALIZATION (Leg A of the follow-up
* zero-norm/unvector-door fix): an entity added with an EXPLICIT
* all-zero vector (any dimension) is normalized to the "unvectored"
* `[]` shape BEFORE the canonical write, the ledger flag, and the index
* ops ever see it the canonical write still succeeds, loudly, and the
* vector-index insert never happens (nothing to index). Supersedes the
* original "canonical keeps the zero vector, only the index refuses"
* shape: a downstream engine's health-report gate reads the canonical
* ledger directly, so leaving a zero-norm vector on the canonical side
* re-opened the exact false-attractor risk this whole fix closes.
* (d) the migrated root never surfaces in `find()` results (it was already
* hidden behind `visibility: 'system'` this pin holds regardless).
*/
@ -142,7 +149,7 @@ describe('VFS root zero-norm cure', () => {
await brain.close()
})
it('(c) the live-write belt: an entity added with an explicit all-zero vector lands its canonical write, but the vector-index insert is refused loudly', async () => {
it('(c) canonical-write normalization: an entity added with an explicit all-zero vector persists UNVECTORED ([]), loudly, and never reaches the vector index', async () => {
const dir = mkTmp()
const brain = openBrain(dir)
await brain.init()
@ -150,20 +157,26 @@ describe('VFS root zero-norm cure', () => {
const warnSpy = vi.spyOn(prodLog, 'warn')
const sizeBefore = (await brain.getIndexStatus()).hnswIndex.size
const ledgerBefore = await brain.storage.getCanonicalCounts()
const zeroVector = new Array(384).fill(0)
const id = await brain.add({ data: 'poisoned entity', type: NounType.Document, vector: zeroVector })
// The canonical write succeeded — the entity is fully readable with its
// (real, all-zero) vector intact.
// The canonical write succeeded — but the zero-norm vector was
// normalized to the "unvectored" `[]` shape BEFORE it was persisted
// (Leg A: a zero-norm vector is not a vector — it never crosses an
// engine boundary, canonical side included).
const entity = await brain.get(id, { includeVectors: true })
expect(entity).not.toBeNull()
expect(entity.vector).toEqual(zeroVector)
expect(entity.vector).toEqual([])
// The vector-index insert was skipped — the index size never moved.
// Nothing to index — the vector-index size never moved, and the
// vectored-noun ledger never counted this row.
const sizeAfter = (await brain.getIndexStatus()).hnswIndex.size
expect(sizeAfter).toBe(sizeBefore)
const ledgerAfter = await brain.storage.getCanonicalCounts()
expect(ledgerAfter.vectors.all).toBe(ledgerBefore.vectors.all)
// The refusal was LOUD and named the entity.
// The normalization was LOUD and named the entity.
const loudCall = warnSpy.mock.calls.find(
(call) => typeof call[0] === 'string' && call[0].includes(id) && call[0].toLowerCase().includes('zero-norm')
)