fix(hnsw): skip unvectored rows on rebuild; refuse empty vectors in the index
Some checks failed
CI / Node 22 (push) Successful in 12m14s
CI / Node 24 (push) Successful in 12m15s
CI / Integration + conformance (Node 22) (push) Failing after 14m55s
CI / Bun (latest) (push) Successful in 12m24s

A canonical row persisted with vector: [] (a system row, a deferred embed
not yet landed, or any other legitimately-unvectored record) is a normal,
enumerable row -- but rebuild()'s storage walk had no guard against it.
storage.getVectorIndexData() derives its answer from the row's own record,
so it returns non-null for any existing noun whether or not that noun was
ever actually indexed -- rebuild() admitted such rows into the live graph
with a length-0 vector. A vector-less node could become the entry point (or
occupy any graph position); the next real insert then ran a distance
calculation against it and blew up with a dimension mismatch.

Fix at two layers in src/hnsw/hnswIndex.ts:
- rebuild() now skips any row whose vector.length === 0 before it ever
  becomes a graph node (one summary count line, never per-row spam), and
  restores the pinned dimension from the first real vector it loads --
  previously the pin stayed null across a restart, since addItem/updateItem
  are the only sites that set it and rebuild() never goes through either.
- addItem/updateItem now refuse a length-0 vector with a typed
  EmptyVectorIndexError instead of ever pinning dimension to 0 or storing a
  vector-less node, so no future fill/rebuild/load path can poison the index
  silently. getVectorSafe's lazy-load "not found" check also missed that an
  empty array is truthy -- tightened to catch it.

IndexOperations.ts's ReplaceInVectorIndexOperation rollback paths now skip
re-adding an oldVector of length 0 (never a legal index member) instead of
attempting an illegal empty re-insert on rollback.

biography.test.ts's final ledger-exactness assertion assumed every noun the
lane creates is vectored, including the VFS root counted in
vfsBaselineNouns -- but the root is deliberately persisted unvectored.
Corrected the expected formula to exclude it.

Adds tests/integration/index-skips-unvectored.test.ts pinning: rebuild()
indexes only vectored rows with the dimension pinned correctly; clear()
then real adds never trip a dimension mismatch; addItem/updateItem refuse a
length-0 vector; and a crash/repair cycle stays dimension-consistent.
This commit is contained in:
David Snelling 2026-08-27 13:29:11 -07:00
parent fd6b4ce4ff
commit 8fc553b126
4 changed files with 371 additions and 8 deletions

View file

@ -382,9 +382,17 @@ describe.sequential('lifecycle — the working store', () => {
},
// Every noun this biography ever adds carries an explicit/computed
// vector (the harness never defers an embed), so the vectored-noun
// scalar tracks nouns.all exactly.
// scalar tracks nouns.all exactly EXCEPT for the VFS root counted
// in `vfsBaselineNouns`: the root is deliberately persisted with
// `vector: []` (the sanctioned "unvectored" shape — see
// VirtualFileSystem.doInitializeRoot()'s zero-norm-avoidance
// comment) so it never pays the WASM engine's cold-compile cost and
// never crosses an engine boundary as a false attractor. It is the
// ONE hidden-tier record `vfsBaselineNouns` represents (see
// biographyHarness's module header), so it is excluded here even
// though it counts toward `nouns.all`.
vectors: {
all: aliveEntities.length + model.vfsFileNouns + model.vfsBaselineNouns
all: aliveEntities.length + model.vfsFileNouns
},
suspect: false
})