fix(vectors): a zero-norm vector is not a vector, canonical side included, plus the sanctioned unvector door
The engine-pair seam law: a zero-norm vector never crosses an engine
boundary. The index belt already refused to insert one, but the canonical
write and the vectored-noun ledger still counted it, so a near-empty store
whose only vectored row was zero-norm read "1 canonical vectored vs 0
indexed" and threw a not-ready error at open, and a store's own legacy
zero-norm VFS root could trip the same gate before its VFS-init-time cure
ever ran.
- add()/update() (single and transact()) now normalize an explicit
real all-zero vector to the unvectored [] shape before the dimension
pin, the ledger flag, and the index ops ever see it (loud, one warn per
write, canonical write still succeeds).
- The legacy counts.json derivation walk (scanVectoredNounCount) excludes
a persisted zero-norm row, matching the live ledger's definition.
- A legacy zero-norm VFS root now migrates at open, before the vector-leg
gate evaluates, via one O(1) fixed-path read (torn-tolerant — skips
rather than aborting init on a torn root, letting the recovery walk
heal it) — independent of whether a VirtualFileSystem is ever
constructed this session.
- update({ id, vector: [] }) (and the same op inside transact()) is now
the sanctioned, idempotent unvector door: index removal, exactly-once
ledger decrement, no re-embed, and it clears a pending deferred-embed
marker rather than leaving it to re-vectorize the row later. The
combination with deferEmbedding is a typed refusal.
- JsHnswVectorIndex.rebuild() now skips a zero-norm/empty persisted
vector when repopulating from canonical (the same belt the live
add/replace paths already had), and health()'s index-parity check now
compares HNSW size against the vectored-noun ledger rather than the
raw metadata-entry count, since a store's VFS root is permanently
unvectored by design.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
fd6b4ce4ff
commit
0de7665930
6 changed files with 810 additions and 53 deletions
|
|
@ -18,9 +18,16 @@
|
|||
* harness reproduces exactly what a pre-fix store looked like on disk)
|
||||
* is rewritten to `[]` on the next `init()`, the ledger is decremented
|
||||
* through the sanctioned path, and a second `init()` is a no-op.
|
||||
* (c) THE BELT at the live provider-write seam: an entity added with an
|
||||
* EXPLICIT all-zero vector (any dimension) still lands its canonical
|
||||
* write, but the vector-index insert is refused loudly.
|
||||
* (c) THE CANONICAL-WRITE NORMALIZATION (Leg A of the follow-up
|
||||
* zero-norm/unvector-door fix): an entity added with an EXPLICIT
|
||||
* all-zero vector (any dimension) is normalized to the "unvectored"
|
||||
* `[]` shape BEFORE the canonical write, the ledger flag, and the index
|
||||
* ops ever see it — the canonical write still succeeds, loudly, and the
|
||||
* vector-index insert never happens (nothing to index). Supersedes the
|
||||
* original "canonical keeps the zero vector, only the index refuses"
|
||||
* shape: a downstream engine's health-report gate reads the canonical
|
||||
* ledger directly, so leaving a zero-norm vector on the canonical side
|
||||
* re-opened the exact false-attractor risk this whole fix closes.
|
||||
* (d) the migrated root never surfaces in `find()` results (it was already
|
||||
* hidden behind `visibility: 'system'` — this pin holds regardless).
|
||||
*/
|
||||
|
|
@ -142,7 +149,7 @@ describe('VFS root zero-norm cure', () => {
|
|||
await brain.close()
|
||||
})
|
||||
|
||||
it('(c) the live-write belt: an entity added with an explicit all-zero vector lands its canonical write, but the vector-index insert is refused loudly', async () => {
|
||||
it('(c) canonical-write normalization: an entity added with an explicit all-zero vector persists UNVECTORED ([]), loudly, and never reaches the vector index', async () => {
|
||||
const dir = mkTmp()
|
||||
const brain = openBrain(dir)
|
||||
await brain.init()
|
||||
|
|
@ -150,20 +157,26 @@ describe('VFS root zero-norm cure', () => {
|
|||
const warnSpy = vi.spyOn(prodLog, 'warn')
|
||||
|
||||
const sizeBefore = (await brain.getIndexStatus()).hnswIndex.size
|
||||
const ledgerBefore = await brain.storage.getCanonicalCounts()
|
||||
const zeroVector = new Array(384).fill(0)
|
||||
const id = await brain.add({ data: 'poisoned entity', type: NounType.Document, vector: zeroVector })
|
||||
|
||||
// The canonical write succeeded — the entity is fully readable with its
|
||||
// (real, all-zero) vector intact.
|
||||
// The canonical write succeeded — but the zero-norm vector was
|
||||
// normalized to the "unvectored" `[]` shape BEFORE it was persisted
|
||||
// (Leg A: a zero-norm vector is not a vector — it never crosses an
|
||||
// engine boundary, canonical side included).
|
||||
const entity = await brain.get(id, { includeVectors: true })
|
||||
expect(entity).not.toBeNull()
|
||||
expect(entity.vector).toEqual(zeroVector)
|
||||
expect(entity.vector).toEqual([])
|
||||
|
||||
// The vector-index insert was skipped — the index size never moved.
|
||||
// Nothing to index — the vector-index size never moved, and the
|
||||
// vectored-noun ledger never counted this row.
|
||||
const sizeAfter = (await brain.getIndexStatus()).hnswIndex.size
|
||||
expect(sizeAfter).toBe(sizeBefore)
|
||||
const ledgerAfter = await brain.storage.getCanonicalCounts()
|
||||
expect(ledgerAfter.vectors.all).toBe(ledgerBefore.vectors.all)
|
||||
|
||||
// The refusal was LOUD and named the entity.
|
||||
// The normalization was LOUD and named the entity.
|
||||
const loudCall = warnSpy.mock.calls.find(
|
||||
(call) => typeof call[0] === 'string' && call[0].includes(id) && call[0].toLowerCase().includes('zero-norm')
|
||||
)
|
||||
|
|
|
|||
Reference in a new issue