fix(hnsw): skip unvectored rows on rebuild; refuse empty vectors in the index
A canonical row persisted with vector: [] (a system row, a deferred embed not yet landed, or any other legitimately-unvectored record) is a normal, enumerable row -- but rebuild()'s storage walk had no guard against it. storage.getVectorIndexData() derives its answer from the row's own record, so it returns non-null for any existing noun whether or not that noun was ever actually indexed -- rebuild() admitted such rows into the live graph with a length-0 vector. A vector-less node could become the entry point (or occupy any graph position); the next real insert then ran a distance calculation against it and blew up with a dimension mismatch. Fix at two layers in src/hnsw/hnswIndex.ts: - rebuild() now skips any row whose vector.length === 0 before it ever becomes a graph node (one summary count line, never per-row spam), and restores the pinned dimension from the first real vector it loads -- previously the pin stayed null across a restart, since addItem/updateItem are the only sites that set it and rebuild() never goes through either. - addItem/updateItem now refuse a length-0 vector with a typed EmptyVectorIndexError instead of ever pinning dimension to 0 or storing a vector-less node, so no future fill/rebuild/load path can poison the index silently. getVectorSafe's lazy-load "not found" check also missed that an empty array is truthy -- tightened to catch it. IndexOperations.ts's ReplaceInVectorIndexOperation rollback paths now skip re-adding an oldVector of length 0 (never a legal index member) instead of attempting an illegal empty re-insert on rollback. biography.test.ts's final ledger-exactness assertion assumed every noun the lane creates is vectored, including the VFS root counted in vfsBaselineNouns -- but the root is deliberately persisted unvectored. Corrected the expected formula to exclude it. Adds tests/integration/index-skips-unvectored.test.ts pinning: rebuild() indexes only vectored rows with the dimension pinned correctly; clear() then real adds never trip a dimension mismatch; addItem/updateItem refuse a length-0 vector; and a crash/repair cycle stays dimension-consistent.
This commit is contained in:
parent
fd6b4ce4ff
commit
8fc553b126
4 changed files with 371 additions and 8 deletions
|
|
@ -324,8 +324,17 @@ export class ReplaceInVectorIndexOperation implements Operation {
|
|||
|
||||
return async () => {
|
||||
// Restore the declared before-state in place (see class JSDoc for
|
||||
// the item-did-not-exist posture).
|
||||
await index.updateItem!({ id: this.id, vector: this.oldVector }, generation)
|
||||
// the item-did-not-exist posture). A length-0 oldVector means the row
|
||||
// was never actually indexed before this op ran (a length-0 vector is
|
||||
// never a legal index member — see EmptyVectorIndexError) — there is
|
||||
// no in-place "restore to empty" for the provider to perform, so
|
||||
// rollback removes the row instead, leaving the same "not indexed"
|
||||
// state the row was in before execute().
|
||||
if (this.oldVector.length > 0) {
|
||||
await index.updateItem!({ id: this.id, vector: this.oldVector }, generation)
|
||||
} else {
|
||||
await this.index.removeItem(this.id, generation)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -336,9 +345,14 @@ export class ReplaceInVectorIndexOperation implements Operation {
|
|||
|
||||
return async () => {
|
||||
// updateItem-style restore via the same adjacent pair, back to the
|
||||
// declared before-state.
|
||||
// declared before-state. Same length-0 carve-out as the updateItem
|
||||
// path above: an empty oldVector was never a legal index member, so
|
||||
// rollback just leaves the row removed rather than attempting an
|
||||
// illegal empty re-add.
|
||||
await this.index.removeItem(this.id, generation)
|
||||
await this.index.addItem({ id: this.id, vector: this.oldVector }, generation)
|
||||
if (this.oldVector.length > 0) {
|
||||
await this.index.addItem({ id: this.id, vector: this.oldVector }, generation)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Reference in a new issue