fix(vectors): a zero-norm vector is not a vector, canonical side included, plus the sanctioned unvector door
The engine-pair seam law: a zero-norm vector never crosses an engine
boundary. The index belt already refused to insert one, but the canonical
write and the vectored-noun ledger still counted it, so a near-empty store
whose only vectored row was zero-norm read "1 canonical vectored vs 0
indexed" and threw a not-ready error at open, and a store's own legacy
zero-norm VFS root could trip the same gate before its VFS-init-time cure
ever ran.
- add()/update() (single and transact()) now normalize an explicit
real all-zero vector to the unvectored [] shape before the dimension
pin, the ledger flag, and the index ops ever see it (loud, one warn per
write, canonical write still succeeds).
- The legacy counts.json derivation walk (scanVectoredNounCount) excludes
a persisted zero-norm row, matching the live ledger's definition.
- A legacy zero-norm VFS root now migrates at open, before the vector-leg
gate evaluates, via one O(1) fixed-path read (torn-tolerant — skips
rather than aborting init on a torn root, letting the recovery walk
heal it) — independent of whether a VirtualFileSystem is ever
constructed this session.
- update({ id, vector: [] }) (and the same op inside transact()) is now
the sanctioned, idempotent unvector door: index removal, exactly-once
ledger decrement, no re-embed, and it clears a pending deferred-embed
marker rather than leaving it to re-vectorize the row later. The
combination with deferEmbedding is a typed refusal.
- JsHnswVectorIndex.rebuild() now skips a zero-norm/empty persisted
vector when repopulating from canonical (the same belt the live
add/replace paths already had), and health()'s index-parity check now
compares HNSW size against the vectored-noun ledger rather than the
raw metadata-entry count, since a store's VFS root is permanently
unvectored by design.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
parent
fd6b4ce4ff
commit
0de7665930
6 changed files with 810 additions and 53 deletions
|
|
@ -19,6 +19,7 @@ import {
|
|||
import { getBrainyVersion } from '../../utils/index.js'
|
||||
import { isAbsentError } from '../../utils/errorClassification.js'
|
||||
import { prodLog } from '../../utils/logger.js'
|
||||
import { isZeroNormVector } from '../../utils/distance.js'
|
||||
import {
|
||||
TornRecordError,
|
||||
isUnparseablePayloadError,
|
||||
|
|
@ -2841,14 +2842,20 @@ export class FileSystemStorage extends BaseStorage {
|
|||
}
|
||||
|
||||
/**
|
||||
* Count canonical nouns holding a REAL (non-empty) vector — the vectored-
|
||||
* noun ledger scalar. UNLIKE {@link scanCanonicalEntities}, presence
|
||||
* cannot be decided from the id-directory listing alone: a deferred-embed
|
||||
* noun's `vectors.json` EXISTS (written at `add()` time with `vector: []`)
|
||||
* until its embed LANDS, so this walk reads every noun's `vectors.json`
|
||||
* CONTENT — O(nouns) reads, not O(ids) listing. Used ONLY for a one-time
|
||||
* legacy-counts.json derivation or a lost/corrupted counts.json recovery;
|
||||
* the result is persisted so this scan never repeats.
|
||||
* Count canonical nouns holding a REAL (non-empty, non-zero-norm) vector —
|
||||
* the vectored-noun ledger scalar. UNLIKE {@link scanCanonicalEntities},
|
||||
* presence cannot be decided from the id-directory listing alone: a
|
||||
* deferred-embed noun's `vectors.json` EXISTS (written at `add()` time
|
||||
* with `vector: []`) until its embed LANDS, so this walk reads every
|
||||
* noun's `vectors.json` CONTENT — O(nouns) reads, not O(ids) listing.
|
||||
* ZERO-NORM LAW: a real all-zero vector is not a vector — it never counts
|
||||
* here either (Brainy's write paths normalize an explicit zero-norm
|
||||
* vector to `[]` at write time, but a store created before that fix may
|
||||
* still carry legacy all-zero rows on disk; this derivation must agree
|
||||
* with the live ledger's definition of "vectored" regardless of when the
|
||||
* row was written). Used ONLY for a one-time legacy-counts.json derivation
|
||||
* or a lost/corrupted counts.json recovery; the result is persisted so
|
||||
* this scan never repeats.
|
||||
*/
|
||||
private async scanVectoredNounCount(): Promise<number> {
|
||||
const base = path.join(this.rootDir, 'entities', 'nouns')
|
||||
|
|
@ -2862,7 +2869,12 @@ export class FileSystemStorage extends BaseStorage {
|
|||
for (const entry of ids) {
|
||||
if (!entry.isDirectory()) continue
|
||||
const record = await this.readEntityVectorRaw(path.join(shardPath, entry.name))
|
||||
if (record && Array.isArray(record.vector) && record.vector.length > 0) {
|
||||
if (
|
||||
record &&
|
||||
Array.isArray(record.vector) &&
|
||||
record.vector.length > 0 &&
|
||||
!isZeroNormVector(record.vector)
|
||||
) {
|
||||
vectored++
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue