fix: spine hardening pass 1 (part) — count symmetry, honest partial-load, flush durability, read-fault propagation
Write/index-spine hardening, first batch of Pass 1. Each fix restores an invariant the surrounding code already intended; every one has a fail-before/pass-after test. - Pattern C, finding 5 (baseStorage): delete now decrements the user-facing scalar total symmetrically — deleteNounMetadata was decrementing only the per-type bucket, deleteVerbMetadata neither the bucket nor the scalar, so getNounCount()/getVerbCount() inflated permanently (the stale scalar wins pagination via Math.max and is persisted). Invariant now holds: scalar total === Σ per-type across add/update/delete and reopen. - Pattern A, finding 3 (graph/lsm/LSMTree): a partial SSTable-load failure no longer publishes the manifest's full relationship count as healthy. Any per-SSTable load failure throws after the batch, which resets to honest-empty and lets the existing size()===0 self-heal rebuild run — size()/isHealthy() can no longer lie about a partial load. - Pattern B, finding 6 (hnsw/hnswIndex): deferred flush() no longer clears dirty nodes whose connections failed to persist — failed nodes stay in the retry set, and flush() throws HnswFlushError instead of returning a lying node count. The immediate-mode first-noun saveHNSWSystem is un-swallowed, so addItem() rejects rather than returning an id for a rootless index. - Pattern B, finding 11 (part — storage reads): new shared isAbsentError() helper (utils/errorClassification, ENOENT-only absence) applied to loadBinaryBlob and readObjectFromPath — a real IO fault (EIO/EACCES/EMFILE) now propagates loudly instead of masquerading as "absent", which had driven needless rebuilds / empty reads (loadBinaryBlob feeds the native provider). Regression: 78 green across the 3 new suites + db-mvcc, generationStore, temporal-vfs, rollback-trapdoor, restore-nondestructive. Full gate runs before the Pass-1 release (David-gated). Remaining Pass 1: finding 11 getNoun/getVerb legs, finding 4 (ColumnStore), finding 8 (pending-flush), finding 10 (degraded), finding 7 (clear). Pattern A guards (1,2,9) as a follow-up release.
This commit is contained in:
parent
eb9c4eb963
commit
119087a75c
8 changed files with 518 additions and 30 deletions
|
|
@ -574,6 +574,7 @@ export class LSMTree {
|
|||
* Load SSTables from storage based on manifest
|
||||
*/
|
||||
private async loadSSTables(): Promise<void> {
|
||||
const failures: string[] = []
|
||||
const loadPromises: Promise<void>[] = []
|
||||
|
||||
this.manifest.sstables.forEach((level, sstableId) => {
|
||||
|
|
@ -598,7 +599,12 @@ export class LSMTree {
|
|||
}
|
||||
}
|
||||
} catch (error) {
|
||||
// A per-SSTable load failure means the persisted adjacency is INCOMPLETE.
|
||||
// Record it and fail the whole load closed (below): a partially-loaded
|
||||
// tree that still publishes its full manifest count via size() would
|
||||
// serve silent-empty traversals as truth (the cold-load swallow class).
|
||||
prodLog.warn(`LSMTree: Failed to load SSTable ${sstableId}`, error)
|
||||
failures.push(sstableId)
|
||||
}
|
||||
})()
|
||||
|
||||
|
|
@ -606,6 +612,20 @@ export class LSMTree {
|
|||
})
|
||||
|
||||
await Promise.all(loadPromises)
|
||||
|
||||
if (failures.length > 0) {
|
||||
// Fail closed. loadManifest()'s catch resets sstables/totalRelationships/
|
||||
// sstablesByLevel to honest-empty, so size() reports 0 and the graph
|
||||
// self-heal (_initializeGraphIndex size()===0 → rebuild) restores the index
|
||||
// from the canonical records. Honest-partial is never published.
|
||||
throw new Error(
|
||||
`LSMTree(${this.config.storagePrefix}): ${failures.length} of ` +
|
||||
`${this.manifest.sstables.size} SSTable(s) failed to load ` +
|
||||
`(${failures.join(', ')}) — failing the load closed so size() reports 0 ` +
|
||||
`and the graph self-heal rebuilds from canonical.`
|
||||
)
|
||||
}
|
||||
|
||||
prodLog.info(`LSMTree: Loaded ${this.manifest.sstables.size} SSTables`)
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue