fix: spine hardening pass 1 (part) — count symmetry, honest partial-load, flush durability, read-fault propagation

Write/index-spine hardening, first batch of Pass 1. Each fix restores an
invariant the surrounding code already intended; every one has a
fail-before/pass-after test.

- Pattern C, finding 5 (baseStorage): delete now decrements the user-facing
  scalar total symmetrically — deleteNounMetadata was decrementing only the
  per-type bucket, deleteVerbMetadata neither the bucket nor the scalar, so
  getNounCount()/getVerbCount() inflated permanently (the stale scalar wins
  pagination via Math.max and is persisted). Invariant now holds:
  scalar total === Σ per-type across add/update/delete and reopen.

- Pattern A, finding 3 (graph/lsm/LSMTree): a partial SSTable-load failure no
  longer publishes the manifest's full relationship count as healthy. Any
  per-SSTable load failure throws after the batch, which resets to honest-empty
  and lets the existing size()===0 self-heal rebuild run — size()/isHealthy()
  can no longer lie about a partial load.

- Pattern B, finding 6 (hnsw/hnswIndex): deferred flush() no longer clears
  dirty nodes whose connections failed to persist — failed nodes stay in the
  retry set, and flush() throws HnswFlushError instead of returning a lying
  node count. The immediate-mode first-noun saveHNSWSystem is un-swallowed, so
  addItem() rejects rather than returning an id for a rootless index.

- Pattern B, finding 11 (part — storage reads): new shared isAbsentError()
  helper (utils/errorClassification, ENOENT-only absence) applied to
  loadBinaryBlob and readObjectFromPath — a real IO fault (EIO/EACCES/EMFILE)
  now propagates loudly instead of masquerading as "absent", which had driven
  needless rebuilds / empty reads (loadBinaryBlob feeds the native provider).

Regression: 78 green across the 3 new suites + db-mvcc, generationStore,
temporal-vfs, rollback-trapdoor, restore-nondestructive. Full gate runs before
the Pass-1 release (David-gated). Remaining Pass 1: finding 11 getNoun/getVerb
legs, finding 4 (ColumnStore), finding 8 (pending-flush), finding 10 (degraded),
finding 7 (clear). Pattern A guards (1,2,9) as a follow-up release.
This commit is contained in:
David Snelling 2026-07-13 08:50:07 -07:00
parent eb9c4eb963
commit 119087a75c
8 changed files with 518 additions and 30 deletions

View file

@ -574,6 +574,7 @@ export class LSMTree {
* Load SSTables from storage based on manifest
*/
private async loadSSTables(): Promise<void> {
const failures: string[] = []
const loadPromises: Promise<void>[] = []
this.manifest.sstables.forEach((level, sstableId) => {
@ -598,7 +599,12 @@ export class LSMTree {
}
}
} catch (error) {
// A per-SSTable load failure means the persisted adjacency is INCOMPLETE.
// Record it and fail the whole load closed (below): a partially-loaded
// tree that still publishes its full manifest count via size() would
// serve silent-empty traversals as truth (the cold-load swallow class).
prodLog.warn(`LSMTree: Failed to load SSTable ${sstableId}`, error)
failures.push(sstableId)
}
})()
@ -606,6 +612,20 @@ export class LSMTree {
})
await Promise.all(loadPromises)
if (failures.length > 0) {
// Fail closed. loadManifest()'s catch resets sstables/totalRelationships/
// sstablesByLevel to honest-empty, so size() reports 0 and the graph
// self-heal (_initializeGraphIndex size()===0 → rebuild) restores the index
// from the canonical records. Honest-partial is never published.
throw new Error(
`LSMTree(${this.config.storagePrefix}): ${failures.length} of ` +
`${this.manifest.sstables.size} SSTable(s) failed to load ` +
`(${failures.join(', ')}) — failing the load closed so size() reports 0 ` +
`and the graph self-heal rebuilds from canonical.`
)
}
prodLog.info(`LSMTree: Loaded ${this.manifest.sstables.size} SSTables`)
}