fix: spine hardening pass 1 (part) — count symmetry, honest partial-load, flush durability, read-fault propagation
Write/index-spine hardening, first batch of Pass 1. Each fix restores an invariant the surrounding code already intended; every one has a fail-before/pass-after test. - Pattern C, finding 5 (baseStorage): delete now decrements the user-facing scalar total symmetrically — deleteNounMetadata was decrementing only the per-type bucket, deleteVerbMetadata neither the bucket nor the scalar, so getNounCount()/getVerbCount() inflated permanently (the stale scalar wins pagination via Math.max and is persisted). Invariant now holds: scalar total === Σ per-type across add/update/delete and reopen. - Pattern A, finding 3 (graph/lsm/LSMTree): a partial SSTable-load failure no longer publishes the manifest's full relationship count as healthy. Any per-SSTable load failure throws after the batch, which resets to honest-empty and lets the existing size()===0 self-heal rebuild run — size()/isHealthy() can no longer lie about a partial load. - Pattern B, finding 6 (hnsw/hnswIndex): deferred flush() no longer clears dirty nodes whose connections failed to persist — failed nodes stay in the retry set, and flush() throws HnswFlushError instead of returning a lying node count. The immediate-mode first-noun saveHNSWSystem is un-swallowed, so addItem() rejects rather than returning an id for a rootless index. - Pattern B, finding 11 (part — storage reads): new shared isAbsentError() helper (utils/errorClassification, ENOENT-only absence) applied to loadBinaryBlob and readObjectFromPath — a real IO fault (EIO/EACCES/EMFILE) now propagates loudly instead of masquerading as "absent", which had driven needless rebuilds / empty reads (loadBinaryBlob feeds the native provider). Regression: 78 green across the 3 new suites + db-mvcc, generationStore, temporal-vfs, rollback-trapdoor, restore-nondestructive. Full gate runs before the Pass-1 release (David-gated). Remaining Pass 1: finding 11 getNoun/getVerb legs, finding 4 (ColumnStore), finding 8 (pending-flush), finding 10 (degraded), finding 7 (clear). Pattern A guards (1,2,9) as a follow-up release.
This commit is contained in:
parent
eb9c4eb963
commit
119087a75c
8 changed files with 518 additions and 30 deletions
76
tests/unit/graph/lsm-partial-load-failclosed.test.ts
Normal file
76
tests/unit/graph/lsm-partial-load-failclosed.test.ts
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
/**
|
||||
* @module tests/unit/graph/lsm-partial-load-failclosed
|
||||
* @description LSMTree partial-load fail-closed guard (Finding 3 of the honest
|
||||
* index-readiness spine plan). `loadSSTables()` used to swallow a per-SSTable
|
||||
* load failure with only a `prodLog.warn`, then `loadManifest()` still published
|
||||
* the FULL persisted `totalRelationships` count — so on a partial cold load,
|
||||
* `size()`/`isHealthy()` reported the full count and "healthy" while a subset of
|
||||
* the tree's edges were silently unqueryable. Fixed by failing the whole load
|
||||
* closed on ANY per-SSTable failure: `loadManifest()`'s existing catch then
|
||||
* resets `sstables`/`totalRelationships`/`sstablesByLevel` to honest-empty, so
|
||||
* `size()` reports 0 and the graph layer's existing `size()===0` self-heal
|
||||
* rebuilds from canonical records — the dishonest partial state can no longer be
|
||||
* observed.
|
||||
*/
|
||||
import { describe, it, expect } from 'vitest'
|
||||
import { LSMTree } from '../../../src/graph/lsm/LSMTree.js'
|
||||
import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js'
|
||||
|
||||
describe('LSMTree partial SSTable load fails closed (honest size())', () => {
|
||||
it('one SSTable fails to load → size() reports 0, not the manifest count', async () => {
|
||||
const storage = new MemoryStorage()
|
||||
await storage.init()
|
||||
const prefix = 'test-lsm-partial'
|
||||
|
||||
const t1 = new LSMTree(storage, { storagePrefix: prefix, enableCompaction: false })
|
||||
await t1.init()
|
||||
await t1.add('a', 'b'); await t1.flush() // SSTable #1
|
||||
await t1.add('c', 'd'); await t1.flush() // SSTable #2
|
||||
expect(t1.size()).toBeGreaterThanOrEqual(2)
|
||||
await t1.close()
|
||||
|
||||
// Find an SSTable id from the persisted manifest.
|
||||
const manifest = await storage.getMetadata(`${prefix}-manifest`)
|
||||
const sstableIds = Object.keys((manifest!.data as any).sstables)
|
||||
expect(sstableIds.length).toBeGreaterThanOrEqual(2)
|
||||
|
||||
// Reopen with ONE SSTable load forced to fail (deterministic).
|
||||
const t2 = new LSMTree(storage, { storagePrefix: prefix, enableCompaction: false })
|
||||
const realGet = storage.getMetadata.bind(storage)
|
||||
;(storage as any).getMetadata = async (key: string) => {
|
||||
if (key === `${prefix}-${sstableIds[1]}`) throw new Error('simulated SSTable load failure')
|
||||
return realGet(key)
|
||||
}
|
||||
await t2.init()
|
||||
|
||||
// BEFORE THE FIX: t2.size() === (full manifest count) ← LIE (partial load, full count)
|
||||
// AFTER THE FIX: t2.size() === 0 ← fail-closed → self-heal rebuilds
|
||||
expect(t2.size()).toBe(0)
|
||||
expect(t2.isHealthy()).toBe(true) // honestly-empty-but-initialized, not "healthy with a lie"
|
||||
|
||||
await t2.close()
|
||||
})
|
||||
|
||||
it('a clean load with no failures still reports the full honest count', async () => {
|
||||
const storage = new MemoryStorage()
|
||||
await storage.init()
|
||||
const prefix = 'test-lsm-clean'
|
||||
|
||||
const t1 = new LSMTree(storage, { storagePrefix: prefix, enableCompaction: false })
|
||||
await t1.init()
|
||||
await t1.add('a', 'b'); await t1.flush()
|
||||
await t1.add('c', 'd'); await t1.flush()
|
||||
const expectedSize = t1.size()
|
||||
expect(expectedSize).toBeGreaterThanOrEqual(2)
|
||||
await t1.close()
|
||||
|
||||
// Reopen with no injected failures — a normal, fully-successful cold load.
|
||||
const t2 = new LSMTree(storage, { storagePrefix: prefix, enableCompaction: false })
|
||||
await t2.init()
|
||||
|
||||
expect(t2.size()).toBe(expectedSize)
|
||||
expect(t2.isHealthy()).toBe(true)
|
||||
|
||||
await t2.close()
|
||||
})
|
||||
})
|
||||
Loading…
Add table
Add a link
Reference in a new issue