fix: honest response when a transaction rollback cannot complete
When a transaction failed and its rollback ALSO failed to undo a canonical write (retries exhausted), the old path logged, continued, threw TransactionRollbackError, and set state='rolled_back' — while the record it could not undo stayed durable on disk. The caller got an error implying the write was undone; a read-back showed the record. A failed rollback had no truthful response (the post-commit response lie). Give a failed rollback a two-branch honest contract, decided by observation: both commit paths already hold byte-identical before-images, so after a failed rollback the store compares current canonical state to them and classifies each touched record as reconciled, an additive orphan (present when it should be gone), or a restorative loss (gone/wrong when it should have been restored). - Adopt-forward: a single-op write whose only damage is a durably-present orphan — the record the caller asked for — is committed forward (its generation buffered) and returns success with a loud warning that the derived index may be incomplete for that id until the next rebuild/repairIndex(). No error, no double-write; the record is durable and get-able immediately. - Fail loud + quarantine: a multi-op batch, or ANY restorative loss, throws the new StoreInconsistentError naming every unreconciled record and its disposition, and puts the brain into write-quarantine (reads keep working, writes refused via assertWritable) until repairIndex() reconciles the derived indexes against canonical and lifts it. The counter is not advanced, and Transaction.rollback now reports state 'inconsistent' instead of the 'rolled_back' lie when any undo failed. New: StoreInconsistentError + UnreconciledRecord exported from the root; repairIndex() forces a rebuild and clears the quarantine. Regression tests/integration/rollback-trapdoor.test.ts injects index-add-throws + canonical-delete-undo-fails and pins adopt-forward (durable, get-able, not quarantined), fail-loud (StoreInconsistentError + quarantine + reads work + repairIndex lifts it), and the error's record naming.
This commit is contained in:
parent
036e56c9f9
commit
711d2f046a
7 changed files with 437 additions and 35 deletions
156
tests/integration/rollback-trapdoor.test.ts
Normal file
156
tests/integration/rollback-trapdoor.test.ts
Normal file
|
|
@ -0,0 +1,156 @@
|
|||
/**
|
||||
* @module tests/integration/rollback-trapdoor
|
||||
* @description Regression for a consumer-reported data-integrity bug: a failed
|
||||
* CANONICAL rollback undo left the record durable while the request errored (a
|
||||
* post-commit response lie). Reproduced by a failure-injection coherence harness:
|
||||
* a late index op throws → rollback runs → an earlier op's canonical undo
|
||||
* (storage.deleteNounMetadata) fails persistently → the record stays on disk
|
||||
* while Transaction.rollback threw TransactionRollbackError AND lied with
|
||||
* state='rolled_back'.
|
||||
*
|
||||
* Fix (David's ruling — adopt-forward when safe, else fail loud):
|
||||
* - SINGLE-op add whose only damage is a durably-present orphan → adopt it
|
||||
* forward: commit the generation (the record IS what add() wanted), return
|
||||
* success + a loud warn; the derived index heals on rebuild/repairIndex.
|
||||
* - MULTI-op batch, OR any restorative LOSS → StoreInconsistentError naming the
|
||||
* unreconciled ids; the brain enters write-quarantine (reads OK, writes
|
||||
* refused) until repairIndex() reconciles and lifts it. Transaction state is
|
||||
* 'inconsistent', never the 'rolled_back' lie.
|
||||
*
|
||||
* The injection mirrors that harness: fault index.addToIndex (trigger rollback) and
|
||||
* storage.deleteNounMetadata (fail the canonical undo).
|
||||
*/
|
||||
import { describe, it, expect, beforeEach, afterEach } from 'vitest'
|
||||
import * as fs from 'node:fs'
|
||||
import * as os from 'node:os'
|
||||
import * as path from 'node:path'
|
||||
import { Brainy } from '../../src/brainy.js'
|
||||
import { NounType } from '../../src/types/graphTypes.js'
|
||||
import { StoreInconsistentError } from '../../src/db/errors.js'
|
||||
|
||||
let seq = 0
|
||||
const freshId = (): string =>
|
||||
`00000000-0000-4000-8000-${(++seq).toString(16).padStart(12, '0')}`
|
||||
|
||||
describe('rollback trapdoor — failed canonical undo (data integrity)', () => {
|
||||
let dir: string
|
||||
let brain: any
|
||||
let restore: Array<() => void>
|
||||
|
||||
beforeEach(async () => {
|
||||
process.env.BRAINY_DETERMINISTIC_EMBEDDINGS = 'true'
|
||||
dir = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-trapdoor-'))
|
||||
brain = new Brainy({
|
||||
requireSubtype: false,
|
||||
storage: { type: 'filesystem', path: dir },
|
||||
dimensions: 384,
|
||||
silent: true
|
||||
})
|
||||
await brain.init()
|
||||
restore = []
|
||||
})
|
||||
|
||||
afterEach(async () => {
|
||||
for (const r of restore) r()
|
||||
try { await brain.close() } catch { /* ignore */ }
|
||||
fs.rmSync(dir, { recursive: true, force: true })
|
||||
})
|
||||
|
||||
/** Fault the metadata-index add so a write's rollback is triggered. */
|
||||
const faultIndexAdd = () => {
|
||||
const idx = brain['metadataIndex']
|
||||
const orig = idx.addToIndex.bind(idx)
|
||||
idx.addToIndex = async () => { throw new Error('injected: index add failed') }
|
||||
restore.push(() => { idx.addToIndex = orig })
|
||||
}
|
||||
/** Fault the canonical additive undo (delete of a just-created record). */
|
||||
const faultCanonicalDelete = () => {
|
||||
const storage = brain['storage']
|
||||
const orig = storage.deleteNounMetadata.bind(storage)
|
||||
storage.deleteNounMetadata = async () => { throw new Error('injected: canonical undo failed') }
|
||||
restore.push(() => { storage.deleteNounMetadata = orig })
|
||||
}
|
||||
|
||||
it('ADOPT-FORWARD: a single-op add whose canonical undo fails commits as a durable, get-able record (no throw, no quarantine)', async () => {
|
||||
const id = freshId()
|
||||
const genBefore = brain.generationStore.generation()
|
||||
|
||||
faultIndexAdd() // triggers rollback
|
||||
faultCanonicalDelete() // the trapdoor: the metadata record can't be un-written
|
||||
|
||||
// add() must NOT throw — the record it wanted is durable, so it is adopted.
|
||||
const returned = await brain.add({ id, data: 'orphan adopted', type: NounType.Thing })
|
||||
expect(returned).toBe(id)
|
||||
|
||||
// Un-fault so reads/subsequent writes use the real methods.
|
||||
for (const r of restore) r()
|
||||
restore = []
|
||||
|
||||
// The record is durable and get-able, and the generation advanced.
|
||||
expect(await brain.get(id)).not.toBeNull()
|
||||
expect(brain.generationStore.generation()).toBe(genBefore + 1)
|
||||
|
||||
// The store is NOT quarantined — a subsequent write succeeds.
|
||||
const ok = await brain.add({ id: freshId(), data: 'still writable', type: NounType.Thing })
|
||||
expect(await brain.get(ok)).not.toBeNull()
|
||||
})
|
||||
|
||||
it('FAIL-LOUD: a multi-op transact whose canonical undo fails throws StoreInconsistentError and write-quarantines the brain', async () => {
|
||||
// A pre-existing record to prove reads keep working under quarantine.
|
||||
const anchor = await brain.add({ id: freshId(), data: 'anchor', type: NounType.Thing })
|
||||
await brain.flush()
|
||||
|
||||
faultIndexAdd()
|
||||
faultCanonicalDelete()
|
||||
|
||||
const a = freshId()
|
||||
const b = freshId()
|
||||
await expect(
|
||||
brain.transact([
|
||||
{ op: 'add', id: a, data: 'batch A', type: NounType.Thing },
|
||||
{ op: 'add', id: b, data: 'batch B', type: NounType.Thing }
|
||||
])
|
||||
).rejects.toBeInstanceOf(StoreInconsistentError)
|
||||
|
||||
// Writes are refused (quarantine); reads still work.
|
||||
await expect(
|
||||
brain.add({ id: freshId(), data: 'blocked', type: NounType.Thing })
|
||||
).rejects.toThrow(/WRITE-QUARANTINED/)
|
||||
expect(await brain.get(anchor)).not.toBeNull()
|
||||
|
||||
// repairIndex() reconciles and lifts the quarantine → writes resume.
|
||||
for (const r of restore) r()
|
||||
restore = []
|
||||
await brain.repairIndex()
|
||||
|
||||
const ok = await brain.add({ id: freshId(), data: 'writable again', type: NounType.Thing })
|
||||
expect(await brain.get(ok)).not.toBeNull()
|
||||
})
|
||||
|
||||
it('the StoreInconsistentError names the unreconciled records with dispositions', async () => {
|
||||
faultIndexAdd()
|
||||
faultCanonicalDelete()
|
||||
|
||||
const a = freshId()
|
||||
const b = freshId()
|
||||
let caught: unknown
|
||||
await brain
|
||||
.transact([
|
||||
{ op: 'add', id: a, data: 'A', type: NounType.Thing },
|
||||
{ op: 'add', id: b, data: 'B', type: NounType.Thing }
|
||||
])
|
||||
.catch((e: unknown) => (caught = e))
|
||||
|
||||
expect(caught).toBeInstanceOf(StoreInconsistentError)
|
||||
const err = caught as StoreInconsistentError
|
||||
expect(err.records.length).toBeGreaterThan(0)
|
||||
// Every unreconciled record here is a durably-present orphan.
|
||||
expect(err.records.every((r) => r.disposition === 'orphan')).toBe(true)
|
||||
expect(err.message).toMatch(/WRITE-QUARANTINED/)
|
||||
|
||||
// Recover so afterEach can close cleanly.
|
||||
for (const r of restore) r()
|
||||
restore = []
|
||||
await brain.repairIndex()
|
||||
})
|
||||
})
|
||||
Loading…
Add table
Add a link
Reference in a new issue