soulcraft/brainy
2
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: honest response when a transaction rollback cannot complete

Browse source 
When a transaction failed and its rollback ALSO failed to undo a canonical
write (retries exhausted), the old path logged, continued, threw
TransactionRollbackError, and set state='rolled_back' — while the record it
could not undo stayed durable on disk. The caller got an error implying the
write was undone; a read-back showed the record. A failed rollback had no
truthful response (the post-commit response lie).

Give a failed rollback a two-branch honest contract, decided by observation:
both commit paths already hold byte-identical before-images, so after a failed
rollback the store compares current canonical state to them and classifies each
touched record as reconciled, an additive orphan (present when it should be
gone), or a restorative loss (gone/wrong when it should have been restored).

- Adopt-forward: a single-op write whose only damage is a durably-present
  orphan — the record the caller asked for — is committed forward (its
  generation buffered) and returns success with a loud warning that the derived
  index may be incomplete for that id until the next rebuild/repairIndex(). No
  error, no double-write; the record is durable and get-able immediately.
- Fail loud + quarantine: a multi-op batch, or ANY restorative loss, throws the
  new StoreInconsistentError naming every unreconciled record and its
  disposition, and puts the brain into write-quarantine (reads keep working,
  writes refused via assertWritable) until repairIndex() reconciles the derived
  indexes against canonical and lifts it. The counter is not advanced, and
  Transaction.rollback now reports state 'inconsistent' instead of the
  'rolled_back' lie when any undo failed.

New: StoreInconsistentError + UnreconciledRecord exported from the root;
repairIndex() forces a rebuild and clears the quarantine. Regression
tests/integration/rollback-trapdoor.test.ts injects index-add-throws +
canonical-delete-undo-fails and pins adopt-forward (durable, get-able, not
quarantined), fail-loud (StoreInconsistentError + quarantine + reads work +
repairIndex lifts it), and the error's record naming.
This commit is contained in:
David Snelling 2026-07-12 12:19:09 -07:00
parent 036e56c9f9
commit 711d2f046a
7 changed files with 437 additions and 35 deletions
Download patch file Download diff file Expand all files Collapse all files

119
src/db/generationStore.ts
View file

@ -32,7 +32,9 @@
*/
import { prodLog } from '../utils/logger.js'
import { GenerationCompactedError, GenerationConflictError } from './errors.js'
import { GenerationCompactedError, GenerationConflictError, StoreInconsistentError } from './errors.js'
import type { UnreconciledRecord } from './errors.js'
import { TransactionRollbackError } from '../transaction/errors.js'
import type {
ChangedIds,
CompactHistoryOptions,
@ -618,18 +620,21 @@ export class GenerationStore {
// before staging; scoped outside the try so an abort can compensate.
let txBlobHashes: string[] = []
// Hoisted so the catch can reconcile canonical state against them after a
// failed rollback (the trapdoor). Empty until populated below.
const nounBefore = new Map<string, GenerationRecord>()
const verbBefore = new Map<string, GenerationRecord>()
try {
// -- 3. Before-images + delta (the durable undo log) ------------------
// Read every before-image FIRST, then run the caller's CAS
// precondition against them, and only then stage to disk — so a
// conflicting batch aborts with zero staging I/O. The maps hold the
// byte-identical records the staged files are written from.
const nounBefore = new Map<string, GenerationRecord>()
for (const id of nouns) {
const prev = await this.storage.readNounRaw(id)
nounBefore.set(id, { kind: 'noun', metadata: prev.metadata, vector: prev.vector })
}
const verbBefore = new Map<string, GenerationRecord>()
for (const id of verbs) {
const prev = await this.storage.readVerbRaw(id)
verbBefore.set(id, { kind: 'verb', metadata: prev.metadata, vector: prev.vector })
@ -737,6 +742,16 @@ export class GenerationStore {
if (crashSimulated) {
throw err
}
// The trapdoor for a batch: if rollback FAILED to fully apply, canonical
// storage may be inconsistent. A batch is never adopted forward (its
// other ops were rolled back — partial commit would break atomicity), so
// any unreconciled record is a fail-loud StoreInconsistentError. Compute
// it here (before the staging cleanup, which is always safe to run) and
// throw it in place of the raw error at the end.
let inconsistent: UnreconciledRecord[] = []
if (err instanceof TransactionRollbackError) {
inconsistent = await this.reconcileFailedRollback(nounBefore, verbBefore)
}
// Failed before the manifest rename: nothing is committed. Remove the
// staging directory; the TransactionManager already restored any
// applied operation byte-identically.
@ -763,11 +778,68 @@ export class GenerationStore {
// Return the reservation when no concurrent bump consumed a later
// number, so a failed transaction leaves generation() unchanged.
if (this.counter === gen) this.counter = gen - 1
// A failed rollback that left canonical records unreconciled surfaces as
// a loud StoreInconsistentError (brainy quarantines writes until repair);
// otherwise the raw error (clean abort, or a derived-only undo failure
// the egress guard + rebuild handle).
if (inconsistent.length > 0) {
throw new StoreInconsistentError(
inconsistent,
(err as TransactionRollbackError).originalError
)
}
throw err
}
})
}
/**
* @description After a transaction's rollback FAILED to fully apply
* (`TransactionRollbackError`), determine which touched records are now
* unreconciled by comparing current canonical state to the before-images the
* commit captured. This observes reality rather than trusting the opaque undo
* closures — the authoritative signal for adopt-forward vs fail-loud.
*
* @param nounBefore - Byte-identical entity before-images captured pre-execute.
* @param verbBefore - Byte-identical relationship before-images.
* @returns Every record whose canonical state no longer matches its
* before-image, tagged `'orphan'` (durably present when it should be gone —
* an add whose delete-undo failed) or `'loss'` (durably gone/wrong when it
* should have been restored — a remove/update whose restore-undo failed). An
* empty array means canonical storage is cleanly rolled back (only a derived
* index undo failed — the egress guard + a rebuild handle that).
*/
private async reconcileFailedRollback(
nounBefore: Map<string, GenerationRecord>,
verbBefore: Map<string, GenerationRecord>
): Promise<UnreconciledRecord[]> {
const out: UnreconciledRecord[] = []
const classify = (
before: GenerationRecord,
current: { metadata: unknown | null; vector: unknown | null }
): 'orphan' | 'loss' | null => {
const beforeEmpty = before.metadata == null && before.vector == null
const currentEmpty = current.metadata == null && current.vector == null
if (beforeEmpty && currentEmpty) return null // add cleanly undone
if (beforeEmpty) return 'orphan' // add's delete-undo failed → still present
if (currentEmpty) return 'loss' // remove's restore-undo failed → gone
// Both present: same value = reconciled; different = restore left a wrong value.
const same =
JSON.stringify({ m: before.metadata, v: before.vector }) ===
JSON.stringify({ m: current.metadata, v: current.vector })
return same ? null : 'loss'
}
for (const [id, before] of nounBefore) {
const d = classify(before, await this.storage.readNounRaw(id))
if (d) out.push({ id, kind: 'noun', disposition: d })
}
for (const [id, before] of verbBefore) {
const d = classify(before, await this.storage.readVerbRaw(id))
if (d) out.push({ id, kind: 'verb', disposition: d })
}
return out
}
// ==========================================================================
// Single-operation commit (Model-B per-write group-commit)
// ==========================================================================
@ -830,7 +902,7 @@ export class GenerationStore {
touched: { nouns?: string[]; verbs?: string[] }
execute: () => Promise<void>
precommit?: (before: CommitBeforeImages) => void
}): Promise<{ generation: number; timestamp: number }> {
}): Promise<{ generation: number; timestamp: number; degraded?: string[] }> {
return this.withMutex(async () => {
const nouns = args.touched.nouns ? [...new Set(args.touched.nouns)] : []
const verbs = args.touched.verbs ? [...new Set(args.touched.verbs)] : []
@ -873,9 +945,42 @@ export class GenerationStore {
await args.execute()
} catch (err) {
this.inTransact = false
// Live write failed: nothing buffered, nothing on disk. Return the
// reservation (when no concurrent bump consumed a later number) so
// generation() is unchanged.
// A failed rollback (TransactionRollbackError) may have left canonical
// storage inconsistent — the trapdoor. Reconcile against the
// before-images to decide the honest response (David's ruling:
// adopt-forward when safe, else fail loud).
if (err instanceof TransactionRollbackError) {
const records = await this.reconcileFailedRollback(nounBefore, verbBefore)
if (records.length > 0 && records.every((r) => r.disposition === 'orphan')) {
// ADOPT FORWARD: the durably-present orphan(s) are exactly what this
// single-op add wrote. Keep + buffer the generation so the record is
// legitimately committed and readable; the derived index may be
// incomplete for these ids until the next rebuild/repairIndex (the
// egress guard prevents wrong results meanwhile). Loud, honest,
// no double-write.
this.pendingBuffer.set(gen, { nouns: nounBefore, verbs: verbBefore, timestamp })
this.pendingGens.push(gen)
this.extendChains(gen, nouns, verbs)
prodLog.warn(
`[GenerationStore] Recovered a failed rollback FORWARD: single-op write ` +
`committed as generation ${gen} because its canonical undo could not be ` +
`applied (record(s) ${records.map((r) => r.id).join(', ')} are durable). ` +
`The derived index may be incomplete for these ids — run repairIndex() to heal.`
)
return { generation: gen, timestamp, degraded: records.map((r) => r.id) }
}
if (records.length > 0) {
// FAIL LOUD: a loss (or mixed) cannot be safely adopted. Return the
// reservation and throw — brainy quarantines writes until repair.
if (this.counter === gen) this.counter = gen - 1
throw new StoreInconsistentError(records, err.originalError)
}
// records.length === 0: canonical is cleanly rolled back (only a
// derived undo failed) — fall through to the clean-abort path below.
}
// Live write failed with canonical cleanly rolled back: nothing buffered,
// nothing durable. Return the reservation (when no concurrent bump
// consumed a later number) so generation() is unchanged.
if (this.counter === gen) this.counter = gen - 1
throw err
}