fix: refuse writes when single-op history cannot be made durable

The async group-commit flush that persists single-op generation history
swallowed persist failures as a bare warn: writes kept succeeding while their
before-images piled up in memory, never durable and unbounded, with no signal.

The generation store now accounts for every failed flush at one place
(flushPendingSingleOps), tolerates a transient blip (retry with capped
exponential backoff), and after PENDING_FLUSH_FAILURE_THRESHOLD consecutive
failures LATCHES a durability failure and refuses further single-op and transact
writes with a typed, exported PendingFlushDurabilityError — rather than promise a
durability it cannot deliver. Live canonical data is untouched; only the
immutable history is stuck. The latch self-heals: the moment a flush succeeds
(a retry, or an explicit flush()/close()) it lifts and writes resume.

Loud errors, never quiet losses.
This commit is contained in:
David Snelling 2026-07-13 09:31:25 -07:00
parent d8301f8d08
commit 54c183668c
4 changed files with 302 additions and 9 deletions

View file

@ -230,3 +230,62 @@ export class StoreInconsistentError extends Error {
this.cause = cause
}
}
/**
* @description Thrown by a write when the store cannot make single-op
* generation **history** durable: the asynchronous group-commit flush that
* persists buffered before-images to disk has failed repeatedly (a real
* storage fault a full disk, an EIO, a permission loss not a transient
* blip). Rather than keep accepting writes whose history silently piles up in
* memory and is never persisted an invisible durability loss, and an
* unbounded leak the store LATCHES this failure and refuses further writes
* until the pending tier drains.
*
* This is the loud, honest response to a broken history-durability path:
* - Live canonical data is unaffected (single-op live bytes are written before
* the history flush; only the immutable before-image history is stuck).
* - The background flush keeps retrying with backoff; when the underlying fault
* clears and a flush succeeds, the latch lifts and writes resume
* automatically. An explicit `flush()` also clears it on success.
* - {@link cause} is the underlying storage error from the last failed flush.
*
* A caller seeing this should treat it exactly like a full disk: stop writing,
* resolve the storage fault, and retry. It is NOT a data-corruption error no
* committed generation is lost it is a refusal to *promise* durability the
* store currently cannot deliver.
*
* @example
* try {
* await brain.add({ ... })
* } catch (err) {
* if (err instanceof PendingFlushDurabilityError) {
* // History can't be persisted right now (disk fault). Resolve storage,
* // then retry — the store self-heals once a flush succeeds.
* console.error('history durability stalled:', err.cause)
* }
* }
*/
export class PendingFlushDurabilityError extends Error {
/** The underlying storage error from the most recent failed flush. */
public override readonly cause: Error
/** How many consecutive flush attempts had failed when the latch tripped. */
public readonly failedAttempts: number
/**
* @param cause - The storage error from the last failed pending-tier flush.
* @param failedAttempts - Consecutive failed flush attempts at latch time.
*/
constructor(cause: Error, failedAttempts: number) {
super(
`Write refused: single-op generation history could not be made durable ` +
`after ${failedAttempts} consecutive flush attempts (${cause.message}). ` +
`Live data is intact, but buffered history is not yet persisted — the ` +
`store refuses further writes rather than silently accumulate ` +
`un-durable history. Resolve the underlying storage fault; the store ` +
`self-heals and resumes writes once a flush succeeds. Cause: ${cause.message}`
)
this.name = 'PendingFlushDurabilityError'
this.cause = cause
this.failedAttempts = failedAttempts
}
}