fix: refuse writes when single-op history cannot be made durable
The async group-commit flush that persists single-op generation history swallowed persist failures as a bare warn: writes kept succeeding while their before-images piled up in memory, never durable and unbounded, with no signal. The generation store now accounts for every failed flush at one place (flushPendingSingleOps), tolerates a transient blip (retry with capped exponential backoff), and after PENDING_FLUSH_FAILURE_THRESHOLD consecutive failures LATCHES a durability failure and refuses further single-op and transact writes with a typed, exported PendingFlushDurabilityError — rather than promise a durability it cannot deliver. Live canonical data is untouched; only the immutable history is stuck. The latch self-heals: the moment a flush succeeds (a retry, or an explicit flush()/close()) it lifts and writes resume. Loud errors, never quiet losses.
This commit is contained in:
parent
d8301f8d08
commit
54c183668c
4 changed files with 302 additions and 9 deletions
|
|
@ -230,3 +230,62 @@ export class StoreInconsistentError extends Error {
|
|||
this.cause = cause
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Thrown by a write when the store cannot make single-op
|
||||
* generation **history** durable: the asynchronous group-commit flush that
|
||||
* persists buffered before-images to disk has failed repeatedly (a real
|
||||
* storage fault — a full disk, an EIO, a permission loss — not a transient
|
||||
* blip). Rather than keep accepting writes whose history silently piles up in
|
||||
* memory and is never persisted — an invisible durability loss, and an
|
||||
* unbounded leak — the store LATCHES this failure and refuses further writes
|
||||
* until the pending tier drains.
|
||||
*
|
||||
* This is the loud, honest response to a broken history-durability path:
|
||||
* - Live canonical data is unaffected (single-op live bytes are written before
|
||||
* the history flush; only the immutable before-image history is stuck).
|
||||
* - The background flush keeps retrying with backoff; when the underlying fault
|
||||
* clears and a flush succeeds, the latch lifts and writes resume
|
||||
* automatically. An explicit `flush()` also clears it on success.
|
||||
* - {@link cause} is the underlying storage error from the last failed flush.
|
||||
*
|
||||
* A caller seeing this should treat it exactly like a full disk: stop writing,
|
||||
* resolve the storage fault, and retry. It is NOT a data-corruption error — no
|
||||
* committed generation is lost — it is a refusal to *promise* durability the
|
||||
* store currently cannot deliver.
|
||||
*
|
||||
* @example
|
||||
* try {
|
||||
* await brain.add({ ... })
|
||||
* } catch (err) {
|
||||
* if (err instanceof PendingFlushDurabilityError) {
|
||||
* // History can't be persisted right now (disk fault). Resolve storage,
|
||||
* // then retry — the store self-heals once a flush succeeds.
|
||||
* console.error('history durability stalled:', err.cause)
|
||||
* }
|
||||
* }
|
||||
*/
|
||||
export class PendingFlushDurabilityError extends Error {
|
||||
/** The underlying storage error from the most recent failed flush. */
|
||||
public override readonly cause: Error
|
||||
/** How many consecutive flush attempts had failed when the latch tripped. */
|
||||
public readonly failedAttempts: number
|
||||
|
||||
/**
|
||||
* @param cause - The storage error from the last failed pending-tier flush.
|
||||
* @param failedAttempts - Consecutive failed flush attempts at latch time.
|
||||
*/
|
||||
constructor(cause: Error, failedAttempts: number) {
|
||||
super(
|
||||
`Write refused: single-op generation history could not be made durable ` +
|
||||
`after ${failedAttempts} consecutive flush attempts (${cause.message}). ` +
|
||||
`Live data is intact, but buffered history is not yet persisted — the ` +
|
||||
`store refuses further writes rather than silently accumulate ` +
|
||||
`un-durable history. Resolve the underlying storage fault; the store ` +
|
||||
`self-heals and resumes writes once a flush succeeds. Cause: ${cause.message}`
|
||||
)
|
||||
this.name = 'PendingFlushDurabilityError'
|
||||
this.cause = cause
|
||||
this.failedAttempts = failedAttempts
|
||||
}
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue