fix: transaction timeouts are a typed no-hot-retry contract; engine-side non-retry pinned; dead transaction path removed

A production incident: a native-provider op ground 38-40s inside a transaction,
blew the apply-phase budget, rolled back, and a downstream pipeline hot-retried
the identical operation into a 6-minute CPU storm. Brainy itself never
auto-retried the timeout; the gap was that TransactionTimeoutError only said
"retryable" in prose, with nothing machine-readable for a caller to branch on.

- TransactionTimeoutError gains two typed, always-true fields: retryable
  (a later attempt may succeed once the slowness resolves or the budget is
  raised) and hotRetryUnsafe (an immediate identical retry re-pays the full
  cost that just timed out and can cascade into a CPU storm -- callers must
  latch and back off, never loop). context's existing telemetry fields
  (timeoutMs, operationIndex, elapsedMs, totalOperations, operationName) are
  now documented as the caller's backoff inputs.
- Updated the "retryable" doc-prose sites (transact()'s timeoutMs option,
  transactionBudgetFloorMs, Transaction.execute()'s contract) to point at
  the new fields instead of bare prose.
- Regression pin (tests/unit/transaction/timeout-never-internally-retried.test.ts):
  an execution counter proves the engine never re-drives a timed-out
  operation, through both the single-op engine TransactionManager/Transaction
  drives for every single-record write, and add()'s upsert-race retry loop
  (which must exit on the first TransactionTimeoutError, never treat it like
  the lost-insert-race signal it retries on).
- Removed TransactionManager.executeTransactionWithResult -- zero callers
  anywhere in the codebase.
This commit is contained in:
David Snelling 2026-07-24 16:01:41 -07:00
parent 22702b81c0
commit 003e2a74ea
7 changed files with 199 additions and 76 deletions

View file

@ -73,14 +73,47 @@ export class InvalidTransactionStateError extends TransactionError {
/**
* Error for transaction timeout
*
* Machine-readable no-hot-retry contract: {@link retryable} and
* {@link hotRetryUnsafe} are both always `true` on this class they exist
* so a caller can branch on the *shape* of the error instead of parsing
* message text. Read them together: the operation may eventually succeed,
* but never by looping on it immediately.
*
* `context` (inherited from {@link TransactionError}) carries the caller's
* backoff inputs see the field docs below.
*/
export class TransactionTimeoutError extends TransactionError {
/**
* The failed operation MAY succeed on a later attempt once the
* underlying slowness resolves (e.g. a cold page cache warms up) or the
* budget is deliberately raised (`transactionBudgetFloorMs`, or a larger
* `timeoutMs` override on the batch). This is a statement about eventual
* retryability, not a license to retry now see {@link hotRetryUnsafe}.
*/
public readonly retryable = true
/**
* An immediate, identical retry re-pays the FULL cost of the work that
* just timed out it does not resume partway. Looping on this error
* (hot-retrying) repeats that full cost every attempt and can cascade
* into a CPU/resource storm on the caller's side. Callers MUST latch: on
* this error, record `{ at: Date.now(), error }`, surface one loud
* failure to their own caller, and hold a cooldown window before any
* re-attempt (clearing the latch only on success). Never retry this error
* in a tight loop.
*/
public readonly hotRetryUnsafe = true
constructor(
timeoutMs: number,
operationIndex: number,
telemetry?: {
/** Milliseconds elapsed in the transaction when the budget tripped. */
elapsedMs?: number
/** Total number of operations in the batch that timed out. */
totalOperations?: number
/** Name of the operation the batch was about to start when it tripped, if named. */
operationName?: string
}
) {
@ -93,8 +126,16 @@ export class TransactionTimeoutError extends TransactionError {
telemetry?.elapsedMs !== undefined ? `${telemetry.elapsedMs}ms elapsed, ` : ''
super(
`Transaction timed out at operation ${progress}${name}${elapsed}budget ${timeoutMs}ms. ` +
`The batch rolled back atomically; retry with a higher timeoutMs or a smaller batch.`,
{ timeoutMs, operationIndex, ...telemetry }
`The batch rolled back atomically; retryable after the underlying slowness resolves or ` +
`the budget is raised, but hot-retry-unsafe — latch and back off, never loop.`,
{
// Caller backoff inputs — all present on every instance:
/** Configured budget (ms) that was exceeded. */
timeoutMs,
/** Index of the operation the batch was about to start when it tripped. */
operationIndex,
...telemetry
}
)
this.name = 'TransactionTimeoutError'
}