fix(storage): counts persistence is single-flight, coalesced, and never races its own temp file
Some checks failed
CI / Node 24 (push) Successful in 12m24s
CI / Node 22 (push) Successful in 12m35s
CI / Integration + conformance (Node 22) (push) Failing after 17m25s
CI / Bun (latest) (push) Successful in 12m19s

persistCounts() was write-through on every count change with no
serialization, and the atomic writer named its temp file with millisecond
granularity. Two persists inside one millisecond shared the temp path: both
wrote it, the first rename consumed it, the second rename found nothing —
ENOENT, roughly 1,500 times a day on a busy production brain, with a full
ledger write per change behind it. No data was lost (the surviving rename
carried a complete ledger and the next change re-persisted), but the race
was real and the write rate absurd.

flushCounts() now runs exactly one persist at a time; requests arriving
during it collapse into one trailing pass that carries the burst's final
state — N changes cost at most two writes. writeFileAtomic() adds a
per-process sequence to the temp name so no two writes can share a path.
Pinned: a 25-change burst → ≤2 ledger writes, zero errors, ledger equal to
memory; parallel real writes land complete; three same-instant atomic
writes own three distinct temp paths.
This commit is contained in:
David Snelling 2026-09-01 09:32:23 -07:00
parent 4014e0f125
commit 5e3b343a0e
3 changed files with 162 additions and 9 deletions

View file

@ -1089,6 +1089,10 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
// Counts changed since the last persist? Drives the write-through flush.
protected pendingCountPersist = false
/** The one persist running right now, if any (single-flight law — see flushCounts). */
private countPersistInFlight: Promise<void> | null = null
/** The one trailing persist a burst has queued behind the in-flight one. */
private countPersistTrailing: Promise<void> | null = null
/**
* Get total noun count - O(1) operation
@ -1341,15 +1345,46 @@ export abstract class BaseStorageAdapter implements StorageAdapter {
return
}
try {
// Persist to storage (implemented by subclass)
await this.persistCounts()
this.pendingCountPersist = false
} catch (error) {
console.error('CRITICAL: Failed to flush counts to storage:', error)
// Keep pending flag set so we retry on next operation
throw error
// SINGLE-FLIGHT, COALESCED. Counts are write-through on every change, so
// a burst of writes used to launch one persist per change, all in flight
// together. Two of them inside the same millisecond shared the atomic
// writer's temp path (`.tmp-<pid>-<ms>`): both wrote it, the first rename
// consumed it, the second rename found nothing — ENOENT, ~1,500 times a
// day on a busy production brain, with a full ledger write per change
// behind it. Now exactly one persist runs at a time; requests that arrive
// while it runs collapse into ONE trailing persist that carries the final
// state. A burst of N changes costs at most two writes and never races
// itself.
if (this.countPersistInFlight) {
// The in-flight write may have already serialised a stale snapshot —
// ask for one more pass after it, and let every caller in this burst
// await that same pass.
if (!this.countPersistTrailing) {
this.countPersistTrailing = this.countPersistInFlight
.catch(() => undefined)
.then(() => {
this.countPersistTrailing = null
return this.flushCounts()
})
}
return this.countPersistTrailing
}
this.countPersistInFlight = (async () => {
try {
// Persist to storage (implemented by subclass)
this.pendingCountPersist = false
await this.persistCounts()
} catch (error) {
// Keep the flag set so the next operation retries.
this.pendingCountPersist = true
console.error('CRITICAL: Failed to flush counts to storage:', error)
throw error
} finally {
this.countPersistInFlight = null
}
})()
return this.countPersistInFlight
}
/**