fix: saveBinaryBlob never acks a durable write that stored nothing

With the unique per-writer temp suffix, a rename ENOENT can no longer mean "a
concurrent idempotent writer already renamed it" — nobody else holds this
writer's temp. It means our just-written temp vanished before the rename, so the
bytes did NOT land. The old code returned success on that ENOENT, acknowledging
a write that persisted nothing; the native provider mmaps these blobs, so a
phantom-acked blob is a silent-loss.

saveBinaryBlob now retries once with a fresh temp (self-healing a transient
external-sweeper/crash-cleanup race), and if the temp vanishes again it throws
loud rather than acknowledging a store-nothing write. Non-ENOENT rename faults
propagate verbatim as before. The unique-temp suffix already eliminated the
concurrent same-key collision that originally motivated the ENOENT shortcut.
This commit is contained in:
David Snelling 2026-07-13 09:35:30 -07:00
parent 54c183668c
commit 7feba49d94
2 changed files with 114 additions and 21 deletions

View file

@ -1178,27 +1178,44 @@ export class FileSystemStorage extends BaseStorage {
await this.ensureInitialized()
const filePath = this.blobPath(key)
await this.ensureDirectoryExists(path.dirname(filePath))
// Unique per-writer temp suffix — matches the pattern used at every other
// atomic-write site in this file (lines 336, 551, 744, 781, 1529, 2908).
// Without a unique suffix, two concurrent saveBinaryBlob() calls for the
// same key collide on `${filePath}.tmp`: both writeFile, the first rename
// succeeds, the second fires against a missing temp and throws ENOENT.
// Reproduced in production: column-store compaction running alongside an
// explicit flush() repeatedly raced on `_column_index/<field>/DELETED.bin`.
const tmpPath = `${filePath}.tmp.${process.pid}.${Date.now()}.${Math.random().toString(36).slice(2)}`
await fs.promises.writeFile(tmpPath, data)
try {
await fs.promises.rename(tmpPath, filePath)
} catch (err) {
const code = (err as NodeJS.ErrnoException).code
// The writes are idempotent for a given key — every caller persists the
// same logical bytes for that key — so ENOENT on rename means the temp
// is already gone (rare with the unique suffix, but defensive against
// crash-resume cleanup paths and external file-system sweepers).
if (code === 'ENOENT') return
// Any other failure: clean up our own temp to avoid orphans before rethrow.
await fs.promises.unlink(tmpPath).catch(() => {})
throw err
// Atomic write via a UNIQUE per-writer temp suffix — matches the pattern
// used at every other atomic-write site in this file (lines 336, 551, 744,
// 781, 1529, 2908). The unique suffix (pid+time+random) means NO other
// writer ever touches this temp, so two concurrent saveBinaryBlob() calls
// for the same key can never collide on the temp path (the shared-`.tmp`
// race that once threw ENOENT — column-store compaction vs an explicit
// flush() on `_column_index/<field>/DELETED.bin` — is gone).
//
// Crucially, BECAUSE the temp is unique, a rename ENOENT can no longer mean
// "a concurrent idempotent writer already renamed it": nobody else has this
// temp. It means OUR just-written temp vanished before the rename, so the
// bytes did NOT land — returning success would acknowledge a write that
// stored nothing (the native provider mmaps these blobs; a phantom-acked
// blob is exactly the silent-loss class). The only way that happens with a
// unique temp is an external sweeper / crash-cleanup removing it mid-write,
// so retry ONCE with a fresh temp; if it vanishes again, FAIL LOUD.
const writeOnce = async (): Promise<'ok' | 'temp-vanished'> => {
const tmpPath = `${filePath}.tmp.${process.pid}.${Date.now()}.${Math.random().toString(36).slice(2)}`
await fs.promises.writeFile(tmpPath, data)
try {
await fs.promises.rename(tmpPath, filePath)
return 'ok'
} catch (err) {
// Clean up our own temp (best-effort) so no failure path orphans it.
await fs.promises.unlink(tmpPath).catch(() => {})
if ((err as NodeJS.ErrnoException).code === 'ENOENT') return 'temp-vanished'
throw err
}
}
if ((await writeOnce()) === 'temp-vanished' && (await writeOnce()) === 'temp-vanished') {
throw new Error(
`saveBinaryBlob('${key}'): the temp file was removed before rename on two ` +
`successive attempts — the blob did NOT persist. Some external process is ` +
`deleting files under ${this.blobsDir} mid-write. Failing loud rather than ` +
`acknowledging a durable write that stored nothing.`
)
}
}