fix: saveBinaryBlob never acks a durable write that stored nothing
With the unique per-writer temp suffix, a rename ENOENT can no longer mean "a concurrent idempotent writer already renamed it" — nobody else holds this writer's temp. It means our just-written temp vanished before the rename, so the bytes did NOT land. The old code returned success on that ENOENT, acknowledging a write that persisted nothing; the native provider mmaps these blobs, so a phantom-acked blob is a silent-loss. saveBinaryBlob now retries once with a fresh temp (self-healing a transient external-sweeper/crash-cleanup race), and if the temp vanishes again it throws loud rather than acknowledging a store-nothing write. Non-ENOENT rename faults propagate verbatim as before. The unique-temp suffix already eliminated the concurrent same-key collision that originally motivated the ENOENT shortcut.
This commit is contained in:
parent
54c183668c
commit
7feba49d94
2 changed files with 114 additions and 21 deletions
|
|
@ -1178,27 +1178,44 @@ export class FileSystemStorage extends BaseStorage {
|
|||
await this.ensureInitialized()
|
||||
const filePath = this.blobPath(key)
|
||||
await this.ensureDirectoryExists(path.dirname(filePath))
|
||||
// Unique per-writer temp suffix — matches the pattern used at every other
|
||||
// atomic-write site in this file (lines 336, 551, 744, 781, 1529, 2908).
|
||||
// Without a unique suffix, two concurrent saveBinaryBlob() calls for the
|
||||
// same key collide on `${filePath}.tmp`: both writeFile, the first rename
|
||||
// succeeds, the second fires against a missing temp and throws ENOENT.
|
||||
// Reproduced in production: column-store compaction running alongside an
|
||||
// explicit flush() repeatedly raced on `_column_index/<field>/DELETED.bin`.
|
||||
const tmpPath = `${filePath}.tmp.${process.pid}.${Date.now()}.${Math.random().toString(36).slice(2)}`
|
||||
await fs.promises.writeFile(tmpPath, data)
|
||||
try {
|
||||
await fs.promises.rename(tmpPath, filePath)
|
||||
} catch (err) {
|
||||
const code = (err as NodeJS.ErrnoException).code
|
||||
// The writes are idempotent for a given key — every caller persists the
|
||||
// same logical bytes for that key — so ENOENT on rename means the temp
|
||||
// is already gone (rare with the unique suffix, but defensive against
|
||||
// crash-resume cleanup paths and external file-system sweepers).
|
||||
if (code === 'ENOENT') return
|
||||
// Any other failure: clean up our own temp to avoid orphans before rethrow.
|
||||
await fs.promises.unlink(tmpPath).catch(() => {})
|
||||
throw err
|
||||
|
||||
// Atomic write via a UNIQUE per-writer temp suffix — matches the pattern
|
||||
// used at every other atomic-write site in this file (lines 336, 551, 744,
|
||||
// 781, 1529, 2908). The unique suffix (pid+time+random) means NO other
|
||||
// writer ever touches this temp, so two concurrent saveBinaryBlob() calls
|
||||
// for the same key can never collide on the temp path (the shared-`.tmp`
|
||||
// race that once threw ENOENT — column-store compaction vs an explicit
|
||||
// flush() on `_column_index/<field>/DELETED.bin` — is gone).
|
||||
//
|
||||
// Crucially, BECAUSE the temp is unique, a rename ENOENT can no longer mean
|
||||
// "a concurrent idempotent writer already renamed it": nobody else has this
|
||||
// temp. It means OUR just-written temp vanished before the rename, so the
|
||||
// bytes did NOT land — returning success would acknowledge a write that
|
||||
// stored nothing (the native provider mmaps these blobs; a phantom-acked
|
||||
// blob is exactly the silent-loss class). The only way that happens with a
|
||||
// unique temp is an external sweeper / crash-cleanup removing it mid-write,
|
||||
// so retry ONCE with a fresh temp; if it vanishes again, FAIL LOUD.
|
||||
const writeOnce = async (): Promise<'ok' | 'temp-vanished'> => {
|
||||
const tmpPath = `${filePath}.tmp.${process.pid}.${Date.now()}.${Math.random().toString(36).slice(2)}`
|
||||
await fs.promises.writeFile(tmpPath, data)
|
||||
try {
|
||||
await fs.promises.rename(tmpPath, filePath)
|
||||
return 'ok'
|
||||
} catch (err) {
|
||||
// Clean up our own temp (best-effort) so no failure path orphans it.
|
||||
await fs.promises.unlink(tmpPath).catch(() => {})
|
||||
if ((err as NodeJS.ErrnoException).code === 'ENOENT') return 'temp-vanished'
|
||||
throw err
|
||||
}
|
||||
}
|
||||
|
||||
if ((await writeOnce()) === 'temp-vanished' && (await writeOnce()) === 'temp-vanished') {
|
||||
throw new Error(
|
||||
`saveBinaryBlob('${key}'): the temp file was removed before rename on two ` +
|
||||
`successive attempts — the blob did NOT persist. Some external process is ` +
|
||||
`deleting files under ${this.blobsDir} mid-write. Failing loud rather than ` +
|
||||
`acknowledging a durable write that stored nothing.`
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue