fix(restore): a restore is an unclean event — the swap runs quiesced and the snapshot's durability stamps never survive it
Two defects with one root, found by the fold-checkpoint work's first integration gate. (1) THE RACE: restore() never quiesced the generation store, so a background flush could write into _system/ while the swap was removing it — observed as ENOTEMPTY mid-swap when a checkpoint stamp landed between readdir and rmdir. The swap now runs inside the store's exclusive section (runStateReplacement): flush timer disarmed, pending tier and checkpoint accumulator discarded BEFORE any directory moves. (2) THE INHERITED ASSERTION: a snapshot carries its source brain's clean-shutdown marker and fold checkpoint, but the restored files were bulk-copied without per-file fsync — the inherited stamps would suppress exactly the recovery fold that cures a post-restore power cut. reopenAfterRestore now deletes both stamps before reopening: the open treats the store as uncleanly shut, folds the restored log into canonical, barrier-syncs what it re-applied, and stamps fresh — the restored state is durably founded at restore time instead of borrowing assertions about bytes this disk never synced. Pinned: restore under in-flight traffic completes; the pre-restore stamp does not survive; the post-restore stamp is the reopen fold's own, at the restored watermark.
This commit is contained in:
parent
ff43de1ada
commit
9ca80667c3
3 changed files with 85 additions and 1 deletions
|
|
@ -3127,6 +3127,28 @@ export class GenerationStore {
|
|||
* are never reissued.
|
||||
* @param floorGeneration - The counter value before the restore.
|
||||
*/
|
||||
/**
|
||||
* @description Run a wholesale state replacement (the restore swap)
|
||||
* EXCLUSIVELY: under the commit mutex, with the pending flush timer
|
||||
* disarmed and the pending tier + fold-checkpoint accumulator discarded
|
||||
* FIRST — so no background flush can write into `_system/` while the
|
||||
* replacement is removing and swapping directories. Observed without this:
|
||||
* a checkpoint stamp raced restore's directory removal and the swap died
|
||||
* ENOTEMPTY mid-flight. The discarded in-memory state describes the store
|
||||
* being replaced — `reopenAfterRestore` (which the caller runs next)
|
||||
* rebuilds everything from the restored bytes.
|
||||
*/
|
||||
async runStateReplacement(replace: () => Promise<void>): Promise<void> {
|
||||
return this.withMutex(async () => {
|
||||
this.clearPendingFlushTimer()
|
||||
this.pendingGens = []
|
||||
this.pendingBuffer.clear()
|
||||
this.checkpointDirtyNouns = new Set()
|
||||
this.checkpointDirtyVerbs = new Set()
|
||||
await replace()
|
||||
})
|
||||
}
|
||||
|
||||
async reopenAfterRestore(floorGeneration: number): Promise<void> {
|
||||
await this.withMutex(async () => {
|
||||
this.deltaCache.clear()
|
||||
|
|
@ -3139,6 +3161,27 @@ export class GenerationStore {
|
|||
this.clearPendingFlushTimer()
|
||||
this.pendingGens = []
|
||||
this.pendingBuffer.clear()
|
||||
// The fold-checkpoint accumulator described the replaced state too.
|
||||
this.checkpointDirtyNouns = new Set()
|
||||
this.checkpointDirtyVerbs = new Set()
|
||||
this.foldCheckpointChainValid = false
|
||||
this.foldCheckpoint = 0
|
||||
// A RESTORE IS AN UNCLEAN EVENT, by construction: the snapshot's files
|
||||
// were just bulk-copied WITHOUT per-file fsync, so a power cut here can
|
||||
// tear them — yet the snapshot may CARRY the source brain's
|
||||
// clean-shutdown marker and fold checkpoint, which would together
|
||||
// suppress exactly the recovery fold that cures such a tear. Delete
|
||||
// both BEFORE reopening: the open below then treats the store as
|
||||
// uncleanly shut, folds the restored log into canonical, barrier-syncs
|
||||
// what it re-applied, and stamps a FRESH checkpoint — the restored
|
||||
// state becomes durably founded at restore time instead of inheriting
|
||||
// the source brain's assertions about bytes this disk never synced.
|
||||
try {
|
||||
await this.storage.deleteRawObject(CLEAN_SHUTDOWN_PATH)
|
||||
} catch { /* absent is fine — same outcome */ }
|
||||
try {
|
||||
await this.storage.deleteRawObject(FOLD_CHECKPOINT_PATH)
|
||||
} catch { /* absent is fine — fold from 0 */ }
|
||||
this.opened = false
|
||||
// open() re-reads counter/manifest and re-registers the bump hook.
|
||||
await this.open()
|
||||
|
|
|
|||
Reference in a new issue