fix(durability): three block-layer power-loss findings from the first fault-injection box run — all cured, matrix 15/15
An internal cross-engine fault-injection run (frozen-platter power-loss capture) surfaced three release-gating findings; each cured in its owning layer, each pinned: 1. WHOLE-LOG REPLAY ON UNCLEAN OPEN (the big one): log-authority replay only covered facts ABOVE the manifest — but live canonical entity writes are tmp+rename without per-file fsync, and the group-commit flush syncs staging + manifest, never the live tree. Power loss could therefore vaporize acked canonical bytes BELOW the manifest while the log held every fact scan-clean (measured: 299 of 301 acks lost). Now: a clean close stamps a clean-shutdown marker (fsynced, written last); every open consumes it; an UNCLEAN open under log authority folds the ENTIRE log into canonical — whole-entity after-images make the re-apply idempotent and byte-safe. Zero cost on the happy path; crash recovery pays one narrated fold. Recovery is replay: a crash is just bigger lag. 2. TORN WRITER LOCK: power loss legally leaves the lock file present but empty; the parse failure read as 'no holder' while the O_EXCL claim EEXISTed forever — a PERMANENT lockout no staleness check could clear. An unparseable lock is stale by definition (no live holder has one): unlink loudly and re-loop; a racer rewriting a valid lock first wins. 3. PAIR GUARD: flush() called metadataIndex.stampWatermark unguarded; a replacement metadata provider without the method killed the pair at first flush. All three stamp calls are optional-chained — a missing stamp is a verdict-side rescan, never a flush crash. Pins: whole-log fold restores rows vanished below the manifest · clean-shutdown marker lifecycle (stamp/consume/re-stamp) · torn-lock recovery with a fresh write after · stampless-provider flush. Gates: unit 2055/2055 · integration 824 · kill-matrix 15/15.
This commit is contained in:
parent
d1698fa5be
commit
67c606be69
4 changed files with 180 additions and 14 deletions
|
|
@ -1785,6 +1785,32 @@ export class FileSystemStorage extends BaseStorage {
|
|||
const now = new Date().toISOString()
|
||||
const existing = await this.readWriterLock()
|
||||
|
||||
// TORN-LOCK RECOVERY: power loss can legally leave the lock file
|
||||
// present but EMPTY/unparseable (the claim's non-atomic write died
|
||||
// mid-flight). readWriterLock() reports it as null — but the O_EXCL
|
||||
// claim below would EEXIST forever, a PERMANENT lockout no staleness
|
||||
// check can clear (staleness needs a parsed PID). A torn lock is
|
||||
// stale BY DEFINITION: no live holder has one (a holder either
|
||||
// completed its write or is dead). Unlink loudly and re-loop; a
|
||||
// racer that rewrites a VALID lock first simply wins the next read.
|
||||
if (existing === null) {
|
||||
try {
|
||||
await fs.promises.access(lockFile)
|
||||
console.warn(
|
||||
`[brainy] Writer lock at ${lockFile} exists but is unreadable/unparseable ` +
|
||||
`(torn write from a previous power loss) — treating as stale and removing.`
|
||||
)
|
||||
try {
|
||||
await fs.promises.unlink(lockFile)
|
||||
} catch (unlinkErr: any) {
|
||||
if (unlinkErr.code !== 'ENOENT') throw unlinkErr
|
||||
}
|
||||
} catch (accessErr: any) {
|
||||
if (accessErr.code !== 'ENOENT') throw accessErr
|
||||
// Absent: the normal fresh-claim path below.
|
||||
}
|
||||
}
|
||||
|
||||
if (existing) {
|
||||
// Same-process re-open: a second Brainy instance in this Node process
|
||||
// (e.g. test "simulate server restart" patterns, or a consumer that
|
||||
|
|
|
|||
Reference in a new issue