fix(durability): three block-layer power-loss findings from the first fault-injection box run — all cured, matrix 15/15
All checks were successful
CI / Node 22 (push) Successful in 12m20s
CI / Node 24 (push) Successful in 12m7s
CI / Bun (latest) (push) Successful in 12m21s

An internal cross-engine fault-injection run (frozen-platter power-loss
capture) surfaced three release-gating findings; each cured in its owning
layer, each pinned:

1. WHOLE-LOG REPLAY ON UNCLEAN OPEN (the big one): log-authority replay
   only covered facts ABOVE the manifest — but live canonical entity
   writes are tmp+rename without per-file fsync, and the group-commit
   flush syncs staging + manifest, never the live tree. Power loss could
   therefore vaporize acked canonical bytes BELOW the manifest while the
   log held every fact scan-clean (measured: 299 of 301 acks lost).
   Now: a clean close stamps a clean-shutdown marker (fsynced, written
   last); every open consumes it; an UNCLEAN open under log authority
   folds the ENTIRE log into canonical — whole-entity after-images make
   the re-apply idempotent and byte-safe. Zero cost on the happy path;
   crash recovery pays one narrated fold. Recovery is replay: a crash is
   just bigger lag.

2. TORN WRITER LOCK: power loss legally leaves the lock file present but
   empty; the parse failure read as 'no holder' while the O_EXCL claim
   EEXISTed forever — a PERMANENT lockout no staleness check could clear.
   An unparseable lock is stale by definition (no live holder has one):
   unlink loudly and re-loop; a racer rewriting a valid lock first wins.

3. PAIR GUARD: flush() called metadataIndex.stampWatermark unguarded;
   a replacement metadata provider without the method killed the pair at
   first flush. All three stamp calls are optional-chained — a missing
   stamp is a verdict-side rescan, never a flush crash.

Pins: whole-log fold restores rows vanished below the manifest ·
clean-shutdown marker lifecycle (stamp/consume/re-stamp) · torn-lock
recovery with a fresh write after · stampless-provider flush.
Gates: unit 2055/2055 · integration 824 · kill-matrix 15/15.
This commit is contained in:
David Snelling 2026-08-10 14:48:32 -07:00
parent d1698fa5be
commit 67c606be69
4 changed files with 180 additions and 14 deletions

View file

@ -1785,6 +1785,32 @@ export class FileSystemStorage extends BaseStorage {
const now = new Date().toISOString()
const existing = await this.readWriterLock()
// TORN-LOCK RECOVERY: power loss can legally leave the lock file
// present but EMPTY/unparseable (the claim's non-atomic write died
// mid-flight). readWriterLock() reports it as null — but the O_EXCL
// claim below would EEXIST forever, a PERMANENT lockout no staleness
// check can clear (staleness needs a parsed PID). A torn lock is
// stale BY DEFINITION: no live holder has one (a holder either
// completed its write or is dead). Unlink loudly and re-loop; a
// racer that rewrites a VALID lock first simply wins the next read.
if (existing === null) {
try {
await fs.promises.access(lockFile)
console.warn(
`[brainy] Writer lock at ${lockFile} exists but is unreadable/unparseable ` +
`(torn write from a previous power loss) — treating as stale and removing.`
)
try {
await fs.promises.unlink(lockFile)
} catch (unlinkErr: any) {
if (unlinkErr.code !== 'ENOENT') throw unlinkErr
}
} catch (accessErr: any) {
if (accessErr.code !== 'ENOENT') throw accessErr
// Absent: the normal fresh-claim path below.
}
}
if (existing) {
// Same-process re-open: a second Brainy instance in this Node process
// (e.g. test "simulate server restart" patterns, or a consumer that