soulcraft/brainy
1
0
Fork 0
Code Issues 1 Pull requests Projects Releases 335 Packages Wiki Activity Actions

fix: release drains in-flight writer-lock heartbeat — no phantom lock after unlink

Browse source 
clearInterval() stops future heartbeat ticks but not one already in
flight: a straggler tick past its ownership guards could land its
atomic lock rewrite AFTER releaseWriterLock()'s unlink, re-creating
the lock file as a phantom that blocks the next writer until the
stale TTL expires (~60s) — the pool-eviction reopen case. Found as an
ENOENT heartbeat warning during benchmark teardown; the quiet variant
is the harmful one.

releaseWriterLock() now awaits the in-flight tick (tracked per tick,
self-clearing) before reading/unlinking, so a straggler's write always
lands BEFORE the unlink and gets removed with everything else. The
heartbeat's ENOENT is also now benign-by-contract (lock or directory
removed under us — the next acquire recreates it); other errors stay
loud.

Pin: straggler-past-guards simulation — lock file absent after close,
directory immediately claimable (fails on the undrained code).
This commit is contained in:
David Snelling 2026-07-19 12:52:24 -07:00
parent 300d9f2a16
commit 70e4bc8a79
2 changed files with 69 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

31
src/storage/adapters/fileSystemStorage.ts
View file

@ -96,6 +96,14 @@ export class FileSystemStorage extends BaseStorage {
private static readonly WRITER_STALE_THRESHOLD_MS = 60_000
private writerLockHeartbeat?: NodeJS.Timeout
private writerLockInfo?: WriterLockInfo
/**
* The currently-executing heartbeat refresh, if any. `releaseWriterLock()`
* awaits it before unlinking: clearInterval() stops FUTURE ticks but not a
* tick already in flight, and a straggler landing after the unlink would
* RE-CREATE the lock file — a phantom lock blocking the next writer until
* the stale TTL expires (the pool-eviction reopen case).
*/
private writerHeartbeatInFlight?: Promise<void>
// Flush-request RPC state. The writer polls `locks/_flush_requests/` for
// new `.req` files and emits `.ack` files in `locks/_flush_responses/` after
@ -1880,8 +1888,18 @@ export class FileSystemStorage extends BaseStorage {
// Heartbeat — rewrite lastHeartbeat every WRITER_HEARTBEAT_MS so other
// processes can tell a live writer from one that crashed without releasing.
this.writerLockHeartbeat = setInterval(() => {
this.refreshWriterLockHeartbeat().catch((err) => {
console.warn('[brainy] Failed to refresh writer lock heartbeat:', err)
const tick = this.refreshWriterLockHeartbeat().catch((err) => {
// ENOENT = the lock (or its directory) vanished mid-refresh — the
// store was released or removed under us; the next acquire recreates
// it. Benign by construction; anything else stays loud.
if ((err as NodeJS.ErrnoException)?.code !== 'ENOENT') {
console.warn('[brainy] Failed to refresh writer lock heartbeat:', err)
}
})
this.writerHeartbeatInFlight = tick.finally(() => {
if (this.writerHeartbeatInFlight === tick) {
this.writerHeartbeatInFlight = undefined
}
})
}, FileSystemStorage.WRITER_HEARTBEAT_MS)
if (typeof this.writerLockHeartbeat.unref === 'function') {
@ -1913,6 +1931,15 @@ export class FileSystemStorage extends BaseStorage {
clearInterval(this.writerLockHeartbeat)
this.writerLockHeartbeat = undefined
}
// Drain an in-flight heartbeat tick BEFORE unlinking: clearInterval stops
// future ticks only, and a straggler write landing after the unlink would
// re-create the lock as a phantom (blocking the next writer until the
// stale TTL). After the drain, any refresh is either fully landed (we
// unlink its output below) or not started (it sees writerLockInfo
// undefined and returns).
if (this.writerHeartbeatInFlight) {
await this.writerHeartbeatInFlight
}
if (!this.writerLockInfo) {
return
}