fix: transact durability barrier — committed transactions are durable on return

A committed transact() reported success while its canonical entity writes were
still only in the OS page cache (tmp+rename, not fsync'd), even though the
generation counter and manifest were fsync'd. A hard kill in that window could
leave the durable counter ahead of the persisted entity bytes — phantom
progress for any generation-based consumer resuming from the counter. Reported
from a downstream migration's crash-lifecycle forensics.

Add an optional transaction durability barrier to GenerationStorage
(beginWriteBarrier / flushWriteBarrier). FileSystemStorage implements it:
writeObjectToPath records each successful canonical write and
deleteObjectFromPath records each delete's parent dir, between begin and flush;
flush fsyncs every recorded write (contents + rename dir entries, via
syncRawObjects) and the parent dir of every delete. commitTransaction opens the
barrier before running the planned operations and flushes it after, BEFORE
persisting the counter and manifest — so the batch's entire canonical footprint
is durable before the generation stamp advances. The barrier is optional: the
generation store calls it through optional chaining, so in-memory and
durable-per-call (cloud object-PUT) adapters treat it as a no-op.

The single-op group-commit path is unchanged (deferred durability is the
Model-B design that avoids a 3-5x per-write fsync regression), but its
durability contract is now documented explicitly on commitSingleOp: transact =
durable on return; single-op = durable at the next flush()/close(), with the
counter buffered alongside the data so a crash loses both together (never a
torn counter-ahead-of-state store).

Regression (tests/integration/transact-durability-barrier.test.ts): the entity
writes fsync in an earlier syncRawObjects batch than the manifest for single-op
and multi-op (add+relate) transactions; a precommit-rejected batch opens no
barrier and advances nothing; MemoryStorage exposes no barrier (optional-chain
no-op).
This commit is contained in:
David Snelling 2026-07-12 08:54:14 -07:00
parent a9fc1f3f9b
commit 3b8fa51161
4 changed files with 270 additions and 0 deletions

View file

@ -686,12 +686,21 @@ export class GenerationStore {
faultPoint('after-staging')
// -- 4. Execute the planned batch -------------------------------------
// Durability barrier: record every canonical write/delete the operations
// make, then fsync them BELOW (before the counter advances). Without
// this, a hard kill after commitTransaction returns could leave the
// fsync'd generation counter ahead of still-page-cached entity bytes —
// phantom progress for any generation-based consumer.
this.storage.beginWriteBarrier?.()
this.inTransact = true
try {
await args.execute()
} finally {
this.inTransact = false
}
// The transaction's entire canonical footprint is now durable, so the
// counter/manifest advance below can never outrun the entity bytes.
await this.storage.flushWriteBarrier?.()
faultPoint('after-execute')
// -- 5. Counter + manifest rename (COMMIT POINT) ----------------------
@ -787,6 +796,20 @@ export class GenerationStore {
* mid-flush is recovered by drop-without-restore
* (see {@link GenerationDelta.groupCommit}).
*
* **Durability contract (single-op vs transact).** A single-op write's live
* canonical bytes are written via tmp+rename but NOT individually fsync'd
* they are durable at the next {@link flushPendingSingleOps} or `close()`, not
* the instant the write resolves. On a hard kill before that flush, a
* single-op write can be lost even though the call returned; the group-commit
* counter is buffered alongside, so counter and data are lost together (no
* counter-ahead-of-state torn store). {@link commitTransaction} is the
* stronger contract: it runs a write barrier that fsyncs the whole batch's
* canonical footprint BEFORE advancing the generation counter, so a committed
* transact is durable on return. Callers that need per-write durability should
* use `transact()` (or `flush()` after a single-op); Model-B group-commit
* deliberately trades single-op fsync latency (a 3-5x write regression) for
* throughput.
*
* The per-write generation is read by graph-index ops through the same
* `generation()` watermark `transact()` uses because this method, like
* {@link commitTransaction}, holds the mutex across the counter bump AND

View file

@ -391,6 +391,24 @@ export interface GenerationStorage {
/** Durability barrier: fsync the given object paths (no-op in memory). */
syncRawObjects(paths: string[]): Promise<void>
/**
* OPTIONAL transaction durability barrier. `commitTransaction` calls
* {@link beginWriteBarrier} immediately before running the planned operations
* and {@link flushWriteBarrier} immediately after BEFORE the generation
* counter and manifest are advanced. An adapter whose canonical writes are
* not synchronously durable (the filesystem adapter's tmp+rename lands in the
* page cache) MUST implement these so a transaction reported "committed" is
* durable before its generation stamp: otherwise a hard kill can leave the
* fsync'd counter ahead of the still-buffered entity bytes (phantom progress
* for any generation-based consumer). Adapters whose writes are already
* durable per-call (cloud object PUT) may leave these undefined the store
* treats them as no-ops. `beginWriteBarrier` also resets any tracking left by
* an aborted prior transaction.
*/
beginWriteBarrier?(): void
/** @see beginWriteBarrier — fsync every canonical write since begin. */
flushWriteBarrier?(): Promise<void>
/** Read an entity's raw stored metadata+vector objects. */
readNounRaw(id: string): Promise<{ metadata: any | null; vector: any | null }>
/** Restore an entity's raw stored objects (`null` part ⇒ delete that file). */