feat: generation fact log — after-image commit records, dual-written at every commit point
Every committed generation now also appends a FACT — an after-image commit record (what each touched entity/relationship became, or a body-less tombstone for a removal) — to an append-only, crc32c-framed segment log under _generations/facts/. The before-image history and the canonical tree remain authoritative; the fact log gives consumers ONE sequential, self-verifying stream (index heals, incremental replays) in place of a per-entity directory walk. - Wire format: positional msgpack facts [generation, timestamp, ops, meta, blobHashes]; op = [kind u8, id bin16, record | nil tombstone]; 32-byte segment header (magic, formatVersion, firstGeneration, zeroed+verified reserved); length+crc32c frame per fact; zero-padded segment names so lexicographic order == generation order; JSON manifest with an atomic rename flip, manifest-first rotation. - Commit protocol: facts append+fsync BEFORE the commit point inside the existing durability window, so a crash can only leave the log AHEAD of committed truth — open() truncates back (torn tails detected by CRC). Absent generation = never committed; a scan can never see an uncommitted fact. transact() facts are durable-on-return; single-op facts ride the group-commit flush exactly like buffered history. A fact-append failure fails the write, loudly — a silent gap would be a lie a later replay discovers. - New public surface: brain.scanFacts() (sequential batches with heal telemetry: head/segments/approx up front, per-batch generation range + bytes + segment id, loud abort on gaps, summary cross-check) and brain.factSegmentPaths() (immutable sealed segments for zero-copy consumers; the mutable tail excluded). Exported types CommitFact, FactOp, FactScanBatch, FactScanHandle. - Storage: optional binary raw-byte primitives (appendRawBytes, readRawBytes, writeRawBytes, rawByteSize) on StorageAdapter — feature-detected; filesystem + memory adapters implement them; an adapter without them hosts no fact log. Fact segments are byte-copied (never hard-linked) into snapshots. The _generations/facts/ namespace is registered as a protected family (rebuildable: false): no sweeper or GC may delete under it. - New crc32c (Castagnoli) utility with RFC known-answer tests.
This commit is contained in:
parent
92299f27be
commit
38b0041464
13 changed files with 1493 additions and 4 deletions
|
|
@ -45,6 +45,7 @@ import type {
|
|||
GenerationStorage,
|
||||
TxLogEntry
|
||||
} from './types.js'
|
||||
import { FactLog, storageSupportsFactLog, type CommitFact, type FactOp } from './factLog.js'
|
||||
|
||||
/**
|
||||
* The byte-identical before-images of every id a commit touches, read UNDER
|
||||
|
|
@ -121,6 +122,16 @@ export interface GenerationStoreOpenResult {
|
|||
export class GenerationStore {
|
||||
private readonly storage: GenerationStorage
|
||||
|
||||
/**
|
||||
* The generation FACT LOG (dual-write transition) — an append-only,
|
||||
* CRC-framed record of every committed generation as an AFTER-IMAGE fact.
|
||||
* `null` when the storage layer lacks the binary raw-byte primitives.
|
||||
* Appends ride the same commit protocol: a fact-append failure FAILS the
|
||||
* write (loud — a silent fact gap would make the log a lie that a later
|
||||
* replay discovers), and open() reconciles the log to committed truth.
|
||||
*/
|
||||
private factLog: FactLog | null = null
|
||||
|
||||
/** Latest reserved/observed generation (≥ {@link committed}). */
|
||||
private counter = 0
|
||||
/** Committed-transaction watermark (manifest generation). */
|
||||
|
|
@ -400,6 +411,19 @@ export class GenerationStore {
|
|||
|
||||
this.opened = true
|
||||
|
||||
// Generation FACT LOG (dual-write transition): when the storage layer
|
||||
// exposes the binary raw-byte primitives, open the after-image fact log
|
||||
// and reconcile it to committed truth — facts are appended BEFORE the
|
||||
// commit point, so a crash can only leave the log AHEAD; open truncates
|
||||
// any fact beyond `committed`. Storage without the primitives simply
|
||||
// hosts no fact log (readers fall back to canonical enumeration).
|
||||
if (storageSupportsFactLog(this.storage)) {
|
||||
this.factLog = new FactLog(this.storage)
|
||||
await this.factLog.open(this.committed)
|
||||
} else {
|
||||
this.factLog = null
|
||||
}
|
||||
|
||||
// Hook single-op write batches so generation() is always meaningful.
|
||||
// Suppressed while a transact batch executes (the batch is ONE generation).
|
||||
if (!options?.readOnly) {
|
||||
|
|
@ -600,6 +624,58 @@ export class GenerationStore {
|
|||
* @returns The committed generation and its commit timestamp.
|
||||
* @throws GenerationConflictError when the CAS expectation fails.
|
||||
*/
|
||||
/**
|
||||
* The generation fact log, or `null` when the storage layer cannot host one.
|
||||
* Consumers scan committed facts through it (`scanFacts` / `segmentPaths`).
|
||||
*/
|
||||
getFactLog(): FactLog | null {
|
||||
return this.factLog
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Build one commit's AFTER-IMAGE fact by reading canonical
|
||||
* state back for every touched id — under the commit mutex, immediately
|
||||
* after the operations applied, so canonical IS the after-image (and the
|
||||
* reads are page-cache-warm: the operations just wrote these files). An
|
||||
* absent id (both legs null) becomes a body-less TOMBSTONE — the delete
|
||||
* fact needs no body, so removal never requires reading the removed thing.
|
||||
* The fact's blobHashes are extracted from the AFTER records (the content
|
||||
* this generation's state references), unlike the history path's
|
||||
* before-image hashes.
|
||||
*/
|
||||
private async buildCommitFact(args: {
|
||||
generation: number
|
||||
timestamp: number
|
||||
nouns: string[]
|
||||
verbs: string[]
|
||||
meta?: Record<string, unknown>
|
||||
}): Promise<CommitFact> {
|
||||
const ops: FactOp[] = []
|
||||
const afterRecords: GenerationRecord[] = []
|
||||
for (const id of args.nouns) {
|
||||
const after = await this.storage.readNounRaw(id)
|
||||
const absent = after.metadata === null && after.vector === null
|
||||
ops.push({ kind: 'noun', id, record: absent ? null : after })
|
||||
if (!absent) afterRecords.push({ kind: 'noun', metadata: after.metadata, vector: after.vector })
|
||||
}
|
||||
for (const id of args.verbs) {
|
||||
const after = await this.storage.readVerbRaw(id)
|
||||
const absent = after.metadata === null && after.vector === null
|
||||
ops.push({ kind: 'verb', id, record: absent ? null : after })
|
||||
if (!absent) afterRecords.push({ kind: 'verb', metadata: after.metadata, vector: after.vector })
|
||||
}
|
||||
const blobHashes = this.storage.extractBlobHashesFromRecords
|
||||
? this.storage.extractBlobHashesFromRecords(afterRecords)
|
||||
: []
|
||||
return {
|
||||
generation: args.generation,
|
||||
timestamp: args.timestamp,
|
||||
ops,
|
||||
...(args.meta ? { meta: args.meta } : {}),
|
||||
...(blobHashes.length > 0 ? { blobHashes } : {})
|
||||
}
|
||||
}
|
||||
|
||||
async commitTransaction(args: {
|
||||
touched: TouchedIds
|
||||
meta?: Record<string, unknown>
|
||||
|
|
@ -733,6 +809,24 @@ export class GenerationStore {
|
|||
await this.storage.flushWriteBarrier?.()
|
||||
faultPoint('after-execute')
|
||||
|
||||
// Fact log (dual-write): append + fsync this generation's AFTER-IMAGE
|
||||
// fact BEFORE the commit point, inside the same durability window —
|
||||
// so a crash can only leave the log AHEAD (open truncates), never a
|
||||
// committed generation without its fact. A real abort below this point
|
||||
// compensates via dropAbove in the catch. An append failure fails the
|
||||
// write, loudly — a silent fact gap would be a lie a replay discovers.
|
||||
if (this.factLog) {
|
||||
const fact = await this.buildCommitFact({
|
||||
generation: gen,
|
||||
timestamp,
|
||||
nouns,
|
||||
verbs,
|
||||
...(args.meta ? { meta: args.meta } : {})
|
||||
})
|
||||
await this.factLog.append(fact)
|
||||
await this.factLog.sync()
|
||||
}
|
||||
|
||||
// -- 5. Counter + manifest rename (COMMIT POINT) ----------------------
|
||||
await this.persistCounterUnlocked()
|
||||
faultPoint('before-manifest-rename')
|
||||
|
|
@ -800,6 +894,13 @@ export class GenerationStore {
|
|||
// over-count-safe; the scrub restores exactness
|
||||
}
|
||||
}
|
||||
// Fact-log compensation: a real (non-crash) abort after the fact was
|
||||
// appended must take the fact back out — the generation never
|
||||
// committed. A crash instead reaches open(), whose truncation does the
|
||||
// same reconcile from disk.
|
||||
if (this.factLog && this.factLog.headGeneration() >= gen) {
|
||||
await this.factLog.dropAbove(gen - 1)
|
||||
}
|
||||
// Return the reservation when no concurrent bump consumed a later
|
||||
// number, so a failed transaction leaves generation() unchanged.
|
||||
if (this.counter === gen) this.counter = gen - 1
|
||||
|
|
@ -991,6 +1092,14 @@ export class GenerationStore {
|
|||
this.pendingBuffer.set(gen, { nouns: nounBefore, verbs: verbBefore, timestamp })
|
||||
this.pendingGens.push(gen)
|
||||
this.extendChains(gen, nouns, verbs)
|
||||
// The adopted generation is committed — it gets its fact like any
|
||||
// other (durability rides the group-commit flush, same as the
|
||||
// buffered history).
|
||||
if (this.factLog) {
|
||||
await this.factLog.append(
|
||||
await this.buildCommitFact({ generation: gen, timestamp, nouns, verbs })
|
||||
)
|
||||
}
|
||||
prodLog.warn(
|
||||
`[GenerationStore] Recovered a failed rollback FORWARD: single-op write ` +
|
||||
`committed as generation ${gen} because its canonical undo could not be ` +
|
||||
|
|
@ -1020,6 +1129,17 @@ export class GenerationStore {
|
|||
this.pendingBuffer.set(gen, { nouns: nounBefore, verbs: verbBefore, timestamp })
|
||||
this.pendingGens.push(gen)
|
||||
this.extendChains(gen, nouns, verbs)
|
||||
// Fact log (dual-write): the acked write's AFTER-IMAGE fact, appended
|
||||
// now (read back warm, under the mutex — group-commit means flush-time
|
||||
// canonical only holds the LATEST state, so each generation's after-image
|
||||
// exists only here). Durability rides the group-commit flush, exactly
|
||||
// like the buffered before-image history: a crash before the flush loses
|
||||
// the fact AND the generation together — never a torn state.
|
||||
if (this.factLog) {
|
||||
await this.factLog.append(
|
||||
await this.buildCommitFact({ generation: gen, timestamp, nouns, verbs })
|
||||
)
|
||||
}
|
||||
this.schedulePendingFlush()
|
||||
return { generation: gen, timestamp }
|
||||
})
|
||||
|
|
@ -1141,6 +1261,12 @@ export class GenerationStore {
|
|||
// ONE fsync for the whole window — the durability-batching win.
|
||||
await this.storage.syncRawObjects(stagedPaths)
|
||||
|
||||
// Fact log (dual-write): make the window's buffered facts durable in the
|
||||
// same batch, BEFORE the commit point below — so a crash can only leave
|
||||
// the log AHEAD of the counter (open truncates), never a committed
|
||||
// generation without its durable fact.
|
||||
await this.factLog?.sync()
|
||||
|
||||
// Test-only crash simulation: a throwing injector here leaves the staged
|
||||
// group-commit generation dirs on disk with NO manifest advance — the
|
||||
// exact "crashed mid-flush" state recovery must DROP-WITHOUT-RESTORE
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue