feat: generation fact log — after-image commit records, dual-written at every commit point

Every committed generation now also appends a FACT — an after-image
commit record (what each touched entity/relationship became, or a
body-less tombstone for a removal) — to an append-only, crc32c-framed
segment log under _generations/facts/. The before-image history and the
canonical tree remain authoritative; the fact log gives consumers ONE
sequential, self-verifying stream (index heals, incremental replays)
in place of a per-entity directory walk.

- Wire format: positional msgpack facts [generation, timestamp, ops,
  meta, blobHashes]; op = [kind u8, id bin16, record | nil tombstone];
  32-byte segment header (magic, formatVersion, firstGeneration,
  zeroed+verified reserved); length+crc32c frame per fact; zero-padded
  segment names so lexicographic order == generation order; JSON
  manifest with an atomic rename flip, manifest-first rotation.
- Commit protocol: facts append+fsync BEFORE the commit point inside
  the existing durability window, so a crash can only leave the log
  AHEAD of committed truth — open() truncates back (torn tails detected
  by CRC). Absent generation = never committed; a scan can never see an
  uncommitted fact. transact() facts are durable-on-return; single-op
  facts ride the group-commit flush exactly like buffered history. A
  fact-append failure fails the write, loudly — a silent gap would be a
  lie a later replay discovers.
- New public surface: brain.scanFacts() (sequential batches with heal
  telemetry: head/segments/approx up front, per-batch generation range
  + bytes + segment id, loud abort on gaps, summary cross-check) and
  brain.factSegmentPaths() (immutable sealed segments for zero-copy
  consumers; the mutable tail excluded). Exported types CommitFact,
  FactOp, FactScanBatch, FactScanHandle.
- Storage: optional binary raw-byte primitives (appendRawBytes,
  readRawBytes, writeRawBytes, rawByteSize) on StorageAdapter —
  feature-detected; filesystem + memory adapters implement them; an
  adapter without them hosts no fact log. Fact segments are byte-copied
  (never hard-linked) into snapshots. The _generations/facts/ namespace
  is registered as a protected family (rebuildable: false): no sweeper
  or GC may delete under it.
- New crc32c (Castagnoli) utility with RFC known-answer tests.
This commit is contained in:
David Snelling 2026-07-15 10:49:02 -07:00
parent 92299f27be
commit 38b0041464
13 changed files with 1493 additions and 4 deletions

View file

@ -222,6 +222,45 @@ export class MemoryStorage extends BaseStorage {
return [...this.txLogLines]
}
// ===========================================================================
// Binary raw-byte primitives — in-memory mirror of the filesystem adapter's
// append-only substrate (the generation fact log's segments), so memory
// brains dual-write facts too and the compat suite runs on both adapters.
// ===========================================================================
/** Raw binary files, keyed by verbatim path. */
private rawBytesStore: Map<string, Uint8Array> = new Map()
/** Append bytes to a raw binary file, creating it when absent. */
public async appendRawBytes(rawPath: string, bytes: Uint8Array): Promise<void> {
const existing = this.rawBytesStore.get(rawPath)
if (!existing) {
this.rawBytesStore.set(rawPath, bytes.slice())
return
}
const merged = new Uint8Array(existing.length + bytes.length)
merged.set(existing, 0)
merged.set(bytes, existing.length)
this.rawBytesStore.set(rawPath, merged)
}
/** Read a raw binary file whole (a copy); absent → null. */
public async readRawBytes(rawPath: string): Promise<Uint8Array | null> {
const bytes = this.rawBytesStore.get(rawPath)
return bytes ? bytes.slice() : null
}
/** Replace a raw binary file (atomic by construction in memory). */
public async writeRawBytes(rawPath: string, bytes: Uint8Array): Promise<void> {
this.rawBytesStore.set(rawPath, bytes.slice())
}
/** Byte size of a raw binary file, or null when absent. */
public async rawByteSize(rawPath: string): Promise<number | null> {
const bytes = this.rawBytesStore.get(rawPath)
return bytes ? bytes.length : null
}
/**
* Serialize the entire in-memory store to a directory in the exact layout
* the filesystem adapter uses (uncompressed JSON objects, `_blobs/*.bin`
@ -354,6 +393,7 @@ export class MemoryStorage extends BaseStorage {
public async clear(): Promise<void> {
this.objectStore.clear()
this.blobStore.clear()
this.rawBytesStore.clear()
this.txLogLines = []
this.statistics = null