feat: generation fact log — after-image commit records, dual-written at every commit point

Every committed generation now also appends a FACT — an after-image
commit record (what each touched entity/relationship became, or a
body-less tombstone for a removal) — to an append-only, crc32c-framed
segment log under _generations/facts/. The before-image history and the
canonical tree remain authoritative; the fact log gives consumers ONE
sequential, self-verifying stream (index heals, incremental replays)
in place of a per-entity directory walk.

- Wire format: positional msgpack facts [generation, timestamp, ops,
  meta, blobHashes]; op = [kind u8, id bin16, record | nil tombstone];
  32-byte segment header (magic, formatVersion, firstGeneration,
  zeroed+verified reserved); length+crc32c frame per fact; zero-padded
  segment names so lexicographic order == generation order; JSON
  manifest with an atomic rename flip, manifest-first rotation.
- Commit protocol: facts append+fsync BEFORE the commit point inside
  the existing durability window, so a crash can only leave the log
  AHEAD of committed truth — open() truncates back (torn tails detected
  by CRC). Absent generation = never committed; a scan can never see an
  uncommitted fact. transact() facts are durable-on-return; single-op
  facts ride the group-commit flush exactly like buffered history. A
  fact-append failure fails the write, loudly — a silent gap would be a
  lie a later replay discovers.
- New public surface: brain.scanFacts() (sequential batches with heal
  telemetry: head/segments/approx up front, per-batch generation range
  + bytes + segment id, loud abort on gaps, summary cross-check) and
  brain.factSegmentPaths() (immutable sealed segments for zero-copy
  consumers; the mutable tail excluded). Exported types CommitFact,
  FactOp, FactScanBatch, FactScanHandle.
- Storage: optional binary raw-byte primitives (appendRawBytes,
  readRawBytes, writeRawBytes, rawByteSize) on StorageAdapter —
  feature-detected; filesystem + memory adapters implement them; an
  adapter without them hosts no fact log. Fact segments are byte-copied
  (never hard-linked) into snapshots. The _generations/facts/ namespace
  is registered as a protected family (rebuildable: false): no sweeper
  or GC may delete under it.
- New crc32c (Castagnoli) utility with RFC known-answer tests.
This commit is contained in:
David Snelling 2026-07-15 10:49:02 -07:00
parent 92299f27be
commit 38b0041464
13 changed files with 1493 additions and 4 deletions

View file

@ -167,6 +167,7 @@ import {
type ImportResult
} from './db/portableGraph.js'
import { GenerationStore, type CommitBeforeImages } from './db/generationStore.js'
import type { FactScanHandle } from './db/factLog.js'
import {
ChangeFeed,
type BrainyChangeEvent,
@ -973,6 +974,24 @@ export class Brainy<T = any> implements BrainyInterface<T> {
readOnly: this.config.mode === 'reader'
})
// The generation fact log is CANONICAL state, not a derived index — no
// sweeper, GC, or blob-lifecycle path may ever delete under it. Declare
// its namespace as a protected family (rebuildable: false — a lost fact
// segment is NOT reconstructable) so the storage layer REFUSES such
// deletes; refusal beats trust. Feature-detected + idempotent per name.
if (
this.config.mode !== 'reader' &&
this.generationStore.getFactLog() &&
typeof this.storage.registerDerivedFamily === 'function'
) {
await this.storage.registerDerivedFamily({
name: 'generation-facts',
members: ['_generations/facts/'],
namespace: true,
rebuildable: false
})
}
// 8.0 ⇄ native-provider version handshake: load the on-disk brain-format
// marker (`_system/brain-format.json`) into an in-memory field NOW —
// after the store-open phase, but BEFORE any derived index or native
@ -7434,6 +7453,59 @@ export class Brainy<T = any> implements BrainyInterface<T> {
await this.removeMigrationBackupSafe()
}
/**
* @description Open a sequential scan over the generation FACT LOG the
* append-only record of every committed generation as an AFTER-IMAGE fact
* (what each touched entity/relationship became, or a body-less tombstone
* for a removal). The scan is the streaming substrate for index heals and
* incremental replays: one sequential read in commit order replaces a
* per-entity directory walk. The handle carries heal-narration telemetry
* (`headGeneration` / `segmentCount` / `approxFactCount` up front; ordered
* batches each stamped with their generation range, byte size, and segment;
* a `summary()` cross-check after iteration). A detected gap or damaged
* segment ABORTS the scan loudly never a silent skip.
*
* Returns `null` when this store hosts no fact log: the storage adapter
* lacks the binary append primitives, or the brain predates the fact log
* (its history began before dual-write facts exist only from the first
* write after upgrade; callers fall back to the canonical enumeration walk).
*
* @param options - `fromGeneration`/`toGeneration` bound the scan (inclusive
* both ends); `kinds` filters ops to `'noun'`/`'verb'`; `batchSize` caps
* facts per yielded batch (default 256).
* @returns The scan handle, or `null` when no fact log exists.
* @example
* const scan = brain.scanFacts({ fromGeneration: 1 })
* if (scan) {
* for await (const batch of scan.batches()) {
* for (const fact of batch.facts) {
* // fact.ops: [{ kind, id, record | null (tombstone) }, ...]
* }
* }
* }
*/
scanFacts(options?: {
fromGeneration?: number
toGeneration?: number
kinds?: Array<'noun' | 'verb'>
batchSize?: number
}): FactScanHandle | null {
const factLog = this.generationStore?.getFactLog()
return factLog ? factLog.scanFacts(options) : null
}
/**
* @description The immutable, sealed fact-log segment files covering
* `fromGeneration` the zero-copy handoff for consumers that map segment
* files directly instead of streaming {@link scanFacts} batches. The
* append-mutable TAIL segment is deliberately excluded (read it via
* `scanFacts`). Paths are storage-root-relative. Empty when no fact log
* exists or nothing is sealed yet.
*/
factSegmentPaths(options?: { fromGeneration?: number }): string[] {
return this.generationStore?.getFactLog()?.segmentPaths(options) ?? []
}
/**
* @description Read the reified transaction log one entry per committed
* generation, carrying the committed generation, the commit timestamp, and