brainy/docs/concepts/generation-fact-log.md

title	slug	public	category	template	order	description	next
The Generation Fact Log	concepts/generation-fact-log	true	concepts	concept	6	Every committed write also appends a self-verifying "fact" — an after-image commit record — to an append-only log. What facts are, the crash-safety model, the scanFacts() streaming surface, family stamps, and how index providers consume the log for sequential heals.	concepts/consistency-model guides/snapshots-and-time-travel

The Generation Fact Log

Since 8.4.0, every committed generation also appends a fact — a compact record of what each touched entity or relationship became — to an append-only, checksummed log under _generations/facts/. Where the generational history answers "what did things look like before?" (before-images, powering asOf() and rollback), the fact log answers "what happened, in order?" — one sequential, self-verifying stream of the store's present being written.

Nothing about querying changes. The fact log exists for three consumers:

1. Index heals and rebuilds — one sequential read in commit order replaces a per-entity directory walk over millions of files.
2. Incremental catch-up — a derived index that knows which generation it reflects reads just the gap, instead of rebuilding from scratch.
3. Replay and audit tooling — anything that wants the store's committed timeline as a stream.

What a fact is

One fact per committed generation:

• generation and timestamp — which commit, when.
• ops — every write in that commit: { kind: 'noun' | 'verb', id, record } where record holds the entity's full after-image (both stored legs), or null for a tombstone — a removal carries no body, by design.
• meta — the transaction metadata transact() was submitted with, when present.
• blobHashes — content-blob references, for exact reclamation accounting.

Facts accumulate from the first write after upgrading — pre-existing history is not retroactively converted, and consumers fall back to the enumeration walk when no log exists.

Crash safety, in one paragraph

Facts are appended and fsynced inside the same durability window as the commit itself, before the commit point — so after a crash, the log can only ever be ahead of committed truth, never behind it with a hole. On open, the store reconciles the log back to the committed watermark: torn tails are detected by per-record checksums and cut; whole records beyond the watermark are truncated. The invariant every reader can rely on: an absent generation was never committed; a present fact was. transact() facts are durable the moment transact() returns; single-op facts share the same group-commit flush as the rest of their generation, so a hard kill loses the fact and the generation together — never a torn state.

Reading the log

const scan = brain.scanFacts({ fromGeneration: 1 })
if (scan) {
  // Telemetry up front — progress bars get a denominator from second zero.
  console.log(scan.headGeneration, scan.segmentCount, scan.approxFactCount)

  for await (const batch of scan.batches()) {
    // Each batch: { facts, firstGeneration, lastGeneration, factCount, byteSize, segmentId }
    for (const fact of batch.facts) {
      for (const op of fact.ops) {
        if (op.record === null) {
          // a tombstone: op.id was removed in this generation
        }
      }
    }
  }

  console.log(scan.summary()) // { factsYielded, segmentsRead } — the cross-check
}

• scanFacts() returns null when the store hosts no fact log (older store, or a storage adapter without binary append support) — fall back to enumerating entities.
• Scans run against a snapshot: facts appended after the scan opens never bleed in, each fact is yielded exactly once, and a detected gap aborts loudly — never a silent skip.
• brain.factSegmentPaths() returns the immutable, sealed segment files for zero-copy consumers (the append-mutable tail is excluded — read it through scanFacts()).

Family stamps: how a projection proves it's current

Anything derived from the store — an index, the entity file tree itself — carries a family stamp: a small JSON record of which committed generation the projection reflects (sourceGeneration) plus the invariants that verify it whole (exact per-file byte sizes for bounded families; rollup invariants like entity counts for unbounded ones). At open, coherence is a comparison, not a walk:

• stamp equals the committed watermark and invariants hold → serve;
• stamp is behind → the projection reads just the gap from the fact log;
• invariants fail → loud, named divergence — brain.repairIndex() rebuilds from canonical and re-stamps.

The verifier is exported (verifyFamilyStamp) so every projection — TypeScript or native — runs literally the same check.

For plugin authors: the storage capability

Index providers receive the storage adapter, not the brain — so the host wires the log onto it. Feature-detect and prefer the stream; fall back to enumeration:

const scan = storage.scanFacts?.({ fromGeneration: stamp.sourceGeneration + 1 })
if (scan) {
  // sequential catch-up from the log
} else {
  // enumeration walk (older store or adapter)
}
const committed = storage.committedGeneration?.() // the watermark stamps compare against

Providers must never construct their own reader over the log's files — the open path belongs to the single writer (it reconciles the log at open); the capability is the sanctioned seam.