brainy/docs/concepts/generation-fact-log.md

---
title: The Generation Fact Log
slug: concepts/generation-fact-log
public: true
category: concepts
template: concept
order: 6
description: Every committed write also appends a self-verifying "fact" — an after-image commit record — to an append-only log. What facts are, the crash-safety model, the scanFacts() streaming surface, family stamps, and how index providers consume the log for sequential heals.
next:
  - concepts/consistency-model
  - guides/snapshots-and-time-travel
---

# The Generation Fact Log

Since 8.4.0, every committed generation also appends a **fact** — a compact record of what each
touched entity or relationship *became* — to an append-only, checksummed log under
`_generations/facts/`. Where the generational history answers *"what did things look like
before?"* (before-images, powering `asOf()` and rollback), the fact log answers *"what happened,
in order?"* — one sequential, self-verifying stream of the store's present being written.

Nothing about querying changes. The fact log exists for three consumers:

1. **Index heals and rebuilds** — one sequential read in commit order replaces a per-entity
   directory walk over millions of files.
2. **Incremental catch-up** — a derived index that knows which generation it reflects reads *just
   the gap*, instead of rebuilding from scratch.
3. **Replay and audit tooling** — anything that wants the store's committed timeline as a stream.

## What a fact is

One fact per committed generation:

- **`generation`** and **`timestamp`** — which commit, when.
- **`ops`** — every write in that commit: `{ kind: 'noun' | 'verb', id, record }` where `record`
  holds the entity's full after-image (both stored legs), or **`null` for a tombstone** — a
  removal carries no body, by design.
- **`meta`** — the transaction metadata `transact()` was submitted with, when present.
- **`blobHashes`** — content-blob references, for exact reclamation accounting.

Facts accumulate **from the first write after upgrading** — pre-existing history is not
retroactively converted, and consumers fall back to the enumeration walk when no log exists.

## Crash safety, in one paragraph

Facts are appended and fsynced **inside the same durability window as the commit itself**, before
the commit point — so after a crash, the log can only ever be *ahead* of committed truth, never
behind it with a hole. On open, the store reconciles the log back to the committed watermark:
torn tails are detected by per-record checksums and cut; whole records beyond the watermark are
truncated. The invariant every reader can rely on: **an absent generation was never committed; a
present fact was.** `transact()` facts are durable the moment `transact()` returns; single-op
facts share the same group-commit flush as the rest of their generation, so a hard kill loses the
fact and the generation *together* — never a torn state.

## Reading the log

```typescript
const scan = brain.scanFacts({ fromGeneration: 1 })
if (scan) {
  // Telemetry up front — progress bars get a denominator from second zero.
  console.log(scan.headGeneration, scan.segmentCount, scan.approxFactCount)

  for await (const batch of scan.batches()) {
    // Each batch: { facts, firstGeneration, lastGeneration, factCount, byteSize, segmentId }
    for (const fact of batch.facts) {
      for (const op of fact.ops) {
        if (op.record === null) {
          // a tombstone: op.id was removed in this generation
        }
      }
    }
  }

  console.log(scan.summary()) // { factsYielded, segmentsRead } — the cross-check
}
```

- `scanFacts()` returns `null` when the store hosts no fact log (older store, or a storage adapter
  without binary append support) — fall back to enumerating entities.
- Scans run against a **snapshot**: facts appended after the scan opens never bleed in, each fact
  is yielded exactly once, and a detected gap aborts loudly — never a silent skip.
- `brain.factSegmentPaths()` returns the immutable, *sealed* segment files for zero-copy consumers
  (the append-mutable tail is excluded — read it through `scanFacts()`).

## Family stamps: how a projection proves it's current

Anything derived from the store — an index, the entity file tree itself — carries a **family
stamp**: a small JSON record of *which committed generation the projection reflects*
(`sourceGeneration`) plus the invariants that verify it whole (exact per-file byte sizes for
bounded families; rollup invariants like entity counts for unbounded ones). At open, coherence is
a **comparison**, not a walk:

- stamp equals the committed watermark and invariants hold → serve;
- stamp is behind → the projection reads just the gap from the fact log;
- invariants fail → loud, named divergence — `brain.repairIndex()` rebuilds from canonical and
  re-stamps.

The verifier is exported (`verifyFamilyStamp`) so every projection — TypeScript or native — runs
literally the same check.

## For plugin authors: the storage capability

Index providers receive the storage adapter, not the brain — so the host wires the log onto it.
Feature-detect and prefer the stream; fall back to enumeration:

```typescript
const scan = storage.scanFacts?.({ fromGeneration: stamp.sourceGeneration + 1 })
if (scan) {
  // sequential catch-up from the log
} else {
  // enumeration walk (older store or adapter)
}
const committed = storage.committedGeneration?.() // the watermark stamps compare against
```

Providers must never construct their own reader over the log's files — the open path belongs to
the single writer (it reconciles the log at open); the capability is the sanctioned seam.