feat: generation fact log — after-image commit records, dual-written at every commit point
Every committed generation now also appends a FACT — an after-image commit record (what each touched entity/relationship became, or a body-less tombstone for a removal) — to an append-only, crc32c-framed segment log under _generations/facts/. The before-image history and the canonical tree remain authoritative; the fact log gives consumers ONE sequential, self-verifying stream (index heals, incremental replays) in place of a per-entity directory walk. - Wire format: positional msgpack facts [generation, timestamp, ops, meta, blobHashes]; op = [kind u8, id bin16, record | nil tombstone]; 32-byte segment header (magic, formatVersion, firstGeneration, zeroed+verified reserved); length+crc32c frame per fact; zero-padded segment names so lexicographic order == generation order; JSON manifest with an atomic rename flip, manifest-first rotation. - Commit protocol: facts append+fsync BEFORE the commit point inside the existing durability window, so a crash can only leave the log AHEAD of committed truth — open() truncates back (torn tails detected by CRC). Absent generation = never committed; a scan can never see an uncommitted fact. transact() facts are durable-on-return; single-op facts ride the group-commit flush exactly like buffered history. A fact-append failure fails the write, loudly — a silent gap would be a lie a later replay discovers. - New public surface: brain.scanFacts() (sequential batches with heal telemetry: head/segments/approx up front, per-batch generation range + bytes + segment id, loud abort on gaps, summary cross-check) and brain.factSegmentPaths() (immutable sealed segments for zero-copy consumers; the mutable tail excluded). Exported types CommitFact, FactOp, FactScanBatch, FactScanHandle. - Storage: optional binary raw-byte primitives (appendRawBytes, readRawBytes, writeRawBytes, rawByteSize) on StorageAdapter — feature-detected; filesystem + memory adapters implement them; an adapter without them hosts no fact log. Fact segments are byte-copied (never hard-linked) into snapshots. The _generations/facts/ namespace is registered as a protected family (rebuildable: false): no sweeper or GC may delete under it. - New crc32c (Castagnoli) utility with RFC known-answer tests.
This commit is contained in:
parent
92299f27be
commit
38b0041464
13 changed files with 1493 additions and 4 deletions
|
|
@ -167,6 +167,7 @@ import {
|
|||
type ImportResult
|
||||
} from './db/portableGraph.js'
|
||||
import { GenerationStore, type CommitBeforeImages } from './db/generationStore.js'
|
||||
import type { FactScanHandle } from './db/factLog.js'
|
||||
import {
|
||||
ChangeFeed,
|
||||
type BrainyChangeEvent,
|
||||
|
|
@ -973,6 +974,24 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
readOnly: this.config.mode === 'reader'
|
||||
})
|
||||
|
||||
// The generation fact log is CANONICAL state, not a derived index — no
|
||||
// sweeper, GC, or blob-lifecycle path may ever delete under it. Declare
|
||||
// its namespace as a protected family (rebuildable: false — a lost fact
|
||||
// segment is NOT reconstructable) so the storage layer REFUSES such
|
||||
// deletes; refusal beats trust. Feature-detected + idempotent per name.
|
||||
if (
|
||||
this.config.mode !== 'reader' &&
|
||||
this.generationStore.getFactLog() &&
|
||||
typeof this.storage.registerDerivedFamily === 'function'
|
||||
) {
|
||||
await this.storage.registerDerivedFamily({
|
||||
name: 'generation-facts',
|
||||
members: ['_generations/facts/'],
|
||||
namespace: true,
|
||||
rebuildable: false
|
||||
})
|
||||
}
|
||||
|
||||
// 8.0 ⇄ native-provider version handshake: load the on-disk brain-format
|
||||
// marker (`_system/brain-format.json`) into an in-memory field NOW —
|
||||
// after the store-open phase, but BEFORE any derived index or native
|
||||
|
|
@ -7434,6 +7453,59 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
await this.removeMigrationBackupSafe()
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Open a sequential scan over the generation FACT LOG — the
|
||||
* append-only record of every committed generation as an AFTER-IMAGE fact
|
||||
* (what each touched entity/relationship became, or a body-less tombstone
|
||||
* for a removal). The scan is the streaming substrate for index heals and
|
||||
* incremental replays: one sequential read in commit order replaces a
|
||||
* per-entity directory walk. The handle carries heal-narration telemetry
|
||||
* (`headGeneration` / `segmentCount` / `approxFactCount` up front; ordered
|
||||
* batches each stamped with their generation range, byte size, and segment;
|
||||
* a `summary()` cross-check after iteration). A detected gap or damaged
|
||||
* segment ABORTS the scan loudly — never a silent skip.
|
||||
*
|
||||
* Returns `null` when this store hosts no fact log: the storage adapter
|
||||
* lacks the binary append primitives, or the brain predates the fact log
|
||||
* (its history began before dual-write — facts exist only from the first
|
||||
* write after upgrade; callers fall back to the canonical enumeration walk).
|
||||
*
|
||||
* @param options - `fromGeneration`/`toGeneration` bound the scan (inclusive
|
||||
* both ends); `kinds` filters ops to `'noun'`/`'verb'`; `batchSize` caps
|
||||
* facts per yielded batch (default 256).
|
||||
* @returns The scan handle, or `null` when no fact log exists.
|
||||
* @example
|
||||
* const scan = brain.scanFacts({ fromGeneration: 1 })
|
||||
* if (scan) {
|
||||
* for await (const batch of scan.batches()) {
|
||||
* for (const fact of batch.facts) {
|
||||
* // fact.ops: [{ kind, id, record | null (tombstone) }, ...]
|
||||
* }
|
||||
* }
|
||||
* }
|
||||
*/
|
||||
scanFacts(options?: {
|
||||
fromGeneration?: number
|
||||
toGeneration?: number
|
||||
kinds?: Array<'noun' | 'verb'>
|
||||
batchSize?: number
|
||||
}): FactScanHandle | null {
|
||||
const factLog = this.generationStore?.getFactLog()
|
||||
return factLog ? factLog.scanFacts(options) : null
|
||||
}
|
||||
|
||||
/**
|
||||
* @description The immutable, sealed fact-log segment files covering
|
||||
* `fromGeneration` — the zero-copy handoff for consumers that map segment
|
||||
* files directly instead of streaming {@link scanFacts} batches. The
|
||||
* append-mutable TAIL segment is deliberately excluded (read it via
|
||||
* `scanFacts`). Paths are storage-root-relative. Empty when no fact log
|
||||
* exists or nothing is sealed yet.
|
||||
*/
|
||||
factSegmentPaths(options?: { fromGeneration?: number }): string[] {
|
||||
return this.generationStore?.getFactLog()?.segmentPaths(options) ?? []
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Read the reified transaction log — one entry per committed
|
||||
* generation, carrying the committed generation, the commit timestamp, and
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue