feat: two-tier history reads + the repacker + generationDigest — D1+D3 wired end-to-end

The packed tier goes live inside GenerationStore:

- Two-tier reads: getDelta / readBeforeImage / readGenerationRecords
  fall through live-tier → sealed segments (live-tier-wins: a crash
  mid-fold leaves a duplicate representation, never a gap). Cold-open
  seeds committedRanges from the segment manifest via interval merge —
  packed generations resolve without their directories existing.
- repackHistory({timeBudgetMs, batchGenerations}): folds cold
  generations (older than the newest 1024) oldest-first into sealed
  segments, deleting per-generation directories only after segment +
  manifest are durable. Public API + automatic time-bounded pass at
  close() (before compaction, so reclaim can drop whole segments);
  re-representation only — the sole history transform under the
  archival profile. Early stop = consistent prefix, next pass resumes.
- compact(): packed generations reclaim logically in the loop and
  physically at whole-segment boundaries via dropSegmentsBelow (the
  frozen partial-segments-wait rule).
- generationDigest(g) (D8): deterministic content digest through g —
  sealed-segment checksum chain + live-tier delta hashes; O(segments +
  live window); RangeError out of range, GenerationCompactedError
  below the horizon (a gate can never silently pin reclaimed history).

Four end-to-end pins: asOf answers byte-identical across fold + cold
reopen with folded dirs physically gone; repack+reclaim composition;
digest reopen-stability/divergence/loud-horizon; budget no-op+resume.
This commit is contained in:
David Snelling 2026-07-19 16:26:10 -07:00
parent d8acb3776b
commit 1201e25543
3 changed files with 454 additions and 8 deletions

View file

@ -8265,6 +8265,27 @@ export class Brainy<T = any> implements BrainyInterface<T> {
return this.generationStore.compact(options)
}
/**
* @description Repack cold generation history into sealed segments
* re-representation, never deletion: every record and delta stays readable
* (`asOf()` unchanged); the physical file count drops by orders of
* magnitude. Runs automatically (time-bounded) at `close()`; call this for
* explicit maintenance windows on long-lived writers. The ONLY history
* transform permitted under the archival profile (`retention: 'all'`).
* @param options - `timeBudgetMs` bounds the pass (early stop = consistent
* prefix, next pass resumes); `batchGenerations` sizes each fold.
* @returns Folded generation count and segments created.
*/
async repackHistory(options?: {
timeBudgetMs?: number
batchGenerations?: number
}): Promise<{ foldedGenerations: number; segmentsCreated: number }> {
this.assertWritable('repackHistory')
await this.ensureInitialized()
await this.generationStore.flushPendingSingleOps()
return this.generationStore.repackHistory(options)
}
/**
* @description Read-only generational-history footprint for fleet audits:
* generation count, total on-disk bytes, generation/timestamp range, the
@ -8292,6 +8313,24 @@ export class Brainy<T = any> implements BrainyInterface<T> {
}
}
/**
* @description A deterministic content digest of the generation log through
* `g` (D8 gate-to-generation provenance): identical history produces the
* identical digest on any machine; divergence produces a different one.
* Release gates and suite verdicts pin `{generation, digest}` and verify
* both at execution time instead of pinning a git commit. O(segments +
* live-tier window), never O(all generations). Throws `RangeError` out of
* range and `GenerationCompactedError` below the horizon a gate can
* never silently pin reclaimed history.
* @example
* const gate = { generation: brain.generation(), digest: await brain.generationDigest(brain.generation()) }
*/
async generationDigest(g: number): Promise<string> {
await this.ensureInitialized()
await this.generationStore.flushPendingSingleOps()
return this.generationStore.generationDigest(g)
}
/**
* @description Drive the adaptive retention byte budget at runtime the
* settable input a machine-level coordinator (e.g. cor's `ResourceManager`,
@ -16192,11 +16231,21 @@ export class Brainy<T = any> implements BrainyInterface<T> {
await this.generationStore.flushPendingSingleOps()
}
// Phase 0b: Auto-compact generational history per config.retention (default
// on) BEFORE the generation store closes below. This is THE auto-compaction
// site (8.9.0 — flush() never compacts): time-bounded per pass, respects
// live Db pins and an explicit autoCompact: false; no-op on read-only
// instances.
// Phase 0b: REPACK cold history into sealed segments (D1+D3 —
// re-representation, never deletion; the only history transform under the
// archival profile), then auto-compact per config.retention. Repack runs
// FIRST so bounded-retention reclaim can drop whole segments. Both are
// time-bounded maintenance passes (8.9.0 law: flush() never pays these);
// both are housekeeping — failures warn, never fail a clean shutdown.
if (!this.isReadOnly && this.generationStore) {
try {
await this.generationStore.repackHistory({ timeBudgetMs: 5_000 })
} catch (error) {
console.warn(
`History repacking failed (non-fatal): ${error instanceof Error ? error.message : String(error)}`
)
}
}
await this.autoCompactHistory()
// Phase 1: Flush ALL components in parallel to persist buffered data