feat: two-tier history reads + the repacker + generationDigest — D1+D3 wired end-to-end
The packed tier goes live inside GenerationStore:
- Two-tier reads: getDelta / readBeforeImage / readGenerationRecords
fall through live-tier → sealed segments (live-tier-wins: a crash
mid-fold leaves a duplicate representation, never a gap). Cold-open
seeds committedRanges from the segment manifest via interval merge —
packed generations resolve without their directories existing.
- repackHistory({timeBudgetMs, batchGenerations}): folds cold
generations (older than the newest 1024) oldest-first into sealed
segments, deleting per-generation directories only after segment +
manifest are durable. Public API + automatic time-bounded pass at
close() (before compaction, so reclaim can drop whole segments);
re-representation only — the sole history transform under the
archival profile. Early stop = consistent prefix, next pass resumes.
- compact(): packed generations reclaim logically in the loop and
physically at whole-segment boundaries via dropSegmentsBelow (the
frozen partial-segments-wait rule).
- generationDigest(g) (D8): deterministic content digest through g —
sealed-segment checksum chain + live-tier delta hashes; O(segments +
live window); RangeError out of range, GenerationCompactedError
below the horizon (a gate can never silently pin reclaimed history).
Four end-to-end pins: asOf answers byte-identical across fold + cold
reopen with folded dirs physically gone; repack+reclaim composition;
digest reopen-stability/divergence/loud-horizon; budget no-op+resume.
This commit is contained in:
parent
d8acb3776b
commit
1201e25543
3 changed files with 454 additions and 8 deletions
|
|
@ -8265,6 +8265,27 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
return this.generationStore.compact(options)
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Repack cold generation history into sealed segments —
|
||||
* re-representation, never deletion: every record and delta stays readable
|
||||
* (`asOf()` unchanged); the physical file count drops by orders of
|
||||
* magnitude. Runs automatically (time-bounded) at `close()`; call this for
|
||||
* explicit maintenance windows on long-lived writers. The ONLY history
|
||||
* transform permitted under the archival profile (`retention: 'all'`).
|
||||
* @param options - `timeBudgetMs` bounds the pass (early stop = consistent
|
||||
* prefix, next pass resumes); `batchGenerations` sizes each fold.
|
||||
* @returns Folded generation count and segments created.
|
||||
*/
|
||||
async repackHistory(options?: {
|
||||
timeBudgetMs?: number
|
||||
batchGenerations?: number
|
||||
}): Promise<{ foldedGenerations: number; segmentsCreated: number }> {
|
||||
this.assertWritable('repackHistory')
|
||||
await this.ensureInitialized()
|
||||
await this.generationStore.flushPendingSingleOps()
|
||||
return this.generationStore.repackHistory(options)
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Read-only generational-history footprint for fleet audits:
|
||||
* generation count, total on-disk bytes, generation/timestamp range, the
|
||||
|
|
@ -8292,6 +8313,24 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @description A deterministic content digest of the generation log through
|
||||
* `g` (D8 — gate-to-generation provenance): identical history produces the
|
||||
* identical digest on any machine; divergence produces a different one.
|
||||
* Release gates and suite verdicts pin `{generation, digest}` and verify
|
||||
* both at execution time instead of pinning a git commit. O(segments +
|
||||
* live-tier window), never O(all generations). Throws `RangeError` out of
|
||||
* range and `GenerationCompactedError` below the horizon — a gate can
|
||||
* never silently pin reclaimed history.
|
||||
* @example
|
||||
* const gate = { generation: brain.generation(), digest: await brain.generationDigest(brain.generation()) }
|
||||
*/
|
||||
async generationDigest(g: number): Promise<string> {
|
||||
await this.ensureInitialized()
|
||||
await this.generationStore.flushPendingSingleOps()
|
||||
return this.generationStore.generationDigest(g)
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Drive the adaptive retention byte budget at runtime — the
|
||||
* settable input a machine-level coordinator (e.g. cor's `ResourceManager`,
|
||||
|
|
@ -16192,11 +16231,21 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
await this.generationStore.flushPendingSingleOps()
|
||||
}
|
||||
|
||||
// Phase 0b: Auto-compact generational history per config.retention (default
|
||||
// on) BEFORE the generation store closes below. This is THE auto-compaction
|
||||
// site (8.9.0 — flush() never compacts): time-bounded per pass, respects
|
||||
// live Db pins and an explicit autoCompact: false; no-op on read-only
|
||||
// instances.
|
||||
// Phase 0b: REPACK cold history into sealed segments (D1+D3 —
|
||||
// re-representation, never deletion; the only history transform under the
|
||||
// archival profile), then auto-compact per config.retention. Repack runs
|
||||
// FIRST so bounded-retention reclaim can drop whole segments. Both are
|
||||
// time-bounded maintenance passes (8.9.0 law: flush() never pays these);
|
||||
// both are housekeeping — failures warn, never fail a clean shutdown.
|
||||
if (!this.isReadOnly && this.generationStore) {
|
||||
try {
|
||||
await this.generationStore.repackHistory({ timeBudgetMs: 5_000 })
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
`History repacking failed (non-fatal): ${error instanceof Error ? error.message : String(error)}`
|
||||
)
|
||||
}
|
||||
}
|
||||
await this.autoCompactHistory()
|
||||
|
||||
// Phase 1: Flush ALL components in parallel to persist buffered data
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue