fix: O(1) adaptive retention accounting + historyStats fleet audit

Under default adaptive retention, every flush() recomputed total history
bytes by walking EVERY committed generation's delta — O(all generations)
with disk re-reads past the 4096-entry delta-cache bound. On a production
brain with 70,000+ accumulated generations this turned every write into a
full-tail scan (60-100s writes, escalating with history growth), even
though the free-RAM budget never tripped and nothing was ever reclaimed
(SELF-GENERATIONS-GROWTH).

historyBytes() now maintains a running total: seeded by one walk on first
use, then updated incrementally at both commit paths (+bytes) and the
compaction reclaim loop (−bytes), dropped on reopenAfterRestore. The
adaptive retention check on every flush is O(1). Invariant regression-
pinned: running total ≡ fresh walk through transact commits, single-op
group commits, and compaction.

New brain.historyStats() (exported HistoryStats): read-only generation
count / bytes / generation+timestamp range / horizon / retention mode /
effective budget — the one-call per-brain fleet audit for retention
exposure.
This commit is contained in:
David Snelling 2026-07-18 10:51:37 -07:00
parent 4fcef7b8ed
commit 6207e48b51
6 changed files with 228 additions and 5 deletions

View file

@ -10,7 +10,31 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so
---
## v8.8.1 — 2026-07-18 (the import dedup off-switch is now honest + lifecycle-safe)
## v8.8.1 — 2026-07-18 (flush no longer walks the whole generation history + the import dedup off-switch is now honest)
### The flush-storm fix (production incident, reported by a long-running deployment)
Under the default adaptive retention, **every `flush()` re-walked the entire committed
generation history** to compute total history bytes for the budget check — O(all
generations) with disk re-reads past the 4,096-entry delta-cache bound. On a brain with
70,000+ accumulated generations that turned every write into a full-tail scan (60-100s
writes), even though the budget (free-RAM-based) never tripped and nothing was ever
reclaimed. Fixed:
- `historyBytes()` now maintains a **running total**: seeded by one walk on first use,
then updated incrementally at every commit and reclaim — the adaptive retention check
on every flush is O(1). Invariant regression-pinned (running total ≡ fresh walk through
both commit paths and compaction).
- New **`brain.historyStats()`** (read-only, exported `HistoryStats`): generation count,
total on-disk bytes, generation/timestamp range, compaction horizon, retention mode,
and the effective adaptive budget — the one-call fleet-audit for sizing retention
exposure per brain.
- Interim guidance for keep-everything deployments already affected: `retention: 'all'`
skips the adaptive accounting entirely (and is the correct policy if you never want
history reclaimed). The accumulated files are harmless at rest; this release removes
the per-write cost of their existence.
### The import dedup off-switch (lifecycle honesty)
The post-import background deduplication pass (a merge-DELETE writer that runs ~5 minutes
after an import, merging entities judged duplicates by id / name / vector similarity) had