fix: O(1) adaptive retention accounting + historyStats fleet audit
Under default adaptive retention, every flush() recomputed total history bytes by walking EVERY committed generation's delta — O(all generations) with disk re-reads past the 4096-entry delta-cache bound. On a production brain with 70,000+ accumulated generations this turned every write into a full-tail scan (60-100s writes, escalating with history growth), even though the free-RAM budget never tripped and nothing was ever reclaimed (SELF-GENERATIONS-GROWTH). historyBytes() now maintains a running total: seeded by one walk on first use, then updated incrementally at both commit paths (+bytes) and the compaction reclaim loop (−bytes), dropped on reopenAfterRestore. The adaptive retention check on every flush is O(1). Invariant regression- pinned: running total ≡ fresh walk through transact commits, single-op group commits, and compaction. New brain.historyStats() (exported HistoryStats): read-only generation count / bytes / generation+timestamp range / horizon / retention mode / effective budget — the one-call per-brain fleet audit for retention exposure.
This commit is contained in:
parent
4fcef7b8ed
commit
6207e48b51
6 changed files with 228 additions and 5 deletions
26
RELEASES.md
26
RELEASES.md
|
|
@ -10,7 +10,31 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so
|
|||
|
||||
---
|
||||
|
||||
## v8.8.1 — 2026-07-18 (the import dedup off-switch is now honest + lifecycle-safe)
|
||||
## v8.8.1 — 2026-07-18 (flush no longer walks the whole generation history + the import dedup off-switch is now honest)
|
||||
|
||||
### The flush-storm fix (production incident, reported by a long-running deployment)
|
||||
|
||||
Under the default adaptive retention, **every `flush()` re-walked the entire committed
|
||||
generation history** to compute total history bytes for the budget check — O(all
|
||||
generations) with disk re-reads past the 4,096-entry delta-cache bound. On a brain with
|
||||
70,000+ accumulated generations that turned every write into a full-tail scan (60-100s
|
||||
writes), even though the budget (free-RAM-based) never tripped and nothing was ever
|
||||
reclaimed. Fixed:
|
||||
|
||||
- `historyBytes()` now maintains a **running total**: seeded by one walk on first use,
|
||||
then updated incrementally at every commit and reclaim — the adaptive retention check
|
||||
on every flush is O(1). Invariant regression-pinned (running total ≡ fresh walk through
|
||||
both commit paths and compaction).
|
||||
- New **`brain.historyStats()`** (read-only, exported `HistoryStats`): generation count,
|
||||
total on-disk bytes, generation/timestamp range, compaction horizon, retention mode,
|
||||
and the effective adaptive budget — the one-call fleet-audit for sizing retention
|
||||
exposure per brain.
|
||||
- Interim guidance for keep-everything deployments already affected: `retention: 'all'`
|
||||
skips the adaptive accounting entirely (and is the correct policy if you never want
|
||||
history reclaimed). The accumulated files are harmless at rest; this release removes
|
||||
the per-write cost of their existence.
|
||||
|
||||
### The import dedup off-switch (lifecycle honesty)
|
||||
|
||||
The post-import background deduplication pass (a merge-DELETE writer that runs ~5 minutes
|
||||
after an import, merging entities judged duplicates by id / name / vector similarity) had
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue