fix: O(1) adaptive retention accounting + historyStats fleet audit

Under default adaptive retention, every flush() recomputed total history
bytes by walking EVERY committed generation's delta — O(all generations)
with disk re-reads past the 4096-entry delta-cache bound. On a production
brain with 70,000+ accumulated generations this turned every write into a
full-tail scan (60-100s writes, escalating with history growth), even
though the free-RAM budget never tripped and nothing was ever reclaimed
(SELF-GENERATIONS-GROWTH).

historyBytes() now maintains a running total: seeded by one walk on first
use, then updated incrementally at both commit paths (+bytes) and the
compaction reclaim loop (−bytes), dropped on reopenAfterRestore. The
adaptive retention check on every flush is O(1). Invariant regression-
pinned: running total ≡ fresh walk through transact commits, single-op
group commits, and compaction.

New brain.historyStats() (exported HistoryStats): read-only generation
count / bytes / generation+timestamp range / horizon / retention mode /
effective budget — the one-call per-brain fleet audit for retention
exposure.
This commit is contained in:
David Snelling 2026-07-18 10:51:37 -07:00
parent 4fcef7b8ed
commit 6207e48b51
6 changed files with 228 additions and 5 deletions

View file

@ -489,4 +489,80 @@ describe('db/GenerationStore', () => {
store.release(2)
})
})
// ==========================================================================
describe('history-bytes running total (the O(1) retention check)', () => {
/** A fresh walk with the cache dropped — ground truth for the invariant. */
async function groundTruthBytes(): Promise<number> {
;(store as any).historyBytesTotal = null
return store.historyBytes()
}
it('is seeded once, then maintained through commits WITHOUT re-walks', async () => {
await commitWrite(ID_A, 1)
await commitWrite(ID_A, 2)
const seeded = await store.historyBytes()
expect(seeded).toBe(await groundTruthBytes())
// From here every read must come from the running total, not a walk:
// getDelta re-reads are the walk's cost — commits must not trigger any.
const getDeltaSpy = vi.spyOn(store as any, 'getDelta')
await commitWrite(ID_B, 1)
const afterCommit = await store.historyBytes()
expect(getDeltaSpy).not.toHaveBeenCalled()
getDeltaSpy.mockRestore()
expect(afterCommit).toBe(await groundTruthBytes())
})
it('stays exact through single-op group commits and compaction', async () => {
await commitWrite(ID_A, 1)
await store.historyBytes() // seed
// Single-op path: buffered generations flushed as one group commit.
await store.commitSingleOp({
touched: { nouns: [ID_B] },
execute: async () => {
await storage.saveNounMetadata(ID_B, metadataFixture(1))
}
})
await store.flushPendingSingleOps()
expect(await store.historyBytes()).toBe(await groundTruthBytes())
await store.historyBytes() // re-seed after ground-truth reset
await store.compact({ maxGenerations: 1 })
expect(await store.historyBytes()).toBe(await groundTruthBytes())
})
it('historyStats reports counts, bytes, range, and horizon read-only', async () => {
await commitWrite(ID_A, 1)
await commitWrite(ID_B, 1)
const stats = await store.historyStats()
expect(stats.generations).toBe(2)
expect(stats.bytes).toBe(await store.historyBytes())
expect(stats.oldestGeneration).toBe(1)
expect(stats.newestGeneration).toBe(2)
expect(stats.oldestTimestamp).toBeLessThanOrEqual(stats.newestTimestamp!)
expect(stats.horizon).toBe(0)
// Read-only: nothing was reclaimed by asking.
expect(store.committedGeneration()).toBe(2)
await store.compact({ maxGenerations: 1 })
const after = await store.historyStats()
expect(after.generations).toBe(1)
expect(after.oldestGeneration).toBe(2)
expect(after.horizon).toBe(1)
})
it('empty history reports null range and zero bytes', async () => {
const stats = await store.historyStats()
expect(stats).toMatchObject({
generations: 0,
bytes: 0,
oldestGeneration: null,
newestGeneration: null,
oldestTimestamp: null,
newestTimestamp: null,
horizon: 0
})
})
})
})