fix: O(1) adaptive retention accounting + historyStats fleet audit
Under default adaptive retention, every flush() recomputed total history bytes by walking EVERY committed generation's delta — O(all generations) with disk re-reads past the 4096-entry delta-cache bound. On a production brain with 70,000+ accumulated generations this turned every write into a full-tail scan (60-100s writes, escalating with history growth), even though the free-RAM budget never tripped and nothing was ever reclaimed (SELF-GENERATIONS-GROWTH). historyBytes() now maintains a running total: seeded by one walk on first use, then updated incrementally at both commit paths (+bytes) and the compaction reclaim loop (−bytes), dropped on reopenAfterRestore. The adaptive retention check on every flush is O(1). Invariant regression- pinned: running total ≡ fresh walk through transact commits, single-op group commits, and compaction. New brain.historyStats() (exported HistoryStats): read-only generation count / bytes / generation+timestamp range / horizon / retention mode / effective budget — the one-call per-brain fleet audit for retention exposure.
This commit is contained in:
parent
4fcef7b8ed
commit
6207e48b51
6 changed files with 228 additions and 5 deletions
|
|
@ -489,4 +489,80 @@ describe('db/GenerationStore', () => {
|
|||
store.release(2)
|
||||
})
|
||||
})
|
||||
|
||||
// ==========================================================================
|
||||
describe('history-bytes running total (the O(1) retention check)', () => {
|
||||
/** A fresh walk with the cache dropped — ground truth for the invariant. */
|
||||
async function groundTruthBytes(): Promise<number> {
|
||||
;(store as any).historyBytesTotal = null
|
||||
return store.historyBytes()
|
||||
}
|
||||
|
||||
it('is seeded once, then maintained through commits WITHOUT re-walks', async () => {
|
||||
await commitWrite(ID_A, 1)
|
||||
await commitWrite(ID_A, 2)
|
||||
const seeded = await store.historyBytes()
|
||||
expect(seeded).toBe(await groundTruthBytes())
|
||||
|
||||
// From here every read must come from the running total, not a walk:
|
||||
// getDelta re-reads are the walk's cost — commits must not trigger any.
|
||||
const getDeltaSpy = vi.spyOn(store as any, 'getDelta')
|
||||
await commitWrite(ID_B, 1)
|
||||
const afterCommit = await store.historyBytes()
|
||||
expect(getDeltaSpy).not.toHaveBeenCalled()
|
||||
getDeltaSpy.mockRestore()
|
||||
expect(afterCommit).toBe(await groundTruthBytes())
|
||||
})
|
||||
|
||||
it('stays exact through single-op group commits and compaction', async () => {
|
||||
await commitWrite(ID_A, 1)
|
||||
await store.historyBytes() // seed
|
||||
// Single-op path: buffered generations flushed as one group commit.
|
||||
await store.commitSingleOp({
|
||||
touched: { nouns: [ID_B] },
|
||||
execute: async () => {
|
||||
await storage.saveNounMetadata(ID_B, metadataFixture(1))
|
||||
}
|
||||
})
|
||||
await store.flushPendingSingleOps()
|
||||
expect(await store.historyBytes()).toBe(await groundTruthBytes())
|
||||
|
||||
await store.historyBytes() // re-seed after ground-truth reset
|
||||
await store.compact({ maxGenerations: 1 })
|
||||
expect(await store.historyBytes()).toBe(await groundTruthBytes())
|
||||
})
|
||||
|
||||
it('historyStats reports counts, bytes, range, and horizon read-only', async () => {
|
||||
await commitWrite(ID_A, 1)
|
||||
await commitWrite(ID_B, 1)
|
||||
const stats = await store.historyStats()
|
||||
expect(stats.generations).toBe(2)
|
||||
expect(stats.bytes).toBe(await store.historyBytes())
|
||||
expect(stats.oldestGeneration).toBe(1)
|
||||
expect(stats.newestGeneration).toBe(2)
|
||||
expect(stats.oldestTimestamp).toBeLessThanOrEqual(stats.newestTimestamp!)
|
||||
expect(stats.horizon).toBe(0)
|
||||
// Read-only: nothing was reclaimed by asking.
|
||||
expect(store.committedGeneration()).toBe(2)
|
||||
|
||||
await store.compact({ maxGenerations: 1 })
|
||||
const after = await store.historyStats()
|
||||
expect(after.generations).toBe(1)
|
||||
expect(after.oldestGeneration).toBe(2)
|
||||
expect(after.horizon).toBe(1)
|
||||
})
|
||||
|
||||
it('empty history reports null range and zero bytes', async () => {
|
||||
const stats = await store.historyStats()
|
||||
expect(stats).toMatchObject({
|
||||
generations: 0,
|
||||
bytes: 0,
|
||||
oldestGeneration: null,
|
||||
newestGeneration: null,
|
||||
oldestTimestamp: null,
|
||||
newestTimestamp: null,
|
||||
horizon: 0
|
||||
})
|
||||
})
|
||||
})
|
||||
})
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue