feat: flush() never compacts — history maintenance moves to close() with bounded passes

flush() is durability work: it must cost what the current window's
deltas cost, never what the history backlog costs. Under adaptive
retention the byte budget derives from free memory, so bulk-load
pressure shrank the budget exactly at peak write volume and flush paid
actual reclaim inline — a production deployment measured single writes
blocked 25-191s behind reclaim-on-flush.

- flush() no longer calls autoCompactHistory(); close() is THE
  auto-compaction site (already ran there; now alone).
- Every auto pass is time-bounded (CLOSE_COMPACTION_BUDGET_MS = 5s):
  reclamation is oldest-first, so an early stop is a consistent prefix
  and the next pass resumes. Explicit compactHistory() gains an
  optional timeBudgetMs for caller-chosen maintenance windows.
- Documented trade stated where operators read: a long-lived writer
  that never closes accumulates history until its next explicit
  compactHistory() — predictable writes, explicit maintenance.

Pins: flush-never-reclaims + close-reclaims-durably (db-mvcc), bounded
pass stops-then-resumes as a consistent prefix (generationStore unit).
This commit is contained in:
David Snelling 2026-07-19 12:04:39 -07:00
parent a16567d626
commit 300d9f2a16
7 changed files with 100 additions and 31 deletions

View file

@ -1280,24 +1280,32 @@ describe('8.0 Db API — generational MVCC', () => {
await expect(reopened.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError)
})
it('Model-B retention — setRetentionBudget drives adaptive reclaim on flush; live data intact', async () => {
// Default brain → ADAPTIVE retention. A coordinator (e.g. cor's ResourceManager)
// pushes a byte budget via setRetentionBudget(); auto-compaction on flush() reclaims
// oldest history down toward it. Each update's before-image carries the full prior
// 384-dim vector (~KBs), so ~13 generations far exceed a few-KB budget.
const { brain } = await openFsBrain()
it('Model-B retention — flush() NEVER compacts (8.9.0); adaptive reclaim runs at close()', async () => {
// Default brain → ADAPTIVE retention with a driven byte budget far below
// the accumulated history (~13 generations of full-vector before-images).
// The 8.9.0 law: flush() is durability-only — it must not reclaim even
// when the budget is exceeded (reclaim-on-flush blocked production writes
// for 25-191s). Maintenance runs at close(), time-bounded.
const { brain, dir } = await openFsBrain()
const a = uid('ret-budget')
await brain.add({ id: a, type: NounType.Document, data: 'v0', vector: vec(1), metadata: { v: 0 } })
for (let v = 1; v <= 12; v++) await brain.update({ id: a, metadata: { v } })
brain.setRetentionBudget(6000) // ~6 KB — well below the accumulated history
await brain.flush() // group-commit + adaptive auto-compaction under the budget
await brain.flush()
// History was reclaimed (the horizon advanced past the oldest generations)…
expect(generationStoreOf(brain).horizon()).toBeGreaterThan(0)
await expect(brain.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError)
// …but the budget reclaims HISTORY only — the live record is untouched.
expect((await brain.get(a))?.metadata?.v).toBe(12)
// flush() paid durability only: nothing reclaimed, all history readable.
expect(generationStoreOf(brain).horizon()).toBe(0)
const probe = await brain.asOf(1) // readable proves nothing was reclaimed…
await probe.release() // …and MUST be released: a held pin would (correctly)
// protect every newer generation through the close() compaction below.
await brain.close() // ← THE auto-compaction site now
// close() reclaimed under the budget; live record intact; horizon durable.
const { brain: reopened } = await openFsBrain(dir)
expect(generationStoreOf(reopened).horizon()).toBeGreaterThan(0)
await expect(reopened.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError)
expect((await reopened.get(a))?.metadata?.v).toBe(12)
})
// ==========================================================================

View file

@ -488,6 +488,20 @@ describe('db/GenerationStore', () => {
expect(result.removedGenerations).toBe(2)
store.release(2)
})
it('timeBudgetMs bounds a pass; the next pass resumes the same prefix', async () => {
await manyGens(4)
// A spent budget (0ms) stops before reclaiming anything — an early stop
// is a consistent prefix, never a partial generation.
const bounded = await store.compact({ timeBudgetMs: 0 })
expect(bounded.removedGenerations).toBe(0)
expect(bounded.horizon).toBe(0)
// The next (unbounded) pass picks up exactly where the bounded one
// stopped and completes the same work.
const resumed = await store.compact()
expect(resumed.removedGenerations).toBe(4)
expect(resumed.horizon).toBe(4)
})
})
// ==========================================================================