feat: flush() never compacts — history maintenance moves to close() with bounded passes
flush() is durability work: it must cost what the current window's deltas cost, never what the history backlog costs. Under adaptive retention the byte budget derives from free memory, so bulk-load pressure shrank the budget exactly at peak write volume and flush paid actual reclaim inline — a production deployment measured single writes blocked 25-191s behind reclaim-on-flush. - flush() no longer calls autoCompactHistory(); close() is THE auto-compaction site (already ran there; now alone). - Every auto pass is time-bounded (CLOSE_COMPACTION_BUDGET_MS = 5s): reclamation is oldest-first, so an early stop is a consistent prefix and the next pass resumes. Explicit compactHistory() gains an optional timeBudgetMs for caller-chosen maintenance windows. - Documented trade stated where operators read: a long-lived writer that never closes accumulates history until its next explicit compactHistory() — predictable writes, explicit maintenance. Pins: flush-never-reclaims + close-reclaims-durably (db-mvcc), bounded pass stops-then-resumes as a consistent prefix (generationStore unit).
This commit is contained in:
parent
a16567d626
commit
300d9f2a16
7 changed files with 100 additions and 31 deletions
|
|
@ -1280,24 +1280,32 @@ describe('8.0 Db API — generational MVCC', () => {
|
|||
await expect(reopened.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError)
|
||||
})
|
||||
|
||||
it('Model-B retention — setRetentionBudget drives adaptive reclaim on flush; live data intact', async () => {
|
||||
// Default brain → ADAPTIVE retention. A coordinator (e.g. cor's ResourceManager)
|
||||
// pushes a byte budget via setRetentionBudget(); auto-compaction on flush() reclaims
|
||||
// oldest history down toward it. Each update's before-image carries the full prior
|
||||
// 384-dim vector (~KBs), so ~13 generations far exceed a few-KB budget.
|
||||
const { brain } = await openFsBrain()
|
||||
it('Model-B retention — flush() NEVER compacts (8.9.0); adaptive reclaim runs at close()', async () => {
|
||||
// Default brain → ADAPTIVE retention with a driven byte budget far below
|
||||
// the accumulated history (~13 generations of full-vector before-images).
|
||||
// The 8.9.0 law: flush() is durability-only — it must not reclaim even
|
||||
// when the budget is exceeded (reclaim-on-flush blocked production writes
|
||||
// for 25-191s). Maintenance runs at close(), time-bounded.
|
||||
const { brain, dir } = await openFsBrain()
|
||||
const a = uid('ret-budget')
|
||||
await brain.add({ id: a, type: NounType.Document, data: 'v0', vector: vec(1), metadata: { v: 0 } })
|
||||
for (let v = 1; v <= 12; v++) await brain.update({ id: a, metadata: { v } })
|
||||
|
||||
brain.setRetentionBudget(6000) // ~6 KB — well below the accumulated history
|
||||
await brain.flush() // group-commit + adaptive auto-compaction under the budget
|
||||
await brain.flush()
|
||||
|
||||
// History was reclaimed (the horizon advanced past the oldest generations)…
|
||||
expect(generationStoreOf(brain).horizon()).toBeGreaterThan(0)
|
||||
await expect(brain.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError)
|
||||
// …but the budget reclaims HISTORY only — the live record is untouched.
|
||||
expect((await brain.get(a))?.metadata?.v).toBe(12)
|
||||
// flush() paid durability only: nothing reclaimed, all history readable.
|
||||
expect(generationStoreOf(brain).horizon()).toBe(0)
|
||||
const probe = await brain.asOf(1) // readable proves nothing was reclaimed…
|
||||
await probe.release() // …and MUST be released: a held pin would (correctly)
|
||||
// protect every newer generation through the close() compaction below.
|
||||
await brain.close() // ← THE auto-compaction site now
|
||||
|
||||
// close() reclaimed under the budget; live record intact; horizon durable.
|
||||
const { brain: reopened } = await openFsBrain(dir)
|
||||
expect(generationStoreOf(reopened).horizon()).toBeGreaterThan(0)
|
||||
await expect(reopened.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError)
|
||||
expect((await reopened.get(a))?.metadata?.v).toBe(12)
|
||||
})
|
||||
|
||||
// ==========================================================================
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue