feat(8.0): Model-B per-write generation-stamping + adaptive retention knob
Every write — transact() AND single-op add/update/remove/relate — is now its
own immutable generation (Model-B), so a now() pin always freezes and
asOf/since/diff/history/transactionLog reflect single-ops exactly like
transacts. Closes the Model-A hole where pins did not freeze against single-op
writes.
Generation-stamping:
- GenerationStore.commitSingleOp: a one-operation commitTransaction with
deferred durability. Wired into add/update/remove/relate/updateRelation/
unrelate + removeMany (the *Many and VFS paths delegate to these).
- Async group-commit (flushPendingSingleOps): the live write is acknowledged
immediately; its before-image is buffered in an in-memory pending tier that
resolveAt/chains/changedBetween/tx-log read like on-disk generations, so the
synchronous now() freezes with no forced flush. One fsync per window
(triggers: size / 50ms timer / flush / close / transact / compactHistory).
- Crash recovery is drop-without-restore for group-commit generations (marked
groupCommit:true): a crash mid-flush discards the partial generation and
never restores its before-images, which would otherwise revert the
already-acknowledged live write.
- Init-time infrastructure (the VFS root) is the un-versioned generation-0
baseline: a fresh brain reports generation()===0 and an empty
transactionLog(); the first user write is generation 1.
- Historical find()/related() overlay bound is the full reserved watermark
(generation()), so un-flushed single-op writes are overlaid too.
Retention knob:
- config `history` -> `retention`: 'all' | 'adaptive' |
{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset ->
adaptive (disk/RAM pressure, zero-config). CompactHistoryOptions floors ->
caps (retainGenerations->maxGenerations, retainMs->maxAge, +maxBytes):
reclaim oldest-unpinned while ANY cap is exceeded; pins always exempt.
- brain.setRetentionBudget(bytes) drives the adaptive byte budget at runtime
(a coordinator's fair-share input). Per-generation bytes recorded in each
delta enable historyBytes() introspection without a storage size API.
Tests: per-write generation resolution, pin freeze vs add/update/remove,
drop-without-restore corruption-trap (fault injector), clean-reopen replay,
maxBytes/maxAge/no-cap reclamation, retention-then-reopen. 107 db/generation/
temporal tests green, tsc clean. Docs (ADR-001, consistency-model, snapshots
guide, api reference, RELEASES) updated to per-write granularity + retention.
This commit is contained in:
parent
afac7f9662
commit
5c3bb2c864
15 changed files with 1207 additions and 218 deletions
|
|
@ -149,12 +149,22 @@ overlay entities carry no embeddings (`with()` never invokes the embedder),
|
|||
so a "full" index query over an overlay would silently exclude the
|
||||
overlay's own entities. Commit with `transact()` to get the full surface.
|
||||
|
||||
**History granularity.** Generation *records* are written per `transact()`
|
||||
batch only. Single-operation writes advance the counter (so watermarks and
|
||||
CAS stay sound) but do not stage before-images: they remain visible through
|
||||
earlier pins and are not reported by `db.since()`. Code that needs pinned
|
||||
isolation across its writes uses `transact()` — that is the documented 8.0
|
||||
contract, stated rather than papered over.
|
||||
**History granularity (Model-B).** EVERY write is its own immutable generation
|
||||
— `transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
|
||||
Single-ops stage a before-image and are reported by `db.since()`/`asOf()`/
|
||||
`diff()`/`history()` exactly like transacts; a pin always freezes against later
|
||||
writes. `transact()` groups several operations into ONE atomic generation.
|
||||
|
||||
Single-op history durability is **async group-commit**: the live write hits
|
||||
canonical storage immediately (acknowledged), while its before-image is buffered
|
||||
and persisted to disk in one batched fsync on a size/timer trigger (or forced by
|
||||
`flush()`/`close()`/`transact()`/`compactHistory()`). The buffer participates in
|
||||
point-in-time resolution exactly like on-disk generations, so the synchronous
|
||||
`now()` freezes with no forced flush. A hard crash before the flush loses only
|
||||
the buffered *history* of the last window — never live data — and a crash
|
||||
*mid-flush* is recovered by **drop-without-restore** (the partial generation's
|
||||
before-images are discarded, never replayed, because the live write was already
|
||||
acknowledged; restoring them would silently revert it).
|
||||
|
||||
### Pinning, retention, compaction
|
||||
|
||||
|
|
@ -162,11 +172,17 @@ contract, stated rather than papered over.
|
|||
`pin(generation)` on every registered `VersionedIndexProvider`, whose
|
||||
explicit pin lifetime overrides any time-based snapshot retention the
|
||||
provider has).
|
||||
- `compactHistory({ retainGenerations?, retainMs? })` reclaims record-sets.
|
||||
A record-set `N` is reclaimed only when `N` is at or below **every** live
|
||||
- The constructor **`retention`** knob governs auto-compaction (on `flush()`/
|
||||
`close()`): unset → ADAPTIVE (disk/RAM-pressure byte budget, zero-config;
|
||||
driven by a coordinator's `budgetBytes` or a local `os.freemem` probe) ·
|
||||
`'all'` → unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit
|
||||
CAPS. `compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
|
||||
manually on the same caps — the oldest unpinned record-sets are reclaimed
|
||||
while ANY supplied cap is exceeded.
|
||||
- A record-set `N` is reclaimed only when `N` is at or below **every** live
|
||||
pin — deleting `N` can only break readers pinned *below* `N`, because
|
||||
resolution reads before-images from generations strictly greater than the
|
||||
pin. With both retention options supplied, both must allow the reclaim.
|
||||
pin. Live pins are ALWAYS exempt, in every retention mode.
|
||||
- The manifest records the **horizon** (highest reclaimed generation).
|
||||
Generations below the horizon are unreachable; `asOf()` on them throws
|
||||
`GenerationCompactedError`. The horizon itself stays reachable, resolved
|
||||
|
|
|
|||
|
|
@ -901,16 +901,18 @@ that generation), once per `Db`, freed on `release()`.
|
|||
|
||||
**Throws:** `GenerationCompactedError` when the generation's records were reclaimed by `compactHistory()`.
|
||||
|
||||
**History granularity:** only `transact()` batches produce historical
|
||||
records; single-operation writes advance the clock but stay visible through
|
||||
earlier pins. See the [consistency model](../concepts/consistency-model.md).
|
||||
**History granularity:** every write is its own immutable generation —
|
||||
`transact()` batches AND single-operation writes — so a pin always freezes and
|
||||
every write is addressable via `asOf()`. See the
|
||||
[consistency model](../concepts/consistency-model.md).
|
||||
|
||||
---
|
||||
|
||||
### `transactionLog(options?)` → `Promise<TxLogEntry[]>`
|
||||
|
||||
Read the reified transaction log — one entry per committed `transact()`
|
||||
batch, newest first: `{ generation, timestamp, meta? }`.
|
||||
Read the reified transaction log — one entry per committed generation (every
|
||||
`transact()` AND single-op write), newest first: `{ generation, timestamp,
|
||||
meta? }`. Single-op generations carry no `meta` (it is a `transact()`-only field).
|
||||
|
||||
```typescript
|
||||
const [latest] = await brain.transactionLog({ limit: 1 })
|
||||
|
|
@ -921,13 +923,15 @@ latest.meta // { author: 'order-service', requestId: 'req-9f2' }
|
|||
|
||||
### `compactHistory(options?)` → `Promise<CompactHistoryResult>`
|
||||
|
||||
Reclaim historical record-sets that no retention rule and no live `Db` pin
|
||||
protects. Pinned reads stay correct across compaction, always.
|
||||
Reclaim historical record-sets that no retention cap and no live `Db` pin
|
||||
protects. Pinned reads stay correct across compaction, always. (Auto-compaction
|
||||
on `flush()`/`close()` is governed by the constructor `retention` knob — unset →
|
||||
adaptive, `'all'` → unbounded, `{ … }` → explicit caps.)
|
||||
|
||||
```typescript
|
||||
await brain.compactHistory({
|
||||
retainGenerations: 100, // keep the 100 most recent commits
|
||||
retainMs: 7 * 24 * 60 * 60 * 1000 // and everything from the last 7 days
|
||||
maxGenerations: 100, // keep at most the 100 most recent generations
|
||||
maxAge: 7 * 24 * 60 * 60 * 1000 // and only those from the last 7 days
|
||||
})
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -212,10 +212,12 @@ Historical records cost disk space, so retention is explicit:
|
|||
- Every live `Db` holds a refcounted **pin**; a record-set is never
|
||||
reclaimed while any pin could need it — pinned reads stay correct across
|
||||
compaction, always.
|
||||
- `brain.compactHistory({ retainGenerations?, retainMs? })` reclaims
|
||||
everything no retention rule and no pin protects, and records the
|
||||
**horizon** — `asOf()` below it throws `GenerationCompactedError`,
|
||||
explicitly, never partial data.
|
||||
- The **`retention`** knob governs auto-compaction (on every `flush()`/
|
||||
`close()`): unset → ADAPTIVE (disk/RAM-pressure, zero-config) · `'all'` →
|
||||
unbounded · `{ maxGenerations?, maxAge?, maxBytes? }` → explicit caps.
|
||||
`brain.compactHistory({ maxGenerations?, maxAge?, maxBytes? })` reclaims
|
||||
manually on the same caps, and records the **horizon** — `asOf()` below it
|
||||
throws `GenerationCompactedError`, explicitly, never partial data.
|
||||
- To keep a state readable forever, `persist()` it first: snapshots are
|
||||
self-contained and unaffected by compaction of the source store.
|
||||
|
||||
|
|
@ -357,9 +359,13 @@ Stated plainly, so nothing surprises you in production:
|
|||
([multi-process model](./multi-process.md)). Transactions are atomic
|
||||
within one writer process — there is no distributed or cross-process
|
||||
transaction coordination.
|
||||
- **History granularity.** Only `transact()` batches produce historical
|
||||
records; single-operation writes between commits stay visible through
|
||||
earlier pins (see above).
|
||||
- **History granularity.** Every write is its own immutable generation —
|
||||
`transact()` batches AND single-operation `add`/`update`/`remove`/`relate`.
|
||||
A pin always freezes against later writes, and every write is addressable
|
||||
via `asOf()`. (`transact()` groups several operations into ONE atomic
|
||||
generation; durability of single-op history is batched via async
|
||||
group-commit — a hard crash can lose only the last un-flushed window's
|
||||
*history*, never live data.)
|
||||
- **Compacted history is gone.** `asOf()` below the compaction horizon
|
||||
fails explicitly; persist what you must keep.
|
||||
- **Counter persistence is coalesced for single-operation writes.** Durable
|
||||
|
|
|
|||
|
|
@ -135,10 +135,12 @@ await after.release()
|
|||
|
||||
Three things to remember:
|
||||
|
||||
- History granularity is `transact()` commits — single-operation writes
|
||||
advance the clock but do not produce historical records (see the
|
||||
[consistency model](../concepts/consistency-model.md)). Use `transact()`
|
||||
for writes you want to travel back through.
|
||||
- History granularity is per-write: EVERY write — `transact()` AND a
|
||||
single-operation `add`/`update`/`remove`/`relate` — is its own immutable
|
||||
generation, so a pin always freezes against later writes and every write is
|
||||
individually addressable via `asOf()` (see the
|
||||
[consistency model](../concepts/consistency-model.md)). Use `transact()` when
|
||||
you want several operations to share ONE atomic generation.
|
||||
- The first index-accelerated query (semantic search, traversal, cursors,
|
||||
aggregation) at a historical generation builds an in-memory index
|
||||
materialization — O(n at that generation), once per `Db`, freed on
|
||||
|
|
@ -336,20 +338,33 @@ For per-entity write coordination (rather than whole-store history), the
|
|||
|
||||
## Keeping history bounded
|
||||
|
||||
Historical records cost disk space. Reclaim what no live pin protects:
|
||||
Under Model-B every write is a generation, so history can grow quickly —
|
||||
Brainy auto-compacts on every `flush()`/`close()` under the **`retention`**
|
||||
knob (configured on the constructor):
|
||||
|
||||
```typescript
|
||||
await brain.compactHistory({
|
||||
retainGenerations: 100, // keep the 100 most recent commits
|
||||
retainMs: 7 * 24 * 60 * 60 * 1000 // and everything from the last 7 days
|
||||
})
|
||||
// Zero-config: ADAPTIVE — keep as much history as free disk/RAM allows,
|
||||
// reclaiming oldest-first under pressure. (This is the default.)
|
||||
new Brainy({ /* retention unset */ })
|
||||
|
||||
// Unbounded — never reclaim history (opt in explicitly):
|
||||
new Brainy({ retention: 'all' })
|
||||
|
||||
// Explicit CAPS — reclaim oldest-unpinned generations while ANY cap is exceeded:
|
||||
new Brainy({ retention: { maxGenerations: 1000, maxAge: 7 * 86_400_000, maxBytes: 512 * 1024 ** 2 } })
|
||||
```
|
||||
|
||||
Reclaim manually at any time (the same caps):
|
||||
|
||||
```typescript
|
||||
await brain.compactHistory({ maxGenerations: 100, maxAge: 7 * 24 * 60 * 60 * 1000 })
|
||||
```
|
||||
|
||||
Compaction never breaks a pinned read — record-sets are reclaimed only when
|
||||
no live `Db` could need them. Release views you are done with (including the
|
||||
ones `transact()` returns), and `persist()` any generation you want to keep
|
||||
beyond the retention window: snapshots are self-contained and unaffected by
|
||||
compaction.
|
||||
no live `Db` could need them (live pins are ALWAYS exempt). Release views you
|
||||
are done with (including the ones `transact()` returns), and `persist()` any
|
||||
generation you want to keep beyond the retention window: snapshots are
|
||||
self-contained and unaffected by compaction.
|
||||
|
||||
## From branches to values
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue