feat: warm contract (warm/warmOnOpen/provider warm hook), configurable transact budget floor, backend-neutral vector index op names
Three pieces addressing the cold-restart-write incident where a production deployment's first writes after every restart (33-35s each on a cold page cache) blew the op-count-scaled transact budget mid-batch: every write is itself a multi-op transaction, so one cold operation consumed the whole budget, the gate before the next operation tripped, and the write rolled back atomically - refused, retried, and refused again until the page cache warmed passively. - The budget's start-gating contract is now explicit and pinned: it gates STARTING the next operation, never rolling back completed work for elapsed time (the shipped schedule since 8.7.0, now stated in contract JSDoc, guarded by code for operation 0, and enforced by regression tests). The 30s floor is configurable via transactionBudgetFloorMs for stores whose cold operations legitimately run long. - New brain.warm() eagerly loads the vector index, metadata index, and graph adjacency so first operations after a cold restart run at steady-state cost. Returns a WarmReport with an honest per-surface outcome (warmed / probed / unavailable) - never reports a probe as a warm. warmOnOpen: true runs it during init(). New optional provider hook warm() on the vector and graph plugin contracts. - Vector-index transaction op classes renamed from the backend-specific AddToHNSWOperation / RemoveFromHNSWOperation to backend-neutral AddToVectorIndexOperation / RemoveFromVectorIndexOperation, stamping the active backend into the emitted op-name string (AddToVectorIndex(js-hnsw) vs a native provider's own identity) so journals never misdirect an operator toward an index that isn't running.
This commit is contained in:
parent
d08679fc84
commit
55b867c998
13 changed files with 1099 additions and 57 deletions
56
RELEASES.md
56
RELEASES.md
|
|
@ -31,6 +31,62 @@ is sometimes cited as a 7.x removal — those methods never existed on 7.x; the
|
|||
|
||||
---
|
||||
|
||||
## Unreleased (the warm contract: cold-restart writes stop paying demand-load latency)
|
||||
|
||||
From a production deployment's cold-restart incident: the FIRST writes after every
|
||||
restart on a large brain measured 33–35s each (page cache cold) against the transact
|
||||
apply budget — Brainy's own op-count-scaled budget, `max(30s, opCount × 2s)`, e.g.
|
||||
32,000ms for a 16-op batch. There is no external deadline in this story, and no
|
||||
post-completion veto either: every write is itself a multi-operation transaction (a
|
||||
single `add()` applies several operations — canonical writes, the vector-index insert,
|
||||
the metadata-index update), so ONE cold operation that legitimately runs ~33s consumes
|
||||
the whole budget, the gate before the NEXT operation trips, and the write rolls back
|
||||
atomically (zero loss, by design) — refused, retried, refused again, until the page
|
||||
cache warms passively (~30 minutes). The cure is not weaker atomicity; it is the two
|
||||
new knobs below — a budget floor sized for cold stores, and a warm contract that pays
|
||||
demand-load cost OFF the transaction path.
|
||||
|
||||
- **The budget's start-gating contract is now explicit, documented, and pinned by
|
||||
regression tests.** The budget gates STARTING the next operation — completed work is
|
||||
never rolled back for elapsed time — and a transaction's first operation now
|
||||
unconditionally starts by code, not merely because elapsed time happens to be ~0 when
|
||||
it is checked. This has been the shipped schedule since 8.7.0 (no behavior change for
|
||||
existing integrations); it is now stated in `Transaction.execute()`'s contract JSDoc
|
||||
and enforced by tests so it cannot silently regress. Mid-batch atomicity is unchanged:
|
||||
a trip before operation `i+1` still rolls back `0..i` and throws a retryable
|
||||
`TransactionTimeoutError`.
|
||||
- **The budget's 30s floor is now configurable**: `new Brainy({ transactionBudgetFloorMs })`
|
||||
raises (or lowers) the floor of `max(transactionBudgetFloorMs, opCount × 2000)` for every
|
||||
internal transact batch. Useful for a store whose cold writes legitimately run past 30s
|
||||
per operation, so a bulk batch gets a proportionally larger runway instead of tripping
|
||||
mid-batch on cold-cache latency.
|
||||
- **New: `brain.warm()`** — eagerly loads/faults-in the vector index, metadata index, and
|
||||
graph adjacency so the first real operation after a cold restart runs at steady-state
|
||||
cost instead of paying demand-load latency on the critical path. Returns a `WarmReport`
|
||||
with one honest outcome per surface — never conflate the first two:
|
||||
- `'warmed'` — the surface's own provider `warm()` hook ran, or a full-hydration seam
|
||||
loaded every shard/field/segment from storage. Steady-state cost is paid.
|
||||
- `'probed'` — no `warm()` hook was available, so a best-effort read (one `search()` call
|
||||
for the vector index) faulted in *some* backing storage as a side effect — real work,
|
||||
but never reported as `'warmed'`.
|
||||
- `'unavailable'` — nothing ran (no hook, no hydration seam, or nothing to probe).
|
||||
- **New config: `warmOnOpen: true`** makes `init()` await `brain.warm()` before it resolves
|
||||
— a deliberate blocking trade-off: startup takes longer, the first request doesn't.
|
||||
Default `false` (unchanged lazy behavior).
|
||||
- **New optional provider hook: `warm?(): Promise<void>`** on the vector and graph
|
||||
acceleration provider contracts (`src/plugin.ts`) — a native provider can implement it to
|
||||
eagerly pretouch its own backing storage (e.g. mmap pretouch); absence means brainy falls
|
||||
back to the probe/hydration behavior above.
|
||||
- **Transaction op-name strings changed in journals/timings**: the vector-index
|
||||
transaction operations were renamed from `AddToHNSW`/`RemoveFromHNSW` to backend-neutral
|
||||
`AddToVectorIndex(...)`/`RemoveFromVectorIndex(...)` — the old names hard-coded an
|
||||
algorithm that may not be the one actually running (a non-HNSW native vector provider
|
||||
emitting `"RemoveFromHNSW"` has sent an operator hunting an index that doesn't exist).
|
||||
The parenthesized suffix names the ACTIVE backend: `js-hnsw` for the built-in engine, or
|
||||
the native provider's own identity when it self-identifies. **If you parse these op-name
|
||||
strings** (log processors, journal tooling), update your matcher from
|
||||
`AddToHNSW`/`RemoveFromHNSW` to `AddToVectorIndex(`/`RemoveFromVectorIndex(`.
|
||||
|
||||
## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close())
|
||||
|
||||
The write path stops paying maintenance costs — the last structural piece of the
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue