feat(8.0): Model-B per-write generation-stamping + adaptive retention knob
Every write — transact() AND single-op add/update/remove/relate — is now its
own immutable generation (Model-B), so a now() pin always freezes and
asOf/since/diff/history/transactionLog reflect single-ops exactly like
transacts. Closes the Model-A hole where pins did not freeze against single-op
writes.
Generation-stamping:
- GenerationStore.commitSingleOp: a one-operation commitTransaction with
deferred durability. Wired into add/update/remove/relate/updateRelation/
unrelate + removeMany (the *Many and VFS paths delegate to these).
- Async group-commit (flushPendingSingleOps): the live write is acknowledged
immediately; its before-image is buffered in an in-memory pending tier that
resolveAt/chains/changedBetween/tx-log read like on-disk generations, so the
synchronous now() freezes with no forced flush. One fsync per window
(triggers: size / 50ms timer / flush / close / transact / compactHistory).
- Crash recovery is drop-without-restore for group-commit generations (marked
groupCommit:true): a crash mid-flush discards the partial generation and
never restores its before-images, which would otherwise revert the
already-acknowledged live write.
- Init-time infrastructure (the VFS root) is the un-versioned generation-0
baseline: a fresh brain reports generation()===0 and an empty
transactionLog(); the first user write is generation 1.
- Historical find()/related() overlay bound is the full reserved watermark
(generation()), so un-flushed single-op writes are overlaid too.
Retention knob:
- config `history` -> `retention`: 'all' | 'adaptive' |
{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset ->
adaptive (disk/RAM pressure, zero-config). CompactHistoryOptions floors ->
caps (retainGenerations->maxGenerations, retainMs->maxAge, +maxBytes):
reclaim oldest-unpinned while ANY cap is exceeded; pins always exempt.
- brain.setRetentionBudget(bytes) drives the adaptive byte budget at runtime
(a coordinator's fair-share input). Per-generation bytes recorded in each
delta enable historyBytes() introspection without a storage size API.
Tests: per-write generation resolution, pin freeze vs add/update/remove,
drop-without-restore corruption-trap (fault injector), clean-reopen replay,
maxBytes/maxAge/no-cap reclamation, retention-then-reopen. 107 db/generation/
temporal tests green, tsc clean. Docs (ADR-001, consistency-model, snapshots
guide, api reference, RELEASES) updated to per-write granularity + retention.
This commit is contained in:
parent
afac7f9662
commit
5c3bb2c864
15 changed files with 1207 additions and 218 deletions
313
src/brainy.ts
313
src/brainy.ts
|
|
@ -121,6 +121,7 @@ import { AggregateMaterializer } from './aggregation/materializer.js'
|
|||
import type { AggregateDefinition, AggregateQueryParams, AggregateResult } from './types/brainy.types.js'
|
||||
import { resolveJsHnswConfig, DEFAULT_RECALL } from './utils/recallPreset.js'
|
||||
import * as fs from 'node:fs'
|
||||
import * as os from 'node:os'
|
||||
import { Db, type DbHost, type HistoricalQueryHandle } from './db/db.js'
|
||||
import {
|
||||
importGraph,
|
||||
|
|
@ -148,7 +149,7 @@ import type {
|
|||
} from './db/types.js'
|
||||
import { stableDeepEqual } from './db/stableEqual.js'
|
||||
import type { VersionedIndexProvider } from './plugin.js'
|
||||
import type { Operation } from './transaction/types.js'
|
||||
import type { Operation, TransactionFunction } from './transaction/types.js'
|
||||
|
||||
/**
|
||||
* Stopwords for semantic highlighting
|
||||
|
|
@ -208,7 +209,7 @@ type ResolvedBrainyConfig = Required<
|
|||
| 'vector'
|
||||
| 'plugins'
|
||||
| 'integrations'
|
||||
| 'history'
|
||||
| 'retention'
|
||||
| 'eagerEmbeddings'
|
||||
>
|
||||
> &
|
||||
|
|
@ -219,7 +220,7 @@ type ResolvedBrainyConfig = Required<
|
|||
| 'vector'
|
||||
| 'plugins'
|
||||
| 'integrations'
|
||||
| 'history'
|
||||
| 'retention'
|
||||
| 'eagerEmbeddings'
|
||||
>
|
||||
|
||||
|
|
@ -365,6 +366,23 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
private _tripleIntelligence?: TripleIntelligenceSystem
|
||||
private _vfs?: VirtualFileSystem
|
||||
private _vfsInitialized = false // Track VFS init completion separately
|
||||
/**
|
||||
* 8.0 MVCC: single-op writes create generations (Model-B) only AFTER init
|
||||
* completes. Init-time infrastructure writes (the VFS root directory, etc.)
|
||||
* form the generation-0 baseline of a freshly-materialized brain and are NOT
|
||||
* versioned — so a brand-new brain reports `generation() === 0` / empty
|
||||
* `transactionLog()`, and the first USER write is generation 1. Enabled at
|
||||
* the tail of `init()`; see {@link persistSingleOp}.
|
||||
*/
|
||||
private _generationStampingActive = false
|
||||
/**
|
||||
* 8.0 MVCC: the coordinator-driven adaptive history byte budget for this
|
||||
* brain (set via {@link setRetentionBudget}; e.g. cor's `ResourceManager`
|
||||
* fair-shares one box-wide budget across co-located instances). Overrides the
|
||||
* local `os.freemem` probe while `retention` is adaptive. `undefined` = use
|
||||
* the probe. See {@link adaptiveHistoryBudgetBytes}.
|
||||
*/
|
||||
private _retentionBudgetBytes?: number
|
||||
private _hub?: IntegrationHub // Integration Hub for external tools
|
||||
private _pendingMigrationRunner?: MigrationRunner // Deferred migration runner for large datasets
|
||||
private _aggregationIndex?: AggregationIndex // Incremental aggregation engine
|
||||
|
|
@ -899,6 +917,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
await this._vfs.init()
|
||||
this._vfsInitialized = true // Mark VFS as fully initialized
|
||||
|
||||
// 8.0 MVCC: infrastructure bootstrap (VFS root, etc.) is now the
|
||||
// generation-0 baseline. From here, every single-op write is a Model-B
|
||||
// generation. (Writers only — readers never stamp; a no-op for them.)
|
||||
if (!this.isReadOnly) {
|
||||
this._generationStampingActive = true
|
||||
}
|
||||
|
||||
// Eager embedding initialization.
|
||||
//
|
||||
// Adaptive default (8.0): the WASM embedding engine eagerly initializes
|
||||
|
|
@ -1233,6 +1258,43 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
return uuidv7()
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Persist one single-operation write (`add`/`update`/`remove`/
|
||||
* `relate`/`updateRelation`/`unrelate` and the per-item calls of their `*Many`
|
||||
* variants) as its OWN immutable generation — the Model-B "every write
|
||||
* versioned" contract. Wraps the write's existing `executeTransaction` batch in
|
||||
* {@link GenerationStore.commitSingleOp}, which captures byte-identical
|
||||
* before-images of `touched`, runs the batch with the generation watermark
|
||||
* pinned to the reserved generation (so this write's graph-index ops stamp cor
|
||||
* with that one distinct per-op generation, via the unchanged
|
||||
* {@link graphWriteGeneration}), and buffers the history for async
|
||||
* group-commit. A crash before the flush loses only that buffered history, not
|
||||
* the acknowledged live write (drop-without-restore recovery).
|
||||
*
|
||||
* @param touched - The entity/relationship ids the write creates, updates, or
|
||||
* deletes — the before-image + per-id-chain set.
|
||||
* @param run - The single-op's existing operation batch builder (the
|
||||
* `tx => {…}` body previously passed straight to `executeTransaction`).
|
||||
*/
|
||||
private async persistSingleOp(
|
||||
touched: { nouns?: string[]; verbs?: string[] },
|
||||
run: TransactionFunction<void>
|
||||
): Promise<void> {
|
||||
if (!this._generationStampingActive) {
|
||||
// Init-time / infrastructure baseline write (e.g. the VFS root): apply
|
||||
// WITHOUT creating a generation. Generation 0 is the freshly-materialized
|
||||
// brain (bootstrap included); the first USER write is generation 1.
|
||||
await this.generationStore.runWithoutGeneration(() =>
|
||||
this.transactionManager.executeTransaction(run)
|
||||
)
|
||||
return
|
||||
}
|
||||
await this.generationStore.commitSingleOp({
|
||||
touched,
|
||||
execute: () => this.transactionManager.executeTransaction(run)
|
||||
})
|
||||
}
|
||||
|
||||
async add(params: AddParams<T>): Promise<string> {
|
||||
this.assertWritable('add')
|
||||
await this.ensureInitialized()
|
||||
|
|
@ -1375,9 +1437,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
}
|
||||
}
|
||||
|
||||
// Execute atomically with transaction system
|
||||
// Execute atomically with transaction system, generation-stamped as one
|
||||
// immutable Model-B generation (before-image = the absent/create sentinel).
|
||||
// All operations succeed or all rollback - prevents partial failures
|
||||
await this.transactionManager.executeTransaction(async (tx) => {
|
||||
await this.persistSingleOp({ nouns: [id] }, async (tx) => {
|
||||
// Operation 1: Save metadata FIRST (TypeAwareStorage caching)
|
||||
// isNew=true: skip pre-read for rollback (entity doesn't exist yet)
|
||||
tx.addOperation(
|
||||
|
|
@ -2248,8 +2311,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
metadata: newMetadata
|
||||
}
|
||||
|
||||
// Execute atomically with transaction system
|
||||
await this.transactionManager.executeTransaction(async (tx) => {
|
||||
// Execute atomically with transaction system, generation-stamped as one
|
||||
// immutable Model-B generation (before-image = the entity's prior state).
|
||||
await this.persistSingleOp({ nouns: [params.id] }, async (tx) => {
|
||||
// Operation 1: Update metadata FIRST (updates type cache)
|
||||
tx.addOperation(
|
||||
new UpdateNounMetadataOperation(this.storage, params.id, updatedMetadata)
|
||||
|
|
@ -2358,8 +2422,12 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
const targetVerbs = await this.storage.getVerbsByTarget(id)
|
||||
const allVerbs = [...verbs, ...targetVerbs]
|
||||
|
||||
// Execute atomically with transaction system
|
||||
await this.transactionManager.executeTransaction(async (tx) => {
|
||||
// Execute atomically with transaction system, generation-stamped as one
|
||||
// immutable Model-B generation covering the entity AND its cascade-deleted
|
||||
// relationships (each id's before-image = its prior state).
|
||||
await this.persistSingleOp(
|
||||
{ nouns: [id], verbs: allVerbs.map((v) => v.id) },
|
||||
async (tx) => {
|
||||
// Operation 1: Remove from vector index
|
||||
if (noun) {
|
||||
tx.addOperation(
|
||||
|
|
@ -2769,8 +2837,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// entities were existence-checked above) and mirror them onto the verb.
|
||||
const { sourceInt, targetInt } = this.resolveVerbEndpointInts(verb)
|
||||
|
||||
// Execute atomically with transaction system
|
||||
await this.transactionManager.executeTransaction(async (tx) => {
|
||||
// Reverse-edge id reserved up front (when bidirectional) so the Model-B
|
||||
// generation's touched-verb set covers both edges this write creates.
|
||||
const reverseId = params.bidirectional ? uuidv4() : undefined
|
||||
|
||||
// Execute atomically with transaction system, generation-stamped as one
|
||||
// immutable Model-B generation (before-image of each new edge = absent).
|
||||
await this.persistSingleOp({ verbs: reverseId ? [id, reverseId] : [id] }, async (tx) => {
|
||||
// Operation 1: Save verb vector data
|
||||
tx.addOperation(
|
||||
new SaveVerbOperation(this.storage, {
|
||||
|
|
@ -2798,8 +2871,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
)
|
||||
|
||||
// Create bidirectional if requested
|
||||
if (params.bidirectional) {
|
||||
const reverseId = uuidv4()
|
||||
if (params.bidirectional && reverseId) {
|
||||
const reverseVerb: GraphVerb = {
|
||||
...verb,
|
||||
id: reverseId,
|
||||
|
|
@ -2868,8 +2940,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// a rollback can re-add through the BigInt addVerb contract.
|
||||
const endpointInts = verb ? this.resolveVerbEndpointInts(verb) : undefined
|
||||
|
||||
// Execute atomically with transaction system
|
||||
await this.transactionManager.executeTransaction(async (tx) => {
|
||||
// Execute atomically with transaction system, generation-stamped as one
|
||||
// immutable Model-B generation (before-image = the relationship's state).
|
||||
await this.persistSingleOp({ verbs: [id] }, async (tx) => {
|
||||
// Operation 1: Remove from graph index
|
||||
if (verb && endpointInts) {
|
||||
tx.addOperation(
|
||||
|
|
@ -2997,7 +3070,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// resolution serves both the remove (rollback re-add) and the re-add.
|
||||
const reindexInts = typeChanged ? this.resolveVerbEndpointInts(verbForIndex) : undefined
|
||||
|
||||
await this.transactionManager.executeTransaction(async (tx) => {
|
||||
await this.persistSingleOp({ verbs: [params.id] }, async (tx) => {
|
||||
tx.addOperation(
|
||||
new UpdateVerbMetadataOperation(this.storage, params.id, updatedMetadata)
|
||||
)
|
||||
|
|
@ -5104,9 +5177,27 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
const chunkQueued: string[] = []
|
||||
const chunkBuilderFailed: Array<{ item: string; error: string }> = []
|
||||
|
||||
// Model-B pre-pass: collect the chunk's touched-id set (entities + their
|
||||
// cascade-deleted relationships) so the generation captures before-images
|
||||
// for all of them. Tolerant by design — the builder below still does the
|
||||
// authoritative per-id load + error handling, and an over-included id whose
|
||||
// delete is skipped simply gets a before-image equal to its live state (a
|
||||
// harmless no-op history entry).
|
||||
const touchedNouns: string[] = [...chunk]
|
||||
const touchedVerbs: string[] = []
|
||||
for (const id of chunk) {
|
||||
try {
|
||||
const srcVerbs = await this.storage.getVerbsBySource(id)
|
||||
const tgtVerbs = await this.storage.getVerbsByTarget(id)
|
||||
for (const v of [...srcVerbs, ...tgtVerbs]) touchedVerbs.push(v.id)
|
||||
} catch {
|
||||
// Ignore — the builder's per-id try/catch is authoritative.
|
||||
}
|
||||
}
|
||||
|
||||
try {
|
||||
// Process chunk in single transaction for atomic deletion
|
||||
await this.transactionManager.executeTransaction(async (tx) => {
|
||||
// Process chunk as ONE atomic Model-B generation (entities + cascade verbs).
|
||||
await this.persistSingleOp({ nouns: touchedNouns, verbs: touchedVerbs }, async (tx) => {
|
||||
for (const id of chunk) {
|
||||
try {
|
||||
// Load entity data
|
||||
|
|
@ -5599,13 +5690,13 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
|
||||
/**
|
||||
* @description Read the reified transaction log — one entry per committed
|
||||
* `transact()` batch, carrying the committed generation, the commit
|
||||
* timestamp, and the `meta` the transaction was submitted with. Entries
|
||||
* are returned newest first. Single-operation writes
|
||||
* (`add`/`update`/`remove`/`relate`/…) advance the generation counter but
|
||||
* do not append log entries (the documented 8.0 history granularity), and
|
||||
* `compactHistory()` never rewrites the log — entries may reference
|
||||
* generations whose record-sets were already reclaimed.
|
||||
* generation, carrying the committed generation, the commit timestamp, and
|
||||
* the `meta` the transaction was submitted with. Entries are returned newest
|
||||
* first. Under Model-B EVERY write is its own generation, so single-operation
|
||||
* writes (`add`/`update`/`remove`/`relate`/…) appear here too (without `meta`
|
||||
* — transaction metadata is a `transact()`-only concept). `compactHistory()`
|
||||
* never rewrites the log — entries may reference generations whose
|
||||
* record-sets were already reclaimed.
|
||||
*
|
||||
* @param options - `from`/`to` (generation number or `Date`) bound the window
|
||||
* to commits in `[from, to]` INCLUSIVE on both ends — a log window names
|
||||
|
|
@ -5754,6 +5845,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
throw new Error('transact(): provide a non-empty array of transaction operations')
|
||||
}
|
||||
|
||||
// Commit any buffered single-op generations FIRST so this transaction's
|
||||
// generation lands strictly after them and the committed-generation
|
||||
// sequence stays gap-free (single-ops reserve generations eagerly).
|
||||
await this.generationStore.flushPendingSingleOps()
|
||||
|
||||
// Early CAS check — avoids expensive planning (embedding) on a stale
|
||||
// expectation. The generation store re-checks authoritatively under the
|
||||
// commit mutex; this one is purely a fast-fail.
|
||||
|
|
@ -5796,9 +5892,9 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* @description Open an immutable `Db` view of PAST state:
|
||||
*
|
||||
* - **`number`** — a generation: pins it on this store; reads resolve
|
||||
* through the generational record layer. History granularity is
|
||||
* `transact()` commits — single-operation writes between commits do not
|
||||
* produce records and remain visible through earlier pins.
|
||||
* through the generational record layer. Under Model-B every write is its
|
||||
* own generation, so single-operation writes are individually addressable
|
||||
* too (a pin always freezes against later single-op writes).
|
||||
* - **`Date`** — a wall-clock instant: resolved via the transaction log to
|
||||
* the newest generation committed at or before it, then pinned as above.
|
||||
* - **`string`** — a snapshot directory previously produced by
|
||||
|
|
@ -6128,57 +6224,114 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* keep (the GC backstop eventually releases leaked ones, but until then
|
||||
* compaction retains the history they protect).
|
||||
*
|
||||
* @param options - `retainGenerations` (keep the N most recent committed
|
||||
* generations) and/or `retainMs` (keep everything committed within the
|
||||
* window). Both supplied = both must allow reclaim. Neither = reclaim
|
||||
* everything unpinned.
|
||||
* @param options - Explicit retention CAPS (`maxGenerations` / `maxAge` /
|
||||
* `maxBytes`). Reclaim the oldest unpinned generations while ANY supplied
|
||||
* cap is exceeded. Neither = reclaim everything unpinned.
|
||||
* @returns Count of reclaimed record-sets and the new horizon.
|
||||
* @example
|
||||
* await brain.compactHistory({ retainGenerations: 100, retainMs: 7 * 86_400_000 })
|
||||
* await brain.compactHistory({ maxGenerations: 100, maxAge: 7 * 86_400_000 })
|
||||
*/
|
||||
async compactHistory(options?: CompactHistoryOptions): Promise<CompactHistoryResult> {
|
||||
this.assertWritable('compactHistory')
|
||||
await this.ensureInitialized()
|
||||
// Persist buffered single-op generations first so compaction reasons over a
|
||||
// complete on-disk history (the pending tier is never itself reclaimable).
|
||||
await this.generationStore.flushPendingSingleOps()
|
||||
return this.generationStore.compact(options)
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Resolve the effective generational-history retention policy
|
||||
* from `config.history`, filling per-field defaults so a partial object
|
||||
* inherits the rest. This is the single read site for the policy referenced
|
||||
* by {@link autoCompactHistory}.
|
||||
*
|
||||
* Defaults (zero-config): keep the 100 most recent committed generations
|
||||
* AND everything committed within the last 7 days; auto-compact on
|
||||
* `flush()` / `close()`. A generation is reclaimed only when it is older
|
||||
* than BOTH bounds and no live `Db` pin protects it.
|
||||
*
|
||||
* @returns The resolved policy: retention bounds plus the `autoCompact` flag.
|
||||
* @description Drive the adaptive retention byte budget at runtime — the
|
||||
* settable input a machine-level coordinator (e.g. cor's `ResourceManager`,
|
||||
* which fair-shares one box-wide history budget across co-located instances)
|
||||
* pushes whenever the box's free-resource picture changes. Takes effect at the
|
||||
* next auto-compaction (flush/close). Only meaningful while `retention` is
|
||||
* adaptive (unset or `'adaptive'`); ignored under `'all'` or explicit caps.
|
||||
* @param bytes - The new per-brain history byte budget (non-negative).
|
||||
* @example brain.setRetentionBudget(2 * 1024 ** 3) // 2 GiB for this brain
|
||||
*/
|
||||
private resolveHistoryPolicy(): {
|
||||
retainGenerations: number
|
||||
retainMs: number
|
||||
setRetentionBudget(bytes: number): void {
|
||||
if (!Number.isFinite(bytes) || bytes < 0) {
|
||||
throw new RangeError(`setRetentionBudget(): bytes must be a non-negative finite number (got ${bytes})`)
|
||||
}
|
||||
this._retentionBudgetBytes = bytes
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Resolve the effective `retention` policy from `config.retention`.
|
||||
* The single read site for the policy {@link autoCompactHistory} applies.
|
||||
*
|
||||
* - `'all'` → `{ mode: 'all' }` (never reclaim history).
|
||||
* - unset / `'adaptive'` → `{ mode: 'adaptive' }` (budget-driven; the budget
|
||||
* is `setRetentionBudget()`/`budgetBytes` when set, else a local probe).
|
||||
* - `{ … }` → `{ mode: 'explicit', maxGenerations?, maxAge?, maxBytes? }`.
|
||||
*
|
||||
* `autoCompact` defaults to `true` in every mode.
|
||||
* @returns The normalized retention policy.
|
||||
*/
|
||||
private resolveRetentionPolicy(): {
|
||||
mode: 'all' | 'adaptive' | 'explicit'
|
||||
maxGenerations?: number
|
||||
maxAge?: number
|
||||
maxBytes?: number
|
||||
budgetBytes?: number
|
||||
autoCompact: boolean
|
||||
} {
|
||||
const history = this.config.history
|
||||
const retention = this.config.retention
|
||||
if (retention === 'all') {
|
||||
return { mode: 'all', autoCompact: true }
|
||||
}
|
||||
if (retention === undefined || retention === 'adaptive') {
|
||||
return { mode: 'adaptive', autoCompact: true }
|
||||
}
|
||||
return {
|
||||
retainGenerations: history?.retainGenerations ?? 100,
|
||||
retainMs: history?.retainMs ?? 604_800_000, // 7 days
|
||||
autoCompact: history?.autoCompact ?? true
|
||||
mode: 'explicit',
|
||||
maxGenerations: retention.maxGenerations,
|
||||
maxAge: retention.maxAge,
|
||||
maxBytes: retention.maxBytes,
|
||||
budgetBytes: retention.budgetBytes,
|
||||
autoCompact: retention.autoCompact ?? true
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Run history compaction with the resolved retention policy if
|
||||
* `config.history.autoCompact` is on (the default). Invoked from `flush()`
|
||||
* and `close()` so generational record-sets cannot accumulate unbounded
|
||||
* across a long-lived writer's lifetime.
|
||||
* @description Compute the adaptive history byte budget for this brain: the
|
||||
* coordinator-driven value when present (`setRetentionBudget()` /
|
||||
* `config.retention.budgetBytes` — e.g. cor's fair-shared box budget), else a
|
||||
* conservative LOCAL probe of free resources. Standalone brainy must not be
|
||||
* cor-dependent, so the fallback uses `os.freemem()` (and is intentionally
|
||||
* generous — a single instance owning the box can keep a lot of history).
|
||||
* @returns The byte budget; `Infinity` only if no signal is available.
|
||||
*/
|
||||
private adaptiveHistoryBudgetBytes(driven?: number): number {
|
||||
if (driven !== undefined) return driven
|
||||
if (this._retentionBudgetBytes !== undefined) return this._retentionBudgetBytes
|
||||
// Local single-instance probe: allot a quarter of currently-free RAM to
|
||||
// history. Bounded, zero-config, and never reclaims pinned generations.
|
||||
try {
|
||||
const free = os.freemem()
|
||||
if (Number.isFinite(free) && free > 0) return Math.floor(free / 4)
|
||||
} catch {
|
||||
// os unavailable (exotic runtime) — fall through to unbounded.
|
||||
}
|
||||
return Infinity
|
||||
}
|
||||
|
||||
/**
|
||||
* @description Run history compaction under the resolved `retention` policy
|
||||
* when `autoCompact` is on (the default). Invoked from `flush()` and
|
||||
* `close()` so generational record-sets cannot accumulate unbounded across a
|
||||
* long-lived writer's lifetime.
|
||||
*
|
||||
* - `'all'` → returns without reclaiming (index compaction for speed still
|
||||
* runs elsewhere; history is decoupled and kept).
|
||||
* - `'adaptive'` → reclaim oldest-unpinned history down to the byte budget.
|
||||
* - `'explicit'` → apply the supplied `maxGenerations`/`maxAge`/`maxBytes` caps.
|
||||
*
|
||||
* Read-only instances and an explicit `autoCompact: false` skip silently.
|
||||
* Pinned generations (live `Db` values) are never reclaimed — the safety
|
||||
* invariant lives in {@link GenerationStore.compact}. Failures are logged
|
||||
* and swallowed: compaction is housekeeping and must never fail a flush or
|
||||
* a clean shutdown.
|
||||
* Pinned generations are never reclaimed ({@link GenerationStore.compact}).
|
||||
* Failures are logged and swallowed — compaction is housekeeping and must
|
||||
* never fail a flush or a clean shutdown.
|
||||
*/
|
||||
private async autoCompactHistory(): Promise<void> {
|
||||
// Nothing to compact on a read-only instance or before init wired up the
|
||||
|
|
@ -6186,15 +6339,22 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
if (this.isReadOnly || !this.generationStore) {
|
||||
return
|
||||
}
|
||||
const policy = this.resolveHistoryPolicy()
|
||||
if (!policy.autoCompact) {
|
||||
const policy = this.resolveRetentionPolicy()
|
||||
if (!policy.autoCompact || policy.mode === 'all') {
|
||||
return
|
||||
}
|
||||
try {
|
||||
await this.generationStore.compact({
|
||||
retainGenerations: policy.retainGenerations,
|
||||
retainMs: policy.retainMs
|
||||
})
|
||||
if (policy.mode === 'adaptive') {
|
||||
const budget = this.adaptiveHistoryBudgetBytes(policy.budgetBytes)
|
||||
if (budget === Infinity) return // no pressure signal → keep everything this pass
|
||||
await this.generationStore.compact({ maxBytes: budget })
|
||||
} else {
|
||||
await this.generationStore.compact({
|
||||
maxGenerations: policy.maxGenerations,
|
||||
maxAge: policy.maxAge,
|
||||
maxBytes: policy.maxBytes
|
||||
})
|
||||
}
|
||||
} catch (error) {
|
||||
console.warn(
|
||||
`Auto-compaction of generational history failed (non-fatal): ${
|
||||
|
|
@ -7964,6 +8124,11 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
console.log('Flushing Brainy indexes and caches to disk...')
|
||||
const startTime = Date.now()
|
||||
|
||||
// 8.0 MVCC: persist the in-memory pending single-op generation history to
|
||||
// disk first (async group-commit), so a flush makes every single-op write's
|
||||
// history durable, not just the live data.
|
||||
await this.generationStore.flushPendingSingleOps()
|
||||
|
||||
// Flush all components in parallel for performance
|
||||
await Promise.all([
|
||||
// 1. Flush storage adapter counts (entity/verb counts by type)
|
||||
|
|
@ -7991,7 +8156,7 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
this.generationStore.persistCounterNow()
|
||||
])
|
||||
|
||||
// 6. Auto-compact generational history per config.history (default on).
|
||||
// 6. Auto-compact generational history per config.retention (default on).
|
||||
// Runs after the flush so durable state is in place; respects live
|
||||
// Db pins and an explicit autoCompact: false.
|
||||
await this.autoCompactHistory()
|
||||
|
|
@ -11730,9 +11895,10 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
// `recall` preset + `persistMode` (folded in from 7.x's hnswPersistMode).
|
||||
// See BRAINY-8.0-RENAME-COORDINATION § A.2 + § G.2.
|
||||
vector: config?.vector ?? undefined,
|
||||
// Generational-history retention — defaults applied at the read site
|
||||
// (resolveHistoryPolicy) so a partial object inherits per-field defaults.
|
||||
history: config?.history ?? undefined,
|
||||
// Generational-history retention — the `retention` knob; defaults applied
|
||||
// at the read site (resolveRetentionPolicy) so a partial object inherits
|
||||
// per-field defaults and `undefined` resolves to ADAPTIVE.
|
||||
retention: config?.retention ?? undefined,
|
||||
// Embedding initialization — left undefined when omitted so the adaptive
|
||||
// default resolves at the init() read site (eager when the WASM embedder
|
||||
// is the active one on a non-reader writer outside unit tests). Explicit
|
||||
|
|
@ -12270,7 +12436,14 @@ export class Brainy<T = any> implements BrainyInterface<T> {
|
|||
* This ensures deferred persistence mode data is saved
|
||||
*/
|
||||
async close(): Promise<void> {
|
||||
// Phase 0: Auto-compact generational history per config.history (default
|
||||
// Phase 0a: Persist buffered single-op generation history (async
|
||||
// group-commit) before anything else, so a clean close never drops history
|
||||
// the caller already observed. No-op when nothing is pending or read-only.
|
||||
if (this.generationStore && !this.isReadOnly) {
|
||||
await this.generationStore.flushPendingSingleOps()
|
||||
}
|
||||
|
||||
// Phase 0b: Auto-compact generational history per config.retention (default
|
||||
// on) BEFORE the generation store closes below. Respects live Db pins and
|
||||
// an explicit autoCompact: false; no-op on read-only instances.
|
||||
await this.autoCompactHistory()
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue