diff --git a/CHANGELOG.md b/CHANGELOG.md index fd0de54e..a7b0044b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,56 +2,6 @@ All notable changes to this project will be documented in this file. See [standard-version](https://github.com/conventional-changelog/standard-version) for commit guidelines. -### [8.9.0](https://github.com/soulcraftlabs/brainy/compare/v8.8.2...v8.9.0) (2026-07-19) - -- docs: measured performance envelopes v1 (per-op p50/p95 at 1k and 10k, pure-JS floor) (5cabd78) -- fix: release drains in-flight writer-lock heartbeat — no phantom lock after unlink (70e4bc8) -- feat: flush() never compacts — history maintenance moves to close() with bounded passes (300d9f2) - - -### [8.8.2](https://github.com/soulcraftlabs/brainy/compare/v8.8.1...v8.8.2) (2026-07-19) - -- fix: one field-resolution law across aggregation hooks, source.where, removeMany, and find() spellings (945d92d) -- chore: push public docs to the soulcraft.com ingest door on release (42037d0) - - -### [8.8.1](https://github.com/soulcraftlabs/brainy/compare/v8.8.0...v8.8.1) (2026-07-18) - -- fix: O(1) adaptive retention accounting + historyStats fleet audit (6207e48) -- fix: import dedup off-switch honesty + brain-owned lifecycle for the background pass (4fcef7b) - - -### [8.8.0](https://github.com/soulcraftlabs/brainy/compare/v8.7.1...v8.8.0) (2026-07-17) - -- feat: OS-limit detection for pool-scale deployments (16a73b8) - - -### [8.7.1](https://github.com/soulcraftlabs/brainy/compare/v8.7.0...v8.7.1) (2026-07-17) - -- fix: race-proof writer-lock acquisition + machine-readable conflict through init (01a3b46) - - -### [8.7.0](https://github.com/soulcraftlabs/brainy/compare/v8.6.0...v8.7.0) (2026-07-17) - -- feat: scaled transact budgets + labeled timeout diagnostics + envelope docs (6ef9fcb) - - -### [8.6.0](https://github.com/soulcraftlabs/brainy/compare/v8.5.2...v8.6.0) (2026-07-17) - -- feat: brain.auditGraph() — read-only graph-truth audit (2a03fae) - - -### [8.5.2](https://github.com/soulcraftlabs/brainy/compare/v8.5.1...v8.5.2) (2026-07-17) - -- fix: exception-safe aggregation backfill + generation-verified adoption + loud open-path guards (a77b064) - - -### [8.5.1](https://github.com/soulcraftlabs/brainy/compare/v8.5.0...v8.5.1) (2026-07-17) - -- fix: aggregation state adoption on reopen + single-flight backfill + query-cap ratchet removal (da55be7) -- docs: external-backups/sparse-storage guide + generation fact log concept (593bb8b) - - ### [8.5.0](https://github.com/soulcraftlabs/brainy/compare/v8.4.0...v8.5.0) (2026-07-15) - test: tolerant timing assertion in the execution-time measure test (4dc0a92) diff --git a/RELEASES.md b/RELEASES.md index 7799c6f6..457b65d0 100644 --- a/RELEASES.md +++ b/RELEASES.md @@ -8,274 +8,8 @@ Full auto-generated changelog: `CHANGELOG.md` · Releases: https://github.com/so - Debugging data, query, or storage behaviour - A new Brainy feature is available that you want to adopt -## Removed APIs — 7.x → 8.x (the complete ledger) - -Every public API removed at the 8.0 major, with its sanctioned replacement. If your code -still calls a left-column name on 8.x it throws (or the config key is rejected) — the -replacement is always a one-line change. (Standing contract from 8.9.0 forward: removals -happen only at majors, after ≥1 minor of loud runtime deprecation naming the replacement.) - -| Removed (7.x) | Replacement (8.x) | -|---|---| -| `brain.search(query, k)` | `find({ query })` — semantic; `find({ query, searchMode })` for hybrid | -| `brain.getRelations({...})` | `related(id, opts)` for adjacency; `find({ connected: {...} })` for scoped traversal | -| `brain.neural()` clustering | `find({ vector })` + aggregation `GROUP BY` | -| `Db.search()` | `db.find({ vector })` | -| Pre-8.0 storage path aliases (`directory`, `basePath`, …) | one `storage.path` key (old aliases throw) | -| Reserved keys inside `metadata` bags (silently remapped in 7.x) | top-level params (`subtype`, `visibility`, `confidence`, `weight`, …) — reserved-in-bag throws | -| 7.x COW branches layout (`branches/main/`) | generational MVCC (`asOf()`, `now()`, `db.persist(path)`) — on-disk migration is automatic at first 8.x open | - -The fork/snapshot family (`brain.snapshot()`, `createSnapshot()`, `restoreSnapshot()`) -is sometimes cited as a 7.x removal — those methods never existed on 7.x; the 8.0 Db API -(`asOf`/`persist`/`restore({confirm})`) is their first real implementation. - --- -## v8.9.0 — 2026-07-19 (flush is durability-only: history maintenance moves to close()) - -The write path stops paying maintenance costs — the last structural piece of the -flush-storm class (a production deployment measured single writes blocked 25–191s behind -history reclaim running inline on flush under memory pressure): - -- **`flush()` never compacts history.** It persists the current window's deltas and - nothing else — its cost no longer depends on history backlog or retention mode, in any - configuration. **`close()` is the auto-compaction site** (time-bounded per pass, ~5s; - an early stop is a consistent prefix and the next pass resumes). -- **`compactHistory()` gains `timeBudgetMs`** — bound your own maintenance windows; the - same resumable-prefix guarantee applies. -- **The documented trade**: a long-lived writer that never closes accumulates history - until its next explicit `compactHistory()`. Predictable writes, explicit maintenance. - If you run bounded retention on an always-on service, schedule a periodic - `compactHistory({ ...caps, timeBudgetMs })` in your maintenance window. -- **New public doc: `docs/performance-envelopes.md`** — measured per-op envelopes - (p50/p95 at stated scales, hardware, and backend, with the measuring script cited). - Refresh rule going forward: any release touching a measured path re-runs that op's - benchmark and updates the envelope in the same release. -- **New in this file: the Removed APIs 7.x→8.x table** (top of this document) — every - removal with its sanctioned replacement, one place, per the engine-currency contract. - Standing from here: removals only at majors, after ≥1 minor of loud runtime deprecation. - -## v8.8.2 — 2026-07-19 (one field-resolution law: reserved-field aggregates stop drifting) - -Four fixes from a consumer conformance audit, all rooted in the same disease — two field-resolution -regimes where there must be one: - -- **Aggregates grouped by a RESERVED field (`subtype`, `visibility`, …) now decrement on - delete.** The delete/update hooks fed the aggregation engine a partial entity view (type, - service, data, metadata only), so a reserved-field `groupBy` resolved to a nonexistent group - on the way DOWN — counts drifted upward forever after any delete, and updates that moved an - entity between reserved-field groups double-counted it. The hooks now pass the full-fidelity - entity view (every reserved field top-level, the same shape the add path uses). If your - deployment derives stats from reserved-field aggregates, re-define those aggregates once - after upgrading (a changed definition triggers one rescan) or run them fresh — the drifted - persisted counts do not self-heal retroactively. -- **Aggregation `source.where` on reserved fields now filters** instead of silently matching - nothing: the matcher resolves fields through the same resolver `groupBy` uses (top-level - standard fields + custom metadata), so `where: { subtype: 'note' }` means what it says. -- **`removeMany()` refuses empty/invalid selectors loudly.** A bare array passed positionally - (`removeMany([id])` instead of `removeMany({ ids: [id] })`), an empty params object, or - `ids: []` used to resolve successfully having deleted nothing. All three now throw. -- **`find()` accepts both where-key spellings.** Metadata is flattened at index time - (`metadata.entry.title` indexes as `entry.title`); a `metadata.`-prefixed where key now - falls back to its flattened spelling when the prefixed one isn't indexed — the - "unindexed field(s), returning []" confusion for storage-shaped spellings is gone. (A - literal nested custom key named `metadata` still wins when indexed as spelled.) - -## v8.8.1 — 2026-07-18 (flush no longer walks the whole generation history + the import dedup off-switch is now honest) - -### The flush-storm fix (production incident, reported by a long-running deployment) - -Under the default adaptive retention, **every `flush()` re-walked the entire committed -generation history** to compute total history bytes for the budget check — O(all -generations) with disk re-reads past the 4,096-entry delta-cache bound. On a brain with -70,000+ accumulated generations that turned every write into a full-tail scan (60-100s -writes), even though the budget (free-RAM-based) never tripped and nothing was ever -reclaimed. Fixed: - -- `historyBytes()` now maintains a **running total**: seeded by one walk on first use, - then updated incrementally at every commit and reclaim — the adaptive retention check - on every flush is O(1). Invariant regression-pinned (running total ≡ fresh walk through - both commit paths and compaction). -- New **`brain.historyStats()`** (read-only, exported `HistoryStats`): generation count, - total on-disk bytes, generation/timestamp range, compaction horizon, retention mode, - and the effective adaptive budget — the one-call fleet-audit for sizing retention - exposure per brain. -- Interim guidance for keep-everything deployments already affected: `retention: 'all'` - skips the adaptive accounting entirely (and is the correct policy if you never want - history reclaimed). The accumulated files are harmless at rest; this release removes - the per-write cost of their existence. - -### The import dedup off-switch (lifecycle honesty) - -The post-import background deduplication pass (a merge-DELETE writer that runs ~5 minutes -after an import, merging entities judged duplicates by id / name / vector similarity) had -three lifecycle defects, all fixed: - -- **`enableDeduplication: false` now actually disables it.** The background pass was - scheduled unconditionally — an import that explicitly opted out could still have - entities auto-removed 5 minutes later. The flag now gates BOTH the inline merge and - the background pass (regression-pinned). -- **One deduplicator per brain, owned by the brain.** Each `import()` call constructed its - own coordinator + deduplicator, so the "debounced" timer never actually debounced across - imports (N imports = N delete timers). The brain now owns a single instance — the - debounce genuinely spans imports — and `close()` cancels pending work, so a delete pass - can never fire against a closed brain. -- **The 5-minute timer is unref'd** — a pending pass no longer holds the process open - (the exit-hang class; this timer had escaped the earlier sweep). - -Retention note for keep-everything deployments: with `enableDeduplication: false` on -import calls and `retention: 'all'` in config, no engine path removes records -automatically. - -## v8.8.0 — 2026-07-17 (OS-limit detection for pool-scale deployments) - -Small minor: brains now detect the two OS limits that bite at pool scale and warn **before** -the incident instead of during it. - -- At open (once per process, Linux-only, measurement-only), Brainy reads `RLIMIT_NOFILE` - (soft/hard, from `/proc/self/limits`) and `vm.max_map_count`, and warns loudly when either - sits below the pool-scale floors (soft NOFILE < 65 536; max_map_count < 262 144) — with the - exact raise commands (`ulimit -n` / `LimitNOFILE=` / `sysctl vm.max_map_count`). On stock - defaults the failure otherwise arrives as `EMFILE` or a failed mmap deep inside an index - open, long after the real cause stopped being visible. An unreadable limit produces **no** - warning — no measurement, no claim (non-Linux platforms stay silent). -- Exported for ops doors: `checkOsLimits()` returns the full `OsLimitsReport` - (values + warnings) programmatically, with the floors exported as constants. - -## v8.7.1 — 2026-07-17 (writer-lock acquisition is race-proof + machine-readable through init) - -Two hardenings of the multi-process writer lock (the `locks/_writer.lock` lease that makes a -second writer on the same brain directory fail loudly): - -- **Lock acquisition claims atomically.** The acquire path used read-then-write, leaving a - window where two processes racing an *absent* lock could both "succeed" — and the loser - kept running unlocked, silently. The claim is now an atomic create-exclusive write - (`O_EXCL`): exactly one racer wins; the loser re-evaluates and either fails loudly with - the winner's details or performs a verified stale-takeover. Bounded retries; contention - beyond them fails loudly rather than degrading into a lockless open. -- **`BRAINY_WRITER_LOCKED` survives `init()`.** The conflict error documents a - machine-readable contract (`err.code`, `err.lockInfo` with the holder's pid/host/ - heartbeat), but init's error wrapping silently stripped both, leaving consumers a message - to regex against. The error now passes through unwrapped. - -Measured while verifying (for operators sizing audits): `brain.auditGraph()` at a -production-consumer scale of ~2,600 relationships / 800 entities costs ~0.1 s warm and -~0.5 s cold, with exact scar counting across reopen. - -## v8.7.0 — 2026-07-17 (bulk-transact ergonomics: scaled budgets + timeout telemetry) - -The bulk-import ergonomics release, from a consumer's measured production incident (a serial -import on network-attached storage at ~2 s/op met a flat 30 s transaction budget): - -- **The transact apply budget now scales with the batch** — `max(30 s, opCount × 2 s)` — or - is exactly what you pass as the new `TransactOptions.timeoutMs`. A flat 30 s cap silently - limited honest bulk work to ~15 operations on slow disks while looking generous for small - batches. Internal batch paths (e.g. `removeMany` chunks) get the same scaling. -- **`TransactionTimeoutError` is a diagnosis, not just a failure**: it now reports the - operation it stopped at as `i/N` with the operation's name, elapsed vs budgeted time, and - states the batch rolled back atomically and is retryable. Its `context` carries the same - fields programmatically. -- **The transact envelope is documented** — batch sizing, budget math, chunking with - `ifAbsent` idempotency, and the precompute pattern (`embedBatch` + per-op `vector`) that - keeps model inference out of the commit path. Guide: `docs/guides/optimistic-concurrency.md`. -- Note: `brain.embed()` / `brain.embedBatch()` (the precompute APIs) already ship — public, - with native-provider passthrough, verified end-to-end (batch and single paths produce - bit-identical vectors; a vector-supplied `add` is fully searchable). Honest measurement: - on the default WASM engine, batch throughput ≈ sequential (~160 ms/text) — the precompute - win is keeping inference out of the budgeted commit path, not raw embedding speed. - -## v8.6.0 — 2026-07-17 (brain.auditGraph — the graph-truth verification instrument) - -A minor release adding one new public API, from the fleet's graph-trust program: a read-only -audit that **proves whether relationship reads return stored truth** on a given brain. - -- **`brain.auditGraph(options?)`** walks every canonical relationship record, queries the same - read path applications use (`related()` / VFS `readdir`) with all visibility tiers included, - and classifies every discrepancy into its failure family: `missingFromReads` (records the - read path omits — a stale adjacency index), `danglingEndpoints` (relationships whose endpoint - entity no longer exists — the historical partial-delete scar class), and `readOnlyVerbIds` - (read-path edges with no stored record — ghosts). Design-hidden internal/system edges are - counted separately so intentional hiding is never misclassified as loss. Counts are exact; - example lists cap at `maxExamples` with an explicit `truncatedExamples` flag; the result is - narrated loudly on incoherence. Mutates nothing — safe on a live brain. -- The operational pairing: audit → if incoherent, `repairIndex()` → audit again. A `coherent` - report after the repair is the verified all-clear. Run it after any engine upgrade, restore, - or migration. Guide: `docs/guides/inspection.md`. -- Also: `getNounIds` pagination now refuses an undecodable resume cursor loudly (the same - contract `getNouns`/`getVerbs` gained in 8.5.2 — the third and final walk brought under it). -- Types exported: `GraphAuditReport`, `GraphAuditDiscrepancy`. - -## v8.5.2 — 2026-07-17 (aggregation backfill: exception-safe, generation-verified, and loud) - -Hardening patch from a migration incident (a byte-copied store on new hardware; the service -entered a silent full-CPU loop at boot). Four changes, all in the aggregation engine's -backfill/adoption path: - -- **Backfill walks are exception-safe and non-destructive.** A rescan now builds into a - staging map and swaps in atomically on completion; a mid-walk failure drops the staging map, - keeps the previous live state serving, and surfaces the storage error to the failing query. - Previously the walk wiped live state *before* a scan that could throw, never cleared the - pending flag on failure, and re-ran a full walk on every subsequent query — a silent - wipe/walk/throw loop at the caller's retry rate. -- **Failed walks are latched.** After a walk fails, retries within a 30-second cooldown rethrow - the recorded error instantly instead of re-walking — a tight caller-side retry loop now costs - one loud error per query, never a full store walk per query. -- **Adoption is generation-verified.** Persisted aggregation state is stamped with the store's - committed generation at flush; reopen adoption requires the stamp to equal the current - watermark. Stale state (unclean shutdown) or over-counting state (a fact-log truncation on a - copied store pulled the watermark back) triggers exactly one loud rescan — never a silent - adopt. Pre-8.5.2 state on generation-aware stores rescans once after upgrade, then is stamped. -- **The path narrates.** Adoption decisions, no-adoptable-state outcomes, walk start/finish - (entity count + duration), and walk failures all log by default; a non-advancing storage - pagination cursor aborts the walk loudly instead of looping forever. - -Plus three guards from a full audit of every loop in the open/init path: - -- **Invalid pagination cursors fail loudly.** A supplied-but-undecodable resume token to - `getNouns`/`getVerbs` used to silently restart the walk at offset 0 — to a `while(hasMore)` - caller that re-serves page 1 forever (an unbounded silent CPU loop). It now throws with a - clear message instead. -- **The graph cold-load verb walk has a stall guard.** `hasMore=true` with a missing or - non-advancing cursor aborts loudly instead of re-reading the same page forever. -- **A derived index AHEAD of the store is named at open.** Brainy already surfaced a provider - generation *behind* the committed watermark; the *ahead* direction (the signature of a - byte-copy of a live service, or a log truncation during crash recovery) now logs a loud - warning explaining what happened and that `brain.repairIndex()` forces a heal — instead of - passing unnamed into whatever the derived index does next. - -## v8.5.1 — 2026-07-17 (aggregation state survives restarts + the query-cap ratchet removed) - -Patch release from a production incident (aggregate/count paths taking 40–90 s on an idle box -while vector search stayed fast, and every `find({ limit: 5000 })` suddenly failing against an -"auto-configured query limit of 1000"). Three fixes, one cosmetic: - -- **Aggregation state is actually adopted on reopen.** The boot pattern `defineAggregate()` → - query raced the engine's async state load: the synchronous define always won, flagged a - backfill, and the first query then wiped the just-loaded persisted state and re-walked the - entire store — every restart, forever. Reopening with an unchanged definition now adopts the - persisted state directly (zero scans); a backfill runs only on a real definition change, a - missing/failed state load, or a write that landed before adoption (exactness wins). Apps that - rely on persisted definitions without re-defining at boot also no longer race a spurious - "Aggregate not defined". -- **Backfills are single-flight and batched.** Concurrent queries on a cold aggregate used to - each wipe the others' partial state and start their own full store walk — under steady query - arrival the store never converged (the 40–90 s loop). Now all concurrent queries share one - walk, and one walk fills every aggregate pending backfill (M aggregates ≠ M scans). -- **The query-cap "learning" ratchet is removed.** `maxLimit` is a memory-protection bound, but - a hidden tuner shrank it 20 % per recorded query while the lifetime-average query time - exceeded 1 s — down to a floor of 1000, below the documented 10 000 auto floor, with no - practical recovery, and the resulting error blamed "available free memory" (stale basis - label). The cap now comes from its construction-time basis (or your explicit - `maxQueryLimit` / `reservedQueryMemory`) alone and never changes at runtime; query timing is - recorded for diagnostics only. -- **Cosmetic:** the engine's own persistence keys (`__aggregation_*`, `brainy:entityIdMapper`) - no longer log `[Storage] Unknown key format` at boot — they were always routed correctly; - they're now recognized before the warning fires. - -Operationally: if a host was bitten, upgrading and restarting is the whole fix — no repair -ritual needed. Setting `maxQueryLimit` explicitly remains the valve that bypasses auto-detection -entirely. - ## v8.5.0 — 2026-07-15 (provider access to the fact log + the shared stamp verifier) Small additive follow-up to 8.4.0, from the native accelerator's first consumption pass: diff --git a/docs/concepts/generation-fact-log.md b/docs/concepts/generation-fact-log.md deleted file mode 100644 index f9b3e974..00000000 --- a/docs/concepts/generation-fact-log.md +++ /dev/null @@ -1,117 +0,0 @@ ---- -title: The Generation Fact Log -slug: concepts/generation-fact-log -public: true -category: concepts -template: concept -order: 6 -description: Every committed write also appends a self-verifying "fact" — an after-image commit record — to an append-only log. What facts are, the crash-safety model, the scanFacts() streaming surface, family stamps, and how index providers consume the log for sequential heals. -next: - - concepts/consistency-model - - guides/snapshots-and-time-travel ---- - -# The Generation Fact Log - -Since 8.4.0, every committed generation also appends a **fact** — a compact record of what each -touched entity or relationship *became* — to an append-only, checksummed log under -`_generations/facts/`. Where the generational history answers *"what did things look like -before?"* (before-images, powering `asOf()` and rollback), the fact log answers *"what happened, -in order?"* — one sequential, self-verifying stream of the store's present being written. - -Nothing about querying changes. The fact log exists for three consumers: - -1. **Index heals and rebuilds** — one sequential read in commit order replaces a per-entity - directory walk over millions of files. -2. **Incremental catch-up** — a derived index that knows which generation it reflects reads *just - the gap*, instead of rebuilding from scratch. -3. **Replay and audit tooling** — anything that wants the store's committed timeline as a stream. - -## What a fact is - -One fact per committed generation: - -- **`generation`** and **`timestamp`** — which commit, when. -- **`ops`** — every write in that commit: `{ kind: 'noun' | 'verb', id, record }` where `record` - holds the entity's full after-image (both stored legs), or **`null` for a tombstone** — a - removal carries no body, by design. -- **`meta`** — the transaction metadata `transact()` was submitted with, when present. -- **`blobHashes`** — content-blob references, for exact reclamation accounting. - -Facts accumulate **from the first write after upgrading** — pre-existing history is not -retroactively converted, and consumers fall back to the enumeration walk when no log exists. - -## Crash safety, in one paragraph - -Facts are appended and fsynced **inside the same durability window as the commit itself**, before -the commit point — so after a crash, the log can only ever be *ahead* of committed truth, never -behind it with a hole. On open, the store reconciles the log back to the committed watermark: -torn tails are detected by per-record checksums and cut; whole records beyond the watermark are -truncated. The invariant every reader can rely on: **an absent generation was never committed; a -present fact was.** `transact()` facts are durable the moment `transact()` returns; single-op -facts share the same group-commit flush as the rest of their generation, so a hard kill loses the -fact and the generation *together* — never a torn state. - -## Reading the log - -```typescript -const scan = brain.scanFacts({ fromGeneration: 1 }) -if (scan) { - // Telemetry up front — progress bars get a denominator from second zero. - console.log(scan.headGeneration, scan.segmentCount, scan.approxFactCount) - - for await (const batch of scan.batches()) { - // Each batch: { facts, firstGeneration, lastGeneration, factCount, byteSize, segmentId } - for (const fact of batch.facts) { - for (const op of fact.ops) { - if (op.record === null) { - // a tombstone: op.id was removed in this generation - } - } - } - } - - console.log(scan.summary()) // { factsYielded, segmentsRead } — the cross-check -} -``` - -- `scanFacts()` returns `null` when the store hosts no fact log (older store, or a storage adapter - without binary append support) — fall back to enumerating entities. -- Scans run against a **snapshot**: facts appended after the scan opens never bleed in, each fact - is yielded exactly once, and a detected gap aborts loudly — never a silent skip. -- `brain.factSegmentPaths()` returns the immutable, *sealed* segment files for zero-copy consumers - (the append-mutable tail is excluded — read it through `scanFacts()`). - -## Family stamps: how a projection proves it's current - -Anything derived from the store — an index, the entity file tree itself — carries a **family -stamp**: a small JSON record of *which committed generation the projection reflects* -(`sourceGeneration`) plus the invariants that verify it whole (exact per-file byte sizes for -bounded families; rollup invariants like entity counts for unbounded ones). At open, coherence is -a **comparison**, not a walk: - -- stamp equals the committed watermark and invariants hold → serve; -- stamp is behind → the projection reads just the gap from the fact log; -- invariants fail → loud, named divergence — `brain.repairIndex()` rebuilds from canonical and - re-stamps. - -The verifier is exported (`verifyFamilyStamp`) so every projection — TypeScript or native — runs -literally the same check. - -## For plugin authors: the storage capability - -Index providers receive the storage adapter, not the brain — so the host wires the log onto it. -Feature-detect and prefer the stream; fall back to enumeration: - -```typescript -const scan = storage.scanFacts?.({ fromGeneration: stamp.sourceGeneration + 1 }) -if (scan) { - // sequential catch-up from the log -} else { - // enumeration walk (older store or adapter) -} -const committed = storage.committedGeneration?.() // the watermark stamps compare against -``` - -Providers must never construct their own reader over the log's files — the open path belongs to -the single writer (it reconciles the log at open); the capability is the sanctioned seam. diff --git a/docs/guides/aggregation.md b/docs/guides/aggregation.md index 11d86ec8..e58eb2ce 100644 --- a/docs/guides/aggregation.md +++ b/docs/guides/aggregation.md @@ -11,13 +11,7 @@ No batch jobs. No scheduled recalculations. Aggregates stay current with every w **Defining over existing data:** if you define an aggregate on a store that already holds matching entities, Brainy backfills it from those entities on the first query (a one-time scan, then purely incremental). So `defineAggregate()` behaves the same whether you define it before -or after the data exists. - -**Reopening a persisted brain:** aggregate state persists across restarts. Re-defining the -same aggregate at boot (the normal declarative pattern) adopts the persisted state directly — -no rescan. A backfill scan runs only when the definition actually changed, when no persisted -state exists, or when the state failed to load; and however many aggregates need backfilling, -they share a single scan. +or after the data exists — including when a persisted brain reopens already populated. ## Quick Start diff --git a/docs/guides/external-backups-and-sparse-storage.md b/docs/guides/external-backups-and-sparse-storage.md deleted file mode 100644 index f28dcfd7..00000000 --- a/docs/guides/external-backups-and-sparse-storage.md +++ /dev/null @@ -1,99 +0,0 @@ ---- -title: External Backups & Sparse Storage -slug: guides/external-backups -public: true -category: guides -template: guide -order: 10 -description: How to back up a brain directory with external tools (tar, rsync, cp) without exploding sparse files — why a store can show 100+ GB "apparent" size on a small disk, which files are sparse, and how persist()/restore() handle it for you. -next: - - guides/snapshots-and-time-travel - - concepts/storage-adapters ---- - -# External Backups & Sparse Storage - -The built-in snapshot path — [`db.persist()` and `brain.restore()`](/docs/guides/snapshots-and-time-travel) — -already handles everything on this page for you. Read this when you back up a brain directory with -**external tools**: `tar`, `rsync`, `cp`, `scp`, or a filesystem-level backup agent. - -## The one-sentence rule - -> **Always use the sparse-aware flag**: `tar czSf` (capital `S`), `rsync --sparse`, -> `cp --sparse=always`. A naive copy can turn a 2 GB store into a 100+ GB one — or fail -> the disk entirely. - -## Why: some files are sparse - -When a native accelerator plugin is active, parts of the index live in **memory-mapped files** -created at a large fixed virtual size — the file's *apparent* size — while the filesystem only -allocates blocks that were actually written. A brand-new id-mapper file can report tens of -gigabytes in `ls -l` while occupying a few megabytes on disk. - -Check the difference yourself: - -```bash -ls -lh brain-data/_id_mapper/ # APPARENT size (can be huge) -du -sh brain-data/ # ALLOCATED size (the real footprint) -``` - -The sparse candidates in a brain directory: - -| Path | What it is | -|---|---| -| `_id_mapper/` | The native id-mapper's mmap files (large fixed virtual size) | -| `_blobs/` | Native index files (vector base, segments) — may be mmap-backed | - -Everything else (entities, `_system`, `_generations`, `_cas` content blobs) is ordinary dense data. - -## Doing it right - -**tar** — the `S` flag detects holes and stores only real data: - -```bash -tar czSf brain-backup.tgz /data/brain -# restore preserves the holes: -tar xzSf brain-backup.tgz -C /data/ -``` - -**rsync**: - -```bash -rsync -a --sparse /data/brain/ backup-host:/backups/brain/ -``` - -**cp**: - -```bash -cp -a --sparse=always /data/brain /backups/brain -``` - -**What goes wrong without the flag:** the copy *materializes* every hole as real zero bytes. -A store whose apparent size exceeds the target disk fails with `ENOSPC` partway through — and a -copy that *does* fit silently costs the full apparent size in storage and transfer time. - -## What the built-in paths do (so you don't have to) - -- **`db.persist(path)`** snapshots via **hard links** — instant and space-shared, since every data - file is immutable-by-rename. The handful of append-in-place files (the transaction log, the - commit fact log's tail segment) and mmap-mutated directories (`_id_mapper/`) are **byte-copied** - instead, so a post-snapshot write can never reach through a shared inode into your backup. -- **`brain.restore(path, { confirm: true })`** is **non-destructive and sparse-aware**: the snapshot - is copied into a staging area *before* any live data is touched (all-zero blocks stay holes), and - only after the copy fully succeeds does an atomic swap move it into place. A failed copy — - including `ENOSPC` — leaves the live store exactly as it was. A crash mid-swap completes forward - on the next open. - -## Live-store caveats for external tools - -1. **Prefer snapshotting a `persist()` output, not the live directory.** `persist()` produces a - crash-consistent, immutable snapshot; running `tar` against a live, actively-written directory - can capture a torn mid-write state. If you must archive live, stop writes first (or accept that - the archive is only as consistent as the moment's flush state). -2. **Never prune or "clean up" files inside a brain directory.** Index files that look stale or - redundant are load-bearing; the store protects its declared index families from in-process - deletion, but an external `rm` bypasses that fence. If space is the concern, `du -sh` first — - the allocated size is usually far smaller than it looks. -3. **Verify restores by opening them.** `Brainy.load(path)` opens any snapshot or restored - directory read-only — the store verifies its own coherence at open and reports loudly if - anything is missing or torn. diff --git a/docs/guides/find-limits.md b/docs/guides/find-limits.md index 4c7fd252..66418b80 100644 --- a/docs/guides/find-limits.md +++ b/docs/guides/find-limits.md @@ -40,10 +40,6 @@ Brainy picks `maxLimit` from the first of these that's available: Worked example: a 4 GB Cloud Run container picks priority 3 → `floor(4 GB × 0.25 / 25 KB) = floor(40 960) = 40 000` results. A 900 MB free-memory box on priority 4 gets `floor(900 MB / 25 KB) = ~36 000`. -The cap is fixed at construction and never changes at runtime. Query timing is recorded -for diagnostics only — a burst of slow queries cannot silently shrink the cap, and the -auto-detected tiers (3 and 4) never go below a floor of 10 000. - > **Calibration note.** Pre-7.30.2 used 100 KB per result instead of 25 KB, which produced caps that were 4× too tight for typical workloads (an 8 KB / result reality). 7.30.2 recalibrated to match observed entity sizes; existing `limit: 10_000` safety patterns now pass silently on any reasonably-sized box. ## What happens when you exceed the cap diff --git a/docs/guides/import-anything.md b/docs/guides/import-anything.md index b1bb15ef..e0cd94ed 100644 --- a/docs/guides/import-anything.md +++ b/docs/guides/import-anything.md @@ -300,10 +300,7 @@ await brain.import(data, { // Deduplication enableDeduplication: true, // Check for duplicate entities (default: true) deduplicationThreshold: 0.85, // Similarity threshold for duplicates (0-1, default: 0.85) - // Notes: false disables BOTH the inline merge and the background pass that - // runs ~5 min after the last import (merged duplicates are deleted). - // The inline pass auto-disables for imports >100 entities (O(n²) cost); - // the background pass still covers those unless the flag is false. + // Note: Auto-disabled for imports >100 entities // Performance chunkSize: 100, // Batch size for processing (default: varies by operation) diff --git a/docs/guides/import-quick-reference.md b/docs/guides/import-quick-reference.md index 7837d49e..5a850a86 100644 --- a/docs/guides/import-quick-reference.md +++ b/docs/guides/import-quick-reference.md @@ -86,22 +86,11 @@ await brain.import(file, { ```typescript await brain.import(file, { - enableDeduplication: true, // Check for duplicates (default: true) + enableDeduplication: true, // Check for duplicates (default: false) deduplicationThreshold: 0.85 // Similarity threshold (default: 0.85) }) ``` -Deduplication merges entities judged duplicates — the non-primary records are -**deleted**. Set `enableDeduplication: false` to disable it entirely: the flag -gates both the inline merge during import and the background pass that runs -about 5 minutes after the last import. - -```typescript -await brain.import(file, { - enableDeduplication: false // No merging, inline or background -}) -``` - ### Import Tracking Track and organize imports by project: diff --git a/docs/guides/inspection.md b/docs/guides/inspection.md index 240e81ae..015c5118 100644 --- a/docs/guides/inspection.md +++ b/docs/guides/inspection.md @@ -166,34 +166,6 @@ brainy inspect diff /data/brain-prod /data/brain-staging Sample-based — for a full diff, dump both with `inspect dump` and compare the JSONL. -## Auditing graph-read truth - -`brain.auditGraph()` (8.6.0+) proves — or disproves — that relationship reads -return canonical truth on a given brain, without mutating anything. It walks -every stored relationship record, asks the same read path your application -uses (`related()`, VFS `readdir`) with every visibility tier included, and -classifies every discrepancy: - -```typescript -const report = await brain.auditGraph() - -report.coherent // true = related()/readdir can be trusted on this brain -report.missingFromReadsCount // records the read path omits — stale index -report.danglingEndpointsCount // relationships whose endpoint entity is gone -report.readOnlyCount // read-path edges with NO stored record — ghosts -report.visibilityHiddenCount // internal/system edges hidden by design (not a fault) -``` - -Counts are always exact; the example lists (`missingFromReads`, -`danglingEndpoints`, `readOnlyVerbIds`) are capped at `maxExamples` -(default 100) and `truncatedExamples` says so when they are. - -Run it after any engine upgrade, restore, or migration. If it reports -discrepancies, run `brain.repairIndex()` and audit again — a `coherent` -report after the repair is the verified statement that the heal worked. -Cost: one relationship-record walk plus one indexed read per distinct -source entity — safe on a live brain. - ## Repairing a corrupted store If invariants fail and you suspect index corruption, `inspect repair` diff --git a/docs/guides/optimistic-concurrency.md b/docs/guides/optimistic-concurrency.md index 268bc5fa..a21a8af0 100644 --- a/docs/guides/optimistic-concurrency.md +++ b/docs/guides/optimistic-concurrency.md @@ -180,35 +180,3 @@ Brainy 8.0 has exactly two write-coordination counters, at two granularities: They compose: a `transact()` batch can carry per-entity `ifRev` checks *and* a whole-store `ifAtGeneration`; any failed check rejects the entire batch before anything is staged. Generations also power snapshots and time travel (`brain.now()`, `brain.asOf()`, `db.persist()`) — see the [consistency model](../concepts/consistency-model.md) and [Snapshots & Time Travel](./snapshots-and-time-travel.md). A snapshot or historical view captures each entity *including* its `_rev` at that moment, so reading the past and writing back with `ifRev` against the live state works exactly as you'd hope: the write fails if the entity moved since the state you copied from. - -## The transact envelope: batch size, budget, and bulk imports - -`transact()` applies its batch atomically under one commit — which means the whole batch -shares one **apply budget**. Since 8.7.0 the budget scales with the batch: -`max(30 s, opCount × 2 s)`, or exactly what you pass as `timeoutMs`. A tripped budget rolls -the entire batch back (nothing partial survives) and throws a retryable -`TransactionTimeoutError` that names the operation it stopped at, the batch size, and the -elapsed vs budgeted time — a diagnosis, not just a failure: - -``` -Transaction timed out at operation 41/120 ('add') — 246012ms elapsed, budget 240000ms. -The batch rolled back atomically; retry with a higher timeoutMs or a smaller batch. -``` - -Practical envelope guidance for bulk work: - -1. **Precompute embeddings outside the commit path.** Embedding inside `transact()` spends - the budget on model inference. Use `brain.embedBatch(texts)` and pass each vector via - the op's `vector` field — the commit then pays only storage costs, and a retried batch - never re-pays inference. (The win is *where* the inference happens, not raw embedding - throughput: on the default WASM engine, batch and sequential embedding measure - comparably, ~160 ms/text; native embedding providers may batch faster.) -2. **Chunk very large imports** into batches of a few hundred ops with one `transact()` - each. You lose whole-import atomicity but keep per-chunk atomicity, bounded memory, and - resumability — pair with `ifAbsent` upserts so a retried chunk is idempotent. -3. **Slow disks change the math, not the contract.** On network-attached storage a single - op can cost ~2 s (canonical write + fsync + index maintenance). The scaled default - absorbs that; pass an explicit `timeoutMs` only when you know better than the scale. -4. **`addMany`/`relateMany` are the convenience tier** — they chunk and batch-embed for - you, with per-item error reporting instead of batch atomicity. Choose by what you need: - atomic-all-or-nothing → `transact()`; resilient bulk load → `addMany`. diff --git a/docs/guides/snapshots-and-time-travel.md b/docs/guides/snapshots-and-time-travel.md index 490aecab..6f7b150a 100644 --- a/docs/guides/snapshots-and-time-travel.md +++ b/docs/guides/snapshots-and-time-travel.md @@ -9,7 +9,6 @@ description: Recipes for the Db API — instant backups with persist(), restore, next: - concepts/consistency-model - guides/optimistic-concurrency - - guides/external-backups --- # Snapshots & Time Travel @@ -42,10 +41,6 @@ bytes. Cross-device targets fall back to per-file byte copies, and persisting an in-memory brain serializes it to the same directory layout — a real, durable store. -> Archiving a brain directory with **external tools** (`tar`, `rsync`, `cp`)? -> Some index files are sparse and can explode to their apparent size under a -> naive copy — see [External Backups & Sparse Storage](/docs/guides/external-backups). - Two things to know: - `persist()` requires the view to still be the store's **latest** @@ -344,12 +339,8 @@ For per-entity write coordination (rather than whole-store history), the ## Keeping history bounded Under Model-B every write is a generation, so history can grow quickly — -Brainy auto-compacts at `close()` (time-bounded per pass) under the -**`retention`** knob (configured on the constructor). Since 8.9.0, `flush()` -never compacts: flushing is durability work and costs only what the current -window's writes cost, regardless of history backlog. A long-lived writer that -never closes keeps its history until its next explicit `compactHistory()` — -schedule one in your maintenance window if you run bounded retention: +Brainy auto-compacts on every `flush()`/`close()` under the **`retention`** +knob (configured on the constructor): ```typescript // Zero-config: ADAPTIVE — keep as much history as free disk/RAM allows, @@ -363,13 +354,10 @@ new Brainy({ retention: 'all' }) new Brainy({ retention: { maxGenerations: 1000, maxAge: 7 * 86_400_000, maxBytes: 512 * 1024 ** 2 } }) ``` -Reclaim manually at any time (the same caps, plus an optional per-pass time -budget for maintenance windows — an early stop is a consistent prefix and the -next pass resumes): +Reclaim manually at any time (the same caps): ```typescript await brain.compactHistory({ maxGenerations: 100, maxAge: 7 * 24 * 60 * 60 * 1000 }) -await brain.compactHistory({ maxBytes: 512 * 1024 ** 2, timeBudgetMs: 10_000 }) ``` Compaction never breaks a pinned read — record-sets are reclaimed only when diff --git a/docs/performance-envelopes.md b/docs/performance-envelopes.md deleted file mode 100644 index d29677e3..00000000 --- a/docs/performance-envelopes.md +++ /dev/null @@ -1,83 +0,0 @@ ---- -title: Performance Envelopes -slug: guides/performance-envelopes -public: true -category: guides -template: guide -order: 40 -description: Measured per-operation latency envelopes at stated scales — what to expect, on what hardware, and exactly how each number was produced. -next: - - guides/find-limits ---- - -# Performance Envelopes - -Every number on this page is **measured, never projected** — produced by the script -cited at the bottom, against the built package (the artifact you install), on the stated -hardware. Each entry says what was measured, at what scale, on which storage backend. -When a release touches a measured path, that operation is re-measured and this page -updates in the same release. - -Two scopes to keep straight: - -- **These envelopes are the pure-JS engine** (no native accelerator registered) on - filesystem storage. This is the floor every deployment gets from `npm install` alone. -- **Accelerated deployments** (the optional native provider) publish their own numbers — - this page never claims them. - -## Read operations - -Reads are where the architecture pays off: after the write path has done its indexing -work, queries answer from purpose-built indexes without scanning. - -| Operation | 1,000 entities | 10,000 entities | Notes | -|---|---|---|---| -| `get(id)` (warm) | p50 < 0.1ms | p50 < 0.1ms | served from cache/metadata index | -| `find` (metadata: indexed equality + range, limit 100) | p50 1.0ms · p95 1.8ms | p50 7.0ms · p95 8.9ms | column-store bitmap paths | -| `related(id)` (per-node adjacency) | p50 < 0.1ms · p95 0.2ms | p50 < 0.1ms | LSM adjacency index — O(degree), scale-independent | -| `find` (semantic: embed + HNSW, 1k docs) | p50 178ms · p95 393ms | — | dominated by WASM query embedding (measured on a machine under concurrent load — treat the p95 as an upper bound); the vector search itself is single-digit ms | - -## Write operations - -Under Model-B **every write is its own durable generation** — a single-op `add` pays -serialization, before-image staging, and fsync before it acks. That durability is priced -into the write path visibly, by design: - -| Operation | 1,000 entities | 10,000 entities | Notes | -|---|---|---|---| -| `add` (single-op) | p50 167ms · p95 171ms | p50 165ms · p95 172ms | full durable generation per write — flat across scale | -| `addMany` (bulk) | ~163ms/entity | ~187ms/entity | **currently per-item commits** — see the honest note below | -| `relateMany` | ~0.8ms/edge | ~0.9ms/edge | edges batch efficiently today | -| `flush` (steady-state, 1 pending write) | p50 8ms · p95 10ms | p50 45ms · p95 52ms | durability-only since 8.9.0 — cost no longer depends on history backlog or retention mode | - -**The honest note on bulk writes:** `addMany` today commits each item as its own -generation (the same durability as single-op `add`, serialized by the single-writer -lock), so bulk-load cost is N × single-op cost. Batched chunk commits (one generation -and one fsync window per chunk, as `removeMany` already does) are designed into the -unified-commit work on the current roadmap. Until that ships, size bulk imports -accordingly — 10k entities is minutes, not seconds, on filesystem storage. - -## Open / close - -| Operation | 1,000 entities | 10,000 entities | Notes | -|---|---|---|---| -| `open` (empty store) | ~560ms | ~190ms | includes embedder initialization | -| `open` (warm, populated, clean shutdown) | 763ms | 4.9s | pure-JS vector index load dominates and grows with entity count; the native accelerator exists precisely to remove this | -| `close` | bounded | bounded | auto-compaction pass is time-bounded (~5s max) since 8.9.0 | - -A store that was NOT cleanly closed pays index rebuilds on top of the warm-open -number (tens of seconds at 10k) — clean shutdown is worth engineering for. - -## How these were produced - -- **Hardware**: Intel Core i9-14900HX (32 threads), 62GB RAM, NVMe, Linux, Node v22. -- **Backend**: `storage: { type: 'filesystem' }`, pure JS (no native providers). -- **Embeddings**: deterministic stub for non-semantic ops (isolates engine cost); - the real WASM embedder for the semantic row (that's what you'll run). -- **Method**: p50/p95 over 50–200 samples per op against the built `dist/`; - the measuring script ships in the repo history and re-runs per release. - -Numbers on different hardware will differ; the *shape* (sub-2ms indexed reads, -~160ms embedding-bound semantic queries, durability-priced writes) is the envelope -you should hold your deployment against. If your measurements diverge from these -shapes by an order of magnitude, something is wrong — file it. diff --git a/package-lock.json b/package-lock.json index fb9262e9..183974d8 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "@soulcraft/brainy", - "version": "8.9.0", + "version": "8.5.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "@soulcraft/brainy", - "version": "8.9.0", + "version": "8.5.0", "license": "MIT", "dependencies": { "@msgpack/msgpack": "^3.1.2", diff --git a/package.json b/package.json index 7366ce98..92b1ba33 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "@soulcraft/brainy", - "version": "8.9.0", + "version": "8.5.0", "description": "Universal Knowledge Protocol™ - World's first Triple Intelligence database unifying vector, graph, and document search in one API. Stage 3 CANONICAL: 42 nouns × 127 verbs covering 96-97% of all human knowledge.", "main": "dist/index.js", "module": "dist/index.js", diff --git a/scripts/push-docs.js b/scripts/push-docs.js deleted file mode 100644 index 699d332b..00000000 --- a/scripts/push-docs.js +++ /dev/null @@ -1,116 +0,0 @@ -#!/usr/bin/env node -/** - * @module scripts/push-docs - * @description Push this repo's PUBLIC docs to the soulcraft.com docs ingest - * door after an npm publish (VENUE-DOCS-RELEASE-PUSH — retires the old - * build-time docs sync). - * - * Contract (mirrors the reference implementation on the serving side): - * POST {base}/api/docs/ingest - * headers: x-service-secret: $DOCS_INGEST_SECRET, Content-Type: application/json - * body: { docs: [{ slug, title, markdown, nav: { order, section } }] } - * batches of 10, idempotent per slug. - * - * A doc is public iff its frontmatter has `public: true` AND a `slug`. The - * frontmatter is stripped; `category` → nav.section, `order` → nav.order. - * - * Deliberately NOT pushed: the combined /docs landing index. It spans BOTH - * engine corpora (this repo's and the native accelerator's), so a per-repo - * push would clobber the union — the index is authored on the serving side. - * - * Env: DOCS_INGEST_SECRET (required), DOCS_INGEST_BASE (default - * https://soulcraft.com). Exits 0 with a LOUD warning when the secret is - * absent (the npm publish has already happened; the serving side runs its - * interim sync on request) and exits 1 when a push actually fails — the docs - * site would silently trail npm otherwise, and that must be visible. - */ -import * as fs from 'node:fs' -import * as path from 'node:path' - -const BASE = (process.env.DOCS_INGEST_BASE || 'https://soulcraft.com').replace(/\/+$/, '') -const SECRET = process.env.DOCS_INGEST_SECRET -const DOCS_DIR = path.join(path.dirname(new URL(import.meta.url).pathname), '..', 'docs') -const BATCH = 10 - -if (!SECRET) { - console.warn( - '⚠️ DOCS PUSH SKIPPED: DOCS_INGEST_SECRET is not set.\n' + - ' soulcraft.com/docs now TRAILS this npm release until docs are pushed.\n' + - ' Either export DOCS_INGEST_SECRET and re-run `node scripts/push-docs.js`,\n' + - ' or ping venue on VENUE-DOCS-RELEASE-PUSH for the interim sync.' - ) - process.exit(0) -} - -/** Minimal frontmatter split — returns [meta, body] or [null, raw]. */ -function parseFrontmatter(raw) { - const m = raw.match(/^---\n([\s\S]*?)\n---\n([\s\S]*)$/) - if (!m) return [null, raw] - const meta = {} - for (const line of m[1].split('\n')) { - const kv = line.match(/^(\w[\w-]*):\s*(.*)$/) - if (kv) meta[kv[1]] = kv[2].trim().replace(/^["']|["']$/g, '') - } - return [meta, m[2]] -} - -const docs = [] -;(function walk(dir) { - for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { - const full = path.join(dir, entry.name) - if (entry.isDirectory()) walk(full) - else if (entry.name.endsWith('.md')) { - const [meta, body] = parseFrontmatter(fs.readFileSync(full, 'utf-8')) - if (!meta || meta.public !== 'true' || !meta.slug) continue - docs.push({ - slug: meta.slug, - title: meta.title || meta.slug, - markdown: body.trim(), - nav: { - order: Number.parseInt(meta.order || '99', 10) || 99, - section: meta.category || 'guides' - } - }) - } - } -})(DOCS_DIR) - -if (docs.length === 0) { - console.error('❌ DOCS PUSH FAILED: zero public docs collected — refusing to push an empty corpus.') - process.exit(1) -} -docs.sort((a, b) => a.slug.localeCompare(b.slug)) -console.log(`Pushing ${docs.length} public docs to ${BASE}/api/docs/ingest …`) - -let failed = false -for (let i = 0; i < docs.length; i += BATCH) { - const batch = docs.slice(i, i + BATCH) - try { - const res = await fetch(`${BASE}/api/docs/ingest`, { - method: 'POST', - headers: { - 'x-service-secret': SECRET, - 'Content-Type': 'application/json', - 'User-Agent': 'brainy-docs-push/1.0' - }, - body: JSON.stringify({ docs: batch }), - signal: AbortSignal.timeout(120_000) - }) - if (!res.ok) { - throw new Error(`HTTP ${res.status}: ${(await res.text()).slice(0, 300)}`) - } - console.log(` batch ${i / BATCH + 1}: ${batch.map((d) => d.slug).join(', ')} → ok`) - } catch (err) { - failed = true - console.error(` batch ${i / BATCH + 1} FAILED: ${err instanceof Error ? err.message : err}`) - } -} - -if (failed) { - console.error( - '❌ DOCS PUSH INCOMPLETE — soulcraft.com/docs may trail npm. ' + - 'Re-run `node scripts/push-docs.js` or ping venue on VENUE-DOCS-RELEASE-PUSH.' - ) - process.exit(1) -} -console.log('✅ Docs pushed.') diff --git a/scripts/release.sh b/scripts/release.sh index 7d860564..0e6a9c43 100755 --- a/scripts/release.sh +++ b/scripts/release.sh @@ -196,18 +196,6 @@ else fi echo -e "${GREEN}✅ GitHub release created${NC}\n" -# Step 12: Push public docs to the soulcraft.com docs ingest door -# (VENUE-DOCS-RELEASE-PUSH). Skips with a loud warning when -# DOCS_INGEST_SECRET is unset; fails loudly (without undoing the publish — -# that already happened) when a push errors, so the docs site never -# silently trails npm. -echo -e "${BLUE}1️⃣2️⃣ Pushing public docs to soulcraft.com/docs...${NC}" -if node scripts/push-docs.js; then - echo -e "${GREEN}✅ Docs push step done${NC}\n" -else - echo -e "${RED}❌ Docs push FAILED — soulcraft.com/docs trails npm until re-run or interim sync${NC}\n" -fi - echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" echo -e "${GREEN}🎉 Release ${NEW_VERSION} complete!${NC}" echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" diff --git a/src/aggregation/AggregationIndex.ts b/src/aggregation/AggregationIndex.ts index f9382218..ecd16228 100644 --- a/src/aggregation/AggregationIndex.ts +++ b/src/aggregation/AggregationIndex.ts @@ -29,7 +29,6 @@ import { matchesMetadataFilter } from '../utils/metadataFilter.js' import { compareCodePoints } from '../utils/collation.js' import { bucketTimestamp } from './timeWindows.js' import { NounType } from '../types/graphTypes.js' -import { prodLog } from '../utils/logger.js' /** Persistence key for aggregate definitions */ const DEFINITIONS_KEY = '__aggregation_definitions__' @@ -88,18 +87,10 @@ function matchesSource(entity: Record, source: AggregateDefinit if (entity.service !== source.service) return false } - // Where filter — resolve each filtered field through resolveEntityField, - // the SAME single source of truth groupBy uses (top-level standard fields - // + custom metadata). Matching only the metadata sub-object made - // where:{subtype}/{visibility}/… a silent no-op: reserved fields never - // live in the custom bag, so those filters could never match anything. + // Metadata where filter — match against the entity's metadata sub-object if (source.where && Object.keys(source.where).length > 0) { - const e = entity as unknown as HNSWNounWithMetadata - const resolved: Record = {} - for (const key of Object.keys(source.where)) { - resolved[key] = resolveEntityField(e, key) - } - if (!matchesMetadataFilter(resolved, source.where)) return false + const metadata = (entity.metadata ?? entity) as Record + if (!matchesMetadataFilter(metadata, source.where)) return false } return true @@ -336,30 +327,6 @@ export class AggregationIndex { /** Track aggregates with stale MIN/MAX (need lazy recompute) */ private staleMinMax = new Map>() - /** Resolves when init() has finished loading persisted definitions/state. */ - private initPromise: Promise | null = null - - /** True once init() has settled (success or failure). */ - private initDone = false - - /** - * Aggregates registered by the app before init() finished loading persisted - * state, awaiting reconciliation: init() adopts the persisted state when the - * definition hash matches; anything left unadopted when init settles resolves - * to a backfill. Deciding backfill eagerly at define time was the boot-order - * bug that wiped valid persisted state on every restart — the synchronous - * defineAggregate() always beats the async init(). - */ - private pendingAdopt = new Set() - - /** - * In-flight rescan targets. While a name has a staging map, ALL - * contributions (the walk's and concurrent write hooks') land there instead - * of the live map; the live map keeps serving until {@link finishBackfill} - * swaps the staging map in atomically. - */ - private backfillStaging = new Map>() - constructor(storage: StorageAdapter, nativeProvider?: AggregationProvider) { this.storage = storage this.nativeProvider = nativeProvider @@ -369,128 +336,21 @@ export class AggregationIndex { /** * Initialize: load persisted definitions and state, detect changes, rebuild stale. - * - * Idempotent — repeated calls return the same promise. Definitions registered - * *before* this completes (the normal boot order: `defineAggregate()` is - * synchronous and always beats this async load) are reconciled rather than - * clobbered: the app's definition wins, and its persisted state is adopted - * when the definition hash matches — backfill happens only on a real change. */ - init(): Promise { - if (!this.initPromise) { - this.initPromise = this.loadPersisted().finally(() => { - this.resolvePendingAdoptToBackfill() - this.initDone = true - }) - } - return this.initPromise - } - - /** - * Await the persisted-state load (if one was started) and settle every - * pending adoption decision. After this resolves, `getPendingBackfills()` - * is authoritative: a name is listed iff it genuinely needs a rescan. - * Query paths must await this before consulting backfill state. - */ - async ready(): Promise { - if (this.initPromise) { - try { - await this.initPromise - } catch { - // The owner already surfaced the load failure loudly; backfill covers. - } - } - this.resolvePendingAdoptToBackfill() - } - - /** - * Any definition still awaiting state adoption has no persisted state to - * adopt (or init never ran / failed) — it must backfill. - */ - private resolvePendingAdoptToBackfill(): void { - if (this.pendingAdopt.size > 0) { - prodLog.info( - `[Aggregation] no adoptable persisted state for: ${Array.from(this.pendingAdopt).join(', ')} — flagged for backfill` - ) - } - for (const name of this.pendingAdopt) this.needsBackfill.add(name) - this.pendingAdopt.clear() - } - - /** - * May this persisted state be ADOPTED? When the store exposes its committed - * watermark, the state's `sourceGeneration` must EQUAL it: behind means - * later writes are missing from the state (unclean shutdown); ahead means - * it counts writes that no longer exist (e.g. a fact-log truncation on a - * copied store pulled the watermark back). Either way: one exact rescan, - * said out loud — never a silent adopt. Stores without the capability (and - * pre-stamp state on them) fall back to hash-only adoption. - */ - private stateGenerationAdoptable(name: string, stateData: unknown): boolean { - const committed = this.storage.committedGeneration?.() ?? null - if (committed === null) return true - const raw = (stateData as Record).sourceGeneration - const stamped = typeof raw === 'number' ? raw : null - if (stamped === committed) return true - prodLog.warn( - `[Aggregation] '${name}': persisted state is at generation ${stamped ?? 'unstamped'} ` + - `but the store's committed generation is ${committed} — rescanning instead of adopting` - ) - return false - } - - private async loadPersisted(): Promise { + async init(): Promise { // Load persisted definitions const savedDefs = await this.storage.getMetadata(DEFINITIONS_KEY) if (savedDefs && typeof savedDefs === 'object' && savedDefs.definitions) { const defs = savedDefs.definitions as Array for (const def of defs) { - const savedHash = def._hash || '' - - if (this.definitions.has(def.name)) { - // The app re-registered this aggregate before the load finished. - // The app's definition wins — never clobber it with the persisted - // copy. Adopt the persisted state when the definition is unchanged - // AND no write has landed for it yet (a landed write would be lost - // by adoption; the hook flips such names to backfill). - const appHash = this.definitionHashes.get(def.name) || '' - if (appHash === savedHash && this.pendingAdopt.has(def.name)) { - const stateData = await this.storage.getMetadata(`${STATE_KEY_PREFIX}${def.name}__`) - if ( - stateData && - stateData.groups && - this.stateGenerationAdoptable(def.name, stateData) - ) { - const groupMap = new Map() - for (const group of stateData.groups as AggregateGroupState[]) { - groupMap.set(serializeGroupKey(group.groupKey), group) - } - this.states.set(def.name, groupMap) - this.pendingAdopt.delete(def.name) - this.needsBackfill.delete(def.name) - prodLog.info( - `[Aggregation] '${def.name}': adopted persisted state (${groupMap.size} groups) — no rescan` - ) - } - // No/invalid persisted state: stays in pendingAdopt and resolves - // to backfill when init settles. - } - continue - } - - // Not registered this session — restore definition + state from - // persistence. this.definitions.set(def.name, def) const currentHash = hashDefinition(def) + const savedHash = def._hash || '' + // Load persisted state const stateData = await this.storage.getMetadata(`${STATE_KEY_PREFIX}${def.name}__`) - if ( - stateData && - stateData.groups && - savedHash === currentHash && - this.stateGenerationAdoptable(def.name, stateData) - ) { + if (stateData && stateData.groups && savedHash === currentHash) { // Definition unchanged — load state const groupMap = new Map() for (const group of stateData.groups as AggregateGroupState[]) { @@ -498,10 +358,6 @@ export class AggregationIndex { groupMap.set(serialized, group) } this.states.set(def.name, groupMap) - this.needsBackfill.delete(def.name) - prodLog.info( - `[Aggregation] '${def.name}': restored definition + adopted persisted state (${groupMap.size} groups)` - ) } else { // Definition changed or no saved state — start fresh and backfill from // existing entities (the owner drains needsBackfill on first query). @@ -542,22 +398,14 @@ export class AggregationIndex { })) await this.storage.saveMetadata(DEFINITIONS_KEY, { definitions: defsToSave }) - // Persist dirty states, stamped with the committed generation they - // reflect. The stamp is what makes reopen-adoption verifiable: state at a - // different generation than the store's committed watermark is stale (an - // unclean shutdown after later writes) or over-counts (a fact-log - // truncation on a copied store pulled the watermark BACK below the - // stamp) — either way the answer is one exact rescan, never a silent - // adopt. Read the generation after collecting groups so any racing - // commit resolves toward rescan, not wrong-adopt. + // Persist dirty states for (const name of this.dirty) { const stateMap = this.states.get(name) if (stateMap) { const groups = Array.from(stateMap.values()) - const sourceGeneration = this.storage.committedGeneration?.() ?? null await this.storage.saveMetadata( `${STATE_KEY_PREFIX}${name}__`, - sourceGeneration === null ? { groups } : { groups, sourceGeneration } + { groups } ) } } @@ -604,19 +452,10 @@ export class AggregationIndex { this.definitions.set(def.name, def) this.definitionHashes.set(def.name, newHash) - // First sight this session, before init() settled: defer the backfill - // decision — init() adopts the persisted state on hash match, and anything - // left unadopted resolves to backfill. Deciding eagerly here wiped valid - // persisted state on every restart. - if (!this.states.has(def.name) && !this.initDone) { - this.states.set(def.name, new Map()) - this.pendingAdopt.add(def.name) - } // Reset state if definition changed or doesn't exist yet, and flag it for // backfill so already-stored entities are counted (write-time hooks only see // future writes). The owner drains this on the next query via getPendingBackfills(). - else if (!this.states.has(def.name) || (oldHash && oldHash !== newHash)) { - this.pendingAdopt.delete(def.name) + if (!this.states.has(def.name) || (oldHash && oldHash !== newHash)) { this.states.set(def.name, new Map()) this.needsBackfill.add(def.name) } @@ -637,8 +476,6 @@ export class AggregationIndex { this.definitionHashes.delete(name) this.states.delete(name) this.staleMinMax.delete(name) - this.pendingAdopt.delete(name) - this.needsBackfill.delete(name) // Notify native provider if (this.nativeProvider?.removeAggregate) { @@ -676,17 +513,9 @@ export class AggregationIndex { return Array.from(this.needsBackfill) } - /** - * Begin a rescan into a STAGING map. The live state is not touched — it - * keeps serving (possibly stale, but flagged pending) until the rescan - * completes and swaps in atomically. A mid-walk failure drops the staging - * map via {@link abortBackfill} and loses nothing: wiping live state before - * a scan that could throw was the destructive-before-durable defect. - * Contributions (walk + concurrent write hooks) land in staging while it - * exists, so the swapped-in result reflects writes that raced the walk. - */ + /** Clear an aggregate's state so a full rescan cannot double-count. */ beginBackfill(name: string): void { - this.backfillStaging.set(name, new Map()) + this.states.set(name, new Map()) // Reset native provider state for this aggregate too, if present. const def = this.definitions.get(name) if (def && this.nativeProvider?.removeAggregate && this.nativeProvider?.defineAggregate) { @@ -695,15 +524,6 @@ export class AggregationIndex { } } - /** - * Abandon an in-flight rescan after a failure: drop the staging map, keep - * the live state serving, leave the aggregate flagged as pending so a later - * attempt rescans. The failure itself must be surfaced loudly by the owner. - */ - abortBackfill(name: string): void { - this.backfillStaging.delete(name) - } - /** Feed one already-stored entity into a single aggregate during backfill. */ backfillEntity(name: string, entity: Record): void { if (isAggregateEntity(entity)) return @@ -717,33 +537,14 @@ export class AggregationIndex { } } - /** Swap the rebuilt staging state in atomically; persists on next flush(). */ + /** Mark an aggregate's backfill complete; rebuilt state persists on next flush(). */ finishBackfill(name: string): void { - const staged = this.backfillStaging.get(name) - if (staged) { - this.states.set(name, staged) - this.backfillStaging.delete(name) - } this.needsBackfill.delete(name) this.dirty.add(name) } // ============= Write-Time Hooks ============= - /** - * A write is landing for an aggregate whose persisted-state adoption is still - * pending — adopting after this write would lose its contribution. Settle the - * decision now: an exact rescan instead of adoption. The window is the few - * milliseconds between a boot-time defineAggregate() and init() completing, - * so this rarely fires; when it does, correctness wins over the walk. - */ - private resolveAdoptOnWrite(name: string): void { - if (this.pendingAdopt.has(name)) { - this.pendingAdopt.delete(name) - this.needsBackfill.add(name) - } - } - /** * Called when an entity is added. Updates all matching aggregates. */ @@ -752,7 +553,6 @@ export class AggregationIndex { for (const [name, def] of this.definitions) { if (!matchesSource(entity, def.source)) continue - this.resolveAdoptOnWrite(name) if (this.nativeProvider) { const results = this.nativeProvider.incrementalUpdate(name, def, entity, 'add') @@ -779,10 +579,6 @@ export class AggregationIndex { const oldMatches = matchesSource(oldEntity, def.source) const newMatches = matchesSource(newEntity, def.source) - if (oldMatches || newMatches) { - this.resolveAdoptOnWrite(name) - } - if (this.nativeProvider && (oldMatches || newMatches)) { const results = this.nativeProvider.incrementalUpdate(name, def, newEntity, 'update', oldEntity) this.applyNativeResults(name, results) @@ -809,7 +605,6 @@ export class AggregationIndex { for (const [name, def] of this.definitions) { if (!matchesSource(entity, def.source)) continue - this.resolveAdoptOnWrite(name) if (this.nativeProvider) { const results = this.nativeProvider.incrementalUpdate(name, def, entity, 'delete') @@ -962,7 +757,7 @@ export class AggregationIndex { def: AggregateDefinition, entity: Record ): void { - const stateMap = (this.backfillStaging.get(aggName) ?? this.states.get(aggName))! + const stateMap = this.states.get(aggName)! // Fan out: an unnest dimension makes one entity contribute to several groups. for (const groupKey of computeGroupKeys(entity, def.groupBy)) { @@ -1020,7 +815,7 @@ export class AggregationIndex { def: AggregateDefinition, entity: Record ): void { - const stateMap = (this.backfillStaging.get(aggName) ?? this.states.get(aggName))! + const stateMap = this.states.get(aggName)! // Fan out: reverse the entity's contribution from every group it joined. for (const groupKey of computeGroupKeys(entity, def.groupBy)) { @@ -1076,7 +871,7 @@ export class AggregationIndex { * Apply results from native provider back into the state maps. */ private applyNativeResults(aggName: string, results: AggregateGroupState[]): void { - const stateMap = (this.backfillStaging.get(aggName) ?? this.states.get(aggName))! + const stateMap = this.states.get(aggName)! for (const group of results) { const serialized = serializeGroupKey(group.groupKey) stateMap.set(serialized, group) diff --git a/src/brainy.ts b/src/brainy.ts index 9ab3acee..d5d4bc33 100644 --- a/src/brainy.ts +++ b/src/brainy.ts @@ -46,10 +46,8 @@ import { pageRank, MinHeap } from './graph/analyticsFallback.js' -import { runGraphAudit, type GraphAuditReport } from './graph/graphAudit.js' import { createPipeline } from './streaming/pipeline.js' import { configureLogger, LogLevel, prodLog } from './utils/logger.js' -import { warnOnLowOsLimits } from './utils/osLimits.js' import { setGlobalCache } from './utils/unifiedCache.js' import type { UnifiedCache } from './utils/unifiedCache.js' import { rankIndicesByScore, reorderByIndices } from './utils/resultRanking.js' @@ -72,7 +70,6 @@ import type { } from './plugin.js' import { ConnectionsCodec } from './hnsw/connectionsCodec.js' import { TransactionManager } from './transaction/TransactionManager.js' -import { transactTimeoutBudget } from './transaction/Transaction.js' import { RevisionConflictError } from './transaction/RevisionConflictError.js' import { EntityNotFoundError, RelationNotFoundError } from './errors/notFound.js' import { @@ -192,7 +189,6 @@ import { MemoryStorage } from './storage/adapters/memoryStorage.js' import type { CompactHistoryOptions, CompactHistoryResult, - HistoryStats, TransactOptions, TransactReceipt, TxLogEntry, @@ -388,23 +384,6 @@ class InsertPreconditionExistsSignal extends Error { */ export type IndexFamily = 'vector' | 'metadata' | 'graph' -/** - * How long a failed aggregation-backfill walk suppresses fresh walk attempts. - * Within the window, queries rethrow the recorded failure instantly (loud, - * cheap); after it, one new attempt is allowed. Bounds the damage of a - * caller-side tight retry loop against a deterministically-failing store. - */ -const AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS = 30_000 - -/** - * Time budget for the auto-compaction pass at close() (8.9.0). Bounds how long - * a clean shutdown spends reclaiming history backlog — an early stop is a - * consistent prefix and the next close/explicit pass resumes. Explicit - * `compactHistory()` calls are unbounded unless the caller passes their own - * `timeBudgetMs` (maintenance windows choose their own budgets). - */ -const CLOSE_COMPACTION_BUDGET_MS = 5_000 - /** * The main Brainy class - Clean, Beautiful, Powerful * REAL IMPLEMENTATION - No stubs, no mocks @@ -619,11 +598,6 @@ export class Brainy implements BrainyInterface { private _hub?: IntegrationHub // Integration Hub for external tools private _pendingMigrationRunner?: MigrationRunner // Deferred migration runner for large datasets private _aggregationIndex?: AggregationIndex // Incremental aggregation engine - private _aggregationBackfillFlight: Promise | null = null // Single-flight backfill walk - // A failed walk latches its error: retries within the cooldown rethrow it - // instantly instead of re-walking, so a tight caller-side retry loop costs - // one loud error per query, never a full store walk per query. - private _aggregationBackfillFailure: { at: number; error: Error } | null = null private _materializer?: AggregateMaterializer // Debounced materialization of aggregate results /** * Fields registered via `brain.trackField()` — drives optional value validation on @@ -946,13 +920,6 @@ export class Brainy implements BrainyInterface { this.storage = await this.setupStorage() await this.storage.init() - // OS-limit detection (once per process, Linux-only, measurement-only): - // warn NOW about RLIMIT_NOFILE / vm.max_map_count values that will bite - // at pool scale, instead of letting the operator meet them as EMFILE or - // a failed mmap deep inside an index open. Fire-and-forget — the check - // never affects open. - void warnOnLowOsLimits() - // Acquire the writer lock for filesystem (and other locking-capable) backends. // Skipped in reader mode and on backends that don't support multi-process locking. // Throws if another live writer holds the directory (unless force: true). @@ -1214,22 +1181,6 @@ export class Brainy implements BrainyInterface { `(storage committed: ${committed}) — provider replays the gap per ` + `the post-commit applier contract` ) - } else if (providerGen > committed) { - // The AHEAD direction is incoherence, not a replay gap: the provider's - // persisted index claims writes the store no longer has — the signature - // of a torn copy or a log truncation that pulled the committed - // watermark back (crash recovery, byte-copy of a live store). A replay - // can never converge on it and index answers may reference vanished - // writes. Name it loudly at open so it is never diagnosed from a - // silent journal; the provider's own coherence check / heal walk (or - // brain.repairIndex()) is the cure. - prodLog.warn( - `[Brainy] Versioned index provider is AHEAD of the store: provider ` + - `generation ${providerGen} vs committed ${committed}. This store was ` + - `likely copied from a live service or truncated during crash recovery. ` + - `Derived-index answers may reference rolled-back writes until the ` + - `provider heals from canonical (brain.repairIndex() forces it).` - ) } } @@ -1391,13 +1342,6 @@ export class Brainy implements BrainyInterface { if (this._readyReject) { this._readyReject(error instanceof Error ? error : new Error(String(error))) } - // Machine-readable init failures pass through UNWRAPPED — the writer-lock - // conflict documents an err.code/err.lockInfo contract ("callers detect - // this case via err.code"), and wrapping in a fresh Error silently - // stripped both, leaving consumers only a message to regex against. - if (error instanceof Error && (error as Error & { code?: string }).code === 'BRAINY_WRITER_LOCKED') { - throw error - } throw new Error(`Failed to initialize Brainy: ${error}`) } } @@ -1778,11 +1722,7 @@ export class Brainy implements BrainyInterface { captureAndCheck({ nouns, verbs } as CommitBeforeImages) } await this.generationStore.runWithoutGeneration(() => - this.transactionManager.executeTransaction(run, { - timeout: transactTimeoutBudget( - (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0) - ) - }) + this.transactionManager.executeTransaction(run) ) const timestamp = Date.now() // Bootstrap writes are not generation-stamped; emit without one. @@ -1794,12 +1734,7 @@ export class Brainy implements BrainyInterface { receipt = await this.generationStore.commitSingleOp({ touched, precommit: captureAndCheck, - execute: () => - this.transactionManager.executeTransaction(run, { - timeout: transactTimeoutBudget( - (touched.nouns?.length ?? 0) + (touched.verbs?.length ?? 0) - ) - }) + execute: () => this.transactionManager.executeTransaction(run) }) } catch (err) { // A failed rollback that left the store inconsistent (a remove/update @@ -1892,29 +1827,6 @@ export class Brainy implements BrainyInterface { } } - /** - * @description Build the AGGREGATION view of an entity from a stored flat - * metadata record — EVERY reserved field mapped to its top-level entity - * name (stored `noun` → `type`), custom metadata in `metadata`. This must - * mirror the add-path `entityForIndexing` shape exactly: the aggregation - * engine resolves groupBy/where fields via `resolveEntityField` - * (top-level standard fields + custom metadata), so a view that drops a - * reserved field makes every aggregate grouped by that field decrement a - * group that does not exist — counts then drift upward forever after - * deletes (SELF-AGGREGATE-DELETE-DRIFT). Do not hand-roll subsets of this. - * @param record - The stored flat metadata record (before-image or pre-delete read). - * @returns The full-fidelity entity view for aggregation hooks. - */ - private entityForAggFromRawRecord(record: Record): Record { - const { reserved, custom } = splitNounMetadataRecord(record) - const { noun, ...rest } = reserved - return { - type: noun, - ...rest, - metadata: custom - } - } - /** * @description Add an entity (noun) to the brain. Embeds `data` into a vector and * indexes the entity across all three intelligences — vector similarity, graph @@ -3157,16 +3069,15 @@ export class Brainy implements BrainyInterface { ] : undefined) - // Aggregation hook (outside transaction — derived data). `existing` is - // the full get() view — every reserved field top-level — and must be - // passed whole: a subset view makes the old-side decrement miss any - // reserved-field group (update would then double-count it). + // Aggregation hook (outside transaction — derived data) if (this._aggregationIndex) { - this._aggregationIndex.onEntityUpdated( - params.id, - entityForIndexing, - existing as unknown as Record - ) + const oldEntityForAgg = { + type: existing.type, + service: existing.service, + data: existing.data, + metadata: existing.metadata + } + this._aggregationIndex.onEntityUpdated(params.id, entityForIndexing, oldEntityForAgg) } } @@ -3278,15 +3189,19 @@ export class Brainy implements BrainyInterface { ] : undefined) - // Aggregation hook (outside transaction — derived data). The view must - // carry EVERY reserved field top-level (not a subset): a groupBy on - // subtype/visibility/etc. otherwise decrements a nonexistent group and - // the real count never comes down. + // Aggregation hook (outside transaction — derived data) if (this._aggregationIndex && metadata) { - this._aggregationIndex.onEntityDeleted( - id, - this.entityForAggFromRawRecord(metadata as Record) - ) + // Reconstruct entity-like object from stored metadata via the + // canonical reserved/custom split (the hand-rolled destructure here + // missed subtype/_rev, leaking them into the aggregation view). + const { reserved, custom } = splitNounMetadataRecord(metadata) + const entityForAgg = { + type: reserved.noun, + service: reserved.service, + data: reserved.data, + metadata: custom + } + this._aggregationIndex.onEntityDeleted(id, entityForAgg) } } @@ -5868,10 +5783,6 @@ export class Brainy implements BrainyInterface { ): Promise { await this.ensureInitialized() this.ensureAggregationIndex() - // Persisted definitions load asynchronously — wait for them before deciding - // the aggregate doesn't exist (an app that relies on persisted definitions - // without re-defining at boot would otherwise race a spurious throw here). - await this._aggregationIndex!.ready() if (!this._aggregationIndex!.hasAggregate(name)) { throw new Error(`Aggregate '${name}' is not defined. Call defineAggregate() first.`) } @@ -6710,13 +6621,10 @@ export class Brainy implements BrainyInterface { /** * Add multiple entities in a single batch operation * - * Uses batch embedding (embedBatch) to pre-compute all vectors before any - * storage write, keeping model inference out of the per-item write path. - * (On the default WASM engine, batch throughput is comparable to sequential - * embed() calls — measured ~160 ms/text either way; a native embedding - * provider may batch faster.) Automatically adapts batch size and - * parallelism to the storage adapter (e.g., smaller batches for cloud - * storage). + * Uses batch embedding (embedBatch) to pre-compute all vectors in a single + * WASM forward pass instead of N individual embed() calls, providing 5-10x + * speedup on bulk inserts. Automatically adapts batch size and parallelism + * to the storage adapter (e.g., smaller batches for cloud storage). * * @param params - Batch add parameters * @param params.items - Array of AddParams (same shape as brain.add()) @@ -6914,30 +6822,6 @@ export class Brainy implements BrainyInterface { this.assertWritable('removeMany') await this.ensureInitialized() - // Loud selector validation: a call with no usable selector used to - // resolve successfully having deleted NOTHING (total: 0) — the classic - // silent no-op being a bare array passed positionally - // (removeMany([id]) instead of removeMany({ ids: [id] })). The caller - // believes the delete happened; every count derived afterwards is "wrong" - // while the engine was never even asked. Refuse instead. - if (Array.isArray(params)) { - throw new Error( - `removeMany() takes a params object, not a bare array — use removeMany({ ids: [...] })` - ) - } - if (!params || (!params.ids && !params.type && !params.where)) { - throw new Error( - `removeMany() requires a selector: { ids } and/or { type, where }. ` + - `An empty selector would silently delete nothing — refusing.` - ) - } - if (params.ids && params.ids.length === 0) { - throw new Error( - `removeMany() received ids: [] — an empty id list deletes nothing. ` + - `Pass the ids to delete, or omit ids and select by { type, where }.` - ) - } - // Determine what to delete let idsToDelete: string[] = [] @@ -7879,17 +7763,11 @@ export class Brainy implements BrainyInterface { ifAtGeneration: options?.ifAtGeneration, precommit: casPrecommit, execute: async () => { - await this.transactionManager.executeTransaction( - async (tx) => { - for (const operation of plan.operations) { - tx.addOperation(operation) - } - }, - // Budget scales with the batch (or the caller's explicit - // timeoutMs): a flat 30s cap silently limited honest bulk work - // to ~15 ops on network disks (~2s/op measured in the field). - { timeout: transactTimeoutBudget(plan.operations.length, options?.timeoutMs) } - ) + await this.transactionManager.executeTransaction(async (tx) => { + for (const operation of plan.operations) { + tx.addOperation(operation) + } + }) } })) } catch (err) { @@ -8265,72 +8143,6 @@ export class Brainy implements BrainyInterface { return this.generationStore.compact(options) } - /** - * @description Repack cold generation history into sealed segments — - * re-representation, never deletion: every record and delta stays readable - * (`asOf()` unchanged); the physical file count drops by orders of - * magnitude. Runs automatically (time-bounded) at `close()`; call this for - * explicit maintenance windows on long-lived writers. The ONLY history - * transform permitted under the archival profile (`retention: 'all'`). - * @param options - `timeBudgetMs` bounds the pass (early stop = consistent - * prefix, next pass resumes); `batchGenerations` sizes each fold. - * @returns Folded generation count and segments created. - */ - async repackHistory(options?: { - timeBudgetMs?: number - batchGenerations?: number - }): Promise<{ foldedGenerations: number; segmentsCreated: number }> { - this.assertWritable('repackHistory') - await this.ensureInitialized() - await this.generationStore.flushPendingSingleOps() - return this.generationStore.repackHistory(options) - } - - /** - * @description Read-only generational-history footprint for fleet audits: - * generation count, total on-disk bytes, generation/timestamp range, the - * compaction horizon, and the retention policy in force. Touches no data and - * changes nothing. First call pays one walk over committed deltas to seed - * the running byte total (subsequent calls — and every adaptive retention - * check — are then O(1)). - * - * A pool operator's exposure check is one call per brain: - * @example - * const stats = await brain.historyStats() - * console.log(`${stats.generations} generations, ${stats.bytes} bytes, mode=${stats.retentionMode}`) - */ - async historyStats(): Promise { - await this.ensureInitialized() - const stats = await this.generationStore.historyStats() - const policy = this.resolveRetentionPolicy() - return { - ...stats, - retentionMode: policy.mode, - effectiveBudgetBytes: - policy.mode === 'adaptive' - ? this.adaptiveHistoryBudgetBytes(policy.budgetBytes) - : null - } - } - - /** - * @description A deterministic content digest of the generation log through - * `g` (D8 — gate-to-generation provenance): identical history produces the - * identical digest on any machine; divergence produces a different one. - * Release gates and suite verdicts pin `{generation, digest}` and verify - * both at execution time instead of pinning a git commit. O(segments + - * live-tier window), never O(all generations). Throws `RangeError` out of - * range and `GenerationCompactedError` below the horizon — a gate can - * never silently pin reclaimed history. - * @example - * const gate = { generation: brain.generation(), digest: await brain.generationDigest(brain.generation()) } - */ - async generationDigest(g: number): Promise { - await this.ensureInitialized() - await this.generationStore.flushPendingSingleOps() - return this.generationStore.generationDigest(g) - } - /** * @description Drive the adaptive retention byte budget at runtime — the * settable input a machine-level coordinator (e.g. cor's `ResourceManager`, @@ -8410,24 +8222,19 @@ export class Brainy implements BrainyInterface { /** * @description Run history compaction under the resolved `retention` policy - * when `autoCompact` is on (the default). Invoked from `close()` ONLY - * (8.9.0) — flush() is durability work and never pays maintenance costs; a - * production deployment measured reclaim-on-flush blocking single writes - * for 25-191s under memory pressure. Long-lived writers that never close - * accumulate history until their next explicit `compactHistory()` — the - * documented trade: predictable writes, explicit maintenance. + * when `autoCompact` is on (the default). Invoked from `flush()` and + * `close()` so generational record-sets cannot accumulate unbounded across a + * long-lived writer's lifetime. * * - `'all'` → returns without reclaiming (index compaction for speed still * runs elsewhere; history is decoupled and kept). * - `'adaptive'` → reclaim oldest-unpinned history down to the byte budget. * - `'explicit'` → apply the supplied `maxGenerations`/`maxAge`/`maxBytes` caps. * - * Every auto pass is TIME-BOUNDED ({@link CLOSE_COMPACTION_BUDGET_MS}) so a - * large backlog can never stall a clean shutdown; the next pass resumes. * Read-only instances and an explicit `autoCompact: false` skip silently. * Pinned generations are never reclaimed ({@link GenerationStore.compact}). * Failures are logged and swallowed — compaction is housekeeping and must - * never fail a clean shutdown. + * never fail a flush or a clean shutdown. */ private async autoCompactHistory(): Promise { // Nothing to compact on a read-only instance or before init wired up the @@ -8443,16 +8250,12 @@ export class Brainy implements BrainyInterface { if (policy.mode === 'adaptive') { const budget = this.adaptiveHistoryBudgetBytes(policy.budgetBytes) if (budget === Infinity) return // no pressure signal → keep everything this pass - await this.generationStore.compact({ - maxBytes: budget, - timeBudgetMs: CLOSE_COMPACTION_BUDGET_MS - }) + await this.generationStore.compact({ maxBytes: budget }) } else { await this.generationStore.compact({ maxGenerations: policy.maxGenerations, maxAge: policy.maxAge, - maxBytes: policy.maxBytes, - timeBudgetMs: CLOSE_COMPACTION_BUDGET_MS + maxBytes: policy.maxBytes }) } } catch (error) { @@ -8477,14 +8280,6 @@ export class Brainy implements BrainyInterface { * replacement. Release them first; a warning is logged when live pins * exist. * - * NON-DESTRUCTIVE STAGING + SPARSE-AWARE: the snapshot is copied into a - * staging area BEFORE any live data is touched (all-zero blocks stay - * holes, so a sparse mmap store restores at its true allocated size — not - * its apparent size). Any copy failure, ENOSPC included, removes only the - * staging and throws; the live store is untouched. Only after the copy - * succeeds does an atomic per-entry swap move it into place; a crash - * mid-swap completes FORWARD on the next open. - * * @param path - Snapshot directory to restore from. * @param options - Must be `{ confirm: true }` — an explicit acknowledgment * that current state is destroyed. @@ -9489,9 +9284,12 @@ export class Brainy implements BrainyInterface { ) plan.touchedNouns.push(params.id) - // The full planGetEntity view, passed whole — a subset view makes the - // old-side decrement miss reserved-field groups (double-count on update). - const oldEntityForAgg = existing as unknown as Record + const oldEntityForAgg = { + type: existing.type, + service: existing.service, + data: existing.data, + metadata: existing.metadata + } plan.postCommit.push(() => { if (this._aggregationIndex) { this._aggregationIndex.onEntityUpdated(params.id, entityForIndexing, oldEntityForAgg) @@ -9612,9 +9410,14 @@ export class Brainy implements BrainyInterface { } if (metadata) { - // Mirror of remove()'s aggregation hook — the FULL reserved view, so - // reserved-field groupBy decrements find their group. - const entityForAgg = this.entityForAggFromRawRecord(metadata as Record) + // Canonical reserved/custom split — mirror of remove()'s aggregation hook. + const { reserved, custom } = splitNounMetadataRecord(metadata) + const entityForAgg = { + type: reserved.noun, + service: reserved.service, + data: reserved.data, + metadata: custom + } plan.postCommit.push(() => { if (this._aggregationIndex) { this._aggregationIndex.onEntityDeleted(id, entityForAgg) @@ -10249,30 +10052,6 @@ export class Brainy implements BrainyInterface { return await coordinator.import(source as Buffer | string | object, options) } - /** Brain-owned background deduplicator (lazy; see getBackgroundDeduplicator). */ - private _backgroundDedup?: import('./import/BackgroundDeduplicator.js').BackgroundDeduplicator - - /** - * The single brain-owned BackgroundDeduplicator, lazily constructed. - * - * Ownership matters here: the post-import dedup timer must outlive the - * per-call ImportCoordinator but never the brain. One instance per brain - * restores the intended cross-import debounce (per-coordinator instances - * each armed their own timer, so the "debounce" never spanned imports) and - * gives close() a handle to cancel pending work — a delete pass must never - * fire against a closed brain. - * @internal - */ - async getBackgroundDeduplicator(): Promise< - import('./import/BackgroundDeduplicator.js').BackgroundDeduplicator - > { - if (!this._backgroundDedup) { - const { BackgroundDeduplicator } = await import('./import/BackgroundDeduplicator.js') - this._backgroundDedup = new BackgroundDeduplicator(this) - } - return this._backgroundDedup - } - /** * Virtual File System API - Knowledge Operating System * @@ -10597,14 +10376,12 @@ export class Brainy implements BrainyInterface { this.generationStore.persistCounterNow() ]) - // NOTE (8.9.0): flush() no longer compacts history. Flush is DURABILITY - // work — it must cost what this window's deltas cost, never what the - // history backlog costs. Auto-compaction (a MAINTENANCE concern) runs at - // close() and via explicit compactHistory(); a production deployment - // measured reclaim-on-flush stalling single writes for 25-191s under - // memory pressure, which is exactly the class this separation ends. + // 6. Auto-compact generational history per config.retention (default on). + // Runs after the flush so durable state is in place; respects live + // Db pins and an explicit autoCompact: false. + await this.autoCompactHistory() - // 6. Stamp the entity tree: which source generation the canonical tree + // 7. Stamp the entity tree: which source generation the canonical tree // reflects + the rollup invariants that verify it whole (the counters // persisted in step 1). Written at flush boundaries — the tree tracks // every commit by construction, so the stamp is a durable checkpoint, @@ -15671,88 +15448,6 @@ export class Brainy implements BrainyInterface { ) } - /** - * Read-only graph-truth audit — proves (or disproves) that relationship - * reads return canonical truth on THIS brain, and classifies every - * discrepancy into its failure family: - * - * - `missingFromReads` — canonical verb records the read path omits - * (PRESENT BUT INVISIBLE: adjacency/membership staleness) - * - `danglingEndpoints` — verbs whose endpoint entity is gone (SCAR class) - * - `readOnlyVerbIds` — read-path edges with no canonical record (GHOSTS) - * - * Design-hidden edges (internal/system visibility) are counted separately — - * the audit reads with every visibility tier included, so intentional - * hiding is never misclassified as index loss. Mutates nothing; safe on a - * live brain (cost: one canonical walk + one indexed read per source). - * Run it after any engine upgrade, restore, or migration; a `coherent` - * report is the verified statement that `related()`/`readdir` can be - * trusted. If it reports discrepancies, `repairIndex()` is the sanctioned - * heal — re-run the audit afterwards to prove the repair. - * - * @param options.maxExamples - Cap per example list (counts stay exact). Default 100. - * @returns The full audit report; also narrated via logs (loud on incoherence). - * @since 8.6.0 - */ - async auditGraph(options: { maxExamples?: number } = {}): Promise { - await this.ensureInitialized({ needs: ['graph'] }) - - const PAGE = 1000 - return runGraphAudit( - { - eachNounId: async (consume) => { - let cursor: string | undefined - let offset = 0 - for (;;) { - const page = await (this.storage as unknown as { - getNounIdsWithPagination(o: { - limit: number - offset?: number - cursor?: string - }): Promise<{ ids: string[]; hasMore: boolean; nextCursor?: string }> - }).getNounIdsWithPagination( - cursor ? { limit: PAGE, cursor } : { limit: PAGE, offset } - ) - for (const id of page.ids) consume(id) - if (!page.hasMore || page.ids.length === 0) break - if (page.nextCursor) cursor = page.nextCursor - else offset += page.ids.length - } - }, - eachVerb: async (consume) => { - let cursor: string | undefined - let offset = 0 - for (;;) { - const page = await this.storage.getVerbs({ - pagination: cursor ? { limit: PAGE, cursor } : { limit: PAGE, offset } - }) - for (const verb of page.items) { - const v = verb as unknown as Record - consume({ - id: String(v.id), - type: String(v.verb ?? 'unknown'), - sourceId: String(v.sourceId), - targetId: String(v.targetId), - visibility: typeof v.visibility === 'string' ? v.visibility : undefined - }) - } - if (!page.hasMore || page.items.length === 0) break - if (page.nextCursor) cursor = page.nextCursor - else offset += page.items.length - } - }, - readRelationsFrom: async (sourceId) => - this.related({ - from: sourceId, - includeInternal: true, - includeSystem: true, - limit: 100000 - }) - }, - options - ) - } - async repairIndex(): Promise { await this.ensureInitialized() @@ -16107,107 +15802,29 @@ export class Brainy implements BrainyInterface { */ private async backfillAggregateIfNeeded(name: string): Promise { const index = this._aggregationIndex - if (!index) return + if (!index || !index.getPendingBackfills().includes(name)) return - // Persisted-state adoption happens inside ready(); after it resolves the - // pending-backfill set is authoritative (an unchanged definition with valid - // persisted state is NOT listed — no walk at all on a clean reopen). - await index.ready() + index.beginBackfill(name) - // Single-flight: concurrent queries share ONE walk instead of each wiping - // the others' partial state and starting their own (the stampede that kept - // a busy store from ever converging). The loop covers the rare case where - // the in-flight walk snapshotted its batch before `name` became pending — - // the next iteration starts a fresh walk that includes it. - while (index.getPendingBackfills().includes(name)) { - // Failure latch: a deterministically-failing walk must not be re-run at - // the caller's retry rate — that is a silent CPU loop wearing a retry - // loop's clothes. Within the cooldown, rethrow the recorded failure - // immediately; after it, one fresh attempt is allowed. - const failure = this._aggregationBackfillFailure - if (failure && Date.now() - failure.at < AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS) { - throw new Error( - `Aggregation backfill for '${name}' is in failure cooldown (retry in ` + - `${Math.ceil((AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS - (Date.now() - failure.at)) / 1000)}s). ` + - `Last failure: ${failure.error.message}` - ) + const PAGE = 500 + let offset = 0 + let cursor: string | undefined + for (;;) { + const page = await this.storage.getNouns({ + pagination: cursor ? { limit: PAGE, cursor } : { limit: PAGE, offset } + }) + for (const noun of page.items) { + index.backfillEntity(name, noun as unknown as Record) } - if (!this._aggregationBackfillFlight) { - this._aggregationBackfillFlight = this.runAggregationBackfillWalk() - .finally(() => { - this._aggregationBackfillFlight = null - }) + if (!page.hasMore || page.items.length === 0) break + if (page.nextCursor) { + cursor = page.nextCursor + } else { + offset += page.items.length } - await this._aggregationBackfillFlight - } - } - - /** - * One store walk fills EVERY aggregate currently pending backfill — M pending - * aggregates cost one enumeration, not M. Only reached when an aggregate - * genuinely needs a rescan (new definition over a populated store, changed - * definition, or failed state load); a clean reopen adopts persisted state - * and never walks. - */ - private async runAggregationBackfillWalk(): Promise { - const index = this._aggregationIndex! - const names = index.getPendingBackfills() - if (names.length === 0) return - - prodLog.info(`[Aggregation] backfill walk starting for: ${names.join(', ')}`) - const startedAt = Date.now() - for (const n of names) index.beginBackfill(n) - - let scanned = 0 - try { - const PAGE = 500 - let offset = 0 - let cursor: string | undefined - for (;;) { - const page = await this.storage.getNouns({ - pagination: cursor ? { limit: PAGE, cursor } : { limit: PAGE, offset } - }) - for (const noun of page.items) { - const record = noun as unknown as Record - for (const n of names) { - index.backfillEntity(n, record) - } - } - scanned += page.items.length - if (!page.hasMore || page.items.length === 0) break - if (page.nextCursor) { - if (page.nextCursor === cursor) { - // A non-advancing cursor with hasMore=true would loop this walk at - // CPU speed forever, silently. That is a storage pagination defect — - // fail the waiting queries loudly instead of spinning. - throw new Error( - `Aggregation backfill aborted: storage pagination returned a non-advancing cursor ` + - `after ${scanned} entities with hasMore=true — the storage adapter's getNouns cursor is broken.` - ) - } - cursor = page.nextCursor - } else { - offset += page.items.length - } - } - } catch (err) { - // Non-destructive failure: drop the staging maps (live state keeps - // serving), keep the aggregates flagged pending, latch the error so - // retries within the cooldown fail fast, and say all of it out loud. - for (const n of names) index.abortBackfill(n) - this._aggregationBackfillFailure = { at: Date.now(), error: err as Error } - prodLog.warn( - `[Aggregation] backfill walk FAILED after ${scanned} entities: ${(err as Error).message} — ` + - `prior aggregate state preserved; retries suppressed for ${AGGREGATION_BACKFILL_RETRY_COOLDOWN_MS / 1000}s` - ) - throw err } - for (const n of names) index.finishBackfill(n) - this._aggregationBackfillFailure = null - prodLog.info( - `[Aggregation] backfill walk finished: ${scanned} entities → ${names.length} aggregate(s) in ${Date.now() - startedAt}ms` - ) + index.finishBackfill(name) } /** @@ -16217,10 +15834,6 @@ export class Brainy implements BrainyInterface { * This ensures deferred persistence mode data is saved */ async close(): Promise { - // Cancel any pending post-import background deduplication FIRST — it is a - // writer (merge-deletes), and no delete pass may start mid- or post-close. - this._backgroundDedup?.cancelPending() - // Change-feed teardown: no events are delivered for or after close(). this._changeFeed.close() @@ -16231,21 +15844,9 @@ export class Brainy implements BrainyInterface { await this.generationStore.flushPendingSingleOps() } - // Phase 0b: REPACK cold history into sealed segments (D1+D3 — - // re-representation, never deletion; the only history transform under the - // archival profile), then auto-compact per config.retention. Repack runs - // FIRST so bounded-retention reclaim can drop whole segments. Both are - // time-bounded maintenance passes (8.9.0 law: flush() never pays these); - // both are housekeeping — failures warn, never fail a clean shutdown. - if (!this.isReadOnly && this.generationStore) { - try { - await this.generationStore.repackHistory({ timeBudgetMs: 5_000 }) - } catch (error) { - console.warn( - `History repacking failed (non-fatal): ${error instanceof Error ? error.message : String(error)}` - ) - } - } + // Phase 0b: Auto-compact generational history per config.retention (default + // on) BEFORE the generation store closes below. Respects live Db pins and + // an explicit autoCompact: false; no-op on read-only instances. await this.autoCompactHistory() // Phase 1: Flush ALL components in parallel to persist buffered data diff --git a/src/db/db.ts b/src/db/db.ts index ac927fc5..62c3de0e 100644 --- a/src/db/db.ts +++ b/src/db/db.ts @@ -945,13 +945,6 @@ export class Db { * {@link SpeculativeOverlayError} (commit them with `brain.transact()` * first). * - * SPARSE FILES: a native accelerator's mmap index files can be sparse — - * huge apparent size, small allocated size. `persist()` handles them - * correctly (hard links share the allocation). But if you then archive the - * snapshot with EXTERNAL tools, use the sparse-aware flags (`tar czSf`, - * `rsync --sparse`, `cp --sparse=always`) or the copy materializes every - * hole — see docs/guides/external-backups-and-sparse-storage.md. - * * @param path - Absolute directory for the snapshot (created; must be * empty or absent). * @throws GenerationConflictError when this view is no longer the latest diff --git a/src/db/factLog.ts b/src/db/factLog.ts index 94e79700..4c5e95fd 100644 --- a/src/db/factLog.ts +++ b/src/db/factLog.ts @@ -102,26 +102,12 @@ export interface FactScanBatch { segmentId: string } -/** - * Liveness bound on a scan's FIRST batch (Stage-2 co-freeze, D1 contract): - * `batches()` must yield its first batch — or fail loudly — within this many - * ms of the first pull. A backlogged or damaged store may be SLOW, but it may - * never be SILENT: a consumer awaiting the first batch is otherwise - * indistinguishable from a wedge (the exact failure shape a production heal - * hit against a generations-backlogged brain). - */ -export const SCANFACTS_FIRST_BATCH_MS = 10_000 - /** The telemetry a scan OPEN returns (frozen shape). */ export interface FactScanHandle { headGeneration: number segmentCount: number approxFactCount: number - /** - * Ordered batches; a detected gap aborts LOUDLY, never a silent skip. - * Liveness contract: the FIRST batch resolves or rejects within - * {@link SCANFACTS_FIRST_BATCH_MS} of the first pull — never a silent hang. - */ + /** Ordered batches; a detected gap aborts LOUDLY, never a silent skip. */ batches: () => AsyncGenerator /** Close telemetry — the invariant cross-check, valid after iteration ends. */ summary: () => { factsYielded: number; segmentsRead: number } @@ -454,8 +440,6 @@ export class FactLog { toGeneration?: number kinds?: Array<'noun' | 'verb'> batchSize?: number - /** Test override for the first-batch liveness bound (default {@link SCANFACTS_FIRST_BATCH_MS}). */ - firstBatchTimeoutMs?: number }): FactScanHandle { const from = options?.fromGeneration ?? 1 const to = options?.toGeneration ?? this.head @@ -530,43 +514,11 @@ export class FactLog { } } - // Liveness wrapper: the FIRST pull races the contract deadline. Only the - // first — the bound is time-to-first-batch (proof the producer is alive), - // not per-batch pacing; and it runs only while a pull is actually pending, - // so consumer think-time between pulls never counts against the producer. - const firstBatchTimeoutMs = options?.firstBatchTimeoutMs ?? SCANFACTS_FIRST_BATCH_MS - async function* batchesWithLiveness(this: void): AsyncGenerator { - const inner = batches() - let timer: NodeJS.Timeout | undefined - try { - const deadline = new Promise((_, reject) => { - timer = setTimeout( - () => - reject( - new Error( - `fact log: scanFacts produced no first batch within ${firstBatchTimeoutMs}ms ` + - `(liveness contract) — the store is wedged or unreadably slow; aborting scan LOUDLY ` + - `instead of hanging the consumer.` - ) - ), - firstBatchTimeoutMs - ) - timer.unref?.() - }) - const first = await Promise.race([inner.next(), deadline]) - if (first.done) return - yield first.value - } finally { - clearTimeout(timer) - } - yield* inner - } - return { headGeneration: this.head, segmentCount: segments.length + (tailSnapshot.length > 0 ? 1 : 0), approxFactCount, - batches: batchesWithLiveness, + batches, summary: () => ({ factsYielded, segmentsRead }) } } diff --git a/src/db/generationSegments.ts b/src/db/generationSegments.ts deleted file mode 100644 index 0c14b60c..00000000 --- a/src/db/generationSegments.ts +++ /dev/null @@ -1,459 +0,0 @@ -/** - * @module db/generationSegments - * @description The generation-segment store — Stage-2 D1+D3+repacking's file - * format (co-frozen 2026-07-19; design: the d1-d3-repacking spec). - * - * Packs CONSECUTIVE cold generations' record-sets (before-images + delta) - * into append-once segment files with derived sidecar indexes, so history - * scales in SEGMENTS (tens) instead of FILES-PER-GENERATION (hundreds of - * thousands), and cold-open reads ONE manifest instead of listing the - * backlog. Layout under `_generations/segments/`: - * - * - `seg-.bgs` — magic "BGS1", then one frame per - * generation: `u32 payloadLen | u32 crc32c | msgpack payload`. Payload is - * POSITIONAL: `[generation, timestamp, delta, records[], flags]` with - * records `[kindByte, id, record]`. `flags` reserves encoding evolution - * (bit 0 = compressed payload — v1 always 0; a future writer upgrade, - * never a format break). Sealed segments are IMMUTABLE — the fact log's - * own law, generalized. - * - `seg-.idx` — DERIVED sidecar (msgpack): per-generation frame - * offsets (point reads = one ranged read, never a listing) + per-id - * generation postings (per-id chain rebuilds read only what they need). - * Corrupt/missing → rebuilt from its segment in one sequential read, - * loudly. - * - `manifest.json` — the segment catalogue + `compactedBelow` (D3's - * horizon marker). Cold-open reads THIS; the packed backlog is never - * listed. - * - * D3 semantics carried here: bounded-retention reclaim drops WHOLE segments - * at boundaries (O(1) per segment, no rewrite); under the archival profile - * (`retention: 'all'`) nothing here is ever dropped — folding is the only - * transform (re-representation, never deletion). - */ - -import { encode as msgpackEncode, decode as msgpackDecode } from '@msgpack/msgpack' -import { crc32c } from '../utils/crc32c.js' -import type { FactLogStorage } from './factLog.js' -import { prodLog } from '../utils/logger.js' - -/** Directory for segment files + manifest, under the generations prefix. */ -export const SEGMENTS_PREFIX = '_generations/segments' - -/** Target sealed-segment size (co-freeze proposal; tunable on evidence). */ -export const SEGMENT_TARGET_BYTES = 64 * 1024 * 1024 - -const MAGIC = new TextEncoder().encode('BGS1') -const FRAME_PREFIX_BYTES = 8 // u32 payloadLen + u32 crc32c -const MANIFEST_PATH = `${SEGMENTS_PREFIX}/manifest.json` - -/** One generation's fold input — exactly what the live tier holds for it. */ -export interface FoldGeneration { - generation: number - timestamp: number - /** The tx.json delta object, carried verbatim. */ - delta: unknown - /** The before-image record-set (empty for record-less generations). */ - records: Array<{ kind: 'noun' | 'verb'; id: string; record: unknown }> -} - -/** Manifest entry for one sealed segment. */ -export interface SegmentMeta { - file: string - firstGeneration: number - lastGeneration: number - frames: number - bytes: number - /** crc32c of the full segment byte stream — the digest chain's link. */ - checksum: number -} - -interface SegmentManifest { - version: 1 - compactedBelow: number - segments: SegmentMeta[] -} - -interface SidecarIndex { - version: 1 - /** [generation, frameOffset, frameLen] ascending by generation. */ - generations: Array<[number, number, number]> - /** `${kindByte}:${id}` → ascending generations holding a record for it. */ - ids: Record -} - -const segmentFileName = (firstGeneration: number): string => - `seg-${String(firstGeneration).padStart(20, '0')}.bgs` -const sidecarFileName = (firstGeneration: number): string => - `seg-${String(firstGeneration).padStart(20, '0')}.idx` - -/** - * The generation-segment store. Owns the packed tier ONLY — the live - * per-generation tier and the routing between tiers belong to - * `GenerationStore`. All mutating entry points here are called under the - * generation store's commit mutex. - */ -export class GenerationSegmentStore { - private readonly storage: FactLogStorage - private manifest: SegmentManifest = { version: 1, compactedBelow: 0, segments: [] } - /** Sidecar cache — segments are immutable, so entries never invalidate. */ - private readonly sidecars = new Map() - - constructor(storage: FactLogStorage) { - this.storage = storage - } - - /** Load the manifest (ONE read — never a directory listing). */ - async open(): Promise { - const raw = (await this.storage.readRawObject(MANIFEST_PATH)) as SegmentManifest | null - if (raw) { - if (raw.version !== 1) { - throw new Error( - `[GenerationSegments] manifest version ${String(raw.version)} is newer than this ` + - `engine understands — refusing to serve partial history. Upgrade the engine.` - ) - } - this.manifest = raw - } - } - - /** The packed tier's catalogue (ascending, immutable snapshot). */ - segments(): readonly SegmentMeta[] { - return this.manifest.segments - } - - /** D3's horizon marker: generations below this were reclaimed (bounded profiles only). */ - compactedBelow(): number { - return this.manifest.compactedBelow - } - - /** The covering sealed segment for `gen`, or null if it lives outside the packed tier. */ - private coveringSegment(gen: number): SegmentMeta | null { - // Manifest is ascending and ranges never overlap — binary search. - const segs = this.manifest.segments - let lo = 0 - let hi = segs.length - 1 - while (lo <= hi) { - const mid = (lo + hi) >> 1 - const s = segs[mid] - if (gen < s.firstGeneration) hi = mid - 1 - else if (gen > s.lastGeneration) lo = mid + 1 - else return s - } - return null - } - - /** True when `gen` is packed (readable from this tier). */ - hasGeneration(gen: number): boolean { - return this.coveringSegment(gen) !== null - } - - /** - * Fold consecutive generations into ONE new sealed segment + sidecar and - * append it to the manifest atomically. Caller guarantees: `gens` is - * ascending, contiguous with the packed tier (first = last packed + 1 when - * segments exist), and already durable in the live tier. Crash between the - * segment write and the caller's live-tier delete leaves a DUPLICATE - * representation — resolved live-tier-wins by the reader; never a gap. - */ - async fold(gens: FoldGeneration[]): Promise { - if (gens.length === 0) { - throw new Error('[GenerationSegments] fold() requires at least one generation') - } - for (let i = 1; i < gens.length; i++) { - if (gens[i].generation <= gens[i - 1].generation) { - throw new Error('[GenerationSegments] fold() input must be strictly ascending') - } - } - const last = this.manifest.segments[this.manifest.segments.length - 1] - if (last && gens[0].generation <= last.lastGeneration) { - throw new Error( - `[GenerationSegments] fold() overlaps the packed tier: ${gens[0].generation} ≤ ` + - `sealed ${last.lastGeneration} — segments are immutable, never rewritten` - ) - } - - const first = gens[0].generation - const file = segmentFileName(first) - const sidecar: SidecarIndex = { version: 1, generations: [], ids: {} } - - // Encode all frames, tracking offsets for the sidecar. - const parts: Uint8Array[] = [MAGIC] - let offset = MAGIC.length - for (const g of gens) { - const payload = msgpackEncode([ - g.generation, - g.timestamp, - g.delta, - g.records.map((r) => [r.kind === 'noun' ? 0 : 1, r.id, r.record]), - 0 // flags: v1 = uncompressed - ]) - const frame = new Uint8Array(FRAME_PREFIX_BYTES + payload.length) - const view = new DataView(frame.buffer) - view.setUint32(0, payload.length, true) - view.setUint32(4, crc32c(payload), true) - frame.set(payload, FRAME_PREFIX_BYTES) - sidecar.generations.push([g.generation, offset, frame.length]) - for (const r of g.records) { - const key = `${r.kind === 'noun' ? 0 : 1}:${r.id}` - ;(sidecar.ids[key] ??= []).push(g.generation) - } - parts.push(frame) - offset += frame.length - } - const total = parts.reduce((n, p) => n + p.length, 0) - const bytes = new Uint8Array(total) - let at = 0 - for (const p of parts) { - bytes.set(p, at) - at += p.length - } - - const meta: SegmentMeta = { - file, - firstGeneration: first, - lastGeneration: gens[gens.length - 1].generation, - frames: gens.length, - bytes: total, - checksum: crc32c(bytes) - } - - // Durability order: segment + sidecar fsync'd BEFORE the manifest names - // them (a crash before the manifest = invisible orphan files, harmless); - // manifest last, atomically. - const segPath = `${SEGMENTS_PREFIX}/${file}` - const idxPath = `${SEGMENTS_PREFIX}/${sidecarFileName(first)}` - await this.storage.writeRawBytes(segPath, bytes) - await this.storage.writeRawBytes(idxPath, msgpackEncode(sidecar)) - await this.storage.syncRawObjects([segPath, idxPath]) - const next: SegmentManifest = { - ...this.manifest, - segments: [...this.manifest.segments, meta] - } - await this.storage.writeRawObject(MANIFEST_PATH, next) - await this.storage.syncRawObjects([MANIFEST_PATH]) - this.manifest = next - this.sidecars.set(file, sidecar) - return meta - } - - /** Load (or rebuild, loudly) a segment's sidecar. */ - private async sidecarFor(meta: SegmentMeta): Promise { - const cached = this.sidecars.get(meta.file) - if (cached) return cached - const idxPath = `${SEGMENTS_PREFIX}/${sidecarFileName(meta.firstGeneration)}` - const raw = await this.storage.readRawBytes(idxPath) - if (raw) { - try { - const idx = msgpackDecode(raw) as SidecarIndex - if (idx.version === 1) { - this.sidecars.set(meta.file, idx) - return idx - } - } catch { - // fall through to rebuild - } - } - // Sidecars are DERIVED: rebuild from the segment, loudly — never serve - // wrong offsets silently. - prodLog.warn( - `[GenerationSegments] sidecar for ${meta.file} missing or unreadable — rebuilding from the segment` - ) - const rebuilt = await this.rebuildSidecar(meta) - await this.storage.writeRawBytes(idxPath, msgpackEncode(rebuilt)) - this.sidecars.set(meta.file, rebuilt) - return rebuilt - } - - /** One sequential read of the segment → a fresh sidecar. Verifies every frame CRC. */ - private async rebuildSidecar(meta: SegmentMeta): Promise { - const frames = await this.readAllFrames(meta) - const idx: SidecarIndex = { version: 1, generations: [], ids: {} } - for (const f of frames) { - idx.generations.push([f.generation, f.offset, f.frameLen]) - for (const r of f.records) { - const key = `${r.kind === 'noun' ? 0 : 1}:${r.id}` - ;(idx.ids[key] ??= []).push(f.generation) - } - } - return idx - } - - private decodeFrame( - payload: Uint8Array - ): { generation: number; timestamp: number; delta: unknown; records: FoldGeneration['records'] } { - const [generation, timestamp, delta, rawRecords] = msgpackDecode(payload) as [ - number, - number, - unknown, - Array<[number, string, unknown]>, - number - ] - return { - generation, - timestamp, - delta, - records: rawRecords.map(([kindByte, id, record]) => ({ - kind: kindByte === 0 ? ('noun' as const) : ('verb' as const), - id, - record - })) - } - } - - private async readAllFrames(meta: SegmentMeta): Promise< - Array & { offset: number; frameLen: number }> - > { - const bytes = await this.storage.readRawBytes(`${SEGMENTS_PREFIX}/${meta.file}`) - if (!bytes) { - throw new Error( - `[GenerationSegments] sealed segment ${meta.file} is MISSING — packed history is damaged; ` + - `refusing to continue silently` - ) - } - const out: Array & { offset: number; frameLen: number }> = [] - let at = MAGIC.length - const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength) - while (at + FRAME_PREFIX_BYTES <= bytes.length) { - const payloadLen = view.getUint32(at, true) - const crc = view.getUint32(at + 4, true) - const payload = bytes.subarray(at + FRAME_PREFIX_BYTES, at + FRAME_PREFIX_BYTES + payloadLen) - if (payload.length !== payloadLen || crc32c(payload) !== crc) { - throw new Error( - `[GenerationSegments] frame CRC mismatch in ${meta.file} at offset ${at} — ` + - `packed history is damaged; refusing to serve it` - ) - } - out.push({ ...this.decodeFrame(payload), offset: at, frameLen: FRAME_PREFIX_BYTES + payloadLen }) - at += FRAME_PREFIX_BYTES + payloadLen - } - return out - } - - /** Read one packed generation's frame via its sidecar offset (one ranged read). */ - private async readFrame( - gen: number - ): Promise | null> { - const meta = this.coveringSegment(gen) - if (!meta) return null - const idx = await this.sidecarFor(meta) - // generations ascending → binary search. - const gens = idx.generations - let lo = 0 - let hi = gens.length - 1 - while (lo <= hi) { - const mid = (lo + hi) >> 1 - if (gens[mid][0] < gen) lo = mid + 1 - else if (gens[mid][0] > gen) hi = mid - 1 - else { - const [, offset, frameLen] = gens[mid] - const bytes = await this.storage.readRawBytes(`${SEGMENTS_PREFIX}/${meta.file}`) - if (!bytes) { - throw new Error(`[GenerationSegments] sealed segment ${meta.file} is MISSING`) - } - const frame = bytes.subarray(offset, offset + frameLen) - const view = new DataView(frame.buffer, frame.byteOffset, frame.byteLength) - const payloadLen = view.getUint32(0, true) - const crc = view.getUint32(4, true) - const payload = frame.subarray(FRAME_PREFIX_BYTES, FRAME_PREFIX_BYTES + payloadLen) - if (payload.length !== payloadLen || crc32c(payload) !== crc) { - throw new Error( - `[GenerationSegments] frame CRC mismatch for generation ${gen} in ${meta.file} — ` + - `packed history is damaged; refusing to serve it` - ) - } - return this.decodeFrame(payload) - } - } - // In the covering range but not present: the packed tier is dense by - // construction (fold packs every generation it is handed, including - // record-less ones) — absence inside a sealed range is damage. - throw new Error( - `[GenerationSegments] generation ${gen} is inside sealed segment ${meta.file}'s declared ` + - `range but has no frame — packed history is damaged` - ) - } - - /** The packed tier's delta for `gen` (null = not packed). */ - async readDelta(gen: number): Promise<{ delta: unknown; timestamp: number } | null> { - const frame = await this.readFrame(gen) - return frame ? { delta: frame.delta, timestamp: frame.timestamp } : null - } - - /** The packed tier's full record-set for `gen` (null = not packed). */ - async readRecords(gen: number): Promise { - const frame = await this.readFrame(gen) - return frame ? frame.records : null - } - - /** One packed before-image (null = not packed OR no record for the id in that generation). */ - async readRecord(gen: number, kind: 'noun' | 'verb', id: string): Promise { - const frame = await this.readFrame(gen) - if (!frame) return null - const hit = frame.records.find((r) => r.kind === kind && r.id === id) - return hit ? hit.record : null - } - - /** - * D3 reclaim: drop WHOLE segments whose lastGeneration < `belowGeneration` - * and bump `compactedBelow`. Partial segments are never dropped — the - * boundary waits. NEVER called under the archival profile (the caller - * enforces retention semantics; this method only executes boundary drops). - */ - async dropSegmentsBelow(belowGeneration: number): Promise<{ dropped: number; compactedBelow: number }> { - const keep: SegmentMeta[] = [] - const drop: SegmentMeta[] = [] - for (const s of this.manifest.segments) { - ;(s.lastGeneration < belowGeneration ? drop : keep).push(s) - } - if (drop.length === 0) { - return { dropped: 0, compactedBelow: this.manifest.compactedBelow } - } - const compactedBelow = Math.max( - this.manifest.compactedBelow, - drop[drop.length - 1].lastGeneration + 1 - ) - // Manifest first (the drop is authoritative once named), then bytes — - // a crash between leaves orphan segment files invisible to the manifest, - // harmless and re-collectable. - const next: SegmentManifest = { ...this.manifest, compactedBelow, segments: keep } - await this.storage.writeRawObject(MANIFEST_PATH, next) - await this.storage.syncRawObjects([MANIFEST_PATH]) - this.manifest = next - for (const s of drop) { - await this.storage.deleteRawObject(`${SEGMENTS_PREFIX}/${s.file}`) - await this.storage.deleteRawObject(`${SEGMENTS_PREFIX}/${sidecarFileName(s.firstGeneration)}`) - this.sidecars.delete(s.file) - } - return { dropped: drop.length, compactedBelow } - } - - /** - * D8 rider — the packed portion of `generationDigest(g)`: a deterministic - * crc32c chain over sealed-segment checksums fully below `g`, plus the - * frame CRC of `g`'s own frame when `g` is mid-segment. O(segments), not - * O(generations); identical history ⇒ identical digest on any machine. - * The live-tier portion is composed by the caller. - */ - async digestThroughPacked(g: number): Promise { - let digest = 0 - let covered = false - for (const s of this.manifest.segments) { - if (s.lastGeneration <= g) { - digest = crc32c(new TextEncoder().encode(`${digest}:${s.checksum}`)) - if (s.lastGeneration === g) covered = true - } else if (s.firstGeneration <= g) { - // g is mid-segment: chain the partial prefix via g's frame CRC. - const frame = await this.readFrame(g) - if (frame === null) return null - const idx = await this.sidecarFor(s) - const upTo = idx.generations.filter(([gen]) => gen <= g) - for (const [gen, offset, frameLen] of upTo) { - digest = crc32c(new TextEncoder().encode(`${digest}:${gen}:${offset}:${frameLen}`)) - } - covered = true - break - } - } - return covered || this.manifest.segments.length > 0 ? digest : null - } -} diff --git a/src/db/generationStore.ts b/src/db/generationStore.ts index aede17a4..0d813f4d 100644 --- a/src/db/generationStore.ts +++ b/src/db/generationStore.ts @@ -46,8 +46,6 @@ import type { TxLogEntry } from './types.js' import { FactLog, storageSupportsFactLog, type CommitFact, type FactOp } from './factLog.js' -import { GenerationSegmentStore, type FoldGeneration } from './generationSegments.js' -import { crc32c } from '../utils/crc32c.js' /** * The byte-identical before-images of every id a commit touches, read UNDER @@ -259,30 +257,6 @@ export class GenerationStore { */ private deltaCacheMax = 4096 - /** - * Running total of on-disk history bytes across committed generations — - * `null` until {@link historyBytes} pays its one seeding walk. Maintained - * incrementally at commit/reclaim so the adaptive retention check on every - * flush() is O(1), never a tail re-walk. Never updated by cache re-reads - * ({@link setDelta} inserts are cache population, not new history). - */ - private historyBytesTotal: number | null = null - - /** - * The packed tier (D1+D3): sealed segments holding folded cold - * generations. Null until {@link open} wires it (and on storage adapters - * without raw-byte primitives — the live tier then carries everything, - * exactly as before the packed tier existed). - */ - private segments: GenerationSegmentStore | null = null - - /** - * Live-tier window: generations newer than `committed - REPACK_LIVE_WINDOW` - * are never folded — the hot tail stays in the per-generation layout the - * write path owns. Matches the resident chain window's scale. - */ - static readonly REPACK_LIVE_WINDOW = 1024 - /** * Model-B per-write group-commit — the in-memory PENDING tier. * @@ -450,33 +424,6 @@ export class GenerationStore { this.factLog = null } - // PACKED TIER (D1+D3): same capability gate as the fact log. Opening - // reads ONE manifest — never a listing of the packed backlog — and seeds - // committedRanges with the sealed ranges so packed generations resolve - // exactly like live ones. - if (storageSupportsFactLog(this.storage)) { - this.segments = new GenerationSegmentStore(this.storage) - await this.segments.open() - const packedRanges = this.segments - .segments() - .map((s): [number, number] => [s.firstGeneration, Math.min(s.lastGeneration, this.committed)]) - .filter(([lo, hi]) => lo <= hi) - if (packedRanges.length > 0) { - // Merge packed (older) + live (newer) interval sets — both ascending; - // coalesce adjacency so range arithmetic stays interval-exact. - const merged: Array<[number, number]> = [] - for (const r of [...packedRanges, ...this.committedRanges].sort((a, b) => a[0] - b[0])) { - const last = merged[merged.length - 1] - if (last && r[0] <= last[1] + 1) last[1] = Math.max(last[1], r[1]) - else merged.push([r[0], r[1]]) - } - this.committedRanges = merged - } - this.horizonGen = Math.max(this.horizonGen, this.segments.compactedBelow() - 1) - } else { - this.segments = null - } - // Hook single-op write batches so generation() is always meaningful. // Suppressed while a transact batch executes (the batch is ONE generation). if (!options?.readOnly) { @@ -536,85 +483,6 @@ export class GenerationStore { return this.horizonGen } - /** - * @description Read-only history footprint for fleet audits: how much - * generational history this store holds on disk. `bytes` pays (and seeds) - * the one-time {@link historyBytes} walk on first call — subsequent calls - * are O(1). The oldest/newest timestamps come from those generations' - * deltas (cache-bounded reads). - * @returns Counts, bytes, generation range, and the compaction horizon. - */ - /** - * @description D8 (gate-to-generation provenance): a deterministic content - * digest of the generation log THROUGH `g` — identical history ⇒ identical - * digest on any machine; any divergence (different records, different - * order, reclaimed range) ⇒ different digest. Composed from the packed - * tier's sealed-segment checksum chain (O(segments)) plus the live tier's - * per-generation delta digests (O(live window at most)). Release gates pin - * {generation, digest} and verify both at execution time. - * @param g - The generation to digest through (≤ committed). - * @returns A hex digest string, stable across reopen and repacking states - * ONLY for fully-packed prefixes — repacking changes representation, so - * the composed digest is defined over CONTENT: live-tier gens hash their - * delta + record ids, packed gens hash via frame CRCs. A gate should pin - * after a repack pass for long-term stability, or re-pin on repack. - */ - async generationDigest(g: number): Promise { - if (!Number.isInteger(g) || g < 1 || g > this.committed) { - throw new RangeError( - `generationDigest(): generation ${g} is out of range [1, ${this.committed}]` - ) - } - if (g <= this.horizonGen) { - throw new GenerationCompactedError(g, this.horizonGen) - } - let digest = 0 - const enc = new TextEncoder() - if (this.segments) { - const packed = await this.segments.digestThroughPacked(g) - if (packed !== null) digest = packed - } - // Live-tier composition: every committed gen ≤ g not covered by a sealed - // segment hashes its delta content in ascending order. - for (const gen of this.committedGensAsc()) { - if (gen > g) break - if (this.segments?.hasGeneration(gen)) continue - const delta = await this.getDelta(gen) - digest = crc32c( - enc.encode( - `${digest}:${gen}:${delta.timestamp}:${[...delta.nouns].sort().join(',')}:${[...delta.verbs].sort().join(',')}` - ) - ) - } - return digest.toString(16).padStart(8, '0') - } - - async historyStats(): Promise<{ - generations: number - bytes: number - oldestGeneration: number | null - newestGeneration: number | null - oldestTimestamp: number | null - newestTimestamp: number | null - horizon: number - }> { - let oldest: number | null = null - let newest: number | null = null - for (const gen of this.committedGensAsc()) { - if (oldest === null) oldest = gen - newest = gen - } - return { - generations: this.committedCount(), - bytes: await this.historyBytes(), - oldestGeneration: oldest, - newestGeneration: newest, - oldestTimestamp: oldest !== null ? (await this.getDelta(oldest)).timestamp : null, - newestTimestamp: newest !== null ? (await this.getDelta(newest)).timestamp : null, - horizon: this.horizonGen - } - } - /** * @description Read one generation's persisted before-image records — the * compaction fallback for generations written before deltas carried @@ -627,17 +495,14 @@ export class GenerationStore { try { paths = await this.storage.listRawObjects(`${GENERATIONS_PREFIX}/${gen}/prev`) } catch { - paths = [] + return [] } const records: GenerationRecord[] = [] for (const p of paths) { const record = (await this.storage.readRawObject(p)) as GenerationRecord | null if (record) records.push(record) } - if (records.length > 0) return records - // Two-tier: folded generations serve their record-set from the segment. - const packed = await this.segments?.readRecords(gen) - return packed ? (packed.map((r) => r.record) as GenerationRecord[]) : [] + return records } /** @@ -984,9 +849,6 @@ export class GenerationStore { timestamp, bytes: delta.bytes ?? 0 }) - if (this.historyBytesTotal !== null) { - this.historyBytesTotal += delta.bytes ?? 0 - } this.extendChains(gen, nouns, verbs) const logEntry: TxLogEntry = { generation: gen, timestamp, ...(args.meta && { meta: args.meta }) } await this.storage.appendTxLogLine(JSON.stringify(logEntry)) @@ -1440,9 +1302,6 @@ export class GenerationStore { timestamp: buf.timestamp, bytes: genBytes.get(gen) ?? 0 }) - if (this.historyBytesTotal !== null) { - this.historyBytesTotal += genBytes.get(gen) ?? 0 - } this.pendingBuffer.delete(gen) } this.pendingGens = [] @@ -1875,15 +1734,9 @@ export class GenerationStore { if (pending) { return (kind === 'noun' ? pending.nouns : pending.verbs).get(id) ?? null } - const live = (await this.storage.readRawObject( + return (await this.storage.readRawObject( `${GENERATIONS_PREFIX}/${gen}/prev/${id}.json` )) as GenerationRecord | null - if (live) return live - // Two-tier: the packed tier serves folded generations (live-tier-wins). - if (this.segments?.hasGeneration(gen)) { - return (await this.segments.readRecord(gen, kind, id)) as GenerationRecord | null - } - return null } /** @@ -2230,21 +2083,6 @@ export class GenerationStore { `${GENERATIONS_PREFIX}/${gen}/tx.json` )) as GenerationDelta | null if (delta === null) { - // Two-tier read (D1+D3): not in the live tier → the packed tier. - // Live-tier-wins ordering (a crash mid-fold leaves a duplicate, never - // a gap), so the segment lookup runs only after the live miss. - const packed = await this.segments?.readDelta(gen) - if (packed) { - const d = packed.delta as GenerationDelta - const entry = { - nouns: new Set(d.nouns), - verbs: new Set(d.verbs), - timestamp: packed.timestamp, - bytes: d.bytes ?? 0 - } - this.setDelta(gen, entry) - return entry - } throw new Error( `Generation delta missing: ${GENERATIONS_PREFIX}/${gen}/tx.json ` + `(store corrupted or records removed outside compactHistory())` @@ -2284,26 +2122,17 @@ export class GenerationStore { /** * @description Total serialized bytes of the ON-DISK generational history — * the sum of every committed generation's recorded `bytes`. Backs the - * `maxBytes` and adaptive retention caps. O(1) after the first call: the - * total is computed by ONE walk over committed deltas, then maintained - * incrementally at every commit (+bytes) and reclaim (−bytes) and dropped on - * a wholesale state replacement (restore). Without the running total, the - * adaptive auto-compaction on every flush() re-walked the ENTIRE history — - * O(committed generations) file reads per flush past the delta-cache bound — - * which is how a 70k-generation production brain turned every write into a - * full-tail scan (SELF-GENERATIONS-GROWTH). Pending (un-flushed) generations - * are excluded (they are not on disk). + * `maxBytes` and adaptive retention caps. Reads each committed generation's + * delta (cached; a re-read only for cache-evicted ones) — O(committed + * generations), bounded by retention itself and invoked only at compaction + * time. Pending (un-flushed) generations are excluded (they are not on disk). * @returns The total on-disk history byte count. */ async historyBytes(): Promise { - if (this.historyBytesTotal !== null) { - return this.historyBytesTotal - } let total = 0 for (const gen of this.committedGensAsc()) { total += (await this.getDelta(gen)).bytes } - this.historyBytesTotal = total return total } @@ -2326,94 +2155,6 @@ export class GenerationStore { * @param options - Retention caps (see {@link CompactHistoryOptions}). * @returns Count of removed record-sets and the new horizon. */ - /** - * @description The REPACKER (D1+D3+repacking): fold cold live-tier - * generations into sealed segments — re-representation, never deletion. - * Every record and delta stays readable (asOf/chains unchanged); the - * per-generation directories are deleted only AFTER their segment is - * durable (crash between = duplicate representation, resolved - * live-tier-wins by every reader; never a gap). This is the transform that - * takes a 70k-file history to tens of segment files, and the ONLY history - * transform permitted under the archival profile. - * - * Folds oldest-first, contiguous from the packed boundary, in batches, and - * stops at the live window ({@link GenerationStore.REPACK_LIVE_WINDOW}) - * or when `timeBudgetMs` is spent — an early stop is a consistent prefix; - * the next pass resumes. - */ - async repackHistory(options?: { timeBudgetMs?: number; batchGenerations?: number }): Promise<{ - foldedGenerations: number - segmentsCreated: number - }> { - if (!this.segments) return { foldedGenerations: 0, segmentsCreated: 0 } - const segments = this.segments - return this.withMutex(async () => { - const deadline = - options?.timeBudgetMs !== undefined ? Date.now() + options.timeBudgetMs : undefined - const batchSize = options?.batchGenerations ?? 512 - const coldCeiling = this.committed - GenerationStore.REPACK_LIVE_WINDOW - const packedThrough = - segments.segments().length > 0 - ? segments.segments()[segments.segments().length - 1].lastGeneration - : 0 - - // Cold, unpacked, committed generations — ascending, contiguous scan. - const eligible: number[] = [] - for (const gen of this.committedGensAsc()) { - if (gen > coldCeiling) break - if (gen <= packedThrough) continue // already packed (dup fold barred) - if (this.pendingBuffer.has(gen)) continue // un-flushed = live by definition - eligible.push(gen) - } - - let folded = 0 - let segmentsCreated = 0 - for (let i = 0; i < eligible.length; i += batchSize) { - if (deadline !== undefined && Date.now() >= deadline) break - const batch = eligible.slice(i, i + batchSize) - const foldInput: FoldGeneration[] = [] - for (const gen of batch) { - const delta = (await this.storage.readRawObject( - `${GENERATIONS_PREFIX}/${gen}/tx.json` - )) as GenerationDelta | null - if (delta === null) { - // Already folded by a prior crashed pass whose dirs were removed, - // or damage — getDelta's two-tier read decides which, loudly, - // when someone asks. Skip; never fold a generation we cannot read. - continue - } - const records: FoldGeneration['records'] = [] - for (const [kind, ids] of [ - ['noun', delta.nouns] as const, - ['verb', delta.verbs] as const - ]) { - for (const id of ids) { - const record = await this.storage.readRawObject( - `${GENERATIONS_PREFIX}/${gen}/prev/${id}.json` - ) - if (record) records.push({ kind, id, record }) - } - } - foldInput.push({ generation: gen, timestamp: delta.timestamp, delta, records }) - } - if (foldInput.length === 0) continue - await segments.fold(foldInput) - segmentsCreated++ - // Segment + manifest durable → the live copies retire. - for (const g of foldInput) { - await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${g.generation}`) - } - folded += foldInput.length - } - if (folded > 0) { - prodLog.info( - `[GenerationStore] repacked ${folded} cold generation(s) into ${segmentsCreated} segment(s) — history preserved, file count reduced` - ) - } - return { foldedGenerations: folded, segmentsCreated } - }) - } - async compact(options?: CompactHistoryOptions): Promise { return this.withMutex(async () => { const minPinned = this.minPinnedGeneration() @@ -2421,11 +2162,6 @@ export class GenerationStore { const maxAge = options?.maxAge const maxBytes = options?.maxBytes const ageCutoff = maxAge !== undefined ? Date.now() - maxAge : undefined - // Bounded maintenance pass (8.9.0): stop reclaiming once the budget is - // spent. Safe mid-loop — reclamation is oldest-first, so an early stop - // leaves a consistent contiguous prefix and the next pass resumes. - const deadline = - options?.timeBudgetMs !== undefined ? Date.now() + options.timeBudgetMs : undefined const noCaps = maxGenerations === undefined && maxAge === undefined && maxBytes === undefined @@ -2439,7 +2175,6 @@ export class GenerationStore { for (const gen of [...this.committedGensAsc()]) { // Pins are always exempt: never reclaim a generation a live pin needs. if (gen > minPinned) break // committedGensAsc ascending → nothing newer is eligible either - if (deadline !== undefined && Date.now() >= deadline) break // budget spent — resume next pass const delta = await this.getDelta(gen) if (!noCaps) { const violatesCount = maxGenerations !== undefined && remainingCount > maxGenerations @@ -2471,9 +2206,6 @@ export class GenerationStore { await this.storage.removeRawPrefix(`${GENERATIONS_PREFIX}/${gen}`) this.deltaCache.delete(gen) - if (this.historyBytesTotal !== null) { - this.historyBytesTotal -= delta.bytes - } // AFTER the record-set is gone (over-count-only crash ordering): // release its history references and reclaim any blob left with zero @@ -2505,16 +2237,6 @@ export class GenerationStore { // Reclaimed generations leave the per-id chains stale → rebuild on next read. this.invalidateChains() this.horizonGen = Math.max(this.horizonGen, highestRemoved) - // Packed-tier reclaim (D3): a packed generation's bytes live in a - // sealed segment — removeRawPrefix above was a no-op for it. Drop - // WHOLE segments now fully below the horizon; a partially-reclaimed - // segment keeps its bytes until the boundary passes it (the frozen - // partial-segments-wait rule; logical reclamation above still holds — - // the generations left committedRanges and asOf below the horizon - // throws regardless). - if (this.segments) { - await this.segments.dropSegmentsBelow(this.horizonGen + 1) - } const manifest: GenerationManifest = { version: 1, generation: this.committed, @@ -2544,9 +2266,6 @@ export class GenerationStore { async reopenAfterRestore(floorGeneration: number): Promise { await this.withMutex(async () => { this.deltaCache.clear() - // The running history-byte total describes the REPLACED store — drop it; - // the next historyBytes() re-seeds with one walk over the new state. - this.historyBytesTotal = null // A wholesale state replacement invalidates any buffered single-op // history — discard the pending tier (its live writes are gone with the // replaced store). diff --git a/src/db/types.ts b/src/db/types.ts index 4c8a4957..2bfd6ac2 100644 --- a/src/db/types.ts +++ b/src/db/types.ts @@ -116,16 +116,6 @@ export interface TransactOptions { * record is staged. */ ifAtGeneration?: number - /** - * Budget (ms) for the atomic apply phase. When omitted, the budget SCALES - * with the batch: `max(30 000, opCount × 2 000)` — production imports on - * network-attached disks measure ~2 s per operation, so a flat 30 s budget - * silently capped honest bulk work at ~15 operations. A tripped budget - * rolls the whole batch back and throws a retryable - * `TransactionTimeoutError` naming the operation it stopped at, the batch - * size, and the elapsed/budget times. - */ - timeoutMs?: number } /** @@ -176,15 +166,6 @@ export interface CompactHistoryOptions { * of each surviving generation's serialized record set (`GenerationDelta.bytes`). */ maxBytes?: number - /** - * Stop reclaiming after this many milliseconds even if caps are still - * exceeded (8.9.0). Compaction is maintenance — a bounded pass keeps - * `close()` (and any explicit maintenance window) from stalling on a large - * backlog; the next pass resumes where this one stopped (reclamation is - * oldest-first, so an early stop is always a consistent prefix). Unset = - * run to completion. - */ - timeBudgetMs?: number } /** @@ -202,36 +183,6 @@ export interface CompactHistoryResult { horizon: number } -/** - * @description Result of `brain.historyStats()` — the read-only generational - * history footprint, for fleet audits and ops doors. A pool operator runs this - * per brain to size retention exposure (how much MVCC history each brain - * carries and under which policy) without touching any data. - */ -export interface HistoryStats { - /** Committed generation record-sets currently on disk. */ - generations: number - /** Total on-disk history bytes across those record-sets. */ - bytes: number - /** Oldest committed generation still on disk (null when history is empty). */ - oldestGeneration: number | null - /** Newest committed generation (null when history is empty). */ - newestGeneration: number | null - /** Commit timestamp (ms) of the oldest on-disk generation. */ - oldestTimestamp: number | null - /** Commit timestamp (ms) of the newest on-disk generation. */ - newestTimestamp: number | null - /** Compaction horizon — generations below it were reclaimed. */ - horizon: number - /** The effective retention mode this brain runs under. */ - retentionMode: 'all' | 'adaptive' | 'explicit' - /** - * The adaptive byte budget in force (coordinator-driven or the local - * free-memory probe); null under 'all' or explicit caps. - */ - effectiveBudgetBytes: number | null -} - // ============================================================================ // Db surfaces // ============================================================================ diff --git a/src/graph/graphAdjacencyIndex.ts b/src/graph/graphAdjacencyIndex.ts index b37391aa..816856b1 100644 --- a/src/graph/graphAdjacencyIndex.ts +++ b/src/graph/graphAdjacencyIndex.ts @@ -282,16 +282,6 @@ export class GraphAdjacencyIndex implements GraphIndexProvider { } hasMore = result.hasMore - if (hasMore && (!result.nextCursor || result.nextCursor === cursor)) { - // A stalled cursor with hasMore=true would re-read the same page - // forever — a silent full-CPU loop at cold open. Abort loudly; a - // graph read failing beats a process that spins without a log line. - throw new Error( - `GraphAdjacencyIndex: verb walk stalled after ${count} verbs — storage returned ` + - `hasMore=true with ${result.nextCursor ? 'a non-advancing' : 'no'} cursor. ` + - `Aborting the cold-load; run brain.repairIndex() if this persists.` - ) - } cursor = result.nextCursor } diff --git a/src/graph/graphAudit.ts b/src/graph/graphAudit.ts deleted file mode 100644 index d0e44cd9..00000000 --- a/src/graph/graphAudit.ts +++ /dev/null @@ -1,214 +0,0 @@ -/** - * @module graph/graphAudit - * @description Read-only graph-truth audit — the graph sibling of `repairIndex()`'s - * diagnosis half. Verifies three layers against each other without mutating anything: - * - * 1. CANONICAL verb records (the storage walk — the source of truth) - * 2. the RELATIONSHIP READ PATH (`related()` with all visibility tiers — exactly - * what application reads like a VFS `readdir` consult) - * 3. ENTITY ENDPOINTS (does each verb's source/target still exist?) - * - * and classifies every discrepancy into the three failure families production - * incidents have shown: - * - * - `missingFromReads` — a canonical verb record the read path does NOT return - * for its source: PRESENT BUT INVISIBLE (adjacency/membership staleness). - * - `danglingEndpoints` — a canonical verb whose endpoint entity is gone: - * the SCAR class (write-path loss / partial delete). - * - `readOnlyVerbIds` — the read path returns an edge with NO canonical - * record: GHOST edges (stale index entries). - * - * `visibilityHiddenCount` is reported separately: an internal/system edge that is - * indexed and present but hidden from DEFAULT reads is working as designed — the - * audit reads with all tiers included so design-hiding is never misclassified as - * index loss. - * - * Full counts are always exact; only the example LISTS are capped (`maxExamples`) - * — a capped report says so via `truncatedExamples`, never silently. - */ - -import { prodLog } from '../utils/logger.js' - -/** One discrepant relationship, identified fully enough to inspect by hand. */ -export interface GraphAuditDiscrepancy { - verbId: string - from: string - to: string - type: string -} - -export interface GraphAuditReport { - /** True iff every discrepancy count is zero — `related()` returns canonical truth. */ - coherent: boolean - verbsInCanonical: number - entitiesInCanonical: number - /** Distinct source entities whose read path was actually consulted (coverage honesty). */ - sourcesChecked: number - - /** PRESENT BUT INVISIBLE: canonical records the read path omits. */ - missingFromReadsCount: number - missingFromReads: GraphAuditDiscrepancy[] - - /** SCAR CLASS: canonical verbs with a missing endpoint entity. */ - danglingEndpointsCount: number - danglingEndpoints: Array - - /** GHOST EDGES: read-path verb ids with no canonical record. */ - readOnlyCount: number - readOnlyVerbIds: string[] - - /** Canonical verbs hidden from DEFAULT reads by design (internal/system visibility). */ - visibilityHiddenCount: number - - /** Example lists above were capped at maxExamples; counts remain exact. */ - truncatedExamples: boolean - durationMs: number -} - -/** A canonical verb record, as the audit needs it. */ -export interface AuditVerbRecord { - id: string - type: string - sourceId: string - targetId: string - visibility?: string -} - -/** The seams the audit runs over — injected so the walk is testable in isolation. */ -export interface GraphAuditDeps { - /** Stream every canonical entity id (id-only; no per-entity reads needed). */ - eachNounId(consume: (id: string) => void): Promise - /** Stream every canonical verb record. */ - eachVerb(consume: (verb: AuditVerbRecord) => void): Promise - /** - * The END-TO-END relationship read for one source, ALL visibility tiers - * included — must be the same path application reads consult. - */ - readRelationsFrom(sourceId: string): Promise> -} - -export interface GraphAuditOptions { - /** Cap on entries per example list (counts stay exact). Default 100. */ - maxExamples?: number -} - -export async function runGraphAudit( - deps: GraphAuditDeps, - options: GraphAuditOptions = {} -): Promise { - const maxExamples = options.maxExamples ?? 100 - const started = Date.now() - - // 1. Canonical entity ids — endpoint existence oracle. - const entityIds = new Set() - await deps.eachNounId((id) => entityIds.add(id)) - - // 2. Canonical verb walk: group by source, check endpoints, note visibility. - const canonicalVerbIds = new Set() - const bySource = new Map() - let verbsInCanonical = 0 - let visibilityHiddenCount = 0 - let danglingEndpointsCount = 0 - const danglingEndpoints: GraphAuditReport['danglingEndpoints'] = [] - - await deps.eachVerb((verb) => { - verbsInCanonical++ - canonicalVerbIds.add(verb.id) - const list = bySource.get(verb.sourceId) - if (list) list.push(verb) - else bySource.set(verb.sourceId, [verb]) - - if (verb.visibility === 'internal' || verb.visibility === 'system') { - visibilityHiddenCount++ - } - - const fromMissing = !entityIds.has(verb.sourceId) - const toMissing = !entityIds.has(verb.targetId) - if (fromMissing || toMissing) { - danglingEndpointsCount++ - if (danglingEndpoints.length < maxExamples) { - danglingEndpoints.push({ - verbId: verb.id, - from: verb.sourceId, - to: verb.targetId, - type: verb.type, - missingEnd: fromMissing && toMissing ? 'both' : fromMissing ? 'from' : 'to' - }) - } - } - }) - - // 3. Per-source read-path comparison. A verb must be returned by the read - // path of ITS OWN source — the exact consult a readdir/traversal makes. - let missingFromReadsCount = 0 - const missingFromReads: GraphAuditDiscrepancy[] = [] - let readOnlyCount = 0 - const readOnlyVerbIds: string[] = [] - const readOnlySeen = new Set() - - for (const [sourceId, verbs] of bySource) { - const readIds = new Set((await deps.readRelationsFrom(sourceId)).map((r) => r.id)) - - for (const verb of verbs) { - if (!readIds.has(verb.id)) { - missingFromReadsCount++ - if (missingFromReads.length < maxExamples) { - missingFromReads.push({ - verbId: verb.id, - from: verb.sourceId, - to: verb.targetId, - type: verb.type - }) - } - } - } - - for (const readId of readIds) { - if (!canonicalVerbIds.has(readId) && !readOnlySeen.has(readId)) { - readOnlySeen.add(readId) - readOnlyCount++ - if (readOnlyVerbIds.length < maxExamples) { - readOnlyVerbIds.push(readId) - } - } - } - } - - const coherent = - missingFromReadsCount === 0 && danglingEndpointsCount === 0 && readOnlyCount === 0 - - const report: GraphAuditReport = { - coherent, - verbsInCanonical, - entitiesInCanonical: entityIds.size, - sourcesChecked: bySource.size, - missingFromReadsCount, - missingFromReads, - danglingEndpointsCount, - danglingEndpoints, - readOnlyCount, - readOnlyVerbIds, - visibilityHiddenCount, - truncatedExamples: - missingFromReadsCount > missingFromReads.length || - danglingEndpointsCount > danglingEndpoints.length || - readOnlyCount > readOnlyVerbIds.length, - durationMs: Date.now() - started - } - - if (coherent) { - prodLog.info( - `[GraphAudit] coherent: ${verbsInCanonical} verbs across ${bySource.size} sources — ` + - `the read path returns canonical truth (${report.durationMs}ms)` - ) - } else { - prodLog.warn( - `[GraphAudit] INCOHERENT: ${missingFromReadsCount} present-but-invisible, ` + - `${danglingEndpointsCount} dangling-endpoint, ${readOnlyCount} ghost ` + - `(of ${verbsInCanonical} canonical verbs, ${bySource.size} sources, ` + - `${visibilityHiddenCount} visibility-hidden by design) — ${report.durationMs}ms` - ) - } - - return report -} diff --git a/src/import/BackgroundDeduplicator.ts b/src/import/BackgroundDeduplicator.ts index 073fbd9e..53b28262 100644 --- a/src/import/BackgroundDeduplicator.ts +++ b/src/import/BackgroundDeduplicator.ts @@ -41,14 +41,6 @@ export interface DeduplicationStats { * - Import-scoped deduplication (no cross-contamination) * - 3-tier strategy (ID → Name → Similarity) * - Uses existing indexes (EntityIdMapper, MetadataIndexManager, TypeAware HNSW) - * - * Lifecycle: ONE instance per brain, owned by Brainy (getBackgroundDeduplicator) - * so the debounce genuinely spans imports and brain.close() cancels pending - * work via cancelPending() — this pass merge-DELETES duplicate entities, so it - * must never fire against a closed brain. The enableDeduplication gate lives - * at the scheduling call site (ImportCoordinator); scheduleDedup itself is - * unconditional. The timer is unref'd — a pending pass never holds the - * process open. */ export class BackgroundDeduplicator { private brain: Brainy @@ -75,15 +67,12 @@ export class BackgroundDeduplicator { clearTimeout(this.debounceTimer) } - // Schedule for 5 minutes from now. unref'd: a pending dedup pass must - // never hold the process open (exit-hang class) — if the process exits - // first, the pass simply never runs; imports are already durable. + // Schedule for 5 minutes from now this.debounceTimer = setTimeout(() => { this.runBatchDedup().catch(error => { prodLog.error('[BackgroundDedup] Batch dedup failed:', error) }) }, 5 * 60 * 1000) - this.debounceTimer.unref?.() } /** diff --git a/src/import/ImportCoordinator.ts b/src/import/ImportCoordinator.ts index 1e1316b7..21d09410 100644 --- a/src/import/ImportCoordinator.ts +++ b/src/import/ImportCoordinator.ts @@ -13,6 +13,7 @@ import { Brainy } from '../brainy.js' import { FormatDetector, SupportedFormat } from './FormatDetector.js' import { ImportHistory, type ImportHistoryEntry } from './ImportHistory.js' +import { BackgroundDeduplicator } from './BackgroundDeduplicator.js' import { SmartExcelImporter } from '../importers/SmartExcelImporter.js' import { SmartPDFImporter } from '../importers/SmartPDFImporter.js' import { SmartCSVImporter } from '../importers/SmartCSVImporter.js' @@ -111,12 +112,7 @@ export interface ValidImportOptions { /** Confidence threshold for entities */ confidenceThreshold?: number - /** - * Enable entity deduplication (default: true). Gates BOTH passes: the - * inline merge during import AND the debounced background pass that runs - * ~5 minutes after the last import (which merge-DELETES duplicate entities). - * Set false for deployments that must never auto-remove records. - */ + /** Enable entity deduplication across imports */ enableDeduplication?: boolean /** Similarity threshold for deduplication (0-1) */ @@ -290,6 +286,7 @@ export class ImportCoordinator { private brain: Brainy private detector: FormatDetector private history: ImportHistory + private backgroundDedup: BackgroundDeduplicator private excelImporter: SmartExcelImporter private pdfImporter: SmartPDFImporter private csvImporter: SmartCSVImporter @@ -303,6 +300,7 @@ export class ImportCoordinator { this.brain = brain this.detector = new FormatDetector() this.history = new ImportHistory(brain) + this.backgroundDedup = new BackgroundDeduplicator(brain) this.excelImporter = new SmartExcelImporter(brain) this.pdfImporter = new SmartPDFImporter(brain) this.csvImporter = new SmartCSVImporter(brain) @@ -1461,16 +1459,9 @@ export class ImportCoordinator { } } - // Schedule background deduplication (debounced 5 minutes, brain-owned so - // close() can cancel it). Honors the same enableDeduplication gate as the - // inline pass — false means NO dedup, inline or background. - if ( - trackingContext && - trackingContext.importId && - options.enableDeduplication !== false - ) { - const backgroundDedup = await this.brain.getBackgroundDeduplicator() - backgroundDedup.scheduleDedup(trackingContext.importId) + // Schedule background deduplication (debounced 5 minutes) + if (trackingContext && trackingContext.importId) { + this.backgroundDedup.scheduleDedup(trackingContext.importId) } return { diff --git a/src/index.ts b/src/index.ts index b978b9fd..bdb5ff73 100644 --- a/src/index.ts +++ b/src/index.ts @@ -28,16 +28,6 @@ export type { FileVersion } from './vfs/types.js' // Export diagnostics result type export type { DiagnosticsResult } from './brainy.js' -export type { - GraphAuditReport, - GraphAuditDiscrepancy -} from './graph/graphAudit.js' -export { - checkOsLimits, - NOFILE_POOL_FLOOR, - MAX_MAP_COUNT_POOL_FLOOR -} from './utils/osLimits.js' -export type { OsLimitsReport } from './utils/osLimits.js' // Export Brainy configuration and types export type { @@ -201,7 +191,6 @@ export type { TxLogEntry, CompactHistoryOptions, CompactHistoryResult, - HistoryStats, ChangedIds, DiffResult, HistoryVersion, @@ -213,7 +202,6 @@ export type { CommitFact, FactOp, FactScanBatch, - SCANFACTS_FIRST_BATCH_MS, FactScanHandle } from './db/factLog.js' // The generalized family stamp — which source generation a projection diff --git a/src/storage/adapters/fileSystemStorage.ts b/src/storage/adapters/fileSystemStorage.ts index 5eb4785a..4f246f67 100644 --- a/src/storage/adapters/fileSystemStorage.ts +++ b/src/storage/adapters/fileSystemStorage.ts @@ -96,14 +96,6 @@ export class FileSystemStorage extends BaseStorage { private static readonly WRITER_STALE_THRESHOLD_MS = 60_000 private writerLockHeartbeat?: NodeJS.Timeout private writerLockInfo?: WriterLockInfo - /** - * The currently-executing heartbeat refresh, if any. `releaseWriterLock()` - * awaits it before unlinking: clearInterval() stops FUTURE ticks but not a - * tick already in flight, and a straggler landing after the unlink would - * RE-CREATE the lock file — a phantom lock blocking the next writer until - * the stale TTL expires (the pool-eviction reopen case). - */ - private writerHeartbeatInFlight?: Promise // Flush-request RPC state. The writer polls `locks/_flush_requests/` for // new `.req` files and emits `.ack` files in `locks/_flush_responses/` after @@ -1772,158 +1764,79 @@ export class FileSystemStorage extends BaseStorage { const lockFile = path.join(this.lockDir, FileSystemStorage.WRITER_LOCK_FILE) const os = await import('node:os') const hostname = os.hostname() - const myPid = typeof process !== 'undefined' && process.pid ? process.pid : 0 + const now = new Date().toISOString() - // Bounded acquire loop. The CLAIM itself is an atomic create-exclusive - // write (O_EXCL) — two processes racing an ABSENT lock can never both - // succeed, which closes the read-then-write window where the loser used - // to keep running unlocked, silently. An EEXIST loser loops, re-reads, - // and handles whatever it finds honestly (fresh foreign lock → loud - // throw; stale/forced → verified takeover). - const MAX_ATTEMPTS = 3 - for (let attempt = 0; attempt < MAX_ATTEMPTS; attempt++) { - const now = new Date().toISOString() - const existing = await this.readWriterLock() - - if (existing) { - // Same-process re-open: a second Brainy instance in this Node process - // (e.g. test "simulate server restart" patterns, or a consumer that - // explicitly re-instantiates without closing first). This isn't the - // dangerous cross-process case the lock exists to prevent — the two - // instances share a memory space and can't silently diverge from each - // other beyond what their callers already see. Warn and take over. - if (existing.pid === myPid && existing.hostname === hostname && !options?.force) { - console.warn( - `[brainy] Re-acquiring writer lock for ${this.rootDir} held by the same process (PID ${existing.pid}). ` + - `If you intended to keep the previous Brainy instance alive, this is a bug — close it first.` - ) - const info: WriterLockInfo = { - pid: myPid, - hostname, - startedAt: now, - lastHeartbeat: now, - version: getBrainyVersion(), - rootDir: this.rootDir - } - await this.writeFileAtomic(lockFile, JSON.stringify(info, null, 2)) - this.installWriterLock(info) - return info - } - - const stale = !options?.force && (await this.isWriterLockStale(existing)) - if (!options?.force && !stale) { + const existing = await this.readWriterLock() + if (existing && !options?.force) { + // Same-process re-open: a second Brainy instance in this Node process + // (e.g. test "simulate server restart" patterns, or a consumer that + // explicitly re-instantiates without closing first). This isn't the + // dangerous cross-process case the lock exists to prevent — the two + // instances share a memory space and can't silently diverge from each + // other beyond what their callers already see. Warn and take over the lock. + if (existing.pid === (typeof process !== 'undefined' ? process.pid : 0) && + existing.hostname === hostname) { + console.warn( + `[brainy] Re-acquiring writer lock for ${this.rootDir} held by the same process (PID ${existing.pid}). ` + + `If you intended to keep the previous Brainy instance alive, this is a bug — close it first.` + ) + } else { + const stale = await this.isWriterLockStale(existing) + if (!stale) { // Consumer-facing error contract: callers detect this case via // err.code and read the holder's details from err.lockInfo. - throw this.writerLockedError(existing) + const err = new Error( + `Another writer holds this Brainy directory.\n` + + ` PID: ${existing.pid} on host ${existing.hostname}\n` + + ` Started: ${existing.startedAt}\n` + + ` Heartbeat: ${existing.lastHeartbeat}\n` + + ` Version: ${existing.version}\n` + + ` Directory: ${this.rootDir}\n\n` + + `For diagnostic queries against this live store, use:\n` + + ` const reader = await Brainy.openReadOnly({ storage: { type: 'filesystem', path: '${this.rootDir}' } })\n\n` + + `If you have verified the existing lock is stale (e.g. a crashed writer on a different host that PID liveness cannot reach), pass { force: true }.` + ) as Error & { code: string; lockInfo: WriterLockInfo } + err.code = 'BRAINY_WRITER_LOCKED' + err.lockInfo = existing + throw err } - console.warn( - options?.force - ? `[brainy] Force-overwriting writer lock for ${this.rootDir} ` + - `(was held by PID ${existing.pid} on ${existing.hostname}).` - : `[brainy] Overwriting stale writer lock for ${this.rootDir} ` + - `(PID ${existing.pid} on ${existing.hostname} appears dead).` + `[brainy] Overwriting stale writer lock for ${this.rootDir} ` + + `(PID ${existing.pid} on ${existing.hostname} appears dead).` ) - // Takeover: verify the file still holds the lock we judged (a live - // successor may have claimed meanwhile), then remove it and fall - // through to the atomic claim below. A racing claimer who beats us to - // the create simply wins — our next loop iteration reads their fresh - // lock and throws honestly. (Advisory file locking has no - // compare-and-delete; staleness requiring a 60s-old heartbeat keeps - // the residual verify-to-unlink window practically unreachable.) - const recheck = await this.readWriterLock() - if ( - recheck && - (recheck.pid !== existing.pid || - recheck.startedAt !== existing.startedAt || - recheck.lastHeartbeat !== existing.lastHeartbeat) - ) { - continue // the lock changed hands while we deliberated — re-evaluate - } - try { - await fs.promises.unlink(lockFile) - } catch (err: any) { - if (err.code !== 'ENOENT') throw err - } } - - const info: WriterLockInfo = { - pid: myPid, - hostname, - startedAt: existing && options?.force ? existing.startedAt : now, - lastHeartbeat: now, - version: getBrainyVersion(), - rootDir: this.rootDir - } - - // The atomic claim: create-exclusive, so exactly ONE racer wins. - try { - await fs.promises.writeFile(lockFile, JSON.stringify(info, null, 2), { flag: 'wx' }) - } catch (err: any) { - if (err.code === 'EEXIST') { - continue // someone else claimed between our read and create — re-evaluate - } - throw err - } - - this.installWriterLock(info) - return info + } else if (existing && options?.force) { + console.warn( + `[brainy] Force-overwriting writer lock for ${this.rootDir} ` + + `(was held by PID ${existing.pid} on ${existing.hostname}).` + ) } - // Attempts exhausted: something is claiming this directory faster than we - // can evaluate it. Read whoever holds it now and fail loudly with their - // details rather than degrading into a lockless open. - const holder = await this.readWriterLock() - if (holder) throw this.writerLockedError(holder) - throw new Error( - `Failed to acquire the writer lock for ${this.rootDir} after ${MAX_ATTEMPTS} attempts — ` + - `the lock file is being contended. Retry, or inspect ${lockFile}.` - ) - } + const info: WriterLockInfo = { + pid: typeof process !== 'undefined' && process.pid ? process.pid : 0, + hostname, + startedAt: existing && options?.force ? existing.startedAt : now, + lastHeartbeat: now, + version: getBrainyVersion(), + rootDir: this.rootDir + } - /** Record lock ownership + start the unref'd heartbeat. */ - private installWriterLock(info: WriterLockInfo): void { + await this.writeFileAtomic(lockFile, JSON.stringify(info, null, 2)) this.writerLockInfo = info // Heartbeat — rewrite lastHeartbeat every WRITER_HEARTBEAT_MS so other // processes can tell a live writer from one that crashed without releasing. this.writerLockHeartbeat = setInterval(() => { - const tick = this.refreshWriterLockHeartbeat().catch((err) => { - // ENOENT = the lock (or its directory) vanished mid-refresh — the - // store was released or removed under us; the next acquire recreates - // it. Benign by construction; anything else stays loud. - if ((err as NodeJS.ErrnoException)?.code !== 'ENOENT') { - console.warn('[brainy] Failed to refresh writer lock heartbeat:', err) - } - }) - this.writerHeartbeatInFlight = tick.finally(() => { - if (this.writerHeartbeatInFlight === tick) { - this.writerHeartbeatInFlight = undefined - } + this.refreshWriterLockHeartbeat().catch((err) => { + console.warn('[brainy] Failed to refresh writer lock heartbeat:', err) }) }, FileSystemStorage.WRITER_HEARTBEAT_MS) if (typeof this.writerLockHeartbeat.unref === 'function') { // Don't keep the event loop alive just for the heartbeat. this.writerLockHeartbeat.unref() } - } - /** The consumer-facing BRAINY_WRITER_LOCKED error, holder details attached. */ - private writerLockedError(existing: WriterLockInfo): Error { - const err = new Error( - `Another writer holds this Brainy directory.\n` + - ` PID: ${existing.pid} on host ${existing.hostname}\n` + - ` Started: ${existing.startedAt}\n` + - ` Heartbeat: ${existing.lastHeartbeat}\n` + - ` Version: ${existing.version}\n` + - ` Directory: ${this.rootDir}\n\n` + - `For diagnostic queries against this live store, use:\n` + - ` const reader = await Brainy.openReadOnly({ storage: { type: 'filesystem', path: '${this.rootDir}' } })\n\n` + - `If you have verified the existing lock is stale (e.g. a crashed writer on a different host that PID liveness cannot reach), pass { force: true }.` - ) as Error & { code: string; lockInfo: WriterLockInfo } - err.code = 'BRAINY_WRITER_LOCKED' - err.lockInfo = existing - return err + return info } public override async releaseWriterLock(): Promise { @@ -1931,15 +1844,6 @@ export class FileSystemStorage extends BaseStorage { clearInterval(this.writerLockHeartbeat) this.writerLockHeartbeat = undefined } - // Drain an in-flight heartbeat tick BEFORE unlinking: clearInterval stops - // future ticks only, and a straggler write landing after the unlink would - // re-create the lock as a phantom (blocking the next writer until the - // stale TTL). After the drain, any refresh is either fully landed (we - // unlink its output below) or not started (it sees writerLockInfo - // undefined and returns). - if (this.writerHeartbeatInFlight) { - await this.writerHeartbeatInFlight - } if (!this.writerLockInfo) { return } diff --git a/src/storage/adapters/memoryStorage.ts b/src/storage/adapters/memoryStorage.ts index 1b1f412e..bab9d4d9 100644 --- a/src/storage/adapters/memoryStorage.ts +++ b/src/storage/adapters/memoryStorage.ts @@ -133,11 +133,6 @@ export class MemoryStorage extends BaseStorage { */ protected async deleteObjectFromPath(path: string): Promise { this.objectStore.delete(path) - // Filesystem parity: on disk, objects and raw BYTE files are both just - // files — unlink removes whichever exists. Without this, deleteRawObject - // on a raw-bytes path (fact-log/generation segments) silently no-ops on - // memory storage: the delete "succeeds" and the bytes remain. - this.rawBytesStore.delete(path) } /** diff --git a/src/storage/baseStorage.ts b/src/storage/baseStorage.ts index 6daf09c0..73b8e859 100644 --- a/src/storage/baseStorage.ts +++ b/src/storage/baseStorage.ts @@ -377,14 +377,7 @@ export abstract class BaseStorage extends BaseStorageAdapter { id.startsWith('statistics_') || id === 'statistics' || id.startsWith('__chunk__') || // Metadata index chunks (roaring bitmap data) - id.startsWith('__sparse_index__') || // Metadata sparse indices (zone maps + bloom filters) - id.startsWith('__aggregation_') || // Aggregation engine definitions + state (routing is - // identical to the unknown-key fallback these keys hit - // before being listed here — this only kills the - // per-boot "Unknown key format" warning) - isSingletonSystemKey(id) // Known singletons (e.g. brainy:entityIdMapper) hit the - // same warn-then-route fallback without this — the - // routing below already handles them identically + id.startsWith('__sparse_index__') // Metadata sparse indices (zone maps + bloom filters) if (isSystemKey) { if (isSingletonSystemKey(id)) { @@ -2068,20 +2061,11 @@ export abstract class BaseStorage extends BaseStorageAdapter { // Cursor (8.0): resume token carrying the (shard, nounId) of the last returned // noun — the noun mirror of getVerbsWithPagination. When present it supersedes // `offset` and resumes the shard walk immediately AFTER that position, so a full - // walk is O(N) instead of the O(N²) of offset paging. (Previously the cursor was - // ignored, which was latent — the only multi-page consumer used a single big - // page — until small chunk sizes needed page 2 and an offset-0-on-every-call - // walk never terminated.) + // walk is O(N) instead of the O(N²) of offset paging. Malformed/foreign tokens + // decode to null → offset fallback. (Previously the cursor was ignored, which + // was latent — the only multi-page consumer used a single big page — until small + // chunk sizes needed page 2 and an offset-0-on-every-call walk never terminated.) const cursor = this.decodeNounWalkCursor(options.cursor) - if (options.cursor && cursor === null) { - // A supplied-but-undecodable resume token must FAIL, not silently restart - // at offset 0 — to a while(hasMore) caller the silent fallback re-serves - // page 1 forever: an unbounded CPU loop wearing pagination's clothes. - throw BrainyError.storage( - `getNouns: invalid pagination cursor '${options.cursor}' — cannot resume this walk. ` + - `Restart it without a cursor.` - ) - } const collected: Array<{ noun: HNSWNounWithMetadata; shard: number }> = [] // Peek one past the window so `hasMore` is decidable. Cursor mode collects one @@ -2230,14 +2214,6 @@ export abstract class BaseStorage extends BaseStorageAdapter { const { limit, offset = 0, filter } = options const cursor = this.decodeNounWalkCursor(options.cursor) - if (options.cursor && cursor === null) { - // Same law as getNouns/getVerbs: an undecodable resume token FAILS - // instead of silently restarting the walk at offset 0. - throw BrainyError.storage( - `getNounIds: invalid pagination cursor '${options.cursor}' — cannot resume this walk. ` + - `Restart it without a cursor.` - ) - } const collected: Array<{ id: string; shard: number }> = [] const peekCount = cursor ? limit + 1 : offset + limit + 1 const startShard = cursor ? cursor.shard : 0 @@ -2383,17 +2359,8 @@ export abstract class BaseStorage extends BaseStorageAdapter { // (shard, verbId) of the last returned verb. When present it SUPERSEDES `offset` // and resumes the shard walk immediately AFTER that position, so a full walk is // O(N) total instead of the O(N²) of offset paging (which re-scans from shard 0 - // every page). + // every page). Malformed / foreign tokens decode to null → offset fallback. const cursor = this.decodeVerbWalkCursor(options.cursor) - if (options.cursor && cursor === null) { - // A supplied-but-undecodable resume token must FAIL, not silently restart - // at offset 0 — to a while(hasMore) caller the silent fallback re-serves - // page 1 forever: an unbounded CPU loop wearing pagination's clothes. - throw BrainyError.storage( - `getVerbs: invalid pagination cursor '${options.cursor}' — cannot resume this walk. ` + - `Restart it without a cursor.` - ) - } // Each collected entry remembers its shard so nextCursor can point at the exact // (shard, id) resume position. diff --git a/src/transaction/Transaction.ts b/src/transaction/Transaction.ts index 092a2a25..75dccc05 100644 --- a/src/transaction/Transaction.ts +++ b/src/transaction/Transaction.ts @@ -37,24 +37,6 @@ const DEFAULT_OPTIONS: Required = { maxRollbackRetries: 3 } -/** - * The apply-phase budget for a batch of `opCount` operations. - * - * An explicit override wins untouched. Otherwise the budget SCALES with the - * batch: `max(30 000 ms, opCount × 2 000 ms)`. The per-op term is calibrated - * from field data — bulk imports on network-attached disks measure ~2 s per - * operation (each op pays canonical writes + fsync + index maintenance) — so - * a flat 30 s budget silently capped honest work at ~15 operations while - * looking generous for small batches. Scaling keeps small transacts - * fast-failing and gives bulk ones a budget proportional to the work they - * actually asked for; a trip still rolls back atomically and throws a - * retryable, fully-labeled TransactionTimeoutError. - */ -export function transactTimeoutBudget(opCount: number, override?: number): number { - if (override !== undefined) return override - return Math.max(30_000, opCount * 2_000) -} - /** * Transaction class */ @@ -132,11 +114,7 @@ export class Transaction implements TransactionContext { // into the catch below and rolls back like any other failure — it must // never bypass rollback. if (Date.now() - this.startTime > this.options.timeout) { - throw new TransactionTimeoutError(this.options.timeout, i, { - elapsedMs: Date.now() - this.startTime, - totalOperations: this.operations.length, - operationName: this.operations[i]?.name - }) + throw new TransactionTimeoutError(this.options.timeout, i) } const operation = this.operations[i] diff --git a/src/transaction/errors.ts b/src/transaction/errors.ts index c270d0ed..c9f6eee1 100644 --- a/src/transaction/errors.ts +++ b/src/transaction/errors.ts @@ -77,24 +77,11 @@ export class InvalidTransactionStateError extends TransactionError { export class TransactionTimeoutError extends TransactionError { constructor( timeoutMs: number, - operationIndex: number, - telemetry?: { - elapsedMs?: number - totalOperations?: number - operationName?: string - } + operationIndex: number ) { - const progress = - telemetry?.totalOperations !== undefined - ? `${operationIndex}/${telemetry.totalOperations}` - : String(operationIndex) - const name = telemetry?.operationName ? ` ('${telemetry.operationName}')` : '' - const elapsed = - telemetry?.elapsedMs !== undefined ? `${telemetry.elapsedMs}ms elapsed, ` : '' super( - `Transaction timed out at operation ${progress}${name} — ${elapsed}budget ${timeoutMs}ms. ` + - `The batch rolled back atomically; retry with a higher timeoutMs or a smaller batch.`, - { timeoutMs, operationIndex, ...telemetry } + `Transaction timed out after ${timeoutMs}ms at operation ${operationIndex}`, + { timeoutMs, operationIndex } ) this.name = 'TransactionTimeoutError' } diff --git a/src/types/brainy.types.ts b/src/types/brainy.types.ts index 1ec4c4c3..c2344375 100644 --- a/src/types/brainy.types.ts +++ b/src/types/brainy.types.ts @@ -1737,10 +1737,7 @@ export interface BrainyConfig { * Under Model-B EVERY write (`transact()` AND single-op `add`/`update`/ * `remove`/`relate`) produces an immutable generation record-set serving * historical reads (`asOf()`, pinned `Db` values). Without compaction those - * accumulate, so Brainy **auto-compacts at `close()`** (time-bounded per - * pass; 8.9.0 removed compaction from `flush()` — flush is durability work - * and never pays maintenance costs). A long-lived writer that never closes - * accumulates history until its next explicit `compactHistory()` call. + * accumulate, so Brainy **auto-compacts on every `flush()` and `close()`**. * Live `Db` pins are ALWAYS exempt from reclamation, in every mode. * * Modes: @@ -1757,7 +1754,7 @@ export interface BrainyConfig { * the oldest unpinned generations while ANY supplied cap is exceeded * (predictable ops). `maxAge` in ms; `maxBytes` total history bytes. * - * `autoCompact: false` disables the automatic close() compaction (manage + * `autoCompact: false` disables the automatic flush/close compaction (manage * manually via `brain.compactHistory()`). `budgetBytes` is the settable * adaptive byte budget a coordinator drives (also via `brain.setRetentionBudget()`). * Long-term archives belong in `db.persist(path)` snapshots, which compaction @@ -1775,7 +1772,7 @@ export interface BrainyConfig { maxBytes?: number /** Adaptive byte budget for this brain, driven by a coordinator (e.g. cor). */ budgetBytes?: number - /** Run compaction automatically at close() (default: true; 8.9.0 — flush() never compacts). */ + /** Run compaction automatically on flush()/close() (default: true). */ autoCompact?: boolean } diff --git a/src/utils/metadataIndex.ts b/src/utils/metadataIndex.ts index c772bce4..fd325319 100644 --- a/src/utils/metadataIndex.ts +++ b/src/utils/metadataIndex.ts @@ -1867,26 +1867,9 @@ export class MetadataIndexManager implements MetadataIndexProvider { // not once per AND-clause inside it. const unindexedFields: string[] = [] - for (const [rawField, condition] of Object.entries(filter)) { + for (const [field, condition] of Object.entries(filter)) { // Skip logical operators - if (rawField === 'allOf' || rawField === 'anyOf' || rawField === 'not') continue - - // Metadata is FLATTENED at index time (metadata.entry.title indexes as - // entry.title), so a `metadata.`-prefixed where key is almost always - // the caller spelling the STORAGE shape rather than the index shape. - // Accept both spellings: when the key as spelled is unindexed but its - // stripped spelling is, query the stripped one. A literal nested - // custom key named `metadata` still wins when indexed as spelled - // (checked first), so that rare shape keeps working. - let field = rawField - if ( - rawField.startsWith('metadata.') && - this.columnStore && - !this.columnStore.hasField(rawField) && - this.columnStore.hasField(rawField.slice('metadata.'.length)) - ) { - field = rawField.slice('metadata.'.length) - } + if (field === 'allOf' || field === 'anyOf' || field === 'not') continue let fieldResults: string[] = [] diff --git a/src/utils/osLimits.ts b/src/utils/osLimits.ts deleted file mode 100644 index 57e1ccf1..00000000 --- a/src/utils/osLimits.ts +++ /dev/null @@ -1,141 +0,0 @@ -/** - * @module utils/osLimits - * @description Detect-and-warn for OS resource limits that bite at POOL scale. - * - * A single brain rarely notices them, but a pool of brains — especially with a - * native accelerator memory-mapping many index files per brain — consumes file - * descriptors and memory mappings multiplicatively. On stock Linux defaults - * (RLIMIT_NOFILE soft 1024, vm.max_map_count 65530) the failure arrives as - * EMFILE or a failed mmap deep inside an index open, long after the real cause - * (the limit) stopped being visible. This module reads the limits at open and - * WARNS ONCE per process with the exact raise commands, so the operator learns - * the fix before the incident instead of from it. - * - * Read-only and Linux-only by construction: both sources are `/proc` files. - * On platforms where they are absent the check reports nulls and stays silent — - * no limit read means no claim made, never a guessed warning. - */ - -import * as fs from 'node:fs' -import { prodLog } from './logger.js' - -/** Soft-NOFILE floor below which pool-scale use is at EMFILE risk. */ -export const NOFILE_POOL_FLOOR = 65536 - -/** vm.max_map_count floor below which mmap-heavy native indexes are at risk. */ -export const MAX_MAP_COUNT_POOL_FLOOR = 262144 - -export interface OsLimitsReport { - /** RLIMIT_NOFILE soft limit (null when unreadable; Infinity for 'unlimited'). */ - nofileSoft: number | null - /** RLIMIT_NOFILE hard limit (null when unreadable; Infinity for 'unlimited'). */ - nofileHard: number | null - /** vm.max_map_count (null when unreadable). */ - maxMapCount: number | null - /** Human-actionable warnings for limits below the pool floors. Empty = fine. */ - warnings: string[] -} - -/** - * Parse the `Max open files` row of a `/proc//limits` document into - * soft/hard values. Returns nulls when the row is absent or malformed. - */ -export function parseProcLimits(content: string): { soft: number | null; hard: number | null } { - const line = content.split('\n').find((l) => l.startsWith('Max open files')) - if (!line) return { soft: null, hard: null } - const m = line.match(/^Max open files\s+(\S+)\s+(\S+)/) - if (!m) return { soft: null, hard: null } - const parse = (v: string): number | null => { - if (v === 'unlimited') return Infinity - const n = Number.parseInt(v, 10) - return Number.isNaN(n) ? null : n - } - return { soft: parse(m[1]), hard: parse(m[2]) } -} - -/** - * Assess readable limits against the pool floors. Pure — feed it any values. - * A null (unreadable) limit produces NO warning: no measurement, no claim. - */ -export function assessOsLimits(limits: { - nofileSoft: number | null - nofileHard: number | null - maxMapCount: number | null -}): string[] { - const warnings: string[] = [] - - if (limits.nofileSoft !== null && limits.nofileSoft < NOFILE_POOL_FLOOR) { - const hardNote = - limits.nofileHard !== null && limits.nofileHard >= NOFILE_POOL_FLOOR - ? ` (the hard limit ${limits.nofileHard === Infinity ? 'unlimited' : limits.nofileHard} already allows it — raise the soft limit only)` - : '' - warnings.push( - `RLIMIT_NOFILE soft limit is ${limits.nofileSoft} — below the ${NOFILE_POOL_FLOOR} recommended ` + - `for pool-scale use (a pool of brains with a native accelerator opens many index files per brain; ` + - `the failure mode is EMFILE deep inside an index open). Raise with \`ulimit -n ${NOFILE_POOL_FLOOR}\` ` + - `or LimitNOFILE=${NOFILE_POOL_FLOOR} in the service unit${hardNote}.` - ) - } - - if (limits.maxMapCount !== null && limits.maxMapCount < MAX_MAP_COUNT_POOL_FLOOR) { - warnings.push( - `vm.max_map_count is ${limits.maxMapCount} — below the ${MAX_MAP_COUNT_POOL_FLOOR} recommended ` + - `for mmap-heavy native indexes at pool scale (each mapped index segment consumes map entries; ` + - `the failure mode is a failed mmap mid-heal). Raise with ` + - `\`sysctl -w vm.max_map_count=${MAX_MAP_COUNT_POOL_FLOOR}\` (persist in /etc/sysctl.d/).` - ) - } - - return warnings -} - -/** - * Read the limits from /proc and assess them. `readFile` is injectable for - * tests; absent/unreadable sources yield nulls (and therefore no warnings). - */ -export async function checkOsLimits( - readFile: (path: string) => Promise = async (p) => fs.promises.readFile(p, 'utf-8') -): Promise { - let nofileSoft: number | null = null - let nofileHard: number | null = null - let maxMapCount: number | null = null - - try { - const parsed = parseProcLimits(await readFile('/proc/self/limits')) - nofileSoft = parsed.soft - nofileHard = parsed.hard - } catch { - // Not Linux (or /proc unavailable) — no measurement, no claim. - } - - try { - const raw = (await readFile('/proc/sys/vm/max_map_count')).trim() - const n = Number.parseInt(raw, 10) - maxMapCount = Number.isNaN(n) ? null : n - } catch { - // Not Linux — same rule. - } - - const warnings = assessOsLimits({ nofileSoft, nofileHard, maxMapCount }) - return { nofileSoft, nofileHard, maxMapCount, warnings } -} - -/** Once-per-process latch so a brain pool warns once, not once per brain. */ -let osLimitsWarned = false - -/** - * Run the check and warn (once per process) about limits below the pool - * floors. Called from brain open; safe everywhere (silent off-Linux). - */ -export async function warnOnLowOsLimits(): Promise { - if (osLimitsWarned) return - osLimitsWarned = true - try { - const report = await checkOsLimits() - for (const warning of report.warnings) { - prodLog.warn(`[Brainy] OS limit check: ${warning}`) - } - } catch { - // The check must never affect open — measurement-only. - } -} diff --git a/src/utils/paramValidation.ts b/src/utils/paramValidation.ts index ca439524..57a9a1c0 100644 --- a/src/utils/paramValidation.ts +++ b/src/utils/paramValidation.ts @@ -209,10 +209,8 @@ export interface ValidationConfigOptions { } /** - * Auto-configured limits based on system resources. - * Derived from memory (explicit overrides > reserved memory > container limit > - * free memory). Query timing is recorded for diagnostics only — it never - * changes the cap (see `recordQuery`). + * Auto-configured limits based on system resources + * These adapt to available memory and observed performance */ export class ValidationConfig { private static instance: ValidationConfig | null = null @@ -325,23 +323,24 @@ export class ValidationConfig { } /** - * Record query timing for diagnostics. Telemetry ONLY — never mutates the cap. - * - * `maxLimit` is a MEMORY-protection bound; query duration says nothing about - * memory-per-result, so duration must never drive it. An earlier version - * "learned" here: while the lifetime-average query time exceeded 1s it shrank - * `maxLimit` by 20% per recorded query down to a floor of 1000 — below the - * documented `MIN_AUTO_QUERY_LIMIT` (10 000) and with no recovery once slow - * samples poisoned the cumulative average. On a production host a burst of - * slow aggregate queries silently strangled every consumer's `find()` to - * 1000 while the error message blamed "available free memory" — exactly the - * silent throttling this module's own contract forbids. The cap now comes - * from its construction-time basis (or explicit overrides) alone. + * Learn from actual usage to adjust limits */ recordQuery(duration: number, resultCount: number) { - void resultCount this.queryCount++ this.avgQueryTime = (this.avgQueryTime * (this.queryCount - 1) + duration) / this.queryCount + + // Only auto-adjust if not using explicit overrides + if (this.limitBasis !== 'override') { + // If queries are consistently fast with large results, increase limits + if (this.avgQueryTime < 100 && resultCount > this.maxLimit * 0.8) { + this.maxLimit = Math.min(this.maxLimit * 1.5, 100000) + } + + // If queries are slow, reduce limits + if (this.avgQueryTime > 1000) { + this.maxLimit = Math.max(this.maxLimit * 0.8, 1000) + } + } } } diff --git a/tests/integration/aggregate-reserved-fields.test.ts b/tests/integration/aggregate-reserved-fields.test.ts deleted file mode 100644 index e81692d6..00000000 --- a/tests/integration/aggregate-reserved-fields.test.ts +++ /dev/null @@ -1,169 +0,0 @@ -/** - * @module tests/integration/aggregate-reserved-fields - * @description One field-resolution law across the whole aggregation + query - * surface (SELF-AGGREGATE-DELETE-DRIFT). Laws: - * (1) aggregates grouped by a RESERVED field (subtype) decrement on delete — - * the delete-side entity view carries every reserved field, so the - * decrement finds its group (counts must never drift from ground truth); - * (2) same for update: moving an entity between reserved-field groups - * decrements the old group and increments the new one (no double-count); - * (3) aggregation source.where on a reserved field (subtype) FILTERS instead - * of silently matching nothing; - * (4) removeMany refuses empty/invalid selectors loudly (bare array, empty - * object, ids: []) instead of resolving as a silent no-op; - * (5) find() accepts both where spellings: flattened (entry.title) and - * storage-shaped (metadata.entry.title) resolve to the same rows. - */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' -import { Brainy } from '../../src/brainy.js' -import { NounType } from '../../src/types/graphTypes.js' - -const stubEmbedding = async (text: string): Promise => { - const hash = text.split('').reduce((acc, char) => acc + char.charCodeAt(0), 0) - return new Array(384).fill(0).map((_, i) => Math.sin(hash + i)) -} - -describe('aggregation + query field-resolution law', () => { - let brain: Brainy - - beforeEach(async () => { - brain = new Brainy({ - requireSubtype: false, - storage: { type: 'memory' as const }, - embeddingFunction: stubEmbedding - }) - await brain.init() - }) - - afterEach(async () => { - await brain.close() - }) - - it('reserved-field groupBy decrements on delete (the drift bug)', async () => { - brain.defineAggregate({ - name: 'by_subtype', - source: { type: NounType.Document }, - groupBy: ['subtype'], - metrics: { count: { op: 'count' } } - }) - - const ids: string[] = [] - for (let i = 0; i < 5; i++) { - ids.push( - await brain.add({ - data: `doc-${i}`, - type: NounType.Document, - subtype: 'note', - metadata: { team: 'alpha' } - }) - ) - } - let groups = await brain.queryAggregate('by_subtype') - expect(groups).toHaveLength(1) - expect(groups[0].groupKey).toEqual({ subtype: 'note' }) - expect(groups[0].metrics.count).toBe(5) - - await brain.remove(ids[0]) - await brain.flush() - - groups = await brain.queryAggregate('by_subtype') - expect(groups[0].metrics.count).toBe(4) - const live = await brain.find({ type: NounType.Document, limit: 100 }) - expect(groups[0].metrics.count).toBe(live.length) - }) - - it('reserved-field groupBy moves between groups on update (no double-count)', async () => { - brain.defineAggregate({ - name: 'by_subtype', - source: { type: NounType.Document }, - groupBy: ['subtype'], - metrics: { count: { op: 'count' } } - }) - const id = await brain.add({ - data: 'doc-move', - type: NounType.Document, - subtype: 'draft' - }) - await brain.update({ id, subtype: 'published' }) - - const groups = await brain.queryAggregate('by_subtype') - const byKey = Object.fromEntries( - groups.map((g) => [String(g.groupKey.subtype), g.metrics.count]) - ) - expect(byKey['published']).toBe(1) - // The old group must be gone or zero — never still counting the entity. - expect(byKey['draft'] ?? 0).toBe(0) - }) - - it('source.where on a reserved field filters instead of matching nothing', async () => { - brain.defineAggregate({ - name: 'notes_only', - source: { type: NounType.Document, where: { subtype: 'note' } }, - groupBy: ['team'], - metrics: { count: { op: 'count' } } - }) - await brain.add({ - data: 'n1', - type: NounType.Document, - subtype: 'note', - metadata: { team: 'alpha' } - }) - await brain.add({ - data: 'd1', - type: NounType.Document, - subtype: 'draft', - metadata: { team: 'alpha' } - }) - - const groups = await brain.queryAggregate('notes_only') - expect(groups).toHaveLength(1) - expect(groups[0].metrics.count).toBe(1) // the note, never the draft - }) - - it('removeMany refuses empty/invalid selectors loudly', async () => { - const id = await brain.add({ data: 'keep-me', type: NounType.Document }) - - // Bare array passed positionally — the classic silent no-op. - await expect( - brain.removeMany([id] as unknown as Parameters[0]) - ).rejects.toThrow(/bare array/) - // Empty selector object. - await expect( - brain.removeMany({} as Parameters[0]) - ).rejects.toThrow(/requires a selector/) - // Explicit empty id list. - await expect(brain.removeMany({ ids: [] })).rejects.toThrow(/ids: \[\]/) - - // Nothing was deleted by any of the refused calls. - expect(await brain.get(id)).toBeTruthy() - }) - - it('find() accepts both flattened and metadata.-prefixed where spellings', async () => { - await brain.add({ - data: 'nested-doc', - type: NounType.Document, - metadata: { entry: { title: 'T1' }, classifier: { contextHints: { vfsPath: '/n/a.md' } } } - }) - await brain.flush() - - const flat = await brain.find({ - type: NounType.Document, - where: { 'entry.title': 'T1' }, - limit: 10 - }) - const prefixed = await brain.find({ - type: NounType.Document, - where: { 'metadata.entry.title': 'T1' }, - limit: 10 - }) - const deepPrefixed = await brain.find({ - type: NounType.Document, - where: { 'metadata.classifier.contextHints.vfsPath': '/n/a.md' }, - limit: 10 - }) - expect(flat).toHaveLength(1) - expect(prefixed).toHaveLength(1) - expect(prefixed[0].id).toBe(flat[0].id) - expect(deepPrefixed).toHaveLength(1) - }) -}) diff --git a/tests/integration/aggregation-state-persistence.test.ts b/tests/integration/aggregation-state-persistence.test.ts deleted file mode 100644 index db8fc088..00000000 --- a/tests/integration/aggregation-state-persistence.test.ts +++ /dev/null @@ -1,311 +0,0 @@ -/** - * @module tests/integration/aggregation-state-persistence - * @description The boot-order contract for the aggregation engine. Five laws: - * (1) STATE ADOPTION — a reopen with an unchanged defineAggregate() adopts the - * persisted state and performs NO store walk. (The pre-fix behavior: the - * synchronous define always beat the async init, flagged a backfill, and - * the first query wiped the just-loaded state and re-walked the whole - * store — every restart, forever.) - * (2) SINGLE-FLIGHT + BATCH — concurrent cold queries across multiple pending - * aggregates share exactly ONE store walk; a query never wipes another's - * partial progress and M pending aggregates cost one enumeration, not M. - * (3) CHANGED DEFINITION — a real definition change still backfills, exactly. - * (4) PERSISTED-ONLY DEFINITIONS — an app that does not re-define at boot can - * query a persisted aggregate without racing a spurious "not defined". - * (5) QUIET KEYS — the engine's persistence keys (__aggregation_*) are - * recognized system keys: no "Unknown key format" warning at boot. - */ -import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' -import * as fs from 'node:fs' -import * as os from 'node:os' -import * as path from 'node:path' -import { Brainy } from '../../src/index.js' -import { NounType } from '../../src/types/graphTypes.js' -import type { AggregateDefinition } from '../../src/types/brainy.types.js' -import { prodLog } from '../../src/utils/logger.js' - -const SPENDING: AggregateDefinition = { - name: 'spending', - source: { type: NounType.Event, where: { domain: 'financial' } }, - groupBy: ['category'], - metrics: { - total: { op: 'sum', field: 'amount' }, - count: { op: 'count' } - } -} - -/** Same name, different metrics — a REAL definition change (hash differs). */ -const SPENDING_CHANGED: AggregateDefinition = { - ...SPENDING, - metrics: { - total: { op: 'sum', field: 'amount' }, - count: { op: 'count' }, - average: { op: 'avg', field: 'amount' } - } -} - -describe('aggregation state persistence — boot-order contract', () => { - let dir: string - - beforeEach(() => { - dir = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-agg-persist-')) - }) - - afterEach(() => { - fs.rmSync(dir, { recursive: true, force: true }) - }) - - const open = async (): Promise => { - const b: any = new Brainy({ - requireSubtype: false, - storage: { type: 'filesystem', path: dir }, - silent: true - }) - await b.init() - return b - } - - /** Count store walks by intercepting the storage adapter's getNouns. */ - const countWalks = (brain: any): { count: () => number } => { - const storage = brain.storage - const orig = storage.getNouns.bind(storage) - let calls = 0 - storage.getNouns = async (opts: unknown) => { - calls++ - return orig(opts) - } - return { count: () => calls } - } - - const seed = async (brain: any): Promise => { - for (let i = 0; i < 12; i++) { - await brain.add({ - data: `tx ${i}`, - type: NounType.Event, - metadata: { - domain: 'financial', - category: i % 2 === 0 ? 'food' : 'transport', - amount: 10 + i - } - }) - } - } - - it('adopts persisted state on reopen with an unchanged definition — zero walks', async () => { - const brain1 = await open() - brain1.defineAggregate(SPENDING) - await seed(brain1) - const before = await brain1.queryAggregate('spending') - expect(before.length).toBe(2) - await brain1.close() - - const brain2 = await open() - brain2.defineAggregate(SPENDING) // the standard declarative boot pattern - await brain2.getNounCount() // settle init paths before counting walks - const walks = countWalks(brain2) - - const after = await brain2.queryAggregate('spending') - - expect(walks.count()).toBe(0) - const key = (r: any) => r.groupKey.category - expect( - after.map((r: any) => [key(r), r.metrics.total, r.metrics.count]).sort() - ).toEqual( - before.map((r: any) => [key(r), r.metrics.total, r.metrics.count]).sort() - ) - await brain2.close() - }) - - it('adopted state keeps accumulating: post-reopen writes land on top of it', async () => { - const brain1 = await open() - brain1.defineAggregate(SPENDING) - await seed(brain1) - await brain1.queryAggregate('spending') - await brain1.close() - - const brain2 = await open() - brain2.defineAggregate(SPENDING) - // A write BEFORE the first query: if it lands before state adoption the - // engine must choose an exact rescan over adoption — either way the - // result must include all 13 entities. - await brain2.add({ - data: 'late tx', - type: NounType.Event, - metadata: { domain: 'financial', category: 'food', amount: 100 } - }) - const rows = await brain2.queryAggregate('spending') - const food = rows.find((r: any) => r.groupKey.category === 'food') - expect(food.metrics.count).toBe(7) // 6 seeded + 1 late - await brain2.close() - }) - - it('a changed definition still backfills — exactly once, with correct results', async () => { - const brain1 = await open() - brain1.defineAggregate(SPENDING) - await seed(brain1) - await brain1.queryAggregate('spending') - await brain1.close() - - const brain2 = await open() - brain2.defineAggregate(SPENDING_CHANGED) - await brain2.getNounCount() - const walks = countWalks(brain2) - - const rows = await brain2.queryAggregate('spending') - - expect(walks.count()).toBe(1) // 12 entities = one page = one getNouns call - const food = rows.find((r: any) => r.groupKey.category === 'food') - expect(food.metrics.count).toBe(6) - expect(food.metrics.average).toBeCloseTo(food.metrics.total / 6) - await brain2.close() - }) - - it('concurrent cold queries across two aggregates share exactly ONE walk', async () => { - const brain = await open() - brain.defineAggregate(SPENDING) - brain.defineAggregate({ - ...SPENDING, - name: 'by_category_count', - metrics: { count: { op: 'count' } } - }) - await seed(brain) - await brain.getNounCount() - const walks = countWalks(brain) - - const results = await Promise.all([ - brain.queryAggregate('spending'), - brain.queryAggregate('by_category_count'), - brain.queryAggregate('spending'), - brain.queryAggregate('by_category_count'), - brain.queryAggregate('spending'), - brain.queryAggregate('by_category_count') - ]) - - expect(walks.count()).toBe(1) // 12 entities = one page; one walk fills both - for (const rows of results) { - const total = rows.reduce((s: number, r: any) => s + r.metrics.count, 0) - expect(total).toBe(12) - } - - // Warm re-query: converged, no further walks. - await brain.queryAggregate('spending') - expect(walks.count()).toBe(1) - await brain.close() - }) - - it('persisted-only definitions are queryable without re-defining at boot', async () => { - const brain1 = await open() - brain1.defineAggregate(SPENDING) - await seed(brain1) - await brain1.queryAggregate('spending') - await brain1.close() - - const brain2 = await open() - // NO defineAggregate — the app relies on the persisted definition. - const walks = countWalks(brain2) - const rows = await brain2.queryAggregate('spending') - expect(walks.count()).toBe(0) // persisted state adopted here too - expect(rows.length).toBe(2) - await brain2.close() - }) - - it('generation-mismatched persisted state is rescanned once, loudly — never adopted', async () => { - const brain1 = await open() - brain1.defineAggregate(SPENDING) - await seed(brain1) - await brain1.queryAggregate('spending') - await brain1.close() - - // Simulate the copied-store incident class: a fact-log truncation (or an - // unclean shutdown) leaves the committed watermark different from the - // generation the flushed state was stamped with. - const tamper: any = await open() - const key = '__aggregation_state_spending__' - const stored = await tamper.storage.getMetadata(key) - expect(typeof stored.sourceGeneration).toBe('number') // the stamp is really persisted - await tamper.storage.saveMetadata(key, { - ...stored, - sourceGeneration: stored.sourceGeneration + 5 - }) - await tamper.close() - - const warnSpy = vi.spyOn(prodLog, 'warn') - const brain2 = await open() - brain2.defineAggregate(SPENDING) - await brain2.getNounCount() - const walks = countWalks(brain2) - - const rows = await brain2.queryAggregate('spending') - - expect(walks.count()).toBe(1) // exactly ONE rescan — no silent adopt, no spin - const food = rows.find((r: any) => r.groupKey.category === 'food') - expect(food.metrics.count).toBe(6) // rescan produced exact results - expect( - warnSpy.mock.calls.some(args => String(args[0]).includes('rescanning instead of adopting')) - ).toBe(true) // and it said so out loud - warnSpy.mockRestore() - await brain2.close() - }) - - it('a failing walk is loud, non-destructive, and latched — never a silent retry loop', async () => { - // Fresh define + seeded writes: the write hooks have populated LIVE state, - // and the first-query rescan is still pending. The incident shape - // (wipe-before-scan + no try/catch + per-query re-walk) would have wiped - // that live state and silently re-walked on every query. - const brain: any = await open() - brain.defineAggregate(SPENDING) - await seed(brain) - expect(brain._aggregationIndex.queryAggregate({ name: 'spending' }).length).toBe(2) - - const storage = brain.storage - const origGetNouns = storage.getNouns.bind(storage) - let walkAttempts = 0 - storage.getNouns = async () => { - walkAttempts++ - throw new Error('injected storage failure') - } - - // First query: the walk fails LOUDLY with the storage error. - await expect(brain.queryAggregate('spending')).rejects.toThrow('injected storage failure') - expect(walkAttempts).toBe(1) - - // Live state was NOT destroyed by the failed walk (staging was dropped). - expect(brain._aggregationIndex.queryAggregate({ name: 'spending' }).length).toBe(2) - - // Second query inside the cooldown: instant loud failure, NO new walk. - await expect(brain.queryAggregate('spending')).rejects.toThrow('failure cooldown') - expect(walkAttempts).toBe(1) - - // Heal the storage + expire the cooldown: one fresh walk succeeds exactly. - storage.getNouns = origGetNouns - brain._aggregationBackfillFailure.at = Date.now() - 60_000 - const rows = await brain.queryAggregate('spending') - const food = rows.find((r: any) => r.groupKey.category === 'food') - expect(food.metrics.count).toBe(6) - await brain.close() - }) - - it('aggregation persistence keys never log "Unknown key format"', async () => { - const warnSpy = vi.spyOn(prodLog, 'warn') - const brain1 = await open() - brain1.defineAggregate(SPENDING) - await seed(brain1) - await brain1.queryAggregate('spending') - await brain1.close() - - const brain2 = await open() - brain2.defineAggregate(SPENDING) - await brain2.queryAggregate('spending') - await brain2.close() - - const offenders = warnSpy.mock.calls - .map(args => String(args[0])) - .filter( - msg => - msg.includes('Unknown key format') && - (msg.includes('__aggregation_') || msg.includes('brainy:entityIdMapper')) - ) - expect(offenders).toEqual([]) - warnSpy.mockRestore() - }) -}) diff --git a/tests/integration/background-dedup-lifecycle.test.ts b/tests/integration/background-dedup-lifecycle.test.ts deleted file mode 100644 index 48c7a4a7..00000000 --- a/tests/integration/background-dedup-lifecycle.test.ts +++ /dev/null @@ -1,85 +0,0 @@ -/** - * @module tests/integration/background-dedup-lifecycle - * @description The post-import background deduplication pass (a merge-DELETE - * writer) obeys the same contract as the inline pass. Laws: - * (1) enableDeduplication:false schedules NO background pass — the brain-owned - * deduplicator is never even constructed; - * (2) by default the pass IS scheduled, brain-owned, with an unref'd timer - * (a pending pass never holds the process open); - * (3) repeated imports debounce into ONE pending batch on ONE instance - * (per-coordinator instances used to arm one timer per import); - * (4) close() cancels pending work — no delete pass can fire after close. - */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' -import { Brainy } from '../../src/brainy.js' - -const ROWS = [ - { name: 'Alice Zephyr', role: 'engineer' }, - { name: 'Bob Quill', role: 'writer' } -] - -// Keep imports fast and deterministic — dedup scheduling is what's under test. -const FAST = { - enableNeuralExtraction: false, - enableRelationshipInference: false, - enableConceptExtraction: false -} as const - -// Deterministic stub embedder (hnsw-rebuild.test.ts pattern) — dedup -// scheduling never inspects vector CONTENT, so skip the WASM model load. -const stubEmbedding = async (text: string): Promise => { - const hash = text.split('').reduce((acc, char) => acc + char.charCodeAt(0), 0) - const vector = new Array(384).fill(0).map((_, i) => Math.sin(hash + i)) - return vector -} - -describe('background dedup lifecycle', () => { - let brain: Brainy - - beforeEach(async () => { - brain = new Brainy({ - requireSubtype: false, - storage: { type: 'memory' as const }, - embeddingFunction: stubEmbedding - }) - await brain.init() - }) - - afterEach(async () => { - await brain.close() - }) - - it('enableDeduplication:false schedules no background pass at all', async () => { - await brain.import(ROWS, { ...FAST, enableDeduplication: false }) - expect((brain as any)._backgroundDedup).toBeUndefined() - }) - - it('default schedules a brain-owned pass with an unref-ed timer', async () => { - await brain.import(ROWS, { ...FAST }) - const dedup = (brain as any)._backgroundDedup - expect(dedup).toBeDefined() - expect(dedup.pendingImports.size).toBe(1) - const timer = dedup.debounceTimer - expect(timer).toBeDefined() - // Node timers expose hasRef(); an unref'd timer must not hold the process. - expect(typeof timer.hasRef).toBe('function') - expect(timer.hasRef()).toBe(false) - }) - - it('imports debounce into one pending batch on one brain-owned instance', async () => { - await brain.import(ROWS, { ...FAST }) - const first = (brain as any)._backgroundDedup - await brain.import([{ name: 'Cara Vex', role: 'analyst' }], { ...FAST }) - expect((brain as any)._backgroundDedup).toBe(first) - expect(first.pendingImports.size).toBe(2) - }) - - it('close() cancels pending background dedup', async () => { - await brain.import(ROWS, { ...FAST }) - const dedup = (brain as any)._backgroundDedup - expect(dedup.debounceTimer).toBeDefined() - await brain.close() - expect(dedup.debounceTimer).toBeUndefined() - expect(dedup.pendingImports.size).toBe(0) - }) -}) diff --git a/tests/integration/db-mvcc.test.ts b/tests/integration/db-mvcc.test.ts index 959d0053..10a158ae 100644 --- a/tests/integration/db-mvcc.test.ts +++ b/tests/integration/db-mvcc.test.ts @@ -1280,32 +1280,24 @@ describe('8.0 Db API — generational MVCC', () => { await expect(reopened.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError) }) - it('Model-B retention — flush() NEVER compacts (8.9.0); adaptive reclaim runs at close()', async () => { - // Default brain → ADAPTIVE retention with a driven byte budget far below - // the accumulated history (~13 generations of full-vector before-images). - // The 8.9.0 law: flush() is durability-only — it must not reclaim even - // when the budget is exceeded (reclaim-on-flush blocked production writes - // for 25-191s). Maintenance runs at close(), time-bounded. - const { brain, dir } = await openFsBrain() + it('Model-B retention — setRetentionBudget drives adaptive reclaim on flush; live data intact', async () => { + // Default brain → ADAPTIVE retention. A coordinator (e.g. cor's ResourceManager) + // pushes a byte budget via setRetentionBudget(); auto-compaction on flush() reclaims + // oldest history down toward it. Each update's before-image carries the full prior + // 384-dim vector (~KBs), so ~13 generations far exceed a few-KB budget. + const { brain } = await openFsBrain() const a = uid('ret-budget') await brain.add({ id: a, type: NounType.Document, data: 'v0', vector: vec(1), metadata: { v: 0 } }) for (let v = 1; v <= 12; v++) await brain.update({ id: a, metadata: { v } }) brain.setRetentionBudget(6000) // ~6 KB — well below the accumulated history - await brain.flush() + await brain.flush() // group-commit + adaptive auto-compaction under the budget - // flush() paid durability only: nothing reclaimed, all history readable. - expect(generationStoreOf(brain).horizon()).toBe(0) - const probe = await brain.asOf(1) // readable proves nothing was reclaimed… - await probe.release() // …and MUST be released: a held pin would (correctly) - // protect every newer generation through the close() compaction below. - await brain.close() // ← THE auto-compaction site now - - // close() reclaimed under the budget; live record intact; horizon durable. - const { brain: reopened } = await openFsBrain(dir) - expect(generationStoreOf(reopened).horizon()).toBeGreaterThan(0) - await expect(reopened.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError) - expect((await reopened.get(a))?.metadata?.v).toBe(12) + // History was reclaimed (the horizon advanced past the oldest generations)… + expect(generationStoreOf(brain).horizon()).toBeGreaterThan(0) + await expect(brain.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError) + // …but the budget reclaims HISTORY only — the live record is untouched. + expect((await brain.get(a))?.metadata?.v).toBe(12) }) // ========================================================================== diff --git a/tests/integration/graph-audit.test.ts b/tests/integration/graph-audit.test.ts deleted file mode 100644 index 74fcc96a..00000000 --- a/tests/integration/graph-audit.test.ts +++ /dev/null @@ -1,164 +0,0 @@ -/** - * @module tests/integration/graph-audit - * @description brain.auditGraph() — the read-only graph-truth instrument. - * Laws under test: - * (1) a healthy brain audits COHERENT: every canonical verb is returned by the - * read path of its source, endpoints exist, no ghosts; - * (2) design-hidden edges (internal/system visibility) are counted separately - * and never misclassified as index loss; - * (3) a verb whose endpoint entity was destroyed at the storage layer (the - * scar class) is flagged as a dangling endpoint, loudly; - * (4) the classification core flags present-but-invisible and ghost edges - * exactly (exercised via injected seams — manufacturing a genuinely stale - * adjacency index end-to-end would require corrupting internals the - * public API rightly refuses to corrupt). - */ -import { describe, it, expect, beforeEach, afterEach } from 'vitest' -import * as fs from 'node:fs' -import * as os from 'node:os' -import * as path from 'node:path' -import { Brainy } from '../../src/index.js' -import { NounType, VerbType } from '../../src/types/graphTypes.js' -import { runGraphAudit, type AuditVerbRecord } from '../../src/graph/graphAudit.js' - -describe('brain.auditGraph() — graph-truth audit', () => { - let dir: string - let brain: any - - beforeEach(async () => { - dir = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-graph-audit-')) - brain = new Brainy({ - requireSubtype: false, - storage: { type: 'filesystem', path: dir }, - silent: true - }) - await brain.init() - }) - - afterEach(async () => { - await brain.close().catch(() => {}) - fs.rmSync(dir, { recursive: true, force: true }) - }) - - async function seedGraph(): Promise<{ ids: string[]; verbIds: string[] }> { - const ids: string[] = [] - for (let i = 0; i < 5; i++) { - ids.push( - await brain.add({ - data: `entity ${i}`, - type: NounType.Concept, - metadata: { n: i } - }) - ) - } - const verbIds: string[] = [] - verbIds.push(await brain.relate({ from: ids[0], to: ids[1], type: VerbType.RelatedTo })) - verbIds.push(await brain.relate({ from: ids[0], to: ids[2], type: VerbType.Contains })) - verbIds.push(await brain.relate({ from: ids[1], to: ids[3], type: VerbType.DependsOn })) - verbIds.push(await brain.relate({ from: ids[3], to: ids[4], type: VerbType.RelatedTo })) - return { ids, verbIds } - } - - it('audits a healthy brain as coherent, with exact counts', async () => { - const { ids } = await seedGraph() - const report = await brain.auditGraph() - - expect(report.coherent).toBe(true) - expect(report.verbsInCanonical).toBe(4) - expect(report.entitiesInCanonical).toBeGreaterThanOrEqual(ids.length) // VFS root etc. may add system nouns - expect(report.sourcesChecked).toBe(3) // ids[0], ids[1], ids[3] - expect(report.missingFromReadsCount).toBe(0) - expect(report.danglingEndpointsCount).toBe(0) - expect(report.readOnlyCount).toBe(0) - expect(report.truncatedExamples).toBe(false) - }) - - it('counts design-hidden edges separately and stays coherent', async () => { - const { ids } = await seedGraph() - await brain.relate({ - from: ids[2], - to: ids[4], - type: VerbType.RelatedTo, - visibility: 'internal' - }) - - const report = await brain.auditGraph() - expect(report.coherent).toBe(true) // hidden-by-design is NOT a discrepancy - expect(report.verbsInCanonical).toBe(5) - expect(report.visibilityHiddenCount).toBeGreaterThanOrEqual(1) - }) - - it('flags a destroyed endpoint as a dangling verb (the scar class)', async () => { - const { ids } = await seedGraph() - // Destroy ids[4] at the STORAGE layer (bypassing remove(), which would - // also delete its verbs) — the historical partial-delete scar shape. - await brain.storage.deleteNoun(ids[4]) - - const report = await brain.auditGraph() - expect(report.coherent).toBe(false) - expect(report.danglingEndpointsCount).toBe(1) - expect(report.danglingEndpoints[0].to).toBe(ids[4]) - expect(report.danglingEndpoints[0].missingEnd).toBe('to') - }) -}) - -describe('runGraphAudit classification core (injected seams)', () => { - const verb = (id: string, from: string, to: string): AuditVerbRecord => ({ - id, - type: 'relatedTo', - sourceId: from, - targetId: to - }) - - const deps = (opts: { - nouns: string[] - verbs: AuditVerbRecord[] - reads: Record // sourceId -> verb ids the read path returns - }) => ({ - eachNounId: async (consume: (id: string) => void) => { - for (const id of opts.nouns) consume(id) - }, - eachVerb: async (consume: (v: AuditVerbRecord) => void) => { - for (const v of opts.verbs) consume(v) - }, - readRelationsFrom: async (sourceId: string) => - (opts.reads[sourceId] ?? []).map((id) => ({ id })) - }) - - it('flags a canonical verb the read path omits — present but invisible', async () => { - const report = await runGraphAudit( - deps({ - nouns: ['A', 'B', 'C'], - verbs: [verb('v1', 'A', 'B'), verb('v2', 'A', 'C')], - reads: { A: ['v1'] } // v2 exists canonically but reads miss it - }) - ) - expect(report.coherent).toBe(false) - expect(report.missingFromReadsCount).toBe(1) - expect(report.missingFromReads[0].verbId).toBe('v2') - }) - - it('flags a read-path edge with no canonical record — a ghost', async () => { - const report = await runGraphAudit( - deps({ - nouns: ['A', 'B'], - verbs: [verb('v1', 'A', 'B')], - reads: { A: ['v1', 'ghost-9'] } - }) - ) - expect(report.coherent).toBe(false) - expect(report.readOnlyCount).toBe(1) - expect(report.readOnlyVerbIds).toEqual(['ghost-9']) - }) - - it('caps example lists but keeps counts exact, and says so', async () => { - const verbs = Array.from({ length: 10 }, (_, i) => verb(`v${i}`, 'A', 'B')) - const report = await runGraphAudit( - deps({ nouns: ['A', 'B'], verbs, reads: { A: [] } }), - { maxExamples: 3 } - ) - expect(report.missingFromReadsCount).toBe(10) - expect(report.missingFromReads.length).toBe(3) - expect(report.truncatedExamples).toBe(true) - }) -}) diff --git a/tests/integration/history-repacking.test.ts b/tests/integration/history-repacking.test.ts deleted file mode 100644 index 2bcee038..00000000 --- a/tests/integration/history-repacking.test.ts +++ /dev/null @@ -1,186 +0,0 @@ -/** - * @module tests/integration/history-repacking - * @description The D1+D3 two-tier history lifecycle end-to-end on a real - * brain. Laws: (1) repacking is RE-REPRESENTATION — after folding, every - * asOf() read below the fold boundary answers exactly as before, across a - * cold reopen; (2) folded per-generation directories are physically gone - * (the file-count cure is real, not cosmetic); (3) repack + reclaim compose: - * bounded retention after repacking drops whole segments and asOf below the - * horizon throws GenerationCompactedError; (4) repackHistory is explicit - * API and time-bounded (spent budget = consistent no-op). - * - * Uses a tiny REPACK_LIVE_WINDOW override so a small history has a cold - * tier at all (the production window is 1024). - */ -import { describe, it, expect, afterEach } from 'vitest' -import * as fs from 'node:fs' -import * as path from 'node:path' -import * as os from 'node:os' -import { Brainy } from '../../src/brainy.js' -import { NounType } from '../../src/types/graphTypes.js' -import { GenerationStore } from '../../src/db/generationStore.js' -import { GenerationCompactedError } from '../../src/db/errors.js' -import { SEGMENTS_PREFIX } from '../../src/db/generationSegments.js' - -const stub = async (text: string): Promise => { - const h = text.split('').reduce((a, c) => a + c.charCodeAt(0), 0) - return new Array(384).fill(0).map((_, i) => Math.sin(h + i)) -} - -const openBrain = async (dir: string): Promise => { - const brain = new Brainy({ - requireSubtype: false, - storage: { type: 'filesystem', path: dir }, - embeddingFunction: stub - }) - await brain.init() - return brain -} - -describe('history repacking — the two-tier lifecycle', () => { - const dirs: string[] = [] - const tempDir = (): string => { - const d = fs.mkdtempSync(path.join(os.tmpdir(), 'brainy-repack-')) - dirs.push(d) - return d - } - const originalWindow = GenerationStore.REPACK_LIVE_WINDOW - - afterEach(() => { - ;(GenerationStore as any).REPACK_LIVE_WINDOW = originalWindow - for (const d of dirs.splice(0)) { - try { - fs.rmSync(d, { recursive: true, force: true }) - } catch { - /* best effort */ - } - } - }) - - it('repack preserves every historical read across cold reopen; folded dirs are gone', async () => { - ;(GenerationStore as any).REPACK_LIVE_WINDOW = 3 - const dir = tempDir() - const brain = await openBrain(dir) - - const id = await brain.add({ - data: 'versioned-entity', - type: NounType.Document, - metadata: { v: 0 } - }) - for (let v = 1; v <= 10; v++) await brain.update({ id, metadata: { v } }) - await brain.flush() - - // Ground truth BEFORE repacking: capture asOf views for early generations. - const before: Record = {} - for (const g of [2, 4, 6]) { - const db = await brain.asOf(g) - before[g] = (await db.get(id))?.metadata?.v as number - await db.release() - } - - const result = await brain.repackHistory() - expect(result.foldedGenerations).toBeGreaterThan(0) - expect(result.segmentsCreated).toBeGreaterThan(0) - - // The folded per-generation directories are PHYSICALLY gone… - const genDirs = fs - .readdirSync(path.join(dir, '_generations'), { withFileTypes: true }) - .filter((e) => e.isDirectory() && /^\d+$/.test(e.name)).length - expect(genDirs).toBeLessThanOrEqual(4) // live window (3) + at most the newest - // …and the segment tier exists (the filesystem adapter stores objects - // gzipped, so the manifest may live at either spelling). - const segDir = path.join(dir, SEGMENTS_PREFIX) - expect( - fs.existsSync(path.join(segDir, 'manifest.json')) || - fs.existsSync(path.join(segDir, 'manifest.json.gz')) - ).toBe(true) - expect(fs.readdirSync(segDir).some((f) => f.endsWith('.bgs'))).toBe(true) - - // Same asOf answers from the packed tier, same process… - for (const g of [2, 4, 6]) { - const db = await brain.asOf(g) - expect((await db.get(id))?.metadata?.v).toBe(before[g]) - await db.release() - } - await brain.close() - - // …and across a COLD REOPEN (manifest discovery, no live dirs to list). - const reopened = await openBrain(dir) - for (const g of [2, 4, 6]) { - const db = await reopened.asOf(g) - expect((await db.get(id))?.metadata?.v).toBe(before[g]) - await db.release() - } - expect((await reopened.get(id))?.metadata?.v).toBe(10) // live state untouched - await reopened.close() - }) - - it('repack + bounded reclaim compose: whole segments drop, horizon is loud', async () => { - ;(GenerationStore as any).REPACK_LIVE_WINDOW = 2 - const dir = tempDir() - const brain = await openBrain(dir) - const id = await brain.add({ data: 'reclaim-probe', type: NounType.Document, metadata: { v: 0 } }) - for (let v = 1; v <= 8; v++) await brain.update({ id, metadata: { v } }) - await brain.flush() - await brain.repackHistory() - - // Reclaim down to the 3 newest generations — packed segments below the - // horizon drop whole; asOf below throws loudly. - const res = await brain.compactHistory({ maxGenerations: 3 }) - expect(res.removedGenerations).toBeGreaterThan(0) - await expect(brain.asOf(1)).rejects.toBeInstanceOf(GenerationCompactedError) - expect((await brain.get(id))?.metadata?.v).toBe(8) - await brain.close() - }) - - it('generationDigest: reopen-stable, divergence-sensitive, loud below the horizon', async () => { - ;(GenerationStore as any).REPACK_LIVE_WINDOW = 2 - const dir = tempDir() - const brain = await openBrain(dir) - const id = await brain.add({ data: 'digest-probe', type: NounType.Document, metadata: { v: 0 } }) - for (let v = 1; v <= 6; v++) await brain.update({ id, metadata: { v } }) - await brain.flush() - await brain.repackHistory() - - const gen = brain.generation() - const atHead = await brain.generationDigest(gen) - const atMid = await brain.generationDigest(3) - expect(atHead).toMatch(/^[0-9a-f]{8}$/) - expect(atMid).not.toBe(atHead) // more history ⇒ different digest - await brain.close() - - // Reopen-stable: same history, same digests (packed prefix stability). - const reopened = await openBrain(dir) - expect(await reopened.generationDigest(gen)).toBe(atHead) - expect(await reopened.generationDigest(3)).toBe(atMid) - - // New history diverges the head digest. - await reopened.update({ id, metadata: { v: 7 } }) - await reopened.flush() - expect(await reopened.generationDigest(reopened.generation())).not.toBe(atHead) - - // Below the horizon: LOUD, never a silent pin of reclaimed history. - await reopened.compactHistory({ maxGenerations: 2 }) - await expect(reopened.generationDigest(1)).rejects.toBeInstanceOf(GenerationCompactedError) - await reopened.close() - }) - - it('a spent time budget is a consistent no-op; the next pass resumes', async () => { - ;(GenerationStore as any).REPACK_LIVE_WINDOW = 2 - const dir = tempDir() - const brain = await openBrain(dir) - const id = await brain.add({ data: 'budget-probe', type: NounType.Document, metadata: { v: 0 } }) - for (let v = 1; v <= 6; v++) await brain.update({ id, metadata: { v } }) - await brain.flush() - - const bounded = await brain.repackHistory({ timeBudgetMs: 0 }) - expect(bounded).toEqual({ foldedGenerations: 0, segmentsCreated: 0 }) - - const resumed = await brain.repackHistory() - expect(resumed.foldedGenerations).toBeGreaterThan(0) - const db = await brain.asOf(3) - expect((await db.get(id))?.metadata?.v).toBeDefined() - await db.release() - await brain.close() - }) -}) diff --git a/tests/integration/multi-process-safety.test.ts b/tests/integration/multi-process-safety.test.ts index 592d7969..6ac5265d 100644 --- a/tests/integration/multi-process-safety.test.ts +++ b/tests/integration/multi-process-safety.test.ts @@ -110,89 +110,6 @@ describe('Multi-process safety + read-only mode', () => { // Don't track `blocked` for afterEach cleanup since init failed. }) - it('takes over a STALE foreign lock (dead PID + old heartbeat) and claims atomically', async () => { - const { mkdirSync, writeFileSync, readFileSync } = await import('node:fs') - const { join } = await import('node:path') - const os = await import('node:os') - mkdirSync(join(dir, 'locks'), { recursive: true }) - const tenMinutesAgo = new Date(Date.now() - 10 * 60 * 1000).toISOString() - writeFileSync(join(dir, 'locks', '_writer.lock'), JSON.stringify({ - pid: 999999999, // no such process — provably dead - hostname: os.hostname(), - startedAt: tenMinutesAgo, - lastHeartbeat: tenMinutesAgo, - version: '8.0.0', - rootDir: dir - })) - - writer = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } }) - await writer.init() // stale takeover must succeed - - const lock = JSON.parse(readFileSync(join(dir, 'locks', '_writer.lock'), 'utf-8')) - expect(lock.pid).toBe(process.pid) // the atomic wx claim installed OUR lock - }) - - it('the writer-locked error carries the machine-readable contract (code + lockInfo)', async () => { - const { mkdirSync, writeFileSync } = await import('node:fs') - const { join } = await import('node:path') - const os = await import('node:os') - mkdirSync(join(dir, 'locks'), { recursive: true }) - const otherPid = (process as any).ppid || 1 - writeFileSync(join(dir, 'locks', '_writer.lock'), JSON.stringify({ - pid: otherPid, - hostname: os.hostname(), - startedAt: new Date().toISOString(), - lastHeartbeat: new Date().toISOString(), - version: '8.7.0', - rootDir: dir - })) - - const blocked = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } }) - const err: any = await blocked.init().catch((e) => e) - expect(err.code).toBe('BRAINY_WRITER_LOCKED') - expect(err.lockInfo?.pid).toBe(otherPid) - }) - - it('release drains an in-flight heartbeat — no phantom lock re-created after unlink', async () => { - // The race (8.9.0): clearInterval stops FUTURE heartbeat ticks, but a - // tick already in flight could land its lock rewrite AFTER release's - // unlink — re-creating the lock as a phantom that blocks the next - // writer until the stale TTL. Simulate the in-flight tick explicitly - // and prove release waits for it. - writer = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } }) - await writer.init() - const storage: any = (writer as any).storage - - // An in-flight refresh that is ALREADY PAST its ownership guards - // (captured the lock info before release ran) and lands its atomic - // rewrite slowly — the exact straggler shape; absent the drain it - // writes after the unlink. - const { join: joinPath } = await import('node:path') - const capturedInfo = { ...storage.writerLockInfo } - const lockPath = joinPath(dir, 'locks', '_writer.lock') - const slowTick = (async () => { - await new Promise((r) => setTimeout(r, 100)) - await storage.writeFileAtomic( - lockPath, - JSON.stringify({ ...capturedInfo, lastHeartbeat: new Date().toISOString() }) - ) - })() - storage.writerHeartbeatInFlight = slowTick.catch(() => {}) - - await writer.close() // → releaseWriterLock must drain slowTick first - await slowTick.catch(() => {}) // both paths fully settled either way - writer = null - - const { existsSync } = await import('node:fs') - const { join } = await import('node:path') - expect(existsSync(join(dir, 'locks', '_writer.lock'))).toBe(false) - - // And the directory is immediately claimable — no stale-TTL wait. - const next = new Brainy({ requireSubtype: false, storage: { type: 'filesystem', path: dir } }) - await expect(next.init()).resolves.toBeUndefined() - await next.close() - }) - it('allows a second in-process writer with a warning (same PID)', async () => { // Two Brainy instances in the same Node process: not the dangerous // cross-process case. Should succeed (with a console warning). diff --git a/tests/regression/metadata-index-cleanup.unit.test.ts b/tests/regression/metadata-index-cleanup.unit.test.ts index 0984d727..266b9a4d 100644 --- a/tests/regression/metadata-index-cleanup.unit.test.ts +++ b/tests/regression/metadata-index-cleanup.unit.test.ts @@ -205,10 +205,10 @@ describe('Metadata index cleanup after remove / removeMany', () => { } }) - it('refuses an empty ids array loudly (a silent no-op is not "graceful")', async () => { - // 8.8.2: an empty selector used to resolve successfully having deleted - // NOTHING — the caller believed the delete happened. Now it throws. - await expect(brain.removeMany({ ids: [] })).rejects.toThrow(/ids: \[\]/) + it('handles empty ids array gracefully', async () => { + const result = await brain.removeMany({ ids: [] }) + expect(result.successful).toHaveLength(0) + expect(result.failed).toHaveLength(0) }) it('handles large batch (> 1 chunk) without leaving stale index entries', async () => { diff --git a/tests/transaction/TransactionManager.unit.test.ts b/tests/transaction/TransactionManager.unit.test.ts index 86b1692c..cc477483 100644 --- a/tests/transaction/TransactionManager.unit.test.ts +++ b/tests/transaction/TransactionManager.unit.test.ts @@ -10,7 +10,6 @@ import { describe, it, expect, beforeEach } from 'vitest' import { TransactionManager } from '../../src/transaction/TransactionManager.js' -import { transactTimeoutBudget } from '../../src/transaction/Transaction.js' import { TransactionError } from '../../src/transaction/errors.js' describe('TransactionManager', () => { @@ -329,45 +328,4 @@ describe('TransactionManager', () => { expect(stats1).toEqual(stats2) // Same values }) }) - - describe('Timeout budget + telemetry', () => { - it('transactTimeoutBudget: explicit override wins; default scales with batch size', () => { - expect(transactTimeoutBudget(1)).toBe(30_000) // small batches keep the 30s floor - expect(transactTimeoutBudget(15)).toBe(30_000) // the old flat cap's break-even point - expect(transactTimeoutBudget(100)).toBe(200_000) // 100 ops × 2s — bulk gets an honest budget - expect(transactTimeoutBudget(1000, 5_000)).toBe(5_000) // caller override is untouched - }) - - it('a tripped budget rolls back and names the operation, progress, and budget', async () => { - const rolledBack: string[] = [] - - const failing = manager.executeTransaction( - async (tx) => { - tx.addOperation({ - name: 'slow-first-op', - execute: async () => { - await new Promise((r) => setTimeout(r, 30)) - return async () => { - rolledBack.push('slow-first-op') - } - } - }) - tx.addOperation({ - name: 'never-reached', - execute: async () => undefined - }) - }, - { timeout: 5 } // the first op's 30ms sleep guarantees the pre-op-2 check trips - ) - - await expect(failing).rejects.toThrow(TransactionError) - const err = await failing.catch((e) => e) - expect(err.name).toBe('TransactionTimeoutError') - expect(err.message).toContain('operation 1/2') // which op, of how many - expect(err.message).toContain("('never-reached')") // its name - expect(err.message).toContain('budget 5ms') // the budget that tripped - expect(err.message).toContain('rolled back') // the retryability statement - expect(rolledBack).toEqual(['slow-first-op']) // the applied op was undone - }) - }) }) diff --git a/tests/unit/aggregation/AggregationIndex.test.ts b/tests/unit/aggregation/AggregationIndex.test.ts index a0c10e62..05f4a62a 100644 --- a/tests/unit/aggregation/AggregationIndex.test.ts +++ b/tests/unit/aggregation/AggregationIndex.test.ts @@ -675,104 +675,4 @@ describe('AggregationIndex', () => { await reloaded.close() }) }) - - // ============= Boot-order reconciliation ============= - // - // The production boot pattern: defineAggregate() is synchronous and always - // beats the async init() that loads persisted state. The old code flagged a - // backfill at define time and init never cleared it — so the loaded state - // was wiped and the whole store re-walked on EVERY restart. - - describe('boot-order reconciliation (define-before-init)', () => { - const DEF: AggregateDefinition = { - name: 'boot_agg', - source: { type: NounType.Event }, - groupBy: ['category'], - metrics: { count: { op: 'count' } } - } - - const entity = (id: string, category: string): Record => ({ - id, - noun: NounType.Event, - metadata: { category }, - createdAt: Date.now(), - updatedAt: Date.now() - }) - - /** Simulate the previous session: define, contribute, flush. */ - async function seedAndFlush(store: MemoryStorage): Promise { - const first = new AggregationIndex(store) - await first.init() - first.defineAggregate(DEF) - first.onEntityAdded('e1', entity('e1', 'food')) - first.onEntityAdded('e2', entity('e2', 'food')) - first.onEntityAdded('e3', entity('e3', 'transport')) - await first.flush() - } - - it('adopts persisted state when define beats init with an unchanged definition', async () => { - const store = new MemoryStorage() - await store.init() - await seedAndFlush(store) - - const second = new AggregationIndex(store) - second.defineAggregate(DEF) // synchronous define FIRST — the real boot order - await second.init() - await second.ready() - - expect(second.getPendingBackfills()).toEqual([]) - const rows = second.queryAggregate({ name: 'boot_agg' }) - const food = rows.find(r => r.groupKey.category === 'food')! - expect(food.metrics.count).toBe(2) - }) - - it('a write landing before adoption forces an exact rescan instead', async () => { - const store = new MemoryStorage() - await store.init() - await seedAndFlush(store) - - const second = new AggregationIndex(store) - second.defineAggregate(DEF) - second.onEntityAdded('e4', entity('e4', 'food')) // lands before init settles - await second.init() - await second.ready() - - // Adoption would lose e4's contribution — the engine must rescan. - expect(second.getPendingBackfills()).toEqual(['boot_agg']) - }) - - it('init never clobbers a changed app definition registered before it', async () => { - const store = new MemoryStorage() - await store.init() - await seedAndFlush(store) - - const CHANGED: AggregateDefinition = { - ...DEF, - metrics: { count: { op: 'count' }, total: { op: 'sum', field: 'amount' } } - } - const second = new AggregationIndex(store) - second.defineAggregate(CHANGED) - await second.init() - await second.ready() - - const def = second.getDefinitions().find(d => d.name === 'boot_agg')! - expect(Object.keys(def.metrics).sort()).toEqual(['count', 'total']) - expect(second.getPendingBackfills()).toEqual(['boot_agg']) - }) - - it('init alone restores persisted definitions with adopted state, no backfill', async () => { - const store = new MemoryStorage() - await store.init() - await seedAndFlush(store) - - const second = new AggregationIndex(store) - await second.init() - await second.ready() - - expect(second.hasAggregate('boot_agg')).toBe(true) - expect(second.getPendingBackfills()).toEqual([]) - const rows = second.queryAggregate({ name: 'boot_agg' }) - expect(rows.reduce((s, r) => s + (r.metrics.count as number), 0)).toBe(3) - }) - }) }) diff --git a/tests/unit/brainy/batch-operations.test.ts b/tests/unit/brainy/batch-operations.test.ts index 58b25744..b2cc7f09 100644 --- a/tests/unit/brainy/batch-operations.test.ts +++ b/tests/unit/brainy/batch-operations.test.ts @@ -533,10 +533,8 @@ describe('Brainy Batch Operations', () => { expect(result.successful).toHaveLength(0) await brain.updateMany({ items: [] }) - // removeMany is the exception (8.8.2): an empty id list is a refused - // selector, not an empty batch — deleting "nothing" silently was the - // bug class (a positional/bare-array call looked identical). - await expect(brain.removeMany({ ids: [] })).rejects.toThrow(/ids: \[\]/) + await brain.removeMany({ ids: [] }) + // Should not throw }) it('should validate batch size limits', async () => { diff --git a/tests/unit/db/fact-log.test.ts b/tests/unit/db/fact-log.test.ts index f1c226cc..abce2dc9 100644 --- a/tests/unit/db/fact-log.test.ts +++ b/tests/unit/db/fact-log.test.ts @@ -186,52 +186,4 @@ describe('fact log — round-trip, framing, reconcile, rotation, scan', () => { await log.sync() expect(log.segmentPaths()).toEqual([]) // only a tail exists — nothing sealed }) - - describe('scanFacts liveness contract (Stage-2 D1)', () => { - it('a wedged store fails LOUDLY within the first-batch bound — never a silent hang', async () => { - // Force a sealed segment (tiny rotateBytes) so the scan must READ from - // storage, then wedge that read: the exact production shape (a - // backlogged brain whose segment read never returned). - const mem: any = new MemoryStorage() - await mem.init() - const wedgeable = new FactLog(mem, { rotateBytes: 1 }) - await wedgeable.open(0) - await wedgeable.append(fact(1)) - await wedgeable.append(fact(2)) // second append rotates → seg 1 sealed - await wedgeable.sync() - - const realRead = mem.readRawBytes.bind(mem) - mem.readRawBytes = (p: string) => - p.includes('facts/seg-') ? new Promise(() => {}) : realRead(p) // hangs forever - - const scan = wedgeable.scanFacts({ firstBatchTimeoutMs: 200 }) - const started = Date.now() - await expect(scan.batches().next()).rejects.toThrow(/no first batch within 200ms/) - expect(Date.now() - started).toBeLessThan(5_000) // bound held, not a hang - }) - - it('a healthy scan is unaffected — first batch well inside the bound, all facts delivered', async () => { - for (let g = 1; g <= 5; g++) await log.append(fact(g)) - await log.sync() - const scan = log.scanFacts({ batchSize: 2 }) - const all: CommitFact[] = [] - for await (const b of scan.batches()) all.push(...b.facts) - expect(all.map((f) => f.generation)).toEqual([1, 2, 3, 4, 5]) - expect(scan.summary().factsYielded).toBe(5) - }) - - it('consumer think-time between pulls never counts against the producer', async () => { - for (let g = 1; g <= 4; g++) await log.append(fact(g)) - await log.sync() - // Bound tighter than the consumer's pause: only the FIRST pull is - // raced, so a slow consumer after batch 1 must not trip the deadline. - const gen = log.scanFacts({ batchSize: 2, firstBatchTimeoutMs: 150 }).batches() - const first = await gen.next() - expect(first.done).toBe(false) - await new Promise((r) => setTimeout(r, 400)) // dawdle past the bound - const second = await gen.next() - expect(second.done).toBe(false) - expect((await gen.next()).done).toBe(true) - }) - }) }) diff --git a/tests/unit/db/generation-segments.test.ts b/tests/unit/db/generation-segments.test.ts deleted file mode 100644 index 27ab85cb..00000000 --- a/tests/unit/db/generation-segments.test.ts +++ /dev/null @@ -1,150 +0,0 @@ -/** - * @module tests/unit/db/generation-segments - * @description The generation-segment store (Stage-2 D1+D3 file format). - * Laws: (1) fold → read round-trips deltas and records byte-faithfully via - * sidecar point-reads; (2) the manifest is the ONLY discovery path — reopen - * reads one file, never a listing; (3) a lost/corrupt sidecar rebuilds from - * its segment loudly, a damaged SEGMENT fails loudly (never silent wrong - * data); (4) D3 reclaim drops whole segments only and bumps compactedBelow; - * (5) the packed digest is deterministic across reopen; (6) immutability — - * fold refuses overlap with sealed ranges. - */ -import { describe, it, expect, beforeEach } from 'vitest' -import { MemoryStorage } from '../../../src/storage/adapters/memoryStorage.js' -import { - GenerationSegmentStore, - SEGMENTS_PREFIX, - type FoldGeneration -} from '../../../src/db/generationSegments.js' - -const UUID = (n: number): string => `00000000-0000-4000-8000-${String(n).padStart(12, '0')}` - -const gen = (g: number, recordCount = 2): FoldGeneration => ({ - generation: g, - timestamp: 1_700_000_000_000 + g, - delta: { generation: g, nouns: [UUID(g)], verbs: [], bytes: 123 + g }, - records: Array.from({ length: recordCount }, (_, i) => ({ - kind: (i % 2 === 0 ? 'noun' : 'verb') as 'noun' | 'verb', - id: UUID(g * 100 + i), - record: { metadata: { noun: 'document', v: g }, vector: { v: [g, i] } } - })) -}) - -describe('db/GenerationSegmentStore — the D1+D3 packed tier', () => { - let storage: MemoryStorage - let store: GenerationSegmentStore - - beforeEach(async () => { - storage = new MemoryStorage() - await storage.init() - store = new GenerationSegmentStore(storage as any) - await store.open() - }) - - it('fold → read round-trips deltas and records via sidecar point-reads', async () => { - const meta = await store.fold([gen(1), gen(2), gen(3)]) - expect(meta).toMatchObject({ firstGeneration: 1, lastGeneration: 3, frames: 3 }) - expect(meta.checksum).toBeGreaterThan(0) - - expect(store.hasGeneration(2)).toBe(true) - expect(store.hasGeneration(4)).toBe(false) - - const d2 = await store.readDelta(2) - expect(d2?.delta).toEqual({ generation: 2, nouns: [UUID(2)], verbs: [], bytes: 125 }) - expect(d2?.timestamp).toBe(1_700_000_000_002) - - const records = await store.readRecords(3) - expect(records).toHaveLength(2) - expect(records![0]).toEqual({ - kind: 'noun', - id: UUID(300), - record: { metadata: { noun: 'document', v: 3 }, vector: { v: [3, 0] } } - }) - // Point read by id, both kinds. - expect(await store.readRecord(3, 'verb', UUID(301))).toEqual({ - metadata: { noun: 'document', v: 3 }, - vector: { v: [3, 1] } - }) - expect(await store.readRecord(3, 'noun', UUID(999))).toBeNull() - }) - - it('reopen discovers everything from the manifest alone — no listing', async () => { - await store.fold([gen(1), gen(2)]) - await store.fold([gen(3), gen(4)]) - - const reopened = new GenerationSegmentStore(storage as any) - await reopened.open() - expect(reopened.segments()).toHaveLength(2) - expect(reopened.hasGeneration(4)).toBe(true) - expect((await reopened.readDelta(1))?.timestamp).toBe(1_700_000_000_001) - }) - - it('a lost sidecar rebuilds from its segment; a damaged segment fails LOUDLY', async () => { - const meta = await store.fold([gen(1), gen(2)]) - const idxPath = `${SEGMENTS_PREFIX}/seg-${String(1).padStart(20, '0')}.idx` - await storage.deleteRawObject(idxPath) - - const reopened = new GenerationSegmentStore(storage as any) - await reopened.open() - // Rebuild path: still serves correct data. - expect((await reopened.readRecords(2))!).toHaveLength(2) - - // Now damage the SEGMENT itself: flip a payload byte → CRC mismatch, loud. - const segPath = `${SEGMENTS_PREFIX}/${meta.file}` - const bytes = (await storage.readRawBytes(segPath))! - bytes[bytes.length - 3] ^= 0xff - await storage.writeRawBytes(segPath, bytes) - const damaged = new GenerationSegmentStore(storage as any) - await damaged.open() - ;(damaged as any).sidecars.clear() - await storage.deleteRawObject(idxPath) // force the sequential rebuild over damaged bytes - await expect(damaged.readRecords(2)).rejects.toThrow(/CRC mismatch|damaged/) - }) - - it('D3 reclaim drops whole segments only and bumps compactedBelow', async () => { - await store.fold([gen(1), gen(2)]) - await store.fold([gen(3), gen(4)]) - await store.fold([gen(5), gen(6)]) - - // Horizon mid-segment-2 (below 4): only segment 1 is FULLY below → drops. - const r1 = await store.dropSegmentsBelow(4) - expect(r1).toEqual({ dropped: 1, compactedBelow: 3 }) - expect(store.hasGeneration(1)).toBe(false) - expect(store.hasGeneration(3)).toBe(true) // partial segment survives whole - - // Bytes actually gone. - expect(await storage.readRawBytes(`${SEGMENTS_PREFIX}/seg-${String(1).padStart(20, '0')}.bgs`)).toBeNull() - - // Horizon past everything: the rest drop; compactedBelow is durable. - const r2 = await store.dropSegmentsBelow(7) - expect(r2.dropped).toBe(2) - const reopened = new GenerationSegmentStore(storage as any) - await reopened.open() - expect(reopened.compactedBelow()).toBe(7) - expect(reopened.segments()).toHaveLength(0) - }) - - it('the packed digest is deterministic across reopen and changes with history', async () => { - await store.fold([gen(1), gen(2), gen(3)]) - const atSeal = await store.digestThroughPacked(3) - const midSegment = await store.digestThroughPacked(2) - expect(atSeal).not.toBeNull() - expect(midSegment).not.toBeNull() - expect(midSegment).not.toBe(atSeal) - - const reopened = new GenerationSegmentStore(storage as any) - await reopened.open() - expect(await reopened.digestThroughPacked(3)).toBe(atSeal) - expect(await reopened.digestThroughPacked(2)).toBe(midSegment) - - await reopened.fold([gen(4)]) - expect(await reopened.digestThroughPacked(4)).not.toBe(atSeal) - }) - - it('sealed segments are immutable — fold refuses overlap, requires ascending input', async () => { - await store.fold([gen(1), gen(2)]) - await expect(store.fold([gen(2), gen(3)])).rejects.toThrow(/overlaps the packed tier/) - await expect(store.fold([gen(4), gen(4)])).rejects.toThrow(/strictly ascending/) - await expect(store.fold([])).rejects.toThrow(/at least one generation/) - }) -}) diff --git a/tests/unit/db/generationStore.test.ts b/tests/unit/db/generationStore.test.ts index 5b667415..184d3974 100644 --- a/tests/unit/db/generationStore.test.ts +++ b/tests/unit/db/generationStore.test.ts @@ -488,95 +488,5 @@ describe('db/GenerationStore', () => { expect(result.removedGenerations).toBe(2) store.release(2) }) - - it('timeBudgetMs bounds a pass; the next pass resumes the same prefix', async () => { - await manyGens(4) - // A spent budget (0ms) stops before reclaiming anything — an early stop - // is a consistent prefix, never a partial generation. - const bounded = await store.compact({ timeBudgetMs: 0 }) - expect(bounded.removedGenerations).toBe(0) - expect(bounded.horizon).toBe(0) - // The next (unbounded) pass picks up exactly where the bounded one - // stopped and completes the same work. - const resumed = await store.compact() - expect(resumed.removedGenerations).toBe(4) - expect(resumed.horizon).toBe(4) - }) - }) - - // ========================================================================== - describe('history-bytes running total (the O(1) retention check)', () => { - /** A fresh walk with the cache dropped — ground truth for the invariant. */ - async function groundTruthBytes(): Promise { - ;(store as any).historyBytesTotal = null - return store.historyBytes() - } - - it('is seeded once, then maintained through commits WITHOUT re-walks', async () => { - await commitWrite(ID_A, 1) - await commitWrite(ID_A, 2) - const seeded = await store.historyBytes() - expect(seeded).toBe(await groundTruthBytes()) - - // From here every read must come from the running total, not a walk: - // getDelta re-reads are the walk's cost — commits must not trigger any. - const getDeltaSpy = vi.spyOn(store as any, 'getDelta') - await commitWrite(ID_B, 1) - const afterCommit = await store.historyBytes() - expect(getDeltaSpy).not.toHaveBeenCalled() - getDeltaSpy.mockRestore() - expect(afterCommit).toBe(await groundTruthBytes()) - }) - - it('stays exact through single-op group commits and compaction', async () => { - await commitWrite(ID_A, 1) - await store.historyBytes() // seed - // Single-op path: buffered generations flushed as one group commit. - await store.commitSingleOp({ - touched: { nouns: [ID_B] }, - execute: async () => { - await storage.saveNounMetadata(ID_B, metadataFixture(1)) - } - }) - await store.flushPendingSingleOps() - expect(await store.historyBytes()).toBe(await groundTruthBytes()) - - await store.historyBytes() // re-seed after ground-truth reset - await store.compact({ maxGenerations: 1 }) - expect(await store.historyBytes()).toBe(await groundTruthBytes()) - }) - - it('historyStats reports counts, bytes, range, and horizon read-only', async () => { - await commitWrite(ID_A, 1) - await commitWrite(ID_B, 1) - const stats = await store.historyStats() - expect(stats.generations).toBe(2) - expect(stats.bytes).toBe(await store.historyBytes()) - expect(stats.oldestGeneration).toBe(1) - expect(stats.newestGeneration).toBe(2) - expect(stats.oldestTimestamp).toBeLessThanOrEqual(stats.newestTimestamp!) - expect(stats.horizon).toBe(0) - // Read-only: nothing was reclaimed by asking. - expect(store.committedGeneration()).toBe(2) - - await store.compact({ maxGenerations: 1 }) - const after = await store.historyStats() - expect(after.generations).toBe(1) - expect(after.oldestGeneration).toBe(2) - expect(after.horizon).toBe(1) - }) - - it('empty history reports null range and zero bytes', async () => { - const stats = await store.historyStats() - expect(stats).toMatchObject({ - generations: 0, - bytes: 0, - oldestGeneration: null, - newestGeneration: null, - oldestTimestamp: null, - newestTimestamp: null, - horizon: 0 - }) - }) }) }) diff --git a/tests/unit/storage/verb-cursor-pagination.test.ts b/tests/unit/storage/verb-cursor-pagination.test.ts index cff41d02..985a650f 100644 --- a/tests/unit/storage/verb-cursor-pagination.test.ts +++ b/tests/unit/storage/verb-cursor-pagination.test.ts @@ -95,15 +95,9 @@ describe('verb cursor pagination (graph-perf #2)', () => { expect(new Set(cursorSeen)).toEqual(new Set(offsetSeen)) }) - it('a foreign/malformed cursor FAILS LOUDLY — never a silent restart from page 1', async () => { - // The old behavior (decode-null → silent offset-0 fallback) re-served page 1 - // forever to any while(hasMore) walker: an unbounded CPU loop with no log - // line. An undecodable resume token now refuses the walk instead. - await expect( - storage.getVerbs({ pagination: { limit: 5, cursor: 'not-a-cv1-token' } }) - ).rejects.toThrow('invalid pagination cursor') - await expect( - storage.getNouns({ pagination: { limit: 5, cursor: 'not-a-cv1-token' } }) - ).rejects.toThrow('invalid pagination cursor') + it('a foreign/malformed cursor falls back gracefully (no throw, starts from the beginning)', async () => { + const page = await storage.getVerbs({ pagination: { limit: 5, cursor: 'not-a-cv1-token' } }) + expect(page.items.length).toBe(5) + expect(page.hasMore).toBe(true) }) }) diff --git a/tests/unit/utils/osLimits.test.ts b/tests/unit/utils/osLimits.test.ts deleted file mode 100644 index a56d593e..00000000 --- a/tests/unit/utils/osLimits.test.ts +++ /dev/null @@ -1,76 +0,0 @@ -/** - * @module tests/unit/utils/osLimits - * @description OS-limit detection for pool-scale use. Laws: - * (1) the /proc/self/limits parser reads soft/hard NOFILE exactly, including - * 'unlimited'; (2) assessment warns ONLY below the pool floors and NEVER - * on an unreadable (null) limit — no measurement, no claim; (3) the full - * check composes both sources and survives unreadable /proc silently. - */ -import { describe, it, expect } from 'vitest' -import { - parseProcLimits, - assessOsLimits, - checkOsLimits, - NOFILE_POOL_FLOOR, - MAX_MAP_COUNT_POOL_FLOOR -} from '../../../src/utils/osLimits.js' - -const SAMPLE_LIMITS = [ - 'Limit Soft Limit Hard Limit Units', - 'Max cpu time unlimited unlimited seconds', - 'Max open files 1024 1048576 files', - 'Max locked memory 8388608 8388608 bytes' -].join('\n') - -describe('osLimits — detect + warn at pool scale', () => { - it('parses soft/hard NOFILE from /proc/self/limits, including unlimited', () => { - expect(parseProcLimits(SAMPLE_LIMITS)).toEqual({ soft: 1024, hard: 1048576 }) - expect( - parseProcLimits('Max open files unlimited unlimited files') - ).toEqual({ soft: Infinity, hard: Infinity }) - expect(parseProcLimits('no such row here')).toEqual({ soft: null, hard: null }) - }) - - it('warns below the floors, stays quiet at or above them', () => { - const low = assessOsLimits({ nofileSoft: 1024, nofileHard: 1048576, maxMapCount: 65530 }) - expect(low).toHaveLength(2) - expect(low[0]).toContain('RLIMIT_NOFILE soft limit is 1024') - expect(low[0]).toContain(`ulimit -n ${NOFILE_POOL_FLOOR}`) - expect(low[0]).toContain('raise the soft limit only') // hard already allows it - expect(low[1]).toContain('vm.max_map_count is 65530') - expect(low[1]).toContain(`vm.max_map_count=${MAX_MAP_COUNT_POOL_FLOOR}`) - - expect( - assessOsLimits({ - nofileSoft: NOFILE_POOL_FLOOR, - nofileHard: Infinity, - maxMapCount: MAX_MAP_COUNT_POOL_FLOOR - }) - ).toEqual([]) - }) - - it('an unreadable limit makes NO claim — nulls never warn', () => { - expect(assessOsLimits({ nofileSoft: null, nofileHard: null, maxMapCount: null })).toEqual([]) - }) - - it('checkOsLimits composes both sources and survives unreadable /proc silently', async () => { - const report = await checkOsLimits(async (p) => { - if (p === '/proc/self/limits') return SAMPLE_LIMITS - if (p === '/proc/sys/vm/max_map_count') return '65530\n' - throw new Error('unexpected path') - }) - expect(report.nofileSoft).toBe(1024) - expect(report.maxMapCount).toBe(65530) - expect(report.warnings).toHaveLength(2) - - const offLinux = await checkOsLimits(async () => { - throw Object.assign(new Error('ENOENT'), { code: 'ENOENT' }) - }) - expect(offLinux).toEqual({ - nofileSoft: null, - nofileHard: null, - maxMapCount: null, - warnings: [] - }) - }) -}) diff --git a/tests/unit/utils/paramValidation.test.ts b/tests/unit/utils/paramValidation.test.ts index 4dc83554..e7113506 100644 --- a/tests/unit/utils/paramValidation.test.ts +++ b/tests/unit/utils/paramValidation.test.ts @@ -270,24 +270,27 @@ describe('Zero-Config Parameter Validation', () => { expect(config.availableMemory).toBeGreaterThan(0) }) - it('never mutates the cap from query timing (telemetry only)', () => { - const initialLimit = getValidationConfig().maxLimit - - // Fast queries with large results: no silent growth. + it('should adapt limits based on query performance', () => { + const initialConfig = getValidationConfig() + const initialLimit = initialConfig.maxLimit + + // Simulate fast queries with large results for (let i = 0; i < 10; i++) { recordQueryPerformance(50, initialLimit * 0.9) } - expect(getValidationConfig().maxLimit).toBe(initialLimit) - - // A burst of catastrophically slow queries must not strangle the cap. - // The removed "learning" ratchet shrank it 20% per recorded query down - // to a floor of 1000 — below the documented MIN_AUTO_QUERY_LIMIT — and - // the error message blamed "available free memory" (a production - // incident: every find({ limit: 5000 }) failed on an idle 23GB-free box). - for (let i = 0; i < 50; i++) { - recordQueryPerformance(90_000, 100) + + const updatedConfig = getValidationConfig() + // Limit might increase if performance is good + expect(updatedConfig.maxLimit).toBeGreaterThanOrEqual(initialLimit) + + // Simulate slow queries + for (let i = 0; i < 10; i++) { + recordQueryPerformance(2000, 100) } - expect(getValidationConfig().maxLimit).toBe(initialLimit) + + const finalConfig = getValidationConfig() + // Limit should decrease if performance is poor + expect(finalConfig.maxLimit).toBeLessThanOrEqual(updatedConfig.maxLimit) }) }) }) \ No newline at end of file