Open Brainy's side of the API contract the accelerated engine published.
DECLARED: package.json carries "brainyContract": 1 and the engine states its
own via contractVersion() / BRAINY_CONTRACT_VERSION — two engines compare an
integer instead of probing prototypes, and a tool reads the package field
without importing the engine. Pinned so the two can never drift apart.
SERVED: hasAll, noneOf and excludes now work on the index path. The defect
underneath was worse than the reported divergence — the metadata index's
operator switch had NO DEFAULT CASE, so any operator without a case left the
field's match set at its initial [] and find() returned an empty page.
Documented, validator-accepted, matcher-implemented operators answering
silently wrong. hasAll intersects each element's posting set (an empty operand
is vacuously true of every row that has the field), noneOf complements their
union, excludes complements contains.
REFUSED BY NAME: startsWith, endsWith, matches and length raise
INVALID_QUERY naming the operator, the field and the reason. An equality/range
posting index cannot evaluate a substring, a pattern or an array length without
reading every row — which is the cost this path exists to avoid — so it refuses
rather than answering an empty page. Both engines now agree on all 25 tokens
and contract 1 has no remaining operator divergence. This is a visible change
for a consumer calling those four through find({ where }): an empty page
becomes a typed refusal.
EMITTED: scripts/emit-contract-manifest.mjs generates docs/api-contract.json
from the BUILT surface — prototype doors, exported error classes, the operator
sets read out of their single definitions, the field-addressing vocabulary, the
health verdicts. Nothing hand-maintained, so a diff between two manifests is a
diff between two engines. `--check` fails on a stale manifest, which makes the
announce-every-addition duty mechanical rather than remembered.
RATIFIED in docs/contract-1-ratification.md: the 41-of-57 required split with
the promise spelled out (a refusal is part of a door; deprecation is not
removal), the serving-withholding list confirmed exhaustive and identical, the
minor/major rule adopted with the announcement duty, the 30 storage seam
methods committed as supported surface until Stage 2, and a finding filed
against the spec — is / isNot / greaterEqual / lessEqual are listed there as
served aliases and have never existed in this engine, which throws
INVALID_QUERY on all four.
Prepares the repo for its new home at soulcraftlabs/open-brainy ahead
of the Forgejo transfer: package name, publish registry, release
script, and every install/import reference across docs, src, tests,
examples, and integrations now point at @soulcraftlabs/brainy on
The Source. The npmjs storefront leg and byte-identity pair
verification are stripped from the release script — The Source is
now the only publish target. README gains an Open Brainy explainer
and a registry note for consumers.
@soulcraft/brainy 10.4.2 was the last release under the old name.
The read gate stops consulting the unnamed isReady() boolean: every provider
may expose healthReport() (sync, O(1), composed from exact ledgers —
HealthReport with a monotonic generation, per-invariant source
ledger|deep|unledgered, missing {count, sample}), and one readiness
authority (assessProviderHealth) derives the verdict. Unledgered families
are UNKNOWN — never healthy, never broken; a report that throws is a loud
not-ready, never a shrug. Reads at the four index choke points refuse with
the typed NotReady errors, narrated once per (provider, generation) — a
read NEVER starts a store walk:
- the first-read lazy build retires (open builds instead, regardless of
size — the ≥10k deferral and the "lazy loading on first query" branch go;
disableAutoRebuild is re-meant honestly in its docs);
- the verify*Live read-path rebuild triggers retire (refuse-or-serve);
- the read-time consistency probe that could launch a dark rebuild from an
ordinary find() retires;
- repairIndex({ rebuild: ['metadata'|'graph'|'vector'] | 'all' }) is the
one explicit door: rebuilds the named leg unconditionally and reports
rebuilt per family; bare repairIndex() stays report-driven.
test(lifecycle): the biography lane — a store's whole life, refereed
tests/lifecycle/: an independent shadow model referees every read after
every chapter (founding, a working day, clean restart, crash, repair,
second life). Chapters 1-3 green. Chapters 4-6 assert the true contract and
are marked .fails as a release-blocking finding (the kill-matrix
convention): after a crash + adopt reopen the metadata index computes its
'catchup' watermark verdict and nothing consumes it — find() serves the
pre-crash index while canonical and counts recover. The catchup wiring is
the cure; a passing .fails will force the marker's removal. The lane runs
in the integration gate (config + coverage guard).
The time-travel recall row moves from envelope-note to contracted: vector
search at a pinned past generation serves the vectors AS THEY STOOD —
a later re-embed never leaks into an earlier pin (byte-exact), tombstones
mask, the deferred-embed pin serves the stub on the vector leg until the
landing generation (text/metadata legs unaffected — triple intelligence by
design), and beyond-head pins refuse typed. Brainy-alone leg = the
documented ephemeral at-generation materialization; the at-scale leg rides
the accelerated provider's as-of index. Registry row added (shared ID
pending the master table).
DP6/DP8 of the Path Registry (BRAINY-PROD-LATENCY-TRIAD, the proven flicker
mechanism): update paths staged RemoveFromVectorIndex then AddToVectorIndex
as two separately-awaited transaction ops — between them a live row was in
NEITHER index (dark to semantic recall, fine in metadata list). The native
pair widened that window to seconds in production before their side's
visibility-commit fix; the structural cure lands here:
- hnswIndex.updateItem: absent → add; SAME vector → pure no-op (the
production shape — a type-only update re-indexed an unchanged vector,
remove+add did pure damage); changed vector → the node NEVER leaves the
index: synchronous vector swap first (every query from that instant sees
correct distances), then unlink/relink at the node's existing level via
shared internals (linkNode/unlinkNodeEdges refactored out of add/remove;
entry point and maxLevel provably unchanged).
- ReplaceInVectorIndexOperation: ONE transaction leg; feature-detects
provider updateItem (native seam flagged — their side ships updateItem,
then the adjacent remove+add fallback is dead code). Both update staging
sites swapped; delete sites untouched.
- LAZY-OPEN GATE (fleet adoption find, SELF-ENGINE-PAIR-STANDARD): under
disableAutoRebuild, ensureIndexesLoaded assessed ONLY the vector index —
a not-ready native METADATA provider never blocked the completion latch
and every find() silently returned [] on a populated store. All three
providers now vote; any not-ready report falls through to the rebuild.
- docs/path-registry.md: brainy's twin table for the 32 shared path IDs —
service class, budgets, lifecycle, narration, and the cited pin per row;
owed rows named (LC4 doors-open migration, MT4 yielding heals, LC7
downgrade contract) per the lifecycle-sprint choreography.
Pins: update-item-atomic 9/9 (visibility-atomic swap, reverse-index parity
vs fresh rebuild, entry-point invariants) · lazy-notready-honor 2/2.
Gates: unit 1928/1928 (148 files) · integration 760 · conformance 27/27.
The write side of the law, ruled 2026-08-03: data is either in main space
where developers can use anything, or it is in system.*.
- The reserved-name write door DIES: add/update/relate/updateRelation
metadata bags accept EVERY name (confidence, type, id, data, level,
content, ...) as ordinary user fields — indexed, filterable, sortable,
aggregatable, identical to any other field. The remap/enforce/warn
machinery, the reservedFieldPolicy config (now a typed init refusal),
and the compile-time metadata key bans are all removed. The one write
refusal left: keys spelled 'system.*' (namespace forgery), now enforced
on all four write doors.
- STORED RECORDS GO NESTED (v2): engine fields top-level, the user bag
nested verbatim under 'metadata', sealed by a format stamp — by-name
storage discrimination is unsound once colliders are admitted. Legacy
flat records stay readable forever through the shape-aware splitters
(sound for them: the old door refused colliders). Time travel rides the
same split (generation store snapshots whole records).
- Name-based index exclusions DIE: user frame indexes every name; the
excludeFields/indexedFields knobs and their silent-[] holes are gone;
bulk-payload protection is value-shape only, uniform across names.
- Consumer-sweep findings fixed in the same wave: per-type counts read
the frozen 'system.type' column (addToIndex sort, affinity tracking,
cold-count rehydration, VFS type bitmaps — legacy 'noun' fallback for
pre-rebuild reads); resolveHiddenIds addresses 'system.visibility'
(bare 'visibility' was a silent no-op under the law — VFS/system
entities leaked into default reads).
- Fidelity fallout fixed in the owning layers: readEntityFieldAddress
reads the bag first (colliders were absent-shadowed by its own guard)
and never serves system addresses from the bag; blob history refs read
the bag shape-aware; migration transforms now receive ONE normalized
view (engine fields + nested bag) regardless of stored era, and stray
flat-habit keys refuse with the fix in the message.
- THE REOPEN-COLLIDER CONFORMANCE CASE (required before any RC counts as
gates-green): all ten collider names + plumbing names written as user
fields, verified verbatim + queryable across live reads, flush+reopen,
a forced epoch rebuild, and asOf time travel; relation mirror; forgery
refusals; legacy flat-record compat. 8/8 green.
Gates: unit 1901/1901 (exit 0) · integration 758 (exit 0) · conformance
27/27 (exit 0) · consumer test sweep migrated (10 files).
First edition of the per-release performance-envelope contract: every
number measured against the built dist on stated hardware, never
projected. Sub-0.1ms get/related (adjacency O(degree), scale-flat),
1-9ms indexed metadata finds, ~178ms semantic (query embedding
dominates), ~167ms durability-priced single-op writes flat across
scale, 8-45ms steady-state flush independent of history backlog
(the 8.9.0 change). Two weak spots stated honestly: addMany commits
per-item today (batched chunk commits belong to the unified-commit
roadmap), and pure-JS warm open grows with corpus (4.9s at 10k) —
the native accelerator's reason to exist. Refresh rule: any release
touching a measured path re-measures in the same release.
flush() is durability work: it must cost what the current window's
deltas cost, never what the history backlog costs. Under adaptive
retention the byte budget derives from free memory, so bulk-load
pressure shrank the budget exactly at peak write volume and flush paid
actual reclaim inline — a production deployment measured single writes
blocked 25-191s behind reclaim-on-flush.
- flush() no longer calls autoCompactHistory(); close() is THE
auto-compaction site (already ran there; now alone).
- Every auto pass is time-bounded (CLOSE_COMPACTION_BUDGET_MS = 5s):
reclamation is oldest-first, so an early stop is a consistent prefix
and the next pass resumes. Explicit compactHistory() gains an
optional timeBudgetMs for caller-chosen maintenance windows.
- Documented trade stated where operators read: a long-lived writer
that never closes accumulates history until its next explicit
compactHistory() — predictable writes, explicit maintenance.
Pins: flush-never-reclaims + close-reclaims-durably (db-mvcc), bounded
pass stops-then-resumes as a consistent prefix (generationStore unit).
The post-import background deduplication pass (a merge-DELETE writer,
debounced ~5 minutes after import) had four lifecycle defects:
- enableDeduplication: false did not gate the background schedule — an
import that explicitly opted out could still have entities auto-removed
minutes later. The flag now gates both the inline and background passes.
- Each import() constructed its own coordinator-owned deduplicator, so the
debounce never spanned imports (N imports = N delete timers). The brain
now owns a single lazy instance (getBackgroundDeduplicator).
- close() never cancelled pending dedup; a delete pass could fire against
a closed brain. close() now cancels it first.
- The 5-minute timer held the process open (exit-hang class); now unref'd.
Four regression tests pin the contract (background-dedup-lifecycle);
import guides document that false disables both passes.
- The transact apply budget scales with the batch — max(30s, opCount x 2s) —
or is exactly the caller's new TransactOptions.timeoutMs. A flat 30s cap
silently limited honest bulk work to ~15 operations on network-attached
storage (~2s/op measured in a production import) while looking generous for
small batches. Internal batch paths (removeMany chunks) get the same
scaling via the shared transactTimeoutBudget helper.
- TransactionTimeoutError now reports the operation it stopped at as i/N with
the operation's name, elapsed vs budgeted time, and states the batch rolled
back atomically and is retryable; context carries the fields
programmatically.
- The transact operational envelope is documented (optimistic-concurrency
guide): budget math, chunking with ifAbsent idempotency, when to reach for
addMany vs transact, and the precompute pattern (embedBatch + per-op
vector) that keeps inference out of the commit path.
- Embedding claims made honest per an end-to-end probe of the built dist:
batch and single embedding paths produce bit-identical vectors and a
vector-supplied add is fully searchable, but batch throughput on the
default WASM engine measures comparable to sequential (~160ms/text) — the
5-10x speedup claim in the addMany JSDoc is replaced with the measured
reality and the actual win (inference outside the budgeted write path).
- New public API: walks every canonical relationship record, queries the same
read path applications use (related() with all visibility tiers included),
and classifies every discrepancy into its failure family: missingFromReads
(records the read path omits — stale adjacency), danglingEndpoints
(endpoint entity gone — the partial-delete scar class), readOnlyVerbIds
(read-path edges with no canonical record — ghosts). Design-hidden
internal/system edges are counted separately so intentional hiding is never
misclassified as loss. Counts exact; example lists capped with an explicit
truncatedExamples flag; loud narration on incoherence; mutates nothing.
- Classification core lives in src/graph/graphAudit.ts behind injected seams
(canonical walks + the end-to-end read) so the discrepancy logic is testable
in isolation; brainy wires the seams to storage walks and related().
- getNounIdsWithPagination now refuses an undecodable resume cursor loudly —
the third and final pagination walk brought under the 8.5.2 cursor contract.
- Types exported: GraphAuditReport, GraphAuditDiscrepancy. Docs: the
inspection guide gains the audit -> repair -> audit verification ritual.
- AggregationIndex: defineAggregate before init no longer forces a backfill.
init reconciles instead of clobbering: the app definition wins, persisted
state is adopted on hash match, a write landing pre-adoption forces an exact
rescan, and a successful state load clears the backfill flag. New ready()
settles every adoption decision before query paths consult backfill state.
- brainy: backfills are single-flight and batched. Concurrent queries share one
store walk and every pending aggregate fills from that same walk; the old
behavior let each concurrent query wipe the others' partial state and start
its own full walk, so a store under steady aggregate traffic never converged.
queryAggregate also waits for persisted definitions before deciding an
aggregate does not exist.
- paramValidation: recordQuery is telemetry-only. The duration-based cap
ratchet (x0.8 per recorded query while lifetime-average exceeded 1s, floored
at 1000 - below the documented 10000 auto floor, reported under a stale
basis label) is removed; the cap comes from its construction-time basis or
explicit overrides alone.
- storage: __aggregation_* and singleton system keys (brainy:entityIdMapper)
are recognized before the unknown-key warning fires; routing is unchanged.
- docs: find-limits cap-immutability note, aggregation reopen/adoption
semantics, RELEASES.md 8.5.1 entry.
- New public guide (guides/external-backups): the sparse-mmap reality
for external backup tooling — why a brain directory can show 100+ GB
apparent size on a small disk, which files are sparse, tar czSf /
rsync --sparse / cp --sparse=always, live-store caveats, and what
persist()/restore() already handle natively.
- New public concept (concepts/generation-fact-log): what facts are
(after-image commit records, body-less tombstones), the crash-safety
model (checksummed frames, reconcile-to-committed, absent = never
committed), the scanFacts()/factSegmentPaths() surfaces with the
telemetry shape, family stamps + open-time coherence, and the
storage capability seam for plugin authors.
- Cross-link from the snapshots guide; sparse notes added to the
public persist()/restore() JSDoc (the operator-facing sites).
The temporal model had a hole exactly where files were concerned: every
entity write is an immutable generation with before-images, but VFS content
BYTES lived under an eager refCount GC left over from the pre-8.0 design —
unlink could physically destroy bytes that in-window history still
referenced, and overwrite never released the old hash at all (an unbounded
silent leak whose accidental byproduct was the only thing "preserving"
history). Reading the past could therefore return a stale field, a dangling
hash, or nothing, depending on luck.
Fix: blob reclamation becomes a HISTORY decision instead of a LIVENESS
decision. Each blob's metadata now carries historyRefCount alongside the
live refCount:
- The commit seam counts one history reference per persisted before-image
record carrying a content hash (commitTransaction staging and the
group-commit flush), recorded BEFORE the record-set persists and carried
in the generation delta (blobHashes — always present on new deltas, so
compaction only falls back to reading records for pre-contract
generations). An aborted transaction compensates best-effort.
- unlink/rmdir/overwrite drop ONLY the live reference (BlobStorage.delete →
release; overwrite finally releases the superseded hash — cancelling the
dedup increment on same-content rewrites and closing the leak), and only
AFTER the canonical mutation commits, so a failed delete can never leave a
live file whose bytes compaction might reclaim.
- History compaction is the ONE reclamation point: after deleting a
generation's record-set it releases that set's references and physically
reclaims any hash at zero live AND zero history references. Pins are
exempt automatically. Crash ordering is over-count-only in every path
(record before persist, release after delete), so a crash can leak until
the scrub recounts but can never reclaim bytes a retained generation
needs. scrubBlobHistoryRefCounts() restores exactness; existing stores get
a one-time marker-gated backfill on open, failing into leak-safe mode
(reclamation disabled) rather than guessing.
On top of the protected history, the temporal API the generational model
always implied:
- vfs.readFile(path, { asOf }) — the exact bytes as of a generation or Date,
materialized from the history (pinned view released so compaction is
never blocked by a read).
- vfs.history(path) — FileVersion[] ascending ({ generation, timestamp,
hash, size, mimeType? }), the newest entry being the live state.
- Overwrites now refresh the file entity's data/embedding text — semantic
search and the data field previously served the FIRST version's text
forever (the stale-field defect a consumer's incident recovery depended
on by luck).
Integration suite (temporal-vfs.test.ts): per-version exact reads +
history listing, leak-fix + history protection on overwrite, rm keeps bytes
readable, compaction reclaims past-window bytes and preserves in-window
(including the cross-file dedup case where an old file's history and a
newer file's removal share one hash), data freshness, and scrub exactness.
Subscribe once and receive one post-commit event per affected record for
EVERY canonical write, regardless of origin: direct calls, batch methods,
transact(), imports, and Virtual Filesystem writes all funnel through the
same commit points the feed is emitted from. This is the authoritative
in-process signal for live UIs, cache invalidation, and realtime sync layers
that forward it over their own transports.
Architecture — the post-commit dual of the ifRev precommit hook: mutation
methods hand lightweight event descriptors to the commit seam
(persistSingleOp / transact's plan), which stamps the committed
{generation, timestamp}, enriches entity deletes with the record's LAST
committed state from the commit's own before-images (free — Model B reads
them anyway; removeMany's id-only deletes gain full payloads this way), and
emits only after the commit succeeds — a losing CAS or rejected batch never
announces anything. Dispatch is a microtask FIFO after the mutex releases:
commit-ordered, a slow listener never delays a write, a throwing listener is
logged and isolated, and with no subscribers the write path constructs no
events at all. Single-point emission was chosen over per-method hooks
because the aggregation-hook pattern demonstrably drifted (relation ops and
removeMany were silently missing from it).
Coverage: add/update/remove (+ cascade unrelate per deleted relationship),
relate (both edges when bidirectional)/unrelate/updateRelation, per-item
events for addMany/updateMany/relateMany/removeMany, per-item events sharing
one generation for transact(), and transitively imports + VFS. clear() and
restore() — wholesale raw-state operations outside the per-record commit
path — emit a single store-level event meaning "refetch everything".
brain.close() drops all listeners.
New module src/events/changeFeed.ts (BrainyChangeEvent + ChangeFeed,
exported from the package root); guide docs/guides/reacting-to-changes.md.
Integration suite pins the contract: per-op payload fidelity, delete
last-state payloads, batch per-item emission, one-generation transact
batches, VFS-origin events, CAS-loser silence, ordering, listener isolation,
unsubscribe, and store-level events.
The 7.x branch system stored Virtual Filesystem content as blobs in its
copy-on-write area (`_cow/`). 8.0 removed that system and stores content blobs
in the content-addressed store (`_cas/`), but the one-time layout migration only
moves entities — it never adopted the `_cow/` content blobs. A store that used
the VFS was left with every VFS-backed read throwing "Blob metadata not found"
and its pages 500ing, with no first-party recovery and the pre-upgrade backup
already removed on entity-migration "success".
Add an on-open recovery pass (`autoAdoptLegacyVfsBlobsIfNeeded`) that runs right
after the layout migration and adopts every orphaned `_cow/` blob into `_cas/` —
copying both the bytes and the metadata via the raw object primitives,
idempotently and non-destructively (the `_cow/` originals are never deleted). It
is a separate phase, not folded into the migration, so it also heals a store
already upgraded by an earlier 8.0.x that stranded the blobs (whose layout-
migration marker is stamped): it is gated on the presence of `_cow/` and its own
`_system/vfs-blob-adoption.json` marker. Filesystem-only; native-8.0 and non-VFS
stores no-op on a cheap existence check.
- `BaseStorage.adoptLegacyCowBlobs()` — the scan/copy primitive, returning
`{ cowBlobs, adopted, alreadyPresent, incomplete }`; skips (does not half-adopt)
a blob missing its bytes or metadata.
- `brain.vfs.adoptOrphanedBlobs()` — the explicit force path; self-initializes.
- Backup retention: the automatic pre-upgrade backup is now kept if any blob
can't be fully adopted (`incomplete > 0`), instead of being removed on
entity-migration success alone — a blob-parity gate the migration signal
cannot provide.
Adds an end-to-end integration test (storage-level adopt with idempotency and
incomplete handling; self-heal on reopen; the explicit API) and an upgrade guide.
- data-storage-architecture.md described transact() as 'the unit of history'
(rc.2 era). Single-op retention has been the model since 8.0 — every write
gets its own generation; transact() only groups several into one. Corrected.
- RELEASE-GUIDE.md now records the hard ordering constraint: no embedding-model
change ships before vector model-version stamping + hard-error-on-mismatch
lands.
A production deployment measured ~48 seconds on EVERY reopen of an
11k-entity brain. Root cause: brainy's rebuild gate decided from in-memory
size()/count, which read 0 for a durable-but-not-resident index, so it
re-read every entity file to rebuild from scratch. At GA we gave only the
GRAPH provider a readiness contract (init() eager cold-load + isReady()
honest signal) so it would never eat that spurious rebuild; the vector and
metadata providers never got it, and brainy never even eager-inited the
vector provider.
Complete the contract symmetrically:
- plugin.ts: VectorIndexProvider gains optional init()+isReady();
MetadataIndexProvider gains isReady() — mirroring GraphIndexProvider.
Additive and optional; a provider that exposes nothing keeps today's
behavior.
- brainy.ts: eager-init every provider that exposes init() (after metadata
init() so the id-mapper is hydrated first), then decide per leg in
precedence order — migrating (skip) -> epoch drift (rebuild) -> isReady()
-> a per-leg empty fallback. The old instant fast-path keyed off
this.index.size()>0, a dishonest proxy that skipped the metadata/graph
checks whenever the vector was warm and never fired on a real cold process
anyway; removed.
The per-leg fallbacks differ because "empty" means different things: the JS
vector's rebuild() IS its load, so size()===0 correctly triggers it; the
id-mapper backs metadata, so totalEntries===0 (past the empty-store return)
is a real load failure; but entities do not imply edges, so a graph
size()===0 is a valid empty state, not a load failure.
- The JS graph now COLD-LOADS its durable LSM instead of re-deriving from a
full canonical verb scan on every boot (baseStorage._initializeGraphIndex
loads the persisted SSTables via a new GraphAdjacencyIndex.init(); it
self-heals from canonical only when the durable state is genuinely missing).
This removes an O(E)-per-open cost every filesystem consumer paid.
- LSMTree.loadManifest loads its SSTables BEFORE publishing the relationship
count, and resets to an honest-empty state on load failure — a tree can no
longer claim persisted relationships while holding none (the silent-empty
cold-load class the query-time guards exist to prevent).
Verified end-to-end against a built brain: a warm reopen (with edges and
edgeless) reloads only the JS vector; the graph and metadata cold-load with
no rebuild, and queries return correct results. New tests in
cold-open-rebuild-gate.test.ts pin the contract (isReady() defers, self-heal
still fires); migration-deference updated to drive size-based deference
through the vector, the leg where empty->rebuild remains correct.
Pairs with the native provider's isReady()/init() implementation — brainy's
gate defers only to a signal the provider exposes.
With plugins unset (the default), init() probes for the first-party
accelerator: not installed means plain brainy with zero noise; installed and
healthy means it loads and announces itself; installed but broken (import
failure, invalid shape, failed activation, version mismatch) makes init()
THROW. An installed accelerator never silently vanishes behind the JS engines
- the anti-drift posture of the explicit list, applied to detection.
plugins: []/false stays a true opt-out (no probe); an explicit list keeps its
required-and-loud semantics. The import runs through an importPluginPackage
seam (variable specifier - bundlers cannot static-resolve the optional
package; tests simulate all outcomes without it installed).
Also: when a plugin activates but registers zero native providers (e.g. a
licensing gate declining to engage), the provider summary now warns loudly
instead of leaving every query silently on the JS engines.
Supersedes 8.0.8's explicit-opt-in wording; README/PLUGINS/types now document
the guarded contract. 7 new tests (tests/unit/plugin-autodetect.test.ts).
A clean-room install smoke of the published GA pair caught the README telling
scale-up users the native provider is auto-detected. It is not, by design: the
loader (loadPlugins) does no auto-detection — undefined means no plugins, and
only an explicit plugins: ['@soulcraft/cor'] loads the native engine (loud
failure if it can't). The stale comment on the plugins config field in the
public types claimed auto-detection and is corrected to match the loader; one
phrase in the plugin-author guide likewise.
The 3.0 native engine ships as @soulcraft/cor (brainy 8.x <-> cor 3.x are a
version-matched pair); the old @soulcraft/cortex package stays on the 2.x/7.x
line. Public docs and .d.ts-visible comments now name the correct package.
Historical 2.x contract references (e.g. the 2.3.1 read-side fallback) keep
the old name deliberately.
Consumer-invisible modernization pass — no public API or runtime-behavior
change; dist for the override-only files is byte-identical.
Toolchain:
- CI: GitHub Actions matrix — Node 22/24 (test:unit) + Bun latest (test:bun).
- engines: node ">=22" (was "22.x"), bun ">=1.1.0".
- tsconfig: isolatedModules + noImplicitOverride; add the 33 `override`
modifiers the flag requires across storage/integrations/vfs/transaction.
- deps: @types/node ^22; add prettier; drop dead standard-version,
@rollup/plugin-* and the redundant embedded eslintConfig (flat
eslint.config.js is the active config — verified identical lint output).
paramValidation: replace the top-level `await import('node:os'/'node:fs')`
with static ESM imports. The top-level-await form poisoned the module graph;
static imports also drop the browser/edge fallback branches no supported
runtime reaches (8.0 is Node/Bun/Deno-only).
Bun positioning: recommend Bun as a runtime (`bun add` / `bun run`), which is
green (test:bun 8/8). Drop single-binary `bun build --compile` as a target —
native addons cannot embed into it, and Bun 1.3.10 has a `--compile` codegen
regression around top-level await. Rename the Bun test to bun-runtime-test.ts
and correct docs that overclaimed single-binary support.
Gates: typecheck 0, build 0, test:unit 1743/1743, test:bun 8/8.
8.0 keeps the canonical operators (eq/ne/gt/gte/lt/lte) and their clean
long-form aliases (equals/notEquals/greaterThan/greaterThanOrEqual/lessThan/
lessThanOrEqual), and drops the four redundant deprecated spellings:
is → eq, isNot → ne, greaterEqual → gte, lessEqual → lte
Removed from every evaluator (metadataIndex criteria + range switches,
metadataFilter, the db whereMatcher egress path) and from the
BrainyFieldOperators type, the unsupported-operator error message, the docs
(QUERY_OPERATORS / api README / VFS projection + semantic guides), and the
whereMatcher alias tests. Also migrated Brainy's own internal use — the VFS
TemporalProjection queried with greaterEqual/lessEqual, which would have
silently broken — to gte/lte.
Full gate green: build, unit 1512, integration 607.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The GA-readiness audit found the public docs had drifted from the shipped
surface and presented uncited performance numbers as measured fact.
- quick-start: `FindResult`→`Result`, `VerbType.BuiltOn`→`DependsOn` (the
canonical getting-started example now compiles).
- noun-verb-taxonomy: rewrote every sample off removed/fictional APIs
(`augment`/`connectModel`/`getVerbs`/two-arg `add`/`like`/`$gte`) onto the
real single-object `add`/`find`/`relate`/`related`; replaced the stale
31-noun/40-verb catalogs with accurate, complete tables (42 nouns, 127 verbs).
- triple-intelligence: `like:`→`query:`, dollar-operators→bare operators, and
several other fictional keys swept to the real `FindParams`.
- FIND_SYSTEM / PERFORMANCE / index-architecture / BATCHING: replaced
fabricated, mutually-inconsistent latency tables and uncited speedup
multipliers with Big-O characterizations, qualitative mechanism descriptions,
and the one genuinely-measured benchmark (graph O(1) neighbor lookup), per the
evidence-based-claims rule.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The dead-code sweep removed `src/utils/deletedItemsIndex.ts` (a utility class
the doc itself noted was "not instantiated"). Remove its "Notes on Other
Indexes" section and the two illustrative `this.deletedItemsIndex.*` lines in
the find()/stats() pseudo-code so the architecture doc no longer references a
deleted, never-wired module.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Every write — transact() AND single-op add/update/remove/relate — is now its
own immutable generation (Model-B), so a now() pin always freezes and
asOf/since/diff/history/transactionLog reflect single-ops exactly like
transacts. Closes the Model-A hole where pins did not freeze against single-op
writes.
Generation-stamping:
- GenerationStore.commitSingleOp: a one-operation commitTransaction with
deferred durability. Wired into add/update/remove/relate/updateRelation/
unrelate + removeMany (the *Many and VFS paths delegate to these).
- Async group-commit (flushPendingSingleOps): the live write is acknowledged
immediately; its before-image is buffered in an in-memory pending tier that
resolveAt/chains/changedBetween/tx-log read like on-disk generations, so the
synchronous now() freezes with no forced flush. One fsync per window
(triggers: size / 50ms timer / flush / close / transact / compactHistory).
- Crash recovery is drop-without-restore for group-commit generations (marked
groupCommit:true): a crash mid-flush discards the partial generation and
never restores its before-images, which would otherwise revert the
already-acknowledged live write.
- Init-time infrastructure (the VFS root) is the un-versioned generation-0
baseline: a fresh brain reports generation()===0 and an empty
transactionLog(); the first user write is generation 1.
- Historical find()/related() overlay bound is the full reserved watermark
(generation()), so un-flushed single-op writes are overlaid too.
Retention knob:
- config `history` -> `retention`: 'all' | 'adaptive' |
{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset ->
adaptive (disk/RAM pressure, zero-config). CompactHistoryOptions floors ->
caps (retainGenerations->maxGenerations, retainMs->maxAge, +maxBytes):
reclaim oldest-unpinned while ANY cap is exceeded; pins always exempt.
- brain.setRetentionBudget(bytes) drives the adaptive byte budget at runtime
(a coordinator's fair-share input). Per-generation bytes recorded in each
delta enable historyBytes() introspection without a storage size API.
Tests: per-write generation resolution, pin freeze vs add/update/remove,
drop-without-restore corruption-trap (fault injector), clean-reopen replay,
maxBytes/maxAge/no-cap reclamation, retention-then-reopen. 107 db/generation/
temporal tests green, tsc clean. Docs (ADR-001, consistency-model, snapshots
guide, api reference, RELEASES) updated to per-write granularity + retention.
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
asOf() answers "state AT a point"; these answer "what happened BETWEEN two
points" and "one entity's whole history", all on the existing generation records.
- Extract resolveAsOfGeneration() as the shared gen|Date→{generation,timestamp}
chokepoint (reachability + Date semantics identical everywhere). asOf()'s
inclusive path is byte-identical — db-mvcc still 25/25.
- asOf(target, { exclusive }) — strict-before (resolved−1, clamped at gen 0,
never a RangeError).
- db.since(Db | generation | Date) — overload; number/Date resolve via the shared
resolver (new DbHost.resolveGeneration); EXCLUSIVE lower bound;
since(db) === since(db.generation). Same-store guard via Db.belongsToStore.
- brain.diff(a, b) → { added, removed, modified } split by nouns/verbs. EARNS its
name: candidate set is ONLY changedBetween(gLow,gHigh), each classified by
existence at both endpoints + a key-order-insensitive value compare
(new src/db/stableEqual.ts) — a touched-but-reverted / born-and-died id is in
no bucket.
- brain.history(id, { from, to }) → every distinct version oldest→newest, each
value === asOf(version.generation).get(id); null = removal; kind auto-detected
(throws on UUID-space collision). New store helper generationsTouching().
- brain.transactionLog({ from, to, limit }) — INCLUSIVE generation/Date window
(contrast since's exclusive lower bound); limit applied last.
- Compaction policy locked: diff/since THROW GenerationCompactedError below the
horizon; history TRUNCATES to it.
Types DiffResult/HistoryVersion/EntityHistory exported from the package root.
Tests: tests/integration/db-temporal.test.ts (8 proofs incl. diff-earns-its-name,
history↔asOf cross-check, the composition proof, granularity, compaction contrast)
+ tests/unit/db/stableEqual.test.ts (6). docs/guides/snapshots-and-time-travel.md
+ RELEASES updated. 1477 unit + db-temporal 8 + db-mvcc 25 green.
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
- New docs/guides/export-and-import.md (public) for the 8.0 surface:
brain.export()/import(), Db composition (asOf/with time-travel + what-if export),
selectors, options, BackupData v1 format, cross-version (7.x→8.0), VFS, and the
generations/persist distinction. Documents only the implemented surface.
- api/README "Export & Import (portable) + Snapshots (native)": adds the portable
brain.export()/import() round-trip alongside the native persist()/asOf() snapshot.
Adds a reserved, top-level `visibility` field (mirrors the subtype rollout):
'public' (default, surfaced) | 'internal' (developer app-internal — hidden from
default find/count/stats, opt-in via includeInternal) | 'system' (Brainy
plumbing, library-set only).
Fixes a real leak: the VFS root entity counted in getNounCount() and appeared in
find() (a fresh brain reported 1 entity). It is now visibility:'system' →
excluded from every user-facing surface. Developers also get a first-class
hidden-unless-asked tier (e.g. learned internals vs user-exposed data).
- Reserved (RESERVED_ENTITY_FIELDS / RESERVED_RELATION_FIELDS) — spoof-proof from metadata.
- Threaded through add/relate/update/transact; surfaced top-level on reads.
- Default exclusion in counts (baseStorage), find()/related() (hard candidate
filter via excludeVisibility — keeps topK/limit correct), and stats;
includeInternal/includeSystem opt-ins.
- VFS root marked 'system'.
Tests: visibility.test.ts 17/17 (fresh-brain getNounCount()===0, internal hidden
+ opt-in, verb symmetry, top-level surfacing, metadata-spoof rejection). Unit
1431 green; count-synchronization integration now passes (off-by-one fixed).
The installation guide asserted a '5.2x geometric mean speedup' linking a /docs/cortex/comparison
page that does not exist (404) and has no backing benchmark — an unbacked performance claim
shipping in the public package, against the evidence rule. Replaced with a qualitative
native-acceleration statement (no number, no dead link) until a measured, reproducible
open-core-vs-native comparison is published. Also dropped the dead 'cortex/comparison' next-link
from PLUGINS.md frontmatter.
Open-core (pure-TS) latencies from tests/benchmarks/find-composition-scale.js: vector and
graph scale ~log(n) (~1.4ms / 0.65ms p50 at 100k); metadata-filtered paths scale with the
match-set size (low-selectivity worst case), which is the path the native provider
accelerates. Composition correctness cited to the triple-composition test. States the
open-core build ceiling (~10^5-10^6) and the native-provider path beyond.
The old config-generation subsystem (src/config/ + autoConfiguration.ts) was
superseded during the 8.0 rework and never wired into init(): it emitted settings
for a partitioning subsystem that no longer exists and probed deleted cloud env
vars. The live zero-config path is inline — recall preset → HNSW knobs, storage
auto-detect, auto persistMode, container-memory-aware cache sizing.
The storage progressive-init / cloud-detection cluster was equally dead after the
cloud adapters were dropped: isCloudStorage() is permanently false (no overriders),
scheduleBackgroundInit/runBackgroundInit were never called (the latter an empty
body), initMode was never assigned, and Brainy.isFullyInitialized()/
awaitBackgroundInit() were always-trivial with zero callers. scheduleCountPersist()
collapses to its only-ever-taken immediate write-through path.
Removed:
- src/config/{index,zeroConfig,storageAutoConfig,modelAutoConfig,sharedConfigManager}.ts
- src/utils/autoConfiguration.ts + the inert BrainyZeroConfig export
- Brainy.isFullyInitialized()/awaitBackgroundInit() (+ BrainyInterface decls)
- InitMode type, isCloudStorage/detectCloudEnvironment/resolveInitMode,
scheduleBackgroundInit/runBackgroundInit/ensureValidatedForWrite and their state
- Dead cloud env-var probes (K_SERVICE/K_REVISION/AWS_LAMBDA_FUNCTION_NAME/
FUNCTIONS_TARGET/AZURE_FUNCTIONS_ENVIRONMENT)
Kept (verified live): production-detection logging (environment.ts), container-
memory cache sizing (memoryDetection/paramValidation), on-disk hash bucketing
(sharding.ts).
Docs: scrubbed deleted-subsystem references (JS quantization knobs, cloud/OPFS
adapters, partitioning, old zero-config API) across 14 files; deleted two wholly-
obsolete feature docs (complete-feature-list, v3-features); rewrote
architecture/zero-config for 8.0.
~3,700 LOC removed. Build clean; 1392 unit + 24 db-mvcc green.
The distributed-clustering subsystem never ran in production: it was inert,
orphaned dead code (faked consensus, stub replication, no live wiring, and it
did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process
library. Scale is single-process + the optional native provider
(@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools +
horizontal read scaling (many reader processes, one writer).
Removed:
- src/distributed/ entirely (coordinator, shardManager, cacheSync,
readWriteSeparation, queryPlanner, healthMonitor, configManager,
hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network
transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts
(slimmed to the live surface).
- src/types/distributedTypes.ts; config.distributed field + JSDoc;
coreTypes distributedConfig; memoryStorage distributedConfig persistence.
- DistributedRole enum + src/config/distributedPresets.ts and the orphaned
src/config/extensibleConfig.ts (config/augmentation registry built on removed
cloud adapters + distributed presets), plus their src/index.ts re-exports.
- 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook;
enableDistributedSearch (dead config flag); the metadata partition field;
the distributed_ reserved key prefix.
- Orphaned src/storage/readOnlyOptimizations.ts (zero importers).
- Tests targeting the subsystem: distributed-demo, distributed-cluster helper,
distributed-transactions, sharding-transactions.
- Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/
shard-manager/multi-node prose from v3-features, enterprise-for-everyone,
augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP,
vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4,
storage-architecture; reframed scale prose to the 8.0 model.
Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via
getShardIdFromUuid — used live by baseStorage, unrelated to clustering);
the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering.
RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0
scale model.
Brainy-owned field names (noun/verb, subtype, createdAt, updatedAt,
confidence, weight, service, data, createdBy, _rev) now have exactly one
home — top level — enforced by three layers driven from a single source of
truth, src/types/reservedFields.ts (RESERVED_ENTITY_FIELDS /
RESERVED_RELATION_FIELDS, exported):
1. Compile time — AddParams/UpdateParams/RelateParams/UpdateRelationParams
metadata (and the transact() ops that extend them) reject a literal
reserved key as a TypeScript error while keeping generic T ergonomics
(typed bags, untyped brains, index-signature shapes, and a documented
exemption for T-declared reserved keys). Pinned by @ts-expect-error
type tests run under vitest typecheck mode on every unit run.
2. Write time — the 7.x update() remap is ported to 8.0 and extended to
every write path: add/update/relate/updateRelation, their transact()
mirrors, and db.with() overlays. User-settable fields lift to their
dedicated param (top-level wins when both are supplied — closes the 7.x
trap where update({metadata:{confidence}}) silently no-oped), and
system-managed fields drop with a one-shot warning naming the right
path. A remapped subtype satisfies subtype-pairing enforcement exactly
like a top-level one.
3. Read time — every storage combine goes through one canonical hydration
helper (hydrateNounWithMetadata / hydrateVerbWithMetadata over
splitNoun/VerbMetadataRecord), so reserved fields surface ONLY top-level
and entity/relation.metadata carry ONLY custom fields on live reads,
batch reads, paginated listings, getRelations by source/target, streamed
verbs, and historical asOf() materialization alike.
Read-path echoes found and fixed (previously the full stored record —
including the verb type key — leaked inside metadata): noun pagination,
verb pagination, getVerbsBySource/ByTarget (adjacency + shard fallback),
getVerbsBySourceBatch (which also dropped subtype/data), and the
filesystem verb stream. getRelations() results now surface
confidence/updatedAt top-level via verbsToRelations, updateRelation() no
longer erases service/createdBy, relate() persists its top-level
confidence/service params, and the dead convertHNSWVerbToGraphVerb echo
path is deleted. Import paths (CLI extract, deduplicator, coordinators,
neural import) write confidence through the dedicated param instead of the
bag. UpdateRelationParams is now exported from the package root.
Documented for consumers in docs/concepts/consistency-model.md ("Reserved
fields") and RELEASES.md. Regression tests ported from the 7.x fix and
extended to the full 8.0 contract (17 runtime tests + 41 type-level
assertions); full unit suite 1427/1427, db-mvcc integration 24/24.
- brain.fillSubtypes(rules): idempotent subtype back-fill for pre-8.0 data.
One rule per NounType/VerbType (literal default or per-entry function);
fills only entries still missing a subtype through the public update()/
updateRelation() paths; returns { scanned, filled, skipped, errors, byType }.
Full unit suite in tests/unit/brainy/fill-subtypes.test.ts.
- Fix getNouns/getVerbs pagination hasMore (peek one past the window) —
was permanently false, silently truncating every multi-page walk.
- find({ near }) without near.id now throws a teaching error instead of an
opaque storage sharding failure; CLI --threshold without --near applies a
plain score floor.
- CLI init/close audit: every one-shot command init()s, close()s, and exits
explicitly; delete the unmaintained interactive REPL; replace the cloud-era
storage subcommands with status/batch-delete; new types/validate commands.
- requireSubtype JSDoc now documents the 8.0 default-on contract; audit()
recommendation points at fillSubtypes.
- Docs: data-storage-architecture rewritten to the real 8.0 on-disk layout;
README storage section reflects filesystem+memory and snapshots; eli5 and
SEMANTIC_VFS /as-of/ semantics corrected; internal tracker IDs and
.strategy references scrubbed from published files.
The legacy backup/import/export/stats facade (src/api/DataAPI.ts) drifted
from the modern entity shape and every job it did now has a first-class
surface. Delete it and brain.data(), and rewire the CLI:
- data-stats → brain.stats() (full BrainyStats report: per-type breakdowns,
indexed fields, index health, storage backend, writer lock, version)
- clean → brain.clear()
- export → alias of snapshot; a db.persist() snapshot is the full-fidelity
export format (open with Brainy.load, load wholesale with brainy restore);
external data ingestion remains brainy import (UniversalImportAPI)
Rewiring clean onto brain.clear() exposed two real bugs, both fixed:
- clear() left this.graphIndex undefined forever — any graph-touching call
afterwards (relate, getNeighbors, stats) crashed. clear() now re-resolves
the graph index exactly as init() does and re-wires the shared UUID↔int
resolver, and re-resolves the metadata index with the same provider
fallback as init().
- storage.clear() reset the legacy totals but not the per-type/subtype
count rollups or id→type caches, so stats() reported phantom counts for
deleted entities. Both adapters now delegate derived-state reset to
reloadDerivedState(), the same path restore-from-snapshot uses.
One-shot CLI commands (data-stats, clean, snapshot/export, restore,
history, generation) now close the brain and exit explicitly — global
cache timers otherwise keep the process alive holding the writer lock.
Verified: build clean, 1383/1383 unit tests, 24/24 db-mvcc integration,
plus an end-to-end CLI smoke (add → data-stats → export → clean →
data-stats).
Historical Db values (now()/asOf() pins that history has moved past) now
serve the COMPLETE query surface - vector/hybrid search, graph traversal,
cursor pagination, and aggregation - by materializing ephemeral in-memory
indexes over the exact at-generation record set. The historical-query
throw is gone; NotYetSupportedAtHistoricalGenerationError is deleted.
Materializer (Brainy.materializeAtGeneration):
- Copies the at-G record set (live bytes for ids untouched since the pin,
immutable before-images otherwise) into a fresh MemoryStorage; a final
reconciliation pass under the commit mutex makes the copy exact even
when transactions commit mid-build.
- Opens a read-only Brainy over the copy: init rebuilds the metadata and
graph-adjacency indexes from the records; the vector index is built by
inserting every at-G vector (the at-G HNSW graph never existed on disk,
so there is nothing to restore). Host embedder and aggregate definitions
are shared - no second model load, aggregates backfill at-G values.
- Cost is the documented contract: O(n at G) time and memory, ONCE per Db
(handle cached; freed by release(), with a FinalizationRegistry backstop
that also closes leaked readers). A native VersionedIndexProvider serves
the same reads from retained segments with no rebuild.
Db routing (src/db/db.ts): metadata-level find()/related() keep the free
record path; index-only dimensions (query/vector/near/connected/cursor/
aggregate/includeRelations/non-metadata modes) route to the cached
materialization; unsupported where-operators on the record path re-route
there too instead of erroring. Speculative with() overlays keep the one
honest boundary - SpeculativeOverlayError (overlay entities carry no
embeddings, so index reads over them would be silently incomplete);
metadata find()/get()/filter related() work on overlays.
UpdateParams.vector contract now honored: an explicit pre-computed vector
applies directly (with dimension validation) in update() and transact
update ops, re-indexing HNSW - previously it was silently ignored unless
data also changed.
GraphAdjacencyIndex: adjacency now derives from the two verb-id LSM trees
filtered through the live-verb tombstone set (entity->entity edge trees
deleted - they carried no verb ids, so removeVerb could never tombstone
them and traversal served stale neighbors forever). Neighbor reads batch-
load live verbs via the unified cache; addVerb seeds the cache.
Proofs (tests/integration/db-mvcc.test.ts, 24 green): historical vector
search finds old vector placement including since-deleted entities;
historical graph traversal walks the old wiring after a rewire; historical
aggregation computes at-G group values; asOf() pins get the same surface;
the materialization builds once per Db and release() closes the ephemeral
reader (it refuses reads afterwards); overlays throw the documented error.
ADR-001 updated to the no-throws historical model.
One mechanism replaces the COW and versioning subsystems: immutable
generation-stamped records behind a Datomic-style database value.
Record layer (src/db/generationStore.ts):
- Monotonic generation counter in _system/generation.json; bumped once per
transact() commit and once per single-operation write (storage hook), so
brain.generation() is always a meaningful watermark.
- Commit protocol: stage before-images + tx.json delta -> fsync -> execute
batch via TransactionManager -> atomic tmp+rename of _system/manifest.json
(the rename IS the commit point) -> append _system/tx-log.jsonl.
- Crash recovery on open rolls uncommitted generations back byte-identically
and forces an index rebuild; refcounted pins gate compactHistory(), which
records a horizon (asOf below it throws GenerationCompactedError).
Db API:
- Db: get/find/search/related pinned at a generation, with() speculative
overlays, since() diffs, persist() hard-link-farm snapshots, timestamp,
generation, release() + FinalizationRegistry backstop.
- Brainy: now() O(1) pin, transact(ops, {meta, ifAtGeneration}) atomic batch
(GenerationConflictError CAS), asOf(generation|Date|path), restore(path,
{confirm}), compactHistory(), generation(), static open() + load().
- get()/metadata find()/related() are fully correct at any reachable pinned
generation; index-accelerated queries at historical generations throw
NotYetSupportedAtHistoricalGenerationError - never silently-wrong results.
- VersionedIndexProvider (generation/isGenerationVisible/pin/release) in
plugin.ts: feature-detected, balanced pin/release in lockstep with Db
lifecycle, post-commit applier + replay-gap model documented.
Storage primitives (BaseStorage + filesystem/memory adapters): raw-object
read/write/list/remove, fsync barrier, noun/verb raw before-image capture,
tx-log append, snapshotToDirectory (hard-link farm; byte-copy fallback and
append-in-place exceptions), restoreFromDirectory + derived-state reload.
Proof suite (tests/integration/db-mvcc.test.ts + tests/unit/db/): isolation
across 200 mutations, batch atomicity under injected execution failure,
ifAtGeneration CAS, snapshot immunity to source mutation, compaction safety
under pins, with() overlay isolation, generation monotonicity across
reopen, crash consistency through the real recovery path, and versioned
provider pin/release balance. Design record in
docs/ADR-001-generational-mvcc.md.
Final cleanup pass for Brainy 8.0. Catches three categories of debt:
A. STEP-7 FOLLOW-THROUGH (rebuild-path collapse)
Step 7's bisect-reset (debugging a flaky test) lost the in-source edits
to three rebuild paths even though the commit message claimed they
shipped. Re-applied now:
- src/utils/metadataIndex.ts — collapsed the `isLocalStorage` /
cloud-pagination branching. Local-load-all-at-once is the only path
in 8.0. Removed ~150 LOC of paginated-cloud branching for both nouns
and verbs, plus the safety counters (`consecutiveEmptyBatches`,
`MAX_ITERATIONS`, etc.).
- src/hnsw/hnswIndex.ts — same simplification for HNSW rebuild. The
paginated cloud path is gone; HNSW now loads all nodes at once.
Removed ~85 LOC.
- src/graph/graphAdjacencyIndex.ts — same simplification for graph
adjacency rebuild. Removed ~50 LOC.
The collapse is safe because cloud adapters were deleted in step 7;
`storageType === 'OPFSStorage'` (and similar) can never match now.
B. CLOUD-ONLY DOCS DELETED
- docs/operations/cost-optimization-aws-s3.md
- docs/operations/cost-optimization-azure.md
- docs/operations/cost-optimization-cloudflare-r2.md
- docs/operations/cost-optimization-gcs.md
- docs/operations/cloud-run-filestore-guide.md
(docs/deployment/* contained no cloud-specific files that needed deletion.)
C. STORAGE-ADAPTERS GUIDE REWRITTEN FOR 8.0
docs/guides/storage-adapters.md → fresh content reflecting the 8.0
reality:
- Two adapters: FileSystemStorage + MemoryStorage. Quick-start matrix.
- Cloud backup section explains the operator-tooling pattern (gsutil /
aws s3 / rclone / azcopy) with the exact commands consumers will run.
- "Why no cloud adapters in 8.0?" section documents the four reasons
per BR-BRAINY-80-STORAGE-SIMPLIFY.
- Migration recipe for 7.x cloud-adapter consumers: mount local disk →
filesystem storage → operator backup cron.
Updated frontmatter description so soulcraft.com/docs renders the
correct preview.
NOT IN THIS COMMIT (deliberate, lower-priority)
- src/storage/cacheManager.ts still references StorageType.S3 /
REMOTE_API / OPFS as dead branches (23 sites). The branches are never
reached in 8.0, but cleaning them would cascade through 5 consumers.
Defer to a follow-up if the dead code surfaces as a real maintenance
issue.
- src/config/storageAutoConfig.ts keeps its StorageType enum + autodetect
for 7.x compat surface. Same reason: rewriting cascades through
zeroConfig, extensibleConfig, sharedConfigManager. Defer.
- docs/MIGRATION-V3-TO-V4.md and docs/DEVELOPER_LEARNING_PATH.md still
reference cloud adapters as historical artefacts. That's accurate —
they describe how things used to be. Left as-is.
- @deprecated audit in src/ (10 files) deferred — audit each individually
in a future polish pass.
VERIFICATION
- npx tsc --noEmit: clean
- npm test: 1408 / 1409 (same pre-existing race-condition outstanding from
step 7; no regressions from this cleanup)
