The shard-scan pagination adapter is offset-based and ignored the cursor, so
getNouns({ pagination: { cursor } }) re-returned the first page on every cursor
call. Harmless until 7.32.1 made totalCount the true dataset total — after which
hasMore correctly stays true until a caller has paged through everything. A
caller that paginates by cursor (cursor = page.nextCursor) — notably aggregate
backfill over an already-populated store — then looped forever, re-walking the
entire entity shard tree each iteration and pegging 1-2 CPU cores permanently on
the JS main thread, with zero queries or traffic. (Pre-7.32.1 the same loop
ended after one page, silently backfilling only the first 500 entities — an
incomplete aggregate.)
getNouns() now treats the cursor as an opaque, advancing offset token, so cursor
pagination advances and terminates exactly like offset pagination — and an
aggregate backfill streams the whole corpus exactly once (no longer truncated,
no longer looping). Hardened the backfill loop to pure offset pagination as
defense in depth.
Regression (reproduces the infinite loop, fails fast on any re-scan):
tests/unit/storage/getNouns-cursor-pagination.test.ts
Every write — transact() AND single-op add/update/remove/relate — is now its
own immutable generation (Model-B), so a now() pin always freezes and
asOf/since/diff/history/transactionLog reflect single-ops exactly like
transacts. Closes the Model-A hole where pins did not freeze against single-op
writes.
Generation-stamping:
- GenerationStore.commitSingleOp: a one-operation commitTransaction with
deferred durability. Wired into add/update/remove/relate/updateRelation/
unrelate + removeMany (the *Many and VFS paths delegate to these).
- Async group-commit (flushPendingSingleOps): the live write is acknowledged
immediately; its before-image is buffered in an in-memory pending tier that
resolveAt/chains/changedBetween/tx-log read like on-disk generations, so the
synchronous now() freezes with no forced flush. One fsync per window
(triggers: size / 50ms timer / flush / close / transact / compactHistory).
- Crash recovery is drop-without-restore for group-commit generations (marked
groupCommit:true): a crash mid-flush discards the partial generation and
never restores its before-images, which would otherwise revert the
already-acknowledged live write.
- Init-time infrastructure (the VFS root) is the un-versioned generation-0
baseline: a fresh brain reports generation()===0 and an empty
transactionLog(); the first user write is generation 1.
- Historical find()/related() overlay bound is the full reserved watermark
(generation()), so un-flushed single-op writes are overlaid too.
Retention knob:
- config `history` -> `retention`: 'all' | 'adaptive' |
{ maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset ->
adaptive (disk/RAM pressure, zero-config). CompactHistoryOptions floors ->
caps (retainGenerations->maxGenerations, retainMs->maxAge, +maxBytes):
reclaim oldest-unpinned while ANY cap is exceeded; pins always exempt.
- brain.setRetentionBudget(bytes) drives the adaptive byte budget at runtime
(a coordinator's fair-share input). Per-generation bytes recorded in each
delta enable historyBytes() introspection without a storage size API.
Tests: per-write generation resolution, pin freeze vs add/update/remove,
drop-without-restore corruption-trap (fault injector), clean-reopen replay,
maxBytes/maxAge/no-cap reclamation, retention-then-reopen. 107 db/generation/
temporal tests green, tsc clean. Docs (ADR-001, consistency-model, snapshots
guide, api reference, RELEASES) updated to per-write granularity + retention.
Dev-only standalone harnesses (run via node --import tsx; not globbed by the
unit/integration vitest configs). model-b-scalability.spike.ts is the evidence
harness referenced by the Model-B build spec — re-validates read-vs-depth,
RAM-vs-depth, reopen, and compaction at scale.
Historical reads (asOf/get) scanned the global committedGens list linearly,
making them O(database-age): a read of an unchanged entity at an old pin scaled
~12x for 10x history depth. Add per-id inverted history chains (nounChains/
verbChains) so resolveAt binary-searches the id's own generation chain instead —
O(log) and flat with depth. Chains build lazily under the commit mutex,
maintain incrementally on commit, and invalidate on compaction.
Also bound deltaCache (LRU cap 4096; getDelta re-reads evicted deltas) so a
long-lived high-write process's heap is O(cap) not O(generations) on the
disk-backed path, and binary-search commitTimestampAtOrBefore (O(log)).
Verified: 373 unit tests green; new tests/unit/db/generation-chain.test.ts
covers chain resolution, eviction re-read, and chain rebuild after compaction.
The public brain.graph.* surface and the GraphAccelerationProvider seam must name
operations by INTENT, not by the algorithm that happens to implement them — so a
plugin can swap algorithms without the contract lying. The JS fallback computes
communities via connected-components and rank via PageRank, but a native provider
(cor) is free to use Louvain/Leiden for communities and personalized-PageRank /
eigenvector-centrality for rank: same intent, different algorithm, same name.
Renamed the (not-yet-implemented) Phase-C analytics methods + their option/result
types on GraphAccelerationProvider:
- pageRank -> rank (PageRankOptions -> RankOptions; dropped the
pagerank-internal damping/maxIterations/tolerance
knobs from the contract — each impl tunes itself)
- connectedComponents -> communities (ComponentsOptions -> CommunitiesOptions,
mode:'weak'|'strong' -> directed?:boolean;
GraphComponents -> GraphCommunities, componentIds/
Count -> communityIds/Count)
- shortestPath -> path (ShortestPathOptions -> PathOptions; weight -> by:'hops'|'weight')
- neighborhoodSample -> sample (NeighborhoodSampleOptions -> SampleOptions)
- topByDegree -> mostConnected (TopByDegreeOptions -> MostConnectedOptions)
traverse / edgesForNode / graphCursorOpen/Next/Close + GraphScores/GraphPath/
OpaqueIdSet/Subgraph are UNCHANGED — everything cor has already built is untouched;
this is a pure pre-implementation rename (cor coordinated). JSDoc reworded to state
the algorithm is the provider's choice. index.ts exports + the native-seam test
mock updated. tsc/unit 1517/integration 599 green.
The 8.0 entry referenced a non-existent '@soulcraft/cortex 3.0' (line 447) while
the graph section already said '@soulcraft/cor 3.0' (line 44) — a self-contradiction.
The native provider was renamed cortex→cor; the 8.0 pairing is @soulcraft/cor 3.0
(the renamed successor to @soulcraft/cortex 2.x). Fixed the 8.0-section refs (the
Cor-compatibility block + the scale-path note); HISTORICAL 2.x refs in the v7.31.2 /
older entries are left intact (renaming them would falsify the changelog).
The brain.graph.subgraph/export NATIVE routing (graphSubgraphNative /
graphExportNative / hydrateNativeSubgraph + provider resolution) had ZERO brainy
CI coverage — in production it's exercised only cross-layer against cor's engine,
so a columnar return-shape or hydration-alignment drift would pass brainy CI
silently. This registers a faithful MOCK GraphAccelerationProvider returning a
columnar Subgraph built from the brain's REAL ints, locking the seam:
- subgraph() routes native (traverse called) and hydrates node int->id, the
nodeDepth column aligned to the node column, node type via batchGet, and edge
verb-int->Relation via verbIntsToIds + getVerbsBatchCached.
- node<->depth alignment is preserved when a node int does NOT resolve
(deleted/unknown) — the unresolvable int is dropped, not collapsed (which would
shift every later depth). This is the exact hydration risk the release audit flagged.
- export() routes to the graph cursor, hydrates chunks, and ALWAYS closes the handle.
Also fixes a latent contract bug found writing the test: graphAccelerationProvider()
only accepted a ready instance, so a provider registered as a (storage)=>provider
FACTORY (the convention graphIndex/metadataIndex/vector use) would fail the duck-test
and the native path would silently never engage. Now resolves instance OR factory,
cached. Covered by the factory-registration test.
brain.graph.export() streams the WHOLE graph in one O(N+E) pass — node chunks
then edge chunks, async-iterable — the right primitive for visualizing all data
(vs. paging per node). Native snapshot-consistent graphCursor when present, else
a TS cursor walk over nouns + verbs. Building it surfaced two latent noun bugs,
both fixed here (each a correctness win beyond export):
- getNounsWithPagination IGNORED its cursor ('offset-based, cursor planned') —
the noun mirror of the verb bug fixed in the cursor-pagination commit. Latent
because the only multi-page consumer (aggregate backfill) uses one big page; a
small chunkSize needed page 2 and, trusting the returned-but-ignored nextCursor,
re-fetched page 0 forever (infinite loop). Ported the proven verb cursor:
opaque cn1:<shard>:<id> token, stable within-shard id order, resume-after,
O(N) at any chunk size. (Generalized verbIdFromVectorPath → idFromVectorPath.)
- hydrateNounWithMetadata DROPPED 'visibility' (noun mirror of the Fix#1 verb
hydration bug) — so getNouns-fed visibility filters saw nothing and leaked
system (VFS root) / internal nodes. Now hydrated.
- New: GraphApi.export + GraphExportOptions (chunkSize, includeInternal/System,
includeNodes/Edges). hydrateNativeSubgraph extracted + shared by subgraph+export.
Test: graph-export.test.ts — full-graph completeness incl. isolated nodes,
default-hides-internal, node/edge include toggles, chunkSize chunking (the
case that exposed the cursor hang). Full gate green.
Introduces the brain.graph namespace and its first op, subgraph() — bounded
multi-hop neighborhood extraction returning a hydrated GraphView (nodes with hop
depth + type/subtype, edges as Relations, a truncated flag). The one-call answer
to 'show me everything around this node, N hops out' the graph-viz path needs.
- brain.graph.subgraph(seeds, { depth, direction, type, subtype, includeInternal,
includeSystem, maxNodes, maxEdges, hydrateNodes }). seeds: id or id[].
- Feature-detect routing: when a GraphAccelerationProvider is registered
(isGraphAccelerationProvider on the 'graphAcceleration' provider) it routes to
native traverse() and hydrates the columnar Subgraph (node ints↔ids paired with
depth position-preserving, edge verb ints → Relations); otherwise a pure-TS BFS
fallback expands each frontier through the O(degree) related() adjacency.
- Both paths produce the identical GraphView, so CI exercises the shape via the
fallback; the native path is cross-layer-tested against cor's provider.
- New public types: GraphView, GraphNode, SubgraphOptions, GraphApi (the surface
grows: export/rank/path/communities follow). isGraphAccelerationProvider guard
added to plugin.ts.
Test: graph-subgraph.test.ts — depth tracking, direction, type filter, default-
hides-internal, node hydration on/off, maxNodes truncation, multi-seed.
related({ from }) / related({ to }) each return one direction; related({ node })
unions both (every edge where the node is source OR target), deduped, in a single
O(degree) call — the indexed equivalent of merging two related() calls by hand.
The 'all edges on a noun' query the graph-viz path needs.
- RelatedParams.node: mutually exclusive with from/to (throws if combined);
composes with type/subtype/visibility filters; natural-key ids resolved.
- Implemented as a parallel out-edge (sourceId) + in-edge (targetId) fetch through
the O(degree) graph-index fast paths, merged + deduped by id (a self-loop appears
once), sorted for a stable page. Full incidence is the intended O(degree) cost;
deep offset paging of a very-high-degree node is best-effort (use the native
edgesForNode / a graph cursor for exhaustive paging).
- db.related() parity: node resolved + honored in relationMatchesParams (matches an
edge incident in either direction) for the historical/overlay paths.
Test: related-node-bidirectional.test.ts — both directions, equals from∪to, self-loop
dedupe, type-filter composition, default-hides-internal, node+from/to throws.
Defines the optional native graph-acceleration provider (cor 3.0) that
brainy feature-detects and routes brain.graph.* / related({node}) /
find({connected}) to, falling back to pure-TS adjacency when absent. This is
the published seam cor wires its native build against (mirrors the existing
VersionedIndexProvider pattern: defined in plugin.ts, exported from index.ts,
implemented externally, feature-detected at runtime).
Contract (locked cross-team in the design thread):
- GraphAccelerationProvider: traverse, edgesForNode, graphCursorOpen/Next/Close,
pageRank, connectedComponents, shortestPath, neighborhoodSample, topByDegree.
Every read op is generation-aware (optional trailing generation?: bigint) so
db.asOf(g).graph.* resolves historically; omitted = now.
- Subgraph: columnar wire format (BigInt64Array node/edge columns, Uint16Array
edgeTypes, optional Uint8Array nodeDepth, truncated flag) — parallel typed
arrays, never array-of-objects; brainy maps u64<->UUID lazily for rendered rows.
- OpaqueIdSet: a find() universe forwarded opaquely into traverse/search for the
zero-crossing query->expand fusion (brainy never inspects it; the provider
version-tags + validates the envelope).
- VectorIndexProvider.search gains allowedIds?: OpaqueIdSet | ReadonlySet<string>
(predicate pushdown — recovers filtered recall lost to post-filtering).
- EntityIdMapperProvider gains entityIntsToUuids(BigInt64Array) — the bigint batch
reverse resolver for Subgraph.nodes; implemented on the JS mapper (TS fallback).
All new public types exported from index.ts. Test: entityIntsToUuids round-trip
+ order-preservation + empty-int sentinel.
getVerbsWithPagination was offset-only: it re-scanned from shard 0 on every page
and early-terminated at offset+limit, so a full edge walk was O(N²) (a consumer
measured ~19k edges → 27s paging at 900/page). It now accepts an opaque resume
token (cv1:<shard>:<verbId>) that resumes the shard walk immediately AFTER the
last returned verb — O(N) total at any page size, no scale cliff.
- Stable within-shard order (sort by verb id); ids come from the path so verbs
skipped by the cursor are never read.
- Cursor supersedes offset when present; offset path preserved for back-compat,
and offset callers now get a usable nextCursor so they can switch to cursor
paging. Foreign/malformed tokens decode to null → offset fallback (no throw).
- The relationships stream generator now uses cursor (was offset → O(N²)).
- Parity with the noun side (getNounsWithPagination already cursored).
Test: tests/unit/storage/verb-cursor-pagination.test.ts — a full cursored walk
visits every edge exactly once (no dup, no skip), matches the offset walk's set,
and a foreign cursor falls back gracefully.
related({from/to}) routes through storage.getVerbs, which has O(degree)
GraphAdjacencyIndex fast paths (getVerbsBySource/Target/Type_internal). But
those fast paths were disqualified whenever excludeVisibility was set —
and every default related() sets it, because visibility defaults to excluding
internal+system. So the common per-node edge lookup fell through to a full O(E)
shard scan (multi-second per node on large graphs; a consumer measured
1.4-4.6s/node and an O(N^2) whole-graph walk).
Root cause two-parter, both fixed here:
- hydrateVerbWithMetadata dropped 'visibility' (mapped subtype but not the
reserved visibility field), so fast-path results couldn't be visibility-filtered
AND related({includeInternal}) couldn't even report which edges were internal
(latent correctness bug — verbsToRelations already mapped it). Now hydrated.
- getVerbs disqualified the fast paths on subtype/excludeVisibility. Now the
fast paths apply both filters on their already-hydrated O(degree) candidate set
via applyVerbMetadataFilters() — matching the full-scan fallback's semantics —
so default related() stays on the index instead of scanning the whole graph.
Filtering semantics are unchanged (same result set as the scan); only the path
that computes it changes. Test: related-visibility-fast-path.test.ts (default
excludes internal both directions, includeInternal surfaces + reports visibility,
subtype filter on the fast path).
Three additive ergonomics from the API-simplification audit (no behavior change
to existing call sites):
- AddParams.upsert: create-or-update in one call. With a custom id, an existing
entity is MERGED via the update path (merges metadata, re-embeds changed data,
bumps _rev, PRESERVES createdAt) instead of the destructive full overwrite a
plain add() does. Mutually exclusive with ifAbsent (throws if both set);
ignored when no id is supplied. Wired into add(), addMany (per-item flag
propagation), and the transact add op (routes to planTxUpdate). Kills the
get()-then-add() round-trip for idempotent writes.
- FindParams.includeVectors: mirror of GetOptions.includeVectors — find() returns
stored vectors when set; default stays empty (the perf contract is preserved).
Honored on both the query and metadata-only where paths, and in db.find().
- removeMany adaptive chunking: replaced the hardcoded chunkSize=10 with
params.chunkSize ?? storageConfig.maxBatchSize, matching addMany/relateMany —
one storage-adaptive batch policy across all *Many methods (no thrash on
high-latency backends).
Tests: tests/unit/brainy/upsert.test.ts (insert/merge/createdAt-preserved/
re-embed/ifAbsent-conflict/no-id/addMany/transact), find-include-vectors.test.ts
(true/default/where-path), batch-operations.test.ts (removeMany >10 items).
An untyped (JS) caller that smuggles a Brainy-reserved field (confidence,
weight, subtype, visibility, service, createdBy, noun/verb, data, createdAt,
updatedAt, _rev) inside a write-path metadata bag previously got a silent
remap-or-drop — a class of bug where confidence-evolution writes no-oped for
weeks before being caught on read-back. 8.0 closes this with no silent failures.
- New BrainyConfig.reservedFieldPolicy: 'throw' | 'warn' | 'remap' (default 'throw').
'throw' rejects the write naming every offending key + its correct write path;
'warn' remaps with a one-shot per-key warning; 'remap' is the legacy silent path.
- Central enforceReservedPolicy gate wired into all four remap methods (add,
update, relate, updateRelation) so live calls AND their transact()/with()
mirrors honor it. Single-source reservedWritePath guidance shared by throw+warn.
- 'warn' now warns for EVERY reserved key (closes the gap where only
system-managed fields warned). Dead warnDropped* helpers removed.
- Import pipeline migrated to route reserved values (confidence/weight/subtype)
through dedicated params and strip reserved keys from extractor/customMetadata
bags via the canonical split*MetadataRecord helpers — imports no longer trip
the default throw.
- Tests: new reservedFieldPolicy matrix (throw/warn/remap across every write
path + transact); remap-correctness suite reframed as opt-in 'remap'; shared
test-factory no longer emits reserved keys in custom metadata.
- id-normalization (#18): `brain.newId()` (UUID v7) + v7 default ids; non-UUID string ids are
transparently normalized to a stable UUID v5 on creation AND every lookup — get/update/remove,
relate(from,to), related, find({connected}), getMany, removeMany, the transact op-handler, and
db.with() overlays — with the caller's original key preserved under `_originalId`. The engine only
ever sees a UUID; real UUIDs pass through untouched. universal/uuid.ts gains v5/v7/isUUID +
BRAINY_ID_NAMESPACE; new utils/idNormalization.ts (coerceNewEntityId / resolveEntityId).
- aggregation min/max delete-safety: `queryAggregate` no longer leaves a stale min/max after
deleting the current extreme — min/max now track a value multiset and recompute in-memory (no
entity scan; that scan was the prior delete-then-hang a consumer reported). Other metrics were
already exact across deletes. Regression test proves resolve-after-delete + correct new min/max.
- release.sh RC-safe: explicit version arg, prerelease detection (npm `--tag rc` + GitHub
`--prerelease`), full `test:ci` gate (unit + integration), current-branch push, npm-access
verification after publish.
- native-engine references point at `@soulcraft/cor` (8.0's partner) in user-facing strings/docs.
Unit 1461/0 · integration 599/0 · tsc clean.
8.0 RC cleanup toward "one place per thing, zero-config, no deprecation":
- Remove the `brain.neural()` clustering namespace (ImprovedNeuralAPI + the dead
legacy NeuralAPI + the neural CLI + neural-only types). Similarity is `find({vector})`
/ `similar({to})`; attribute grouping is the aggregation `GROUP BY` engine. The separate
entity-extraction / smart-import feature (NeuralImport, NeuralEntityExtractor, SmartExtractor,
NaturalLanguageProcessor, `brain.extract()`/`brain.nlp()`) is kept.
- Remove `Db.search()`; `find()` is the one query verb (accepts a bare string or FindParams).
Fix the bundled MCP client, which called a non-existent `brain.search(query, limit)` →
now `find({ query, limit })`.
- Storage config: collapse to one canonical top-level `path` key. The pre-8.0 aliases
(`rootDirectory`, `options.*`, `fileSystemStorage.*`) are removed and now THROW with the
exact rename instead of silently defaulting to `./brainy-data` on upgrade. A single resolver
feeds createStorage, the 7.x→8.0 migration probe, and the plugin-factory handoff, so a native
storage provider resolves the identical root (no split-brain).
- Fix `similar({ threshold })`: the min-similarity filter was silently dropped; it is now
applied as a post-filter on `result.score` (the documented way to bound semantic results).
- Fix `vfs.rename()` on a directory: child path updates spread the entity vector into `update()`
and failed dimension validation; they are metadata-only updates now.
- Fix `vfs.move()`: copy+delete orphaned the content-addressed content blob (the destination
shared the source hash, then unlink removed it). `move()` now delegates to `rename()` — an
in-place path change that preserves the blob and the entity id, for files and directories.
- Fix streaming import: the bulk fast path never flushed mid-import nor signalled queryability.
Entity writes are now chunked by a progressive flush interval (100 → 1000 → 5000); each chunk
flushes and emits `progress.queryable`, so imported data is queryable during the import.
- Sweep all docs, comments, and JSDoc for the removed/changed APIs.
Integration suite: 49 files / 588 passed / 0 failed. Unit: 80 files / 1456 passed, no type errors.
Adds a reserved, top-level `visibility` field (mirrors the subtype rollout):
'public' (default, surfaced) | 'internal' (a consumer's app-internal data — hidden
from default find()/getRelations()/counts/stats, opt-in via includeInternal) |
'system' (Brainy plumbing, library-set only; the add()/relate() param narrows to
'public' | 'internal').
Fixes a real leak: the VFS root entity counted in getNounCount() and appeared in
find() (a fresh brain reported 1 entity). It is now visibility:'system' → excluded
from every user-facing surface (fresh brain reports 0).
- Reserved (STANDARD_ENTITY_FIELDS / STANDARD_VERB_FIELDS) — surfaced top-level on
reads, never in the custom metadata bag; a 'public'/'internal' value smuggled
through metadata is lifted to the field, 'system' dropped with a one-shot warning.
- Threaded through add/relate/update/updateRelation + their internal mirrors;
stored only when not 'public' so the common case stays lean.
- Default exclusion in counts (baseStorage, isCountedVisibility), find()/getRelations()
(hard candidate filter via excludeVisibility — keeps top-K/limit/offset correct),
and rebuildCounts; includeInternal/includeSystem opt-ins.
- VFS root marked 'system'.
Same on-disk shape as the 8.0 line, so it carries forward unchanged. Backward-compatible:
absent === 'public', so all existing data stays counted + returned.
Tests: tests/unit/brainy/visibility.test.ts 17/17 (fresh-brain getNounCount()===0,
internal hidden + opt-in, verb symmetry, top-level surfacing, metadata-spoof rejection).
1522 unit green; count-synchronization integration green.
7.x stored every object branch-scoped under branches/<branch>/<basePath>; 8.0
stores the IDENTICAL entity structure at the storage ROOT plus the generational
_system/_generations layer. GenerationStore.open tolerates a missing manifest
(opens at gen 0), so a naive 8.0 open of a 7.x directory reported ZERO entities
and SILENTLY LOST ALL DATA. The fix is blocked in the normal init order — cor 3.0's
storage-factory legacy guard fires inside setupStorage(), five steps before the
old migration check — so a new phase runs BEFORE setupStorage().
legacyLayoutMigrationPhase() (init, between loadPlugins and setupStorage):
- Fast-path: returns immediately when there's no top-level branches/ dir (every
native 8.0 brain + fresh dir pays only one fs.existsSync on the init hot path).
- Runs on a temporary BUILT-IN FileSystemStorage (never the plugin adapter, whose
guard would throw). Memory/cloud/pre-built-adapter stores are a strict no-op.
- Detect via _system/migration-layout.json marker + branches/<head>/entities/.
- autoMigrate:false on a legacy layout THROWS explicit guidance (no silent loss).
- Acquire the writer lock, then COLLAPSE branches/<head>/entities/* → entities/*
via the .gz-transparent raw primitives (read→write→delete, idempotent/resume-safe;
NOT fs.rename — logical paths + memory-adapter incompatibility).
- Rebuild persisted counts (rebuildCounts → totalNounCount/counts.json;
rebuildTypeCounts/rebuildSubtypeCounts → per-type stats) — the three indexes are
rebuilt by the normal init's rebuildIndexesIfNeeded afterward.
- Finalize: stamp the flat-v8 marker (re-open no-op), drain the head branch
(non-head branches = 7.x version history 8.0's MVCC does not import; left as-is).
Tests (tests/integration/migration-7x-to-8x.test.ts): the data-loss LOCK (a
built-in FileSystemStorage on a reshaped 7.x dir reports 0 nouns), a byte-equal
ROUND TRIP (find/related/counts equal the pre-reshape reference), IDEMPOTENCY,
the autoMigrate:false GUARD throw, and the native-flat NO-OP. RELEASES upgrade
checklist rewritten (the old note documented the opposite, lossy behavior).
1477 unit + migration 5 + db-mvcc 25 green; no init-path regression.
Follow-ups (hardening, not correctness for the common case): backupTo config
plumbing (currently a loud warning), a crash-mid-collapse resume test, and a
boundary-safe fake-plugin ordering test.
asOf() answers "state AT a point"; these answer "what happened BETWEEN two
points" and "one entity's whole history", all on the existing generation records.
- Extract resolveAsOfGeneration() as the shared gen|Date→{generation,timestamp}
chokepoint (reachability + Date semantics identical everywhere). asOf()'s
inclusive path is byte-identical — db-mvcc still 25/25.
- asOf(target, { exclusive }) — strict-before (resolved−1, clamped at gen 0,
never a RangeError).
- db.since(Db | generation | Date) — overload; number/Date resolve via the shared
resolver (new DbHost.resolveGeneration); EXCLUSIVE lower bound;
since(db) === since(db.generation). Same-store guard via Db.belongsToStore.
- brain.diff(a, b) → { added, removed, modified } split by nouns/verbs. EARNS its
name: candidate set is ONLY changedBetween(gLow,gHigh), each classified by
existence at both endpoints + a key-order-insensitive value compare
(new src/db/stableEqual.ts) — a touched-but-reverted / born-and-died id is in
no bucket.
- brain.history(id, { from, to }) → every distinct version oldest→newest, each
value === asOf(version.generation).get(id); null = removal; kind auto-detected
(throws on UUID-space collision). New store helper generationsTouching().
- brain.transactionLog({ from, to, limit }) — INCLUSIVE generation/Date window
(contrast since's exclusive lower bound); limit applied last.
- Compaction policy locked: diff/since THROW GenerationCompactedError below the
horizon; history TRUNCATES to it.
Types DiffResult/HistoryVersion/EntityHistory exported from the package root.
Tests: tests/integration/db-temporal.test.ts (8 proofs incl. diff-earns-its-name,
history↔asOf cross-check, the composition proof, granularity, compaction contrast)
+ tests/unit/db/stableEqual.test.ts (6). docs/guides/snapshots-and-time-travel.md
+ RELEASES updated. 1477 unit + db-temporal 8 + db-mvcc 25 green.
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.
Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.
The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.
1471 unit green; build clean; db-portable-graph.test.ts 17/17.
Parity with the 8.0 rename, backported to the 7.x line. The brain.data()
export()/import() document type was BackupData, but it is the portable, versioned
interchange representation of a graph (entities + relations + optional vectors),
NOT a backup — exported for transport between instances, versions, and products.
The actual backup is the native snapshot, so "Backup*" mis-signalled.
Rename every developer-visible symbol, JSDoc, comment and doc:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
BackupRelation→PortableGraphRelation, BACKUP_FORMAT[_VERSION]→
PORTABLE_GRAPH_FORMAT[_VERSION], internal toBackup* helpers→toPortableGraph*.
- src/api/DataAPI.ts, src/index.ts, src/cli/commands/core.ts, docs, and the test
(renamed data-backup.test.ts → data-portable-graph.test.ts).
The on-the-wire `format` tag is also renamed 'brainy-backup' → 'brainy-portable-graph':
the export/import format was introduced in 7.32.0 and has not been adopted by any
consumer, so there are no stored documents to stay compatible with — a clean rename
beats carrying a legacy tag forward. No deprecated aliases (nothing to alias).
1505 unit green; build clean; data-portable-graph.test.ts 20/20.
Two independent restart/multi-instance correctness bugs.
VFS path cache (A.3): PathResolver keyed the PROCESS-GLOBAL path cache on
`vfs:path:<path>` with no instance scope. Multiple Brainys per process (a
supported pattern) over DIFFERENT storage collided — instance A's `/x → id_A`
satisfied instance B's `stat('/x')` against unrelated storage → stale id →
"Entity not found" (surfaced by metadata-only-comprehensive's VFS stat() once
another VFS test had seeded the cache). Scope every `vfs:path:` key by a
monotonic per-process instance token (the VFS root id is a fixed sentinel, so it
can't disambiguate; the global cache is itself process-scoped so the token
suffices). Scoped the invalidate/prefix-delete paths too, so a branch-switch /
clear / fork on one brain no longer wipes another brain's cache. Cross-instance
sharing was only an optimization and was the collision itself; each instance
keeps its own local pathCache.
Verb totalCount (verb mirror of b2005ff): getVerbsWithPagination returned
`collectedVerbs.length` (the peeked page size, bounded by offset+limit+1) as
totalCount, so getVerbs({limit:1}).totalCount read 1/2 for any non-empty brain
on the unfiltered path. Now returns the authoritative O(1) totalVerbCount
(isNew-gated, visibility-filtered, rehydrated on init); filtered scans keep the
collected length. Regression: tests/unit/storage/getVerbs-totalCount.test.ts
(warm + cold reopen, mirrors the noun test).
metadata-only-comprehensive + vfs-api-wiring integration green; 1471 unit green.
addToIndex builds a fieldsMap from the extracted {field,value} entries, but
only __words__ accumulated — every other repeated field did fieldsMap[field]=value,
so a multi-valued field (tags:['a','b','c'] → three 'tags' entries from
extractIndexableFields) collapsed to last-value-wins. columnStore.addEntity
expands an array to one indexed entry per element, but it only ever received the
final scalar, so `contains` matched only the last element and missed the rest.
Accumulate any repeated non-__words__ field into an array before addEntity
(promote scalar→array on the second occurrence). __words__ keeps its always-array
special-case so its multiValue manifest flag stays set even for single-word docs.
find-unified-integration.test.ts → 0 failures (was 4):
- 'array contains' (A.4): now queries first/middle/last element — all match
(previously only the last-indexed element did).
- 'complete find workflow': $contains→contains (8.0 operator) AND fixed the test
premise — find() hard-ANDs vector∩graph∩metadata, and connected:{from:X} returns
X's NEIGHBOURS not X, so the graph anchor must be 'earth' (which the ML concept
relates to), not the concept itself. Multi-signal scores are reciprocal-rank
fusion (~1/61), not [0,1] cosine, so assert a positive fusion score, not >0.5.
- 'nonsense vector query': cosine search always returns nearest neighbours, so
assert the structural array/bound contract, not length===0 (a Tier-2 concern).
- 'hard-ANDs signals': a nonexistent connected.from is an empty graph signal that
zeros the intersection — assert length===0 (documented AND semantics), was >0.
1469 unit + full find-unified integration green.
getIdsForRange computed includeMin/includeMax correctly for every operator
(gt→false/true, lt→true/false, between→true/true) but DROPPED them when the
column store served the query — it called columnStore.rangeQuery(field,min,max)
with no flags, and the column-store path was inclusive-only. So whenever a field
lived in the column store, strict lessThan/greaterThan silently behaved as
lte/gte. The sparse-fallback path already forwarded the flags, so this was
column-store-only.
Thread includeMin/includeMax end to end:
- ColumnSegmentCursor: add textbook lowerBound/upperBound helpers and express
binarySearchRange in terms of them. Inclusive/inclusive is byte-identical to
the old impl (lowerBound(lo) == old binarySearchValue(lo).index; upperBound(hi)
== old "first position after hi"). Exclusive lower advances past ALL duplicates
of the boundary value; exclusive upper stops before them.
- ColumnStore.rangeQuery: accept the flags, apply them in the segment cursor AND
the tail-buffer linear scan (strict > / < when exclusive). A bound taken from a
segment's own min/max stays inclusive — it is a real stored value.
- VectorIndex/types interface + getIdsForRange call site updated.
Surfaced by brainy-complete dual-bound test (year/popularity exclusive both ends
→ ['Express','Vue.js']; React@95 and Angular@75 correctly excluded). Added 5
column-store unit tests: exclusive lower, exclusive upper, both, duplicate
boundary values, and the unflushed tail-buffer path. 1469 unit green.
lazyLoadCounts() read the `__sparse_index__noun` blob, but the sparse-index
WRITE path was removed in 7.20.0 — new workspaces persist the 'noun' field
ONLY to the column store. So on close()+reopen the sparse load found nothing
and left every per-type count at 0: counts.byType / byTypeEnum / topTypes /
allNounTypeCounts all read empty, while find() / getNounCount() (different
sources) stayed correct.
Rehydrate from the column store's 'noun' field instead. Its per-value
cardinality matches the warm updateTypeFieldAffinity counts exactly because
both are driven from the same addToIndex field set, in lockstep, with no
visibility gate on either — so syncTypeCountsToFixed (called right after in
init) reproduces the warm fixed-array values precisely. Legacy chunked sparse
index kept as a fallback for pre-7.20.0 workspaces.
Ground-truth verified: 12 Person + 5 Document → cold reopen now reports
person:12 / document:5 (was 0), topTypes [person, document, collection].
Tests: un-skipped the intentionally-failing phase1c "warm cache on init"
reopen test and strengthened it to exact persisted counts; added a warm==cold
element-for-element equality test. 1464 unit + count-sync/multi-process/
clear-persistence integration green.
A wrong or missing accelerator (@soulcraft/cor) used to silently degrade to the
default JS engine — the #1 source of invisible cross-repo drift. Brainy does no
plugin auto-detection (a registered plugin is always explicitly requested via
config.plugins or brain.use()), so any mismatch/failure is now fatal:
- BrainyPlugin gains optional `brainyRange` (semver range of brainy it supports);
init() throws if the running brainy is outside it. New dep-free, prerelease-safe
matcher pluginRangeSatisfies() (8.0.0-rc1 satisfies >=8.0.0).
- activateAll(): a plugin that THROWS during activate now re-throws (was swallowed
to a JS fallback); a graceful decline (activate()→false) stays non-fatal but logs
a loud warning.
- loadPlugins(): an explicitly-listed package that can't load — or isn't a valid
plugin — throws (was a silent skip).
- Provider-key cliff: a pre-8.0 plugin that registers a legacy vector key
('hnsw'/'diskann') but not the 8.0 'vector' key throws (brainy 8.x reads only
'vector', so it would otherwise run JS vectors invisibly).
Tests: tests/unit/plugin-version-coupling.test.ts (range matcher incl. rc1; range
mismatch / activate-throw / cliff / missing-package all throw; decline non-fatal).
Updated two plugin.test.ts cases that asserted the old silent-swallow behavior.
Build green; 1464 unit green. cor declares its brainyRange per handoff LV.1 #2.
The open/closed separation test now forbids the proprietary native package under
BOTH names (@soulcraft/cor + legacy @soulcraft/cortex) — in any dependency field
and any static import under src/ or tests/. Renamed boundary-no-cortex →
boundary-no-native. This is load-bearing for the lockstep model: because brainy
cannot depend on cor, the combined brainy-8.0 × cor-3.0 integration matrix lives
in the cor repo (which devDeps brainy), not here.
nounCountsByType (read by stats().entitiesByType / counts.byTypeEnum) was
incremented in saveNoun_internal(), which runs on every noun write — including
the HNSW index re-saving a node whenever its neighbor links change. So per-type
counts grew with graph connectivity instead of entity count (an internal report
saw 8 documents read as 44). Moved the increment into saveNounMetadata_internal(),
gated on isNew + visibility, exactly parallel to the authoritative total; the
visibility-flip path adjusts it in lockstep too.
Tests: tests/unit/storage/stats-count-accuracy.test.ts (30 dense-type entities ->
exactly 30 across stats/counts/getNounCount) + de-theatricalized the >=2 assertion
in brainy-core.unit.test.ts to exact equality. Build green; 1455 unit green.
Part of the 8.0 readiness audit Phase 1; cold-reopen count rehydration is still WIP.
getNounsWithPagination returned collectedNouns.length as totalCount, but the
type-first shard scan early-terminates at offset+limit+1 (peeks one for hasMore)
— so getNouns({ pagination: { limit: 1 } }).totalCount was the page size for any
non-empty brain. The index-rebuild gate calls exactly that, so cold starts
mis-logged "Small dataset (N items)". Now reports the authoritative O(1) noun
counter (rehydrated on init) as the unfiltered total; 8.0's peek-based hasMore is
already correct and unchanged. Filtered scans unchanged.
Regression: tests/unit/storage/getNouns-totalCount.test.ts. Full 8.0 unit suite green (1453).
getNounsWithPagination returned collectedNouns.length as totalCount, but the
type-first shard scan early-terminates at offset+limit — so
getNouns({ pagination: { limit: 1 } }).totalCount was 1 for any non-empty brain.
The index-rebuild gate calls exactly that, so cold starts logged
"Small dataset (1 items) - rebuilding all indexes" and rebuilt from scratch
regardless of corpus size (a production deployment saw this for an ~8,800-entity
brain). Now reports the authoritative O(1) noun counter (maintained on add/delete,
rehydrated from counts.json on init) as the unfiltered total and derives hasMore
from it. Filtered scans unchanged. Layout-independent (branch/COW included).
Also downgrade the "mmap-vector backend not wired" console.log to prodLog.debug:
it is benign in the native-vector-index model (the native provider owns its own
vector storage and has no setVectorBackend hook), but it fired on every init and
was repeatedly mistaken for the cold-start cause.
Regression: tests/unit/storage/getNouns-totalCount.test.ts. Full unit suite green (1505).
Three of the five bugs the rot pass surfaced (correct inputs → wrong output), fixed:
1. where-filter dropped all but the last operator on a field. getIdsForFilter()'s
per-operator loop overwrote fieldResults each iteration, so
{ year: { greaterThan: 2009, lessThan: 2020 } } kept only lessThan. Now each
operator's match set is AND-intersected (multi-operator-per-field works).
(metadataIndex.ts)
2. related({ offset }) ignored the offset — getVerbs() zeroed it and smuggled it via
an unimplemented cursor, so every page returned items [0, limit). Now the real
offset is passed through to getVerbsWithPagination (which slices [offset,+limit)).
(baseStorage.ts) — verified: get-relations-fix pagination test passes.
3. relate() never persisted updatedAt, so reads fabricated a fresh Date.now() each
call (non-idempotent). Now createdAt and updatedAt share one timestamp at create.
(brainy.ts) — verified: get-relations-fix equivalence test passes.
Follow-ups (precisely diagnosed, not papered over): exclusive range bounds —
ColumnStore.rangeQuery(field,min,max) is inclusive-only, so getIdsForRange() drops
includeMin/includeMax (metadataIndex.ts:996) and lessThan/greaterThan behave as
lte/gte when the column store is active (the dual-bound test's popularity:95 boundary
stays red); counts not rehydrating after restart; unscoped VFS path-cache.
Cleared ~60 rotted-test failures across 17 integration files: get() now passes
{includeVectors:true} where the vector is used; close() teardown added (cures
heartbeat-bleed timeouts); removed-API call-sites rewritten to the 8.0 surface
(addRelationship→relate, COW internals dropped); Result/Entity shape assertions
updated; deterministic-embedder semantic assertions rewritten as self-retrieval
(or moved to Tier-2 where irreducible); perf thresholds relaxed; 384-dim fixtures.
Adds the Tier-2 (real-model) scaffolding: tests/setup-semantic.ts +
tests/configs/vitest.semantic.config.ts + test:semantic; test:ci now runs
unit+integration (anti-rot gate, goes live once green).
Remaining 17 failures are REAL 8.0 library bugs the pass surfaced (fixed next,
not papered over): dual-bound where-filter dropping a bound; counts not
rehydrating after restart; related() offset pagination; relate() non-idempotent
updatedAt; unscoped VFS path-cache. Plus find-unified finish + a few stragglers.
distinctCount previously routed through getNumericField, so it silently returned 0 for
string/categorical fields — its primary use case (distinct categories / users / tags).
It now tracks the raw value (any type, keyed by string form) on both the add and remove
contribution paths; numeric ops (sum/avg/min/max/stddev/variance/percentile) are unchanged.
Also adds regression coverage confirming two long-standing query behaviors hold on the 8.0
engine: find({ where: { field: { missing: true } } }) matches a never-registered field, and
find({ type: [...], orderBy, limit }) returns the full set on the first call after
mutate+query+get. (percentile/median were already correct.)
Tests: tests/unit/brainy/find-agg-edge-cases.test.ts + existing aggregation suite green.
- New docs/guides/export-and-import.md (public) for the 8.0 surface:
brain.export()/import(), Db composition (asOf/with time-travel + what-if export),
selectors, options, BackupData v1 format, cross-version (7.x→8.0), VFS, and the
generations/persist distinction. Documents only the implemented surface.
- api/README "Export & Import (portable) + Snapshots (native)": adds the portable
brain.export()/import() round-trip alongside the native persist()/asOf() snapshot.
Re-adds the portable graph round-trip on 8.0 (deleted with DataAPI), now on the
immutable Db so it composes with the generational model.
- src/db/backup.ts: BackupData v1 engine (identical wire format to the 7.32.0 line).
exportGraph() reads through a Db at its pinned generation; importGraph() applies the
whole backup as ONE atomic transact() (a single generation).
- Db.export(selector?, options?) — read at this view's generation, so
brain.asOf(g).export() is a time-travel export and brain.now().with(ops).export() a
what-if export.
- brain.export(selector?, options?) — sugar for now().export().
- brain.import() is polymorphic: a BackupData document -> graph round-trip; a
file/buffer -> existing foreign-file ingestion (dispatched on the 'brainy-backup'
tag; zero migration for ingestion callers).
- Selectors (ids/collection/connected/vfsPath/predicate/whole) + edge policy
(induced/incident/none) + includeVectors/includeContent/includeSystem; system
entities excluded by visibility. onConflict merge/replace/skip, reembed auto/never,
remapIds. DbHost gains storage (VFS blob bytes).
- Types exported from the package root: BackupData/BackupEntity/BackupRelation/
ExportSelector/ExportOptions/ImportOptions/ImportResult + isBackupData.
Tests: tests/unit/db/db-backup.test.ts (10, green) — round-trip, selectors, edges,
vectors+re-embed, merge, asOf/with composition, validation. Full unit suite green.
Follow-ups: docs (8.0 guide + RELEASES + api/README), completeness adds
(since(prior).export() delta, exportStream/importStream, validate()), includeContent
filesystem test, ids-at-asOf get() refinement, @soulcraft/formats Zod schema mirror.
brain.data() now serializes part or all of a brain to a versioned, portable
JSON document (BackupData) and restores it — the path for partial backups,
cross-environment moves, and 7.x→8.0 migration.
- export(selector?, options?): select by ids, collection (+transitive Contains),
connected neighbourhood, vfsPath subtree, predicate, or whole brain; structural
and predicate selectors compose. Options: includeVectors, includeContent (VFS
blobs), includeSystem, edges ('induced'|'incident'|'none').
- import(backup, options?): onConflict 'merge' (dedup-by-id) | 'replace' | 'skip';
reembed 'auto' (re-embed from data when no vector carried) | 'never'; remapIds
to clone a subgraph under fresh ids.
- BackupData v1: format/formatVersion/brainyVersion/createdAt/embedding/entities/
relations/blobs?/danglingIds?/stats. Reserved fields (subtype, data, confidence,
weight, service) top-level; metadata custom-only. Current-state, no generations.
- Replaces the prior flat-entity export() (dropped relations) with a graph-complete
document. Distinct from brain.import(file) ingestion, which is unchanged.
- Export BackupData/BackupEntity/BackupRelation/ExportSelector/ExportOptions/
ImportOptions/ImportResult/DataAPI from the package root. CLI `brainy export`
now writes a BackupData document.
Guide: docs/guides/backup-and-export.md. Tests: tests/unit/api/data-backup.test.ts.
Adds a reserved, top-level `visibility` field (mirrors the subtype rollout):
'public' (default, surfaced) | 'internal' (developer app-internal — hidden from
default find/count/stats, opt-in via includeInternal) | 'system' (Brainy
plumbing, library-set only).
Fixes a real leak: the VFS root entity counted in getNounCount() and appeared in
find() (a fresh brain reported 1 entity). It is now visibility:'system' →
excluded from every user-facing surface. Developers also get a first-class
hidden-unless-asked tier (e.g. learned internals vs user-exposed data).
- Reserved (RESERVED_ENTITY_FIELDS / RESERVED_RELATION_FIELDS) — spoof-proof from metadata.
- Threaded through add/relate/update/transact; surfaced top-level on reads.
- Default exclusion in counts (baseStorage), find()/related() (hard candidate
filter via excludeVisibility — keeps topK/limit correct), and stats;
includeInternal/includeSystem opt-ins.
- VFS root marked 'system'.
Tests: visibility.test.ts 17/17 (fresh-brain getNounCount()===0, internal hidden
+ opt-in, verb symmetry, top-level surfacing, metadata-spoof rejection). Unit
1431 green; count-synchronization integration now passes (off-by-one fixed).
