Commit graph

64 commits

Author SHA1 Message Date
c0c68ac6a6 docs: RELEASES.md entry for 8.3.1 (full-removal deletes + family-scoped gate) 2026-07-14 10:12:00 -07:00
7692c6f4ef docs: RELEASES.md entry for 8.3.0 (heal-cost + ADR-004 Pass 2/3)
Consumer summary of the 8.3.0 minor: 16-way parallel canonical enumeration +
id-only getNounIdsWithPagination (index-heal speedup, standalone); the ADR-004
cross-layer integrity contract (validateInvariants delegation + repairIndex
native rebuild); and the registered-blob family contract (declared index blobs
undeletable). The two contract pieces are inert until a native provider
implements the matching hooks.
2026-07-13 15:18:09 -07:00
d0f69c731f fix: honest index readiness — no silently-empty queries on a cold index
Closes the "dishonest readiness proxy" anti-pattern (Pattern A): size()>0 /
isInitialized were treated as "this index serves queries", but a cold native
index can load its COUNT before its SERVING structure, so a query returned a
silent [] indistinguishable from "no such data". A shared assessIndexReadiness()
now reads only the provider's honest isReady() signal (never size()), applied at
every site:

- Vector: a one-shot verifyVectorLive() guard on the semantic/proximity search
  path (a pure semantic find({query}) has no filter, so nothing guarded it). It
  prefers isReady(), else a known-vector self-match probe; self-heals via rebuild
  or throws the new VectorIndexNotReadyError instead of a silent [].
- Graph: getVerbsBySource/ByTarget skip the fast path when the provider reports
  not-ready (falling to the canonical shard scan), plus a one-shot probe that
  self-heals a no-isReady provider whose adjacency did not cold-load.
- getIndexStatus(): folds in per-index honest `ready` (making `populated`
  honest) + rebuildFailed/rebuildError/degradedIds, so a readiness probe never
  200s a brain that is still warming up or degraded.

Unblocked by the native providers now reporting serving-truth (graph via
SSTable-residency readiness, vector via durableBaseLoadFailed). New export:
VectorIndexNotReadyError. getIndexStatus gains additive fields. No breaking API.
13 new tests; existing readiness guards green.
2026-07-13 13:17:56 -07:00
b6c7039769 fix: restore loadBinaryBlob fault-propagation (native column-store lockstep)
The Pass-1 hardening that makes loadBinaryBlob distinguish genuine absence
(ENOENT -> null) from a real fault (EIO/EACCES/... -> throw) was held out of
8.2.6 because the native accelerator's two column-store read sites still relied
on null-on-error. That accelerator release has now hardened those sites to
handle the throw (a faulted segment read marks the field unavailable and throws
a named error), so the fault-propagating form is restored and ships lockstep.

A present-but-unreadable index blob no longer masquerades as absent (which drove
needless rebuilds / empty reads). Adds the loadBinaryBlob leg to the blob
durability suite (absent -> null; present -> bytes; real fault -> throws).
2026-07-13 12:09:55 -07:00
a873852e61 docs: RELEASES.md entry for 8.2.6 (write/index-spine hardening)
Consumer-facing summary of the Pass-1 durability + integrity release: durable
blob-write honesty, single-op history durability (PendingFlushDurabilityError),
degraded-index surfacing, full-footprint clear(), count symmetry, and loud
index-maintenance / aggregation failures. loadBinaryBlob fault-propagation is
held for the native lockstep and deliberately omitted here.
2026-07-13 10:45:39 -07:00
a7c7aa5102 docs: RELEASES.md entry for 8.2.5 (honest rollback-failure response) 2026-07-12 12:19:09 -07:00
457469593a docs: RELEASES.md entry for 8.2.4 (non-destructive restore) 2026-07-12 09:10:35 -07:00
be5ce0bcc3 docs: RELEASES.md entry for 8.2.3 (transact durability barrier) 2026-07-12 08:54:14 -07:00
ed97006eb9 docs: RELEASES.md entry for 8.2.2 (transaction timeout rollback) 2026-07-11 13:44:15 -07:00
708978210a docs: RELEASES.md entry for 8.2.1 (transact forward-ref parity fix) 2026-07-10 18:06:37 -07:00
98ceadca7c docs: RELEASES.md entry for 8.2.0 (temporal VFS) 2026-07-10 16:43:48 -07:00
4e9be08f44 docs: RELEASES.md entry for 8.1.0 (brain.onChange change feed) 2026-07-10 11:24:32 -07:00
6b8b9cba18 docs: RELEASES.md entry for 8.0.17 (canonical count recovery + dead-machinery sweep) 2026-07-08 15:49:19 -07:00
54e7c0eced docs: RELEASES.md entry for 8.0.16 (atomic ifAbsent/upsert + exact blob refCounts) 2026-07-08 15:13:26 -07:00
b1fe25a339 docs: RELEASES.md entry for 8.0.15 (atomic ifRev CAS) 2026-07-08 14:53:34 -07:00
64188a33d4 docs: RELEASES.md entry for 8.0.14 (migration preserves branch-scoped non-entity state) 2026-07-07 16:07:10 -07:00
38e8de5e48 docs: RELEASES.md entry for 8.0.13 (accurate boot log for established stores) 2026-07-07 15:49:49 -07:00
d9017e7dde docs: RELEASES.md entry for 8.0.12 (7→8 VFS recovery, zero-rebuild cold open, strict query operators) 2026-07-07 12:23:36 -07:00
4fde94bc2f docs: RELEASES.md entry for 8.0.11 (exit-hang class closed for every op shape) 2026-07-02 17:38:34 -07:00
2da2736ac6 docs: RELEASES.md entry for 8.0.10 (clean process exit after close) 2026-07-02 17:04:06 -07:00
588267be7f feat: guarded plugin auto-detection — installing @soulcraft/cor is the opt-in
With plugins unset (the default), init() probes for the first-party
accelerator: not installed means plain brainy with zero noise; installed and
healthy means it loads and announces itself; installed but broken (import
failure, invalid shape, failed activation, version mismatch) makes init()
THROW. An installed accelerator never silently vanishes behind the JS engines
- the anti-drift posture of the explicit list, applied to detection.

plugins: []/false stays a true opt-out (no probe); an explicit list keeps its
required-and-loud semantics. The import runs through an importPluginPackage
seam (variable specifier - bundlers cannot static-resolve the optional
package; tests simulate all outcomes without it installed).

Also: when a plugin activates but registers zero native providers (e.g. a
licensing gate declining to engage), the provider summary now warns loudly
instead of leaving every query silently on the JS engines.

Supersedes 8.0.8's explicit-opt-in wording; README/PLUGINS/types now document
the guarded contract. 7 new tests (tests/unit/plugin-autodetect.test.ts).
2026-07-02 16:19:55 -07:00
e420369850 docs: plugins are explicit opt-in — correct the README scale section and plugins config comment
A clean-room install smoke of the published GA pair caught the README telling
scale-up users the native provider is auto-detected. It is not, by design: the
loader (loadPlugins) does no auto-detection — undefined means no plugins, and
only an explicit plugins: ['@soulcraft/cor'] loads the native engine (loud
failure if it can't). The stale comment on the plugins config field in the
public types claimed auto-detection and is corrected to match the loader; one
phrase in the plugin-author guide likewise.
2026-07-02 15:47:31 -07:00
5db2c41f87 docs: GA version is 8.0.7 — npm retired 8.0.0-8.0.6 (January dev-cycle unpublishes) 2026-07-02 15:18:57 -07:00
e44620e91d docs: flagship README for the 8.0 GA; GA version is 8.0.1
Rewrites the README around the feature-showcase structure: write->index->query
table, runnable quick start (subtype-correct on the 8.0 strict default, real
VerbTypes), feature tour with per-pillar code, the scale-up path to
@soulcraft/cor, and measured-only performance claims with benchmark citations.

GA retargets to 8.0.1: npm permanently retired the 8.0.0 version number after
a January development-cycle publish/unpublish, so the first stable 8.x release
is 8.0.1. RELEASES.md documents this; the phantom 8.0.0 CHANGELOG entry is
removed (the release run regenerates it as 8.0.1).
2026-07-02 15:11:41 -07:00
4584d0b8b8 docs: RELEASES.md 8.0.0 GA entry (RC notes become history) 2026-07-02 14:41:24 -07:00
3a33987136 docs(8.0): RELEASES.md — rc.9 (migration LOCK + 6x cosine + ES2023/Node22 floor) 2026-07-01 11:52:52 -07:00
5af48a925c docs(8.0): RELEASES.md — rc.8 (no-freeze online whole-brain auto-upgrade) 2026-06-30 13:38:34 -07:00
1ddc786c22 docs(8.0): RELEASES.md — rc.7 (cold-graph self-heal + billion-scale RAM + version handshake) 2026-06-30 10:31:36 -07:00
6daa70ed6a docs(8.0): RELEASES.md — rc.6 (perf + native-provider contract + test hygiene)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 12:11:56 -07:00
6c9a43816c docs(8.0): RELEASES.md — rc.5 hardening + the breaking operator removal
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-29 10:30:44 -07:00
1c363e8c4b feat(8.0): asOf at-gen vector defer — provider-served historical semantic search (#35)
A filtered semantic read at a historical generation (db.asOf(g).find({ query, where }))
no longer unconditionally rebuilds an ephemeral JS-HNSW over every at-g vector
(O(n@G), the OOM ceiling the 2026-06-24 scaling audit flagged). When the vector
index is a VersionedIndexProvider that advertises isGenerationVisible(g), Brainy:
  1. resolves the at-g metadata∩graph universe from the record-overlay path (no
     materialization — it's the metadata-only historical find),
  2. routes the vector leg to the provider with { allowedIds, generation }, and
  3. composes the at-g entities ranked by the provider's at-g vector distance.
Without a versioned provider (the JS index, or a native one that refuses the
generation) it falls through to the existing materialization — unchanged.

- Seam (A): VectorIndexProvider.search gains an optional `generation?: bigint`
  ("omitted = now"), the vector mirror of the graph provider's trailing-gen + the
  #46 allowedIds field. JsHnswVectorIndex accepts and ignores it (no per-gen
  segments → refuse/fall-back posture; Brainy never routes a historical read there).
- Gate: Db.find tries the native at-gen vector path before materialize(); host
  exposes canServeVectorAtGeneration + vectorSearchAtGeneration.
- generation is Brainy's u64 commit counter — the same value handed to the graph
  index on writes (graphWriteGeneration), so it maps 1:1 to the native side.

Decided lockstep with the cor team (handoff #35 thread: (A) + vector-only). The
gen-g candidate-vector supply for cor's exact-rerank (part-3) is a separate,
non-blocking seam still being confirmed; the honesty guard keeps this path inactive
(falls through to materialize) until cor's native at-gen rerank is live.

Tested: provider routing + at-gen universe correctness (excludes future-born,
applies the filter) + page window via a mock versioned provider; seam-ignore on the
JS index; 38 db-mvcc/db-temporal historical-read tests green (materialize fallback).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 15:52:09 -07:00
450084b6ce perf(8.0): bound find({ where, orderBy }) sort to the page (CTX-BR-FIND-ORDERBY)
A filtered + ordered find() materialized the FULL sorted match set before slicing
the page — passing k = filteredIds.length to the column store's filteredSortTopK.
A broad filter + orderBy returning 20 rows at billion scale produced hundreds of
millions of sorted ids to discard all but the page (O(matches) heap).

Thread the page bound (offset + limit + the hidden-tier over-fetch) into
getSortedIdsForFilter → filteredSortTopK / sortTopK / the sparse-path slice, so
the sort produces only the page. Ordering and pagination are unchanged.

From cor's 2026-06-24 co-release scaling audit (item 1).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 15:01:49 -07:00
82dde92077 feat(8.0): brain.graph.subgraph(query) query→expand fusion (#61)
The subgraph seed selector now accepts not just entity id(s) but a find() result
or a FindParams query — brain.graph.subgraph({ where: { team: 'platform' } },
{ depth: 1 }) runs the query and expands the neighborhood of every match in one
call. Existing id-seeded calls are unchanged.

The win is the native query→expand path: for a metadata-only query, the matched
universe is forwarded to traverse() as an OpaqueIdSet (a roaring Buffer) with NO
id materialization in TypeScript — the find() result never leaves the engine's
representation (the O(1)-crossing the cor 3.0 contract is built around). The
pure-JS path (or any query carrying vector/text/proximity criteria) materializes
the matched ids via find() and seeds the traversal from them, capped at a bounded
default when the caller pins no limit.

- New public `SubgraphSelector<T>` union (id | id[] | Result[] | FindParams).
- find()'s metadata filter-builder extracted to a shared `buildMetadataFilter`
  (reused by the opaque universe producer); behavior unchanged.
- graphSubgraphNative generalized to accept `bigint[] | OpaqueIdSet` seeds.

Tested: JS query→expand + Result[] selector + empty-match in graph-subgraph, and
the native opaque pass-through (Buffer reaches traverse unmaterialized) vs the
find()-materialized bigint[] seeds via the mock provider in graph-native-routing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 13:30:30 -07:00
dd325f2f94 feat(8.0): vector allowedIds predicate-pushdown into find() (#46)
Filtered semantic search now keeps its recall. A find({ query, where }) that
pairs vector search with a metadata filter restricts the HNSW beam walk to the
matching candidates INSIDE the traversal (walk-all, collect-allowed) rather than
filtering the top-k afterward — so a query whose nearest vectors are all filtered
out still returns the best matches that DO pass the filter, instead of empty.

- JS HNSW search() honors the 8.0 `allowedIds: OpaqueIdSet | ReadonlySet<string>`
  contract param (ANDs with candidateIds; ignores the opaque Buffer form it can't
  decode — only a native provider consumes that).
- MetadataIndexProvider gains an OPTIONAL `getIdSetForFilter(filter): OpaqueIdSet`
  producer — the native (cor) metadata index returns its roaring filter result as
  a serialized buffer; the JS index does not implement it.
- find() forwards the matched universe to the vector walk: the opaque buffer
  (zero id materialization) when the native producer is present, alongside the
  materialized visibility-precise candidateIds for the JS index. Visibility stays
  correct via the existing post-search hard filter.

Tested: JS recall/restriction/AND/opaque-ignore on the index directly, plus
find() forwarding the opaque universe through to the beam walk via a stubbed
native producer.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 13:18:34 -07:00
632d90aac5 feat(8.0): graph analytics — brain.graph.rank / communities / path
Adds three intent-level graph reads to the `brain.graph` namespace, each
native-dispatched to the optional `@soulcraft/cor` 3.0 graph engine when present
and served from pure-TS kernels otherwise (identical public shapes, default
visibility filter respected on both paths):

- `rank(opts?)` → `{ id, score }[]` descending — importance / centrality.
  TS fallback: PageRank power-iteration with dangling-mass redistribution.
- `communities(opts?)` → `{ groups, count }` — connected grouping. TS fallback:
  union-find weakly-connected components, or iterative Tarjan SCC when
  `{ directed: true }`.
- `path(from, to, opts?)` → `{ nodes, relationships, cost } | null` — best route.
  TS fallback: BFS for fewest hops, Dijkstra (min-heap) for least summed edge
  weight (`by: 'weight'`); on-demand frontier expansion so short paths terminate
  early. `direction` / `type` / `maxDepth` filters apply.

These are intent contracts, not algorithm contracts — the question is the
promise, the algorithm is the engine's choice.

Pure kernels live in src/graph/analyticsFallback.ts (PageRank, connected
components, Tarjan SCC, MinHeap) — unit-tested in isolation. The full surface is
tested end-to-end through the TS fallback, and the native dispatch + int↔uuid
hydration paths are covered by a mock provider in graph-native-routing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-23 12:32:03 -07:00
5c3bb2c864 feat(8.0): Model-B per-write generation-stamping + adaptive retention knob
Every write — transact() AND single-op add/update/remove/relate — is now its
own immutable generation (Model-B), so a now() pin always freezes and
asOf/since/diff/history/transactionLog reflect single-ops exactly like
transacts. Closes the Model-A hole where pins did not freeze against single-op
writes.

Generation-stamping:
- GenerationStore.commitSingleOp: a one-operation commitTransaction with
  deferred durability. Wired into add/update/remove/relate/updateRelation/
  unrelate + removeMany (the *Many and VFS paths delegate to these).
- Async group-commit (flushPendingSingleOps): the live write is acknowledged
  immediately; its before-image is buffered in an in-memory pending tier that
  resolveAt/chains/changedBetween/tx-log read like on-disk generations, so the
  synchronous now() freezes with no forced flush. One fsync per window
  (triggers: size / 50ms timer / flush / close / transact / compactHistory).
- Crash recovery is drop-without-restore for group-commit generations (marked
  groupCommit:true): a crash mid-flush discards the partial generation and
  never restores its before-images, which would otherwise revert the
  already-acknowledged live write.
- Init-time infrastructure (the VFS root) is the un-versioned generation-0
  baseline: a fresh brain reports generation()===0 and an empty
  transactionLog(); the first user write is generation 1.
- Historical find()/related() overlay bound is the full reserved watermark
  (generation()), so un-flushed single-op writes are overlaid too.

Retention knob:
- config `history` -> `retention`: 'all' | 'adaptive' |
  { maxGenerations?, maxAge?, maxBytes?, budgetBytes?, autoCompact? }; unset ->
  adaptive (disk/RAM pressure, zero-config). CompactHistoryOptions floors ->
  caps (retainGenerations->maxGenerations, retainMs->maxAge, +maxBytes):
  reclaim oldest-unpinned while ANY cap is exceeded; pins always exempt.
- brain.setRetentionBudget(bytes) drives the adaptive byte budget at runtime
  (a coordinator's fair-share input). Per-generation bytes recorded in each
  delta enable historyBytes() introspection without a storage size API.

Tests: per-write generation resolution, pin freeze vs add/update/remove,
drop-without-restore corruption-trap (fault injector), clean-reopen replay,
maxBytes/maxAge/no-cap reclamation, retention-then-reopen. 107 db/generation/
temporal tests green, tsc clean. Docs (ADR-001, consistency-model, snapshots
guide, api reference, RELEASES) updated to per-write granularity + retention.
2026-06-22 15:19:58 -07:00
96d9c0b92c docs(8.0): RELEASES — native provider is @soulcraft/cor 3.0 (fix cortex 3.0 self-contradiction)
The 8.0 entry referenced a non-existent '@soulcraft/cortex 3.0' (line 447) while
the graph section already said '@soulcraft/cor 3.0' (line 44) — a self-contradiction.
The native provider was renamed cortex→cor; the 8.0 pairing is @soulcraft/cor 3.0
(the renamed successor to @soulcraft/cortex 2.x). Fixed the 8.0-section refs (the
Cor-compatibility block + the scale-path note); HISTORICAL 2.x refs in the v7.31.2 /
older entries are left intact (renaming them would falsify the changelog).
2026-06-22 09:39:14 -07:00
18f27cb16e docs(8.0): RELEASES rc.2 additions — graph engine + additive wins + correctness fixes 2026-06-21 10:33:02 -07:00
1bc709d31b docs(8.0): note reserved-field default-throw in RELEASES rc additions 2026-06-20 15:38:06 -07:00
ae3fe82fd9 docs: mark 8.0.0-rc.1 published (npm tag rc) + note rc.1 additions 2026-06-20 14:55:16 -07:00
0c4a51c24e feat(8.0): 7.x→8.0 layout migration — fix silent total data loss on first open
7.x stored every object branch-scoped under branches/<branch>/<basePath>; 8.0
stores the IDENTICAL entity structure at the storage ROOT plus the generational
_system/_generations layer. GenerationStore.open tolerates a missing manifest
(opens at gen 0), so a naive 8.0 open of a 7.x directory reported ZERO entities
and SILENTLY LOST ALL DATA. The fix is blocked in the normal init order — cor 3.0's
storage-factory legacy guard fires inside setupStorage(), five steps before the
old migration check — so a new phase runs BEFORE setupStorage().

legacyLayoutMigrationPhase() (init, between loadPlugins and setupStorage):
- Fast-path: returns immediately when there's no top-level branches/ dir (every
  native 8.0 brain + fresh dir pays only one fs.existsSync on the init hot path).
- Runs on a temporary BUILT-IN FileSystemStorage (never the plugin adapter, whose
  guard would throw). Memory/cloud/pre-built-adapter stores are a strict no-op.
- Detect via _system/migration-layout.json marker + branches/<head>/entities/.
- autoMigrate:false on a legacy layout THROWS explicit guidance (no silent loss).
- Acquire the writer lock, then COLLAPSE branches/<head>/entities/* → entities/*
  via the .gz-transparent raw primitives (read→write→delete, idempotent/resume-safe;
  NOT fs.rename — logical paths + memory-adapter incompatibility).
- Rebuild persisted counts (rebuildCounts → totalNounCount/counts.json;
  rebuildTypeCounts/rebuildSubtypeCounts → per-type stats) — the three indexes are
  rebuilt by the normal init's rebuildIndexesIfNeeded afterward.
- Finalize: stamp the flat-v8 marker (re-open no-op), drain the head branch
  (non-head branches = 7.x version history 8.0's MVCC does not import; left as-is).

Tests (tests/integration/migration-7x-to-8x.test.ts): the data-loss LOCK (a
built-in FileSystemStorage on a reshaped 7.x dir reports 0 nouns), a byte-equal
ROUND TRIP (find/related/counts equal the pre-reshape reference), IDEMPOTENCY,
the autoMigrate:false GUARD throw, and the native-flat NO-OP. RELEASES upgrade
checklist rewritten (the old note documented the opposite, lossy behavior).

1477 unit + migration 5 + db-mvcc 25 green; no init-path regression.

Follow-ups (hardening, not correctness for the common case): backupTo config
plumbing (currently a loud warning), a crash-mid-collapse resume test, and a
boundary-safe fake-plugin ordering test.
2026-06-19 13:44:03 -07:00
2c84f86815 feat(8.0): temporal range verbs — diff, history, since(gen|Date), asOf{exclusive}, transactionLog window
asOf() answers "state AT a point"; these answer "what happened BETWEEN two
points" and "one entity's whole history", all on the existing generation records.

- Extract resolveAsOfGeneration() as the shared gen|Date→{generation,timestamp}
  chokepoint (reachability + Date semantics identical everywhere). asOf()'s
  inclusive path is byte-identical — db-mvcc still 25/25.
- asOf(target, { exclusive }) — strict-before (resolved−1, clamped at gen 0,
  never a RangeError).
- db.since(Db | generation | Date) — overload; number/Date resolve via the shared
  resolver (new DbHost.resolveGeneration); EXCLUSIVE lower bound;
  since(db) === since(db.generation). Same-store guard via Db.belongsToStore.
- brain.diff(a, b) → { added, removed, modified } split by nouns/verbs. EARNS its
  name: candidate set is ONLY changedBetween(gLow,gHigh), each classified by
  existence at both endpoints + a key-order-insensitive value compare
  (new src/db/stableEqual.ts) — a touched-but-reverted / born-and-died id is in
  no bucket.
- brain.history(id, { from, to }) → every distinct version oldest→newest, each
  value === asOf(version.generation).get(id); null = removal; kind auto-detected
  (throws on UUID-space collision). New store helper generationsTouching().
- brain.transactionLog({ from, to, limit }) — INCLUSIVE generation/Date window
  (contrast since's exclusive lower bound); limit applied last.
- Compaction policy locked: diff/since THROW GenerationCompactedError below the
  horizon; history TRUNCATES to it.

Types DiffResult/HistoryVersion/EntityHistory exported from the package root.
Tests: tests/integration/db-temporal.test.ts (8 proofs incl. diff-earns-its-name,
history↔asOf cross-check, the composition proof, granularity, compaction contrast)
+ tests/unit/db/stableEqual.test.ts (6). docs/guides/snapshots-and-time-travel.md
+ RELEASES updated. 1477 unit + db-temporal 8 + db-mvcc 25 green.
2026-06-19 13:21:02 -07:00
373a48122d refactor(8.0): rename BackupData → PortableGraph (the type is interchange, not a backup)
The export()/import() document type was named BackupData, but it is not a backup:
it is the portable, versioned, partial-or-whole interchange representation of a
graph (entities + relations + optional vectors/blobs) — the unit Brainy exports
for transport between instances, versions, and products, and the payload other
artifacts embed. The actual backup is persist()/load() (the native whole-brain,
generation-preserving snapshot), so "Backup*" actively collided with that concept.

Rename every developer-visible symbol, file, doc, JSDoc and comment:
- BackupData→PortableGraph, BackupEntity→PortableGraphEntity,
  BackupRelation→PortableGraphRelation, BackupReader/Writer→PortableGraphReader/Writer,
  BackupValidation→PortableGraphValidation, isBackupData→isPortableGraph,
  validateBackup→validatePortableGraph, BACKUP_FORMAT[_VERSION]→PORTABLE_GRAPH_FORMAT[_VERSION].
- src/db/backup.ts → src/db/portableGraph.ts; test → db-portable-graph.test.ts.
- Guide, api/README, RELEASES updated.

The on-the-wire `format` tag is renamed 'brainy-backup' → 'brainy-portable-graph':
the format was introduced in 7.32.0 and has not been adopted by any consumer, so
there are no stored documents to stay compatible with — a clean rename beats
carrying a legacy tag. No deprecated aliases (nothing to alias). 7.x shipped the
same rename as 7.32.2.

1471 unit green; build clean; db-portable-graph.test.ts 17/17.
2026-06-19 12:37:23 -07:00
4741e23d45 docs(8.0): RELEASES — portable export/import (BackupData v1) + distinctCount any-type section 2026-06-17 12:05:00 -07:00
f12ca68b82 docs(8.0): RELEASES.md — record removed BrainyZeroConfig + isFullyInitialized/awaitBackgroundInit in the breaking-change inventory 2026-06-15 11:12:26 -07:00
00d3203d68 refactor(8.0)!: remove distributed clustering subsystem — inert/orphaned, scale is single-process + native provider
The distributed-clustering subsystem never ran in production: it was inert,
orphaned dead code (faked consensus, stub replication, no live wiring, and it
did not interoperate with the 8.0 Db API). Brainy 8.0 is a single-process
library. Scale is single-process + the optional native provider
(@soulcraft/cortex, on-disk DiskANN to 10B+ vectors) + per-tenant pools +
horizontal read scaling (many reader processes, one writer).

Removed:
- src/distributed/ entirely (coordinator, shardManager, cacheSync,
  readWriteSeparation, queryPlanner, healthMonitor, configManager,
  hashPartitioner, shardMigration, domainDetector, storageDiscovery, http/network
  transports). ReaderMode/HybridMode relocated to src/storage/operationalModes.ts
  (slimmed to the live surface).
- src/types/distributedTypes.ts; config.distributed field + JSDoc;
  coreTypes distributedConfig; memoryStorage distributedConfig persistence.
- DistributedRole enum + src/config/distributedPresets.ts and the orphaned
  src/config/extensibleConfig.ts (config/augmentation registry built on removed
  cloud adapters + distributed presets), plus their src/index.ts re-exports.
- 13 BRAINY_* cluster env vars; the storage setDistributedComponents hook;
  enableDistributedSearch (dead config flag); the metadata partition field;
  the distributed_ reserved key prefix.
- Orphaned src/storage/readOnlyOptimizations.ts (zero importers).
- Tests targeting the subsystem: distributed-demo, distributed-cluster helper,
  distributed-transactions, sharding-transactions.
- Docs: EXTENDING_STORAGE.md (deleted); scrubbed distributed/cluster/Raft/
  shard-manager/multi-node prose from v3-features, enterprise-for-everyone,
  augmentations-actual, complete-feature-list, vfs/README, vfs/ROADMAP,
  vfs/VFS_CORE, capacity-planning, transactions, MIGRATION-V3-TO-V4,
  storage-architecture; reframed scale prose to the 8.0 model.

Kept: src/storage/sharding.ts (local-disk 256-bucket directory sharding via
getShardIdFromUuid — used live by baseStorage, unrelated to clustering);
the multi-process mode: 'reader' | 'writer' roles; semantic/HNSW clustering.

RELEASES.md: added a removed-surfaces row documenting the cut and the 8.0
scale model.
2026-06-15 10:37:39 -07:00
f8e0079d3f feat(8.0): zero-config finalize + cut JS quantization (config.vector = recall + persistMode) 2026-06-15 10:08:51 -07:00
1f7e365a4e chore(8.0): final pre-RC1 sweep — API consistency, named errors, orphans, zero-cast codebase 2026-06-11 14:51:00 -07:00
970e08c466 feat(8.0): reserved-field contract — one canonical location, typed prevention, unified read/write
Brainy-owned field names (noun/verb, subtype, createdAt, updatedAt,
confidence, weight, service, data, createdBy, _rev) now have exactly one
home — top level — enforced by three layers driven from a single source of
truth, src/types/reservedFields.ts (RESERVED_ENTITY_FIELDS /
RESERVED_RELATION_FIELDS, exported):

1. Compile time — AddParams/UpdateParams/RelateParams/UpdateRelationParams
   metadata (and the transact() ops that extend them) reject a literal
   reserved key as a TypeScript error while keeping generic T ergonomics
   (typed bags, untyped brains, index-signature shapes, and a documented
   exemption for T-declared reserved keys). Pinned by @ts-expect-error
   type tests run under vitest typecheck mode on every unit run.

2. Write time — the 7.x update() remap is ported to 8.0 and extended to
   every write path: add/update/relate/updateRelation, their transact()
   mirrors, and db.with() overlays. User-settable fields lift to their
   dedicated param (top-level wins when both are supplied — closes the 7.x
   trap where update({metadata:{confidence}}) silently no-oped), and
   system-managed fields drop with a one-shot warning naming the right
   path. A remapped subtype satisfies subtype-pairing enforcement exactly
   like a top-level one.

3. Read time — every storage combine goes through one canonical hydration
   helper (hydrateNounWithMetadata / hydrateVerbWithMetadata over
   splitNoun/VerbMetadataRecord), so reserved fields surface ONLY top-level
   and entity/relation.metadata carry ONLY custom fields on live reads,
   batch reads, paginated listings, getRelations by source/target, streamed
   verbs, and historical asOf() materialization alike.

Read-path echoes found and fixed (previously the full stored record —
including the verb type key — leaked inside metadata): noun pagination,
verb pagination, getVerbsBySource/ByTarget (adjacency + shard fallback),
getVerbsBySourceBatch (which also dropped subtype/data), and the
filesystem verb stream. getRelations() results now surface
confidence/updatedAt top-level via verbsToRelations, updateRelation() no
longer erases service/createdBy, relate() persists its top-level
confidence/service params, and the dead convertHNSWVerbToGraphVerb echo
path is deleted. Import paths (CLI extract, deduplicator, coordinators,
neural import) write confidence through the dedicated param instead of the
bag. UpdateRelationParams is now exported from the package root.

Documented for consumers in docs/concepts/consistency-model.md ("Reserved
fields") and RELEASES.md. Regression tests ported from the 7.x fix and
extended to the full 8.0 contract (17 runtime tests + 41 type-level
assertions); full unit suite 1427/1427, db-mvcc integration 24/24.
2026-06-11 13:13:09 -07:00
c44678390e feat(8.0): brain.fillSubtypes migration helper + pre-RC1 gap closure
- brain.fillSubtypes(rules): idempotent subtype back-fill for pre-8.0 data.
  One rule per NounType/VerbType (literal default or per-entry function);
  fills only entries still missing a subtype through the public update()/
  updateRelation() paths; returns { scanned, filled, skipped, errors, byType }.
  Full unit suite in tests/unit/brainy/fill-subtypes.test.ts.
- Fix getNouns/getVerbs pagination hasMore (peek one past the window) —
  was permanently false, silently truncating every multi-page walk.
- find({ near }) without near.id now throws a teaching error instead of an
  opaque storage sharding failure; CLI --threshold without --near applies a
  plain score floor.
- CLI init/close audit: every one-shot command init()s, close()s, and exits
  explicitly; delete the unmaintained interactive REPL; replace the cloud-era
  storage subcommands with status/batch-delete; new types/validate commands.
- requireSubtype JSDoc now documents the 8.0 default-on contract; audit()
  recommendation points at fillSubtypes.
- Docs: data-storage-architecture rewritten to the real 8.0 on-disk layout;
  README storage section reflects filesystem+memory and snapshots; eli5 and
  SEMANTIC_VFS /as-of/ semantics corrected; internal tracker IDs and
  .strategy references scrubbed from published files.
2026-06-11 10:53:55 -07:00